Scenario
The fictional Relay Agent runs as a systemd service and forwards events over TLS. The service is running, but events have stopped arriving.
What This Sample Demonstrates
This sample demonstrates Linux administration, systemd and journalctl, DNS, TCP, TLS, credential and permission checks, diagnostic sequencing, and the evidence an administrator should collect before escalation.
Symptom
Use this procedure when the service appears active but events are missing, reconnecting, queueing, or failing delivery. Do not reinstall the agent first, because doing so can erase evidence without fixing the cause.
1. Confirm Service State
systemctl status relay-agent --no-pager
journalctl -u relay-agent --since "30 minutes ago" --no-pager
A running process proves only that the process exists, not that it can deliver events.
2. Find a Concrete Delivery Error
journalctl -u relay-agent --since "30 minutes ago" --no-pager | \
grep -Ei "error|fail|timeout|tls|certificate|dns|unauthorized|forbidden|queue"
Record the timestamp, exact error, and any request or correlation ID.
3. Check Local Queueing
du -sh /var/lib/relay-agent/queue
find /var/lib/relay-agent/queue -type f | wc -l
A growing queue points toward destination connectivity, TLS, or authentication; an empty queue points toward the local input.
4. Confirm the Destination
grep -E '^(endpoint|server|port)=' /etc/relay-agent/relay-agent.conf
Verify hostname, port, and environment without printing secrets.
5. Test DNS
getent ahosts collector.example.invalid
resolvectl status
If resolution fails, fix or escalate DNS/network configuration before changing the agent.
6. Test TCP Reachability
nc -vz collector.example.invalid 443
A timeout suggests routing, firewall, VPN, proxy, or availability; refusal means the host is reachable but the port is not accepting.
7. Validate TLS
openssl s_client \
-connect collector.example.invalid:443 \
-servername collector.example.invalid \
-verify_return_error </dev/null
Check expiration, hostname, issuer/chain, and protocol. Do not disable verification.
8. Check Authentication Safely
namei -l /etc/relay-agent/credentials/token
stat -c '%U %G %a %n' /etc/relay-agent/credentials/token
Confirm existence, service-account access, validity, and credential type without printing the secret.
9. Check Required File Access
systemctl show relay-agent -p User -p Group
namei -l /etc/relay-agent/relay-agent.conf
namei -l /var/lib/relay-agent/queue
A permission failure can occur on a parent directory.
10. Restart Only With Evidence
sudo systemctl restart relay-agent
sudo systemctl status relay-agent --no-pager
journalctl -u relay-agent -f
After a correction, confirm both successful delivery and arrival of a known test event.
Escalate When
- DNS and TCP work but TLS fails with an issue you do not own.
- Authentication fails with a known-current credential and correct permissions.
- The endpoint returns repeated server-side 5xx errors.
- The local queue grows despite successful application responses.
Include OS and agent versions, destination and port without secrets, start time and timezone, relevant logs and IDs, DNS/TCP/TLS results, and changes already made.
Troubleshooting Logic
- Is the service running? If not, inspect logs and exit reason.
- Are events queueing? If not, check the local input path.
- Does DNS resolve? If not, fix or escalate network configuration.
- Is the TCP port reachable? If not, check routing, firewall, VPN, proxy, or endpoint.
- Does TLS validate? If not, check hostname, chain, expiry, and trust store.
- Does authentication succeed? If not, check validity and file access.
- Otherwise, inspect the application response and queue behavior.
Why the Procedure Is Shaped This Way
The sequence moves through process health, input health, name resolution, transport, TLS trust, authentication, and application behavior so every test narrows the problem and determines the next action.