Administrator Documentation · Original Demonstration

Linux Service Troubleshooting Runbook

This fictional, original runbook investigates a systemd service that is still running but no longer delivering events. Each check preserves evidence and narrows the next decision.

Scenario

The fictional Relay Agent runs as a systemd service and forwards events over TLS. The process still exists, but events have stopped arriving. That distinction matters because a green service state can hide failures in input, queueing, DNS, transport, trust, credentials, or the receiving application.

What This Sample Demonstrates

This sample demonstrates Linux administration, systemd and journalctl, DNS, TCP, TLS, credential and permission checks, diagnostic sequencing, and the evidence an administrator should collect before escalation. It is organized around eliminating classes of failure rather than accumulating commands.

Symptom

Use this procedure when the service appears active but events are missing, reconnecting, queueing, or failing delivery. Do not reinstall the agent first, because doing so can erase evidence without fixing the cause.

Before You Begin

Prerequisites

  • Authorized shell access to the Linux host running Relay Agent.
  • The approximate time delivery stopped, including the timezone.
  • The expected destination hostname, port, and environment.
  • A way to send a known test event and confirm its arrival in the receiving application.
  • Access to the approved credential system, or a credential owner who can confirm validity without revealing the secret.
  • systemctl, journalctl, grep, du, find, getent, resolvectl, nc, openssl, namei, and stat installed on the host. If a command is unavailable, use the organization's approved equivalent rather than installing software during the incident without authorization.

Required Permissions

Use the least-privileged account that can complete the checks. You need:

  • Permission to view the service state and unit properties.
  • Permission to read the service journal. Depending on the distribution, this may require sudo or membership in an approved journal-reading group such as systemd-journal or adm.
  • Read and directory-traversal permission for the agent configuration, credential path metadata, and queue. You do not need to display the credential value.
  • Permission to perform DNS lookups and make an outbound TCP connection to the configured destination and port.
  • Approved access to view credential status in the credential system.
  • Change authorization and sudo permission to restart relay-agent in Step 10.

If a command returns Permission denied, record that result and follow the organization's access-escalation process. Do not grant broader permissions or switch to an unrestricted account solely to continue the procedure.

1. Confirm Service State

Open a terminal on the host running Relay Agent.

Type the following:

systemctl status relay-agent --no-pager
journalctl -u relay-agent --since "30 minutes ago" --no-pager

Look for:

  • An active (running) service state.
  • Recent restarts, warnings, or errors in the journal.
  • Log activity during the period when delivery stopped.

An active service proves only that the process exists, not that it can deliver events.

2. Find a Concrete Delivery Error

In the same terminal, filter the recent journal for delivery-related messages.

Type the following:

journalctl -u relay-agent --since "30 minutes ago" --no-pager | \
  grep -Ei "error|fail|timeout|tls|certificate|dns|unauthorized|forbidden|queue"

Look for:

  • The first error near the time delivery stopped.
  • Repeated timeout, DNS, TLS, authentication, or queue messages.
  • A request or correlation ID that connects related entries.

Record the timestamp, exact error, and any request or correlation ID. If the filter returns nothing, preserve the unfiltered journal output from Step 1 and continue.

3. Check Local Queueing

Check the queue size and file count, then repeat the check after a few minutes.

Type the following:

du -sh /var/lib/relay-agent/queue
find /var/lib/relay-agent/queue -type f | wc -l

Look for:

  • An increasing size or file count, which points toward destination connectivity, TLS, or authentication.
  • An empty or unchanged queue while events are expected, which points toward the local input.

4. Confirm the Destination

Read only the destination settings from the agent configuration.

Type the following:

grep -E '^(endpoint|server|port)=' /etc/relay-agent/relay-agent.conf

Look for:

  • The expected hostname and port for the current environment.
  • Missing, duplicate, or unexpected destination values.

Do not print or copy secrets into the case record.

5. Test DNS

Resolve the configured destination and inspect the host's resolver state.

Type the following:

getent ahosts collector.example.invalid
resolvectl status

Look for:

  • One or more IP addresses for the configured hostname.
  • The expected DNS servers, search domains, and active interface.
  • Resolution errors or an empty result.

If resolution fails, fix or escalate DNS or network configuration before changing the agent.

6. Test TCP Reachability

Test whether the configured destination accepts a connection on its configured port.

Type the following:

nc -vz collector.example.invalid 443

Look for:

  • A successful or open connection, which confirms basic TCP reachability.
  • A timeout, which suggests routing, firewall, VPN, proxy, or availability problems.
  • A refusal, which means the host is reachable but the port is not accepting connections.

7. Validate TLS

Inspect the TLS handshake without sending application data.

Type the following:

openssl s_client \
  -connect collector.example.invalid:443 \
  -servername collector.example.invalid \
  -verify_return_error </dev/null

Look for:

  • Verify return code: 0 (ok).
  • A certificate whose hostname, expiration, issuer, and chain match the destination.
  • A negotiated protocol supported by the agent and endpoint.

Do not disable certificate verification to bypass a failure.

8. Check Authentication Safely

Inspect the credential path and file metadata without displaying the credential.

Type the following:

namei -l /etc/relay-agent/credentials/token
stat -c '%U %G %a %n' /etc/relay-agent/credentials/token

Look for:

  • A token file at the expected path.
  • Execute permission on each parent directory for the service account.
  • File ownership and mode that allow the service account to read the token without exposing it to other users.

In the approved credential system, also confirm that the credential is current and is the type expected by the endpoint. Do not print the secret.

9. Check Required File Access

Identify the service account, then inspect access to the configuration and queue paths.

Type the following:

systemctl show relay-agent -p User -p Group
namei -l /etc/relay-agent/relay-agent.conf
namei -l /var/lib/relay-agent/queue

Look for:

  • The expected service user and group.
  • Read access to the configuration file.
  • The required read and write access to the queue.
  • Execute permission on every parent directory in both paths.

A permission failure can occur on a parent directory even when the final file's mode appears correct.

10. Restart Only With Evidence

After correcting an identified problem and preserving the evidence, restart the service and watch its journal.

Type the following:

sudo systemctl restart relay-agent
sudo systemctl status relay-agent --no-pager
journalctl -u relay-agent -f

Look for:

  • The service returning to active (running) without entering a restart loop.
  • A successful connection and delivery message in the journal.
  • No recurrence of the original error.

Service state alone does not prove recovery.

11. Verify End-to-End Recovery

Send a known test event through the normal input path, then confirm its arrival in the receiving application.

Look for:

  • The test event appearing once with the expected timestamp and content.
  • A successful delivery entry in the Relay Agent journal.
  • The local queue remaining empty or decreasing.
  • No recurrence of the original error during the approved observation period.

Record the test event ID, delivery time, relevant journal entry, queue result, and observation period. If the process is running but the event does not arrive, recovery is not complete. Return to the evidence from Steps 2 through 9 or escalate with the collected results.

Escalate When

  • DNS and TCP work but TLS fails with an issue you do not own.
  • Authentication fails with a known-current credential and correct permissions.
  • The endpoint returns repeated server-side 5xx errors.
  • The local queue grows despite successful application responses.

Include OS and agent versions, destination and port without secrets, start time and timezone, relevant logs and IDs, DNS/TCP/TLS results, and changes already made.

Troubleshooting Logic

  1. Is the service running? If not, inspect logs and exit reason.
  2. Are events queueing? If not, check the local input path.
  3. Does DNS resolve? If not, fix or escalate network configuration.
  4. Is the TCP port reachable? If not, check routing, firewall, VPN, proxy, or endpoint.
  5. Does TLS validate? If not, check hostname, chain, expiry, and trust store.
  6. Does authentication succeed? If not, check validity and file access.
  7. Otherwise, inspect the application response and queue behavior.

Why the Procedure Is Shaped This Way

The sequence moves through process health, input health, name resolution, transport, TLS trust, authentication, and application behavior. Each result narrows the problem and determines the next action.

Restarting comes late for a reason. A restart can restore service without explaining the failure, or erase the evidence needed to understand it. The administrator preserves that evidence first, then verifies end-to-end delivery after any correction. A running process alone does not establish recovery.