Your application cannot reach a service that should be up. Walk me through diagnosing it from the shell.

Get a precise failure first

Before touching a tool, get the exact failure text and the exact target. "The service is down" is not a symptom; "connect to 10.20.4.11:8443 timed out after 30s" is. The distinction matters because the shape of the failure already discards half the search space, and a candidate who asks for it before typing anything has demonstrated the most valuable habit in this whole exercise.

Three failure modes cover nearly everything. A resolution error means you never got as far as a packet. Connection refused means a packet reached a host and something at the other end answered with a TCP RST, so DNS, routing and firewalling are all working and the problem is at the destination. A timeout means your packets went somewhere and nothing came back, which is the ambiguous case and the one that needs the layers walked properly.

Name to address

# What the DNS server says.
dig +short api.internal.example A

# What this host will actually use, via nsswitch: /etc/hosts, then DNS.
getent hosts api.internal.example

dig talks to a nameserver directly. getent hosts goes through the glibc resolver, which consults /etc/nsswitch.conf and therefore /etc/hosts first. When those two disagree you have found your bug: a stale entry in /etc/hosts on one host, or a search domain in /etc/resolv.conf turning a short name into something unintended. If dig returns NXDOMAIN the name does not exist; if it returns NOERROR with an empty answer section the name exists but has no record of the type you asked for, which is a different mistake and often means an A lookup against a CNAME-only or AAAA-only name.

A resolution failure that only appears intermittently is usually a resolver reachability problem rather than a records problem. dig @10.0.0.2 api.internal.example against each nameserver in /etc/resolv.conf in turn tells you whether one of them is dead, and the timeout you feel is the client waiting out the first one before falling back.

The resolver you test with is often not the resolver the application uses

This is the check that catches the case where every command you ran succeeded and the application still fails. dig is a DNS client and nothing more; it does not read /etc/hosts, it does not honour nsswitch, and it does not share the application's cache. A JVM caches successful lookups in-process, so a service that resolved a name before a failover keeps the pre-failover address long after dig reports the new one. Containers built on Alpine use musl rather than glibc, whose resolver behaviour around search domains and multiple nameservers differs from what you get on the host. And anything routed through a local caching resolver on 127.0.0.53 gives you a cached answer while dig @ an upstream gives you a fresh one.

So resolve the way the application resolves. getent hosts for anything glibc-based, and where you cannot, bypass the question entirely with curl --resolve so a single test tells you whether the address works when the name is taken out of the picture.

# Pins the name to an address, so a failure here is NOT DNS.
curl -v --resolve api.internal.example:8443:10.20.4.11 https://api.internal.example/healthz

Route and reachability

# Which interface and which source address the kernel picks for this destination.
ip route get 10.20.4.11

This one command answers questions people usually guess at. It names the egress interface, the next hop, and the source address that will appear on the packet, which is what the far-end firewall or security group will match on. A missing route shows up here immediately as Network is unreachable. A route out of the wrong interface, common on hosts with several networks or a VPN, explains a timeout that looks inexplicable at every other layer.

ping is worth one attempt and no more than that. A reply proves the path works in both directions and narrows the fault to the port or the process. Silence proves very little, because ICMP is filtered on plenty of otherwise healthy networks, so do not conclude the host is down from a failed ping. traceroute -T -p 8443 10.20.4.11 is more useful than plain traceroute for exactly this reason: it probes with TCP to the port you care about rather than with a protocol nobody allows.

Is anything listening, and on what

# Listening TCP sockets, numeric, with the owning process.
ss -ltnp

# Every socket to this destination, including half-open ones.
ss -tnp state all dst 10.20.4.11

Run the first on the server. The distinction that matters is the local address column: a socket bound to 127.0.0.1:8443 is reachable only from that host no matter how correct everything else is, whereas 0.0.0.0:8443 or [::]:8443 accepts from anywhere the network permits. A service that works when you curl localhost on the box and refuses from elsewhere is almost always this, and it is a configuration change rather than a network one.

Two subtler readings of ss output are worth knowing. On a listening socket, Recv-Q is the number of connections completed but not yet accepted and Send-Q is the configured backlog, so a Recv-Q sitting at the backlog limit means the process is alive but too slow to accept, which presents to clients as intermittent timeouts rather than refusals. And on the client side, sockets stuck in SYN-SENT towards the target confirm the handshake is being started and never answered, which is the signature of a drop rather than a reject.

When you still cannot tell, capture

A capture is not the first tool, because everything above is cheaper and answers more narrowly. It is the right tool once you have a timeout you cannot attribute, because it distinguishes "my packets never left" from "they left and nothing came back".

# Both directions, all interfaces, no name resolution to avoid its own delays.
tcpdump -i any -nn 'host 10.20.4.11 and tcp port 8443'

Read it by what is absent. Repeated SYNs with no response at all means the packet is being dropped silently, which points at a DROP firewall rule, a security group or network policy, or a route that gets the packet somewhere with no way back. A SYN answered by an immediate RST is a reject, and that is the same signal as Connection refused: something is there and declining. SYN, SYN-ACK, ACK followed by nothing means the TCP layer is entirely fine and your problem is the application or TLS above it, which is where curl -v and openssl s_client -connect host:8443 -servername host take over to show whether the handshake completes and which certificate is presented.

The strongest version of this step is capturing on both ends at once. If the SYN leaves the client and never arrives at the server, the loss is in the middle. If it arrives and the server sends a SYN-ACK that the client never sees, the return path is asymmetric or something stateful in the middle has dropped the flow. That two-sided capture is what turns "the network is broken" into a claim you can hand to a network team with evidence.

The asymmetry that this order buys you

Each step above is chosen because a negative result excludes a category rather than merely failing. Resolution excludes naming. ip route get excludes local routing and pins the source address. ss on the server excludes the bind address and the process. The failure mode of the connect attempt excludes either reject or drop but never both. By the time you are running a packet capture you should be able to state, out loud, the one hypothesis it is meant to confirm. A candidate who cannot say what they expect to see in the capture is not diagnosing, they are watching.