Build the mental model
The last project is not about a protocol, it is about a method. Everything you learned earlier gets turned into an ordered list of questions, and the program's only real job is to ask them bottom-up and stop at the first no.
The ordering is the whole idea. CHECKS runs from the host's own configuration upward: does this machine have a usable address at all, is its gateway inside its own subnet, does the gateway answer, does the name resolve, where would a packet to that address actually go, and only then does the connection complete. Once something fails, every later check is marked SKIP rather than run, because their results would be meaningless -- a DNS timeout on a host whose mask is wrong tells you nothing whatsoever about DNS.
Three earlier concepts do the detecting. Subnet math powers the sharpest check in the tool: gw not in net catches a misconfigured mask, which is the failure that looks most like 'the network is down' and is in fact entirely local. In the first scenario the host has a /26, so its subnet stops at 192.168.1.63, and a gateway at .100 is unreachable no matter how healthy the switch is. The routing decision -- is the target on-link or off-link -- is the same containment comparison, and it tells you whether the gateway is even involved in this conversation. And the transport layer supplies the single most useful distinction in troubleshooting: a RST means the host answered and nothing is listening, while silence means something dropped the SYN. In the second scenario DNS resolved and the SYN came back with a RST, so the name is fine and the route is fine, and what you are looking at is a dead service.
BOTTOM-UP DIAGNOSTIC LADDER
---------------------------
OBSERVATIONS CHECK LADDER FIRST FAILURE
------------ ------------ -------------
host ip + mask ----> [L3] host address & mask
| PASS
v
gateway ip ----> [L3] gateway inside subnet --FAIL--> wrong mask
| PASS
v
arp / ping ----> [L2] gateway reachable --FAIL--> cable / VLAN
| PASS
v
resolver ----> [L7] DNS resolution --FAIL--> name problem
| PASS
v
routing table ----> [L3] on-link or via gateway
|
v
connect() ----> [L4] TCP connect --FAIL--> RST = no
| PASS listener
v drop = firewall
ALL LAYERS OKConnect it to a real scenario
This is the shape of an on-call runbook rendered as code. The value is not that a computer runs the checks -- you can run them by hand -- it is that the order is fixed and the stopping rule enforced, so a tired engineer at 3am does not skip ahead to the interesting layer.
The failure mode this prevents is the most common one in real incident response: starting at the top. Someone reports that the app cannot reach the database, and the instinct is to look at the app, the connection string, the database logs. Hours later it turns out the host picked up a /26 from a bad DHCP scope, or a security group was tightened in a deploy. The ladder finds either one in seconds, and it produces evidence you can hand to somebody else. 'L2 and L3 pass, DNS resolves to 10.20.9.14, and the SYN gets a RST' is a sentence the database team can act on immediately; 'it is not working' is not.
The scenario table also demonstrates the second habit worth building: always keep a known-good baseline. Scenario three exists so you know what a clean run looks like, and without it an unfamiliar PASS line is as suspicious as a FAIL. In production this is why teams keep a jump host or canary in every subnet -- half of diagnosis is having something that works to compare against.
Try the working example
import ipaddress
import textwrap
PASS, FAIL, INFO = "PASS", "FAIL", "INFO"
# Hardcoded observations. In a real tool these come from `ip addr`, `ip route`,
# a resolver call and a connect() attempt -- here they are frozen so the
# diagnosis is reproducible.
SCENARIOS = [
{
"name": "laptop-a (freshly re-imaged)",
"host_ip": "192.168.1.50", "mask": "255.255.255.192",
"gateway": "192.168.1.100",
"gateway_reachable": False,
"target_name": "api.example.com", "target_port": 443,
"dns_result": "93.184.216.34",
"tcp_result": "timeout",
},
{
"name": "web-tier-03 (deploy just failed)",
"host_ip": "10.20.5.31", "mask": "255.255.255.0",
"gateway": "10.20.5.1",
"gateway_reachable": True,
"target_name": "db.internal", "target_port": 5432,
"dns_result": "10.20.9.14",
"tcp_result": "refused",
},
{
"name": "workstation-7 (baseline, known good)",
"host_ip": "172.16.4.9", "mask": "255.255.255.0",
"gateway": "172.16.4.1",
"gateway_reachable": True,
"target_name": "cache.internal", "target_port": 6379,
"dns_result": "172.16.4.60",
"tcp_result": "open",
},
]
def check_host_address(obs, ctx):
iface = ipaddress.ip_interface("%s/%s" % (obs["host_ip"], obs["mask"]))
ctx["network"] = iface.network
if iface.ip in ipaddress.ip_network("169.254.0.0/16"):
return FAIL, "link-local address -- DHCP never answered", \
"Fix DHCP or assign a static address."
return PASS, "%s in %s" % (iface.ip, iface.network), None
def check_gateway_local(obs, ctx):
gw = ipaddress.ip_address(obs["gateway"])
net = ctx["network"]
if gw not in net:
return FAIL, "gateway %s is OUTSIDE %s" % (gw, net), \
("The mask is wrong. /%d only spans %s-%s, so the host can never "
"ARP for its own gateway." % (net.prefixlen, net[0], net[-1]))
return PASS, "gateway %s is inside %s" % (gw, net), None
def check_gateway_reachable(obs, ctx):
if not obs["gateway_reachable"]:
return FAIL, "no reply from gateway %s" % obs["gateway"], \
"Layer 1/2 problem: cable, VLAN, or a down switch port."
return PASS, "gateway %s replies" % obs["gateway"], None
def check_dns(obs, ctx):
if not obs["dns_result"]:
return FAIL, "%s did not resolve" % obs["target_name"], \
"Name problem, not a reachability problem. Check the resolver."
ctx["target_ip"] = ipaddress.ip_address(obs["dns_result"])
return PASS, "%s -> %s" % (obs["target_name"], obs["dns_result"]), None
def check_route(obs, ctx):
target, net = ctx["target_ip"], ctx["network"]
if target in net:
return INFO, "%s is on-link, delivered directly" % target, None
return INFO, "%s is off-link, sent via %s" % (target, obs["gateway"]), None
def check_tcp(obs, ctx):
result = obs["tcp_result"]
if result == "open":
return PASS, "handshake completed on port %d" % obs["target_port"], None
if result == "refused":
return FAIL, "RST on port %d" % obs["target_port"], \
("The host is reachable and answered -- nothing is listening on "
"that port. A dead service, not a network fault.")
return FAIL, "no SYN-ACK on port %d" % obs["target_port"], \
"Silent drop: a firewall or ACL is discarding the SYN."
CHECKS = [
("L3", "Host address and mask", check_host_address),
("L3", "Gateway within subnet", check_gateway_local),
("L2", "Gateway reachable", check_gateway_reachable),
("L7", "DNS resolution", check_dns),
("L3", "Routing decision", check_route),
("L4", "TCP connect", check_tcp),
]
def diagnose(obs):
print("SCENARIO: %s" % obs["name"])
print("-" * 70)
print(" target %s:%d" % (obs["target_name"], obs["target_port"]))
print()
ctx = {}
root_cause = None
for layer, label, fn in CHECKS:
if root_cause is not None:
print(" %-3s %-24s %-6s %s" % (layer, label, "SKIP", "not reached"))
continue
status, detail, advice = fn(obs, ctx)
print(" %-3s %-24s %-6s %s" % (layer, label, status, detail))
if status == FAIL:
root_cause = (layer, label, advice)
print()
if root_cause is None:
print(" VERDICT: healthy -- every layer checks out.")
else:
layer, label, advice = root_cause
print(" VERDICT: first failure at %s -- %s" % (layer, label))
for i, line in enumerate(textwrap.wrap(advice, 60)):
print(" %s %s" % ("CAUSE :" if i == 0 else " ", line))
print()
print("LAYERED NETWORK DIAGNOSTIC REPORT")
print("=" * 70)
print()
for scenario in SCENARIOS:
diagnose(scenario)
LAYERED NETWORK DIAGNOSTIC REPORT
======================================================================
SCENARIO: laptop-a (freshly re-imaged)
----------------------------------------------------------------------
target api.example.com:443
L3 Host address and mask PASS 192.168.1.50 in 192.168.1.0/26
L3 Gateway within subnet FAIL gateway 192.168.1.100 is OUTSIDE 192.168.1.0/26
L2 Gateway reachable SKIP not reached
L7 DNS resolution SKIP not reached
L3 Routing decision SKIP not reached
L4 TCP connect SKIP not reached
VERDICT: first failure at L3 -- Gateway within subnet
CAUSE : The mask is wrong. /26 only spans 192.168.1.0-192.168.1.63,
so the host can never ARP for its own gateway.
SCENARIO: web-tier-03 (deploy just failed)
----------------------------------------------------------------------
target db.internal:5432
L3 Host address and mask PASS 10.20.5.31 in 10.20.5.0/24
L3 Gateway within subnet PASS gateway 10.20.5.1 is inside 10.20.5.0/24
L2 Gateway reachable PASS gateway 10.20.5.1 replies
L7 DNS resolution PASS db.internal -> 10.20.9.14
L3 Routing decision INFO 10.20.9.14 is off-link, sent via 10.20.5.1
L4 TCP connect FAIL RST on port 5432
VERDICT: first failure at L4 -- TCP connect
CAUSE : The host is reachable and answered -- nothing is listening
on that port. A dead service, not a network fault.
SCENARIO: workstation-7 (baseline, known good)
----------------------------------------------------------------------
target cache.internal:6379
L3 Host address and mask PASS 172.16.4.9 in 172.16.4.0/24
L3 Gateway within subnet PASS gateway 172.16.4.1 is inside 172.16.4.0/24
L2 Gateway reachable PASS gateway 172.16.4.1 replies
L7 DNS resolution PASS cache.internal -> 172.16.4.60
L3 Routing decision INFO 172.16.4.60 is on-link, delivered directly
L4 TCP connect PASS handshake completed on port 6379
VERDICT: healthy -- every layer checks out.
5-minute try-it
Add a fourth scenario in which the host holds a link-local address, 169.254.10.7 with a /16 mask, meaning DHCP never answered, while every other observation looks healthy. The report should stop at the very first check and mark all five remaining checks as SKIP. Then try a scenario with dns_result set to None and confirm that the TCP check genuinely does not run once DNS has failed -- that stopping rule is the entire point of the tool, so it is worth proving to yourself that it holds.
One important caution
Running every remaining check after one has already failed and printing a long list of failures. The later 'failures' are consequences, not causes -- a host with a wrong mask will also fail DNS and fail TCP, and the report then blames DNS for what is a layer 3 problem. Stop at the first FAIL and mark the rest explicitly as SKIP so nobody chases a downstream symptom.
Comparing gateway to host by eyeballing the first three octets. '192.168.1.50 and 192.168.1.100 are obviously in the same subnet' is true for a /24 and completely false for a /26. Always run the containment test against the network derived from the real mask, via ip_interface(...).network, rather than against the dotted-quad prefix. This bug tends to stay hidden until the day someone moves off /24.
RFC 1122 -- Requirements for Internet Hosts: Communication Layers — Computer Networking