Build the mental model
This parser is where encapsulation stops being a diagram and becomes a byte offset. The script builds forty bytes and then takes them apart exactly the way a NIC driver, tcpdump, or a firewall does, with no knowledge of what the bytes mean beyond the layouts the RFCs define.
Four earlier ideas do the work. Encapsulation gives the shape: the TCP header does not sit somewhere in the packet, it sits immediately after the IPv4 header, and you find it by reading the IHL nibble out of byte zero and multiplying by four. That is why the code says packet[ip_len:] rather than packet[20:] -- twenty is only correct when there are no IP options, and hardcoding it is exactly how parsers break on the first packet that carries them. The header layouts give the format strings: !BBHHHBBH4s4s is literally the IPv4 header drawn left to right, and !HHIIHHHH is the TCP one, with the leading exclamation mark meaning network byte order, big-endian, which is the whole reason struct is the right tool and manual shifting is not.
Bit masking gives the flags. TCP packs a four-bit data offset and nine flag bits into a single sixteen-bit word, so offset_flags >> 12 recovers the header length and offset_flags & 0x01FF recovers the flags, after which each flag is one AND against its own bit. Finally the four-tuple falls out for free: source and destination IP came from layer three, source and destination port from layer four, and together with the protocol they are what every NAT table and connection tracker along the path keys on. Verifying the checksum is the detail that makes the report feel real -- recomputing the Internet checksum over a correct header yields zero, because the stored value is already the one's complement of everything else.
PACKET PARSER: BYTES IN, FIELDS OUT
-----------------------------------
RAW BYTES (40 total)
+----------------------------------------------------------+
| 4500 0028 1c46 4000 4006 26fd c0a8 010a 5db8 d822 ... |
+----------------------------------------------------------+
|
| byte 0 low nibble = IHL = 5 words = 20 bytes
v
+----------------------------+----------------------------+
| IPv4 HEADER [0:20] | TCP HEADER [20:40] |
+----------------------------+----------------------------+
| |
| struct.unpack | struct.unpack
| "!BBHHHBBH4s4s" | "!HHIIHHHH"
v v
+----------------------------+ +----------------------------+
| version, IHL, total_length | | sport, dport, seq, ack |
| ttl, protocol, checksum | | offset_flags, window |
| src IP, dst IP | +----------------------------+
+----------------------------+ |
| | & 0x001 .. 0x100
v v
+----------------------------+ +------------------------+
| checksum16(header) == 0 ? | | FIN SYN RST PSH ACK .. |
+----------------------------+ +------------------------+
| |
+----------------+-------------------+
v
THE 4-TUPLE: src IP : sport -> dst IP : dport / TCPConnect it to a real scenario
You will not replace tcpdump with this. You write a parser like this when the summary line is not enough -- when you are staring at a capture and need to answer a question no tool phrases for you: are these retransmits or is the sequence number going backwards, is the DF bit set on the packets a smaller-MTU tunnel is dropping. The habit it builds is reading a packet as bytes plus a layout -- the same habit every binary protocol you meet later will demand.
The concrete stakes show up in two places. The first is MTU and fragmentation debugging: a connection that completes the handshake and then hangs on the first large transfer is the classic 'DF set, packet too big, ICMP being filtered' story, and you diagnose it by reading total_length and the DF bit off real packets. The second is anything touching the four-tuple -- NAT, load balancers, connection tracking. When two flows collide, or a firewall log makes no sense, the answer comes from lining up those exact four values.
The naive alternative is to slice at fixed offsets and skip the checksum entirely. That works on the packets you tested and fails on the ones you did not: options push every TCP field four or eight bytes right, and without a checksum you cannot tell 'the sender is doing something surprising' from 'I am parsing garbage'.
Try the working example
import struct
import ipaddress
import binascii
PROTOCOLS = {1: "ICMP", 6: "TCP", 17: "UDP"}
# TCP flag bits, low bit first, as they sit in the 9-bit flags field.
TCP_FLAGS = [
(0x001, "FIN"), (0x002, "SYN"), (0x004, "RST"), (0x008, "PSH"),
(0x010, "ACK"), (0x020, "URG"), (0x040, "ECE"), (0x080, "CWR"),
(0x100, "NS"),
]
def checksum16(data):
"""The standard Internet checksum: one's complement of the one's
complement sum of all 16-bit words."""
if len(data) % 2:
data += b"\x00"
total = 0
for (word,) in struct.iter_unpack("!H", data):
total += word
while total >> 16:
total = (total & 0xFFFF) + (total >> 16)
return ~total & 0xFFFF
def build_ipv4_header(src, dst, payload_len, ttl=64, proto=6, ident=0x1C46):
version_ihl = (4 << 4) | 5 # version 4, IHL 5 words = 20 bytes
total_len = 20 + payload_len
flags_frag = 0x4000 # Don't Fragment, offset 0
header = struct.pack(
"!BBHHHBBH4s4s",
version_ihl, 0, total_len, ident, flags_frag, ttl, proto, 0,
ipaddress.IPv4Address(src).packed,
ipaddress.IPv4Address(dst).packed,
)
csum = checksum16(header) # computed with the checksum field zeroed
return header[:10] + struct.pack("!H", csum) + header[12:]
def build_tcp_header(sport, dport, seq, ack, flags, window=64240):
offset_reserved = (5 << 12) | flags # data offset 5 words = 20 bytes
return struct.pack("!HHIIHHHH", sport, dport, seq, ack,
offset_reserved, window, 0, 0)
def parse_ipv4(data):
(version_ihl, tos, total_len, ident, flags_frag, ttl, proto,
csum, src, dst) = struct.unpack("!BBHHHBBH4s4s", data[:20])
return {
"version": version_ihl >> 4,
"ihl_words": version_ihl & 0x0F,
"tos": tos,
"total_length": total_len,
"id": ident,
"df": bool(flags_frag & 0x4000),
"mf": bool(flags_frag & 0x2000),
"ttl": ttl,
"protocol": proto,
"checksum": csum,
"src": str(ipaddress.IPv4Address(src)),
"dst": str(ipaddress.IPv4Address(dst)),
}
def parse_tcp(data):
(sport, dport, seq, ack, offset_flags, window, csum, urg) = \
struct.unpack("!HHIIHHHH", data[:20])
return {
"sport": sport,
"dport": dport,
"seq": seq,
"ack": ack,
"offset_words": offset_flags >> 12,
"flags": offset_flags & 0x01FF,
"window": window,
"checksum": csum,
"urgent": urg,
}
def flag_names(flags):
return [name for bit, name in TCP_FLAGS if flags & bit]
def report(packet):
ip = parse_ipv4(packet)
ip_len = ip["ihl_words"] * 4
tcp = parse_tcp(packet[ip_len:])
print("PACKET HEADER REPORT")
print("=" * 64)
print("Raw bytes (%d total):" % len(packet))
hexs = binascii.hexlify(packet).decode()
for i in range(0, len(hexs), 32):
print(" %04x %s" % (i // 2, " ".join(
hexs[j:j + 4] for j in range(i, min(i + 32, len(hexs)), 4))))
print()
print("-- LAYER 3: IPv4 " + "-" * 47)
print(" Version : %d" % ip["version"])
print(" IHL : %d words (%d bytes)" % (ip["ihl_words"], ip_len))
print(" Total length : %d bytes (payload %d)" % (ip["total_length"],
ip["total_length"] - ip_len))
print(" Identification : 0x%04x" % ip["id"])
print(" Flags : DF=%d MF=%d" % (ip["df"], ip["mf"]))
print(" TTL : %d" % ip["ttl"])
print(" Protocol : %d (%s)" % (ip["protocol"],
PROTOCOLS.get(ip["protocol"], "unknown")))
print(" Header checksum : 0x%04x" % ip["checksum"])
# A correct header re-checksums to zero, because the stored value is
# already the complement of the sum of every other word.
verify = checksum16(packet[:ip_len])
print(" Checksum verify : %s (recomputed 0x%04x)" %
("VALID" if verify == 0 else "INVALID", verify))
print(" Source IP : %s" % ip["src"])
print(" Destination IP : %s" % ip["dst"])
print()
print("-- LAYER 4: TCP " + "-" * 48)
print(" Source port : %d" % tcp["sport"])
print(" Destination port : %d" % tcp["dport"])
print(" Sequence number : %d (0x%08x)" % (tcp["seq"], tcp["seq"]))
print(" Ack number : %d" % tcp["ack"])
print(" Data offset : %d words (%d bytes)" % (tcp["offset_words"],
tcp["offset_words"] * 4))
print(" Flags : 0x%03x -> %s" % (tcp["flags"],
" ".join(flag_names(tcp["flags"])) or "none"))
for bit, name in TCP_FLAGS:
print(" %-4s %d" % (name, 1 if tcp["flags"] & bit else 0))
print(" Window : %d" % tcp["window"])
print()
print("-- THE 4-TUPLE " + "-" * 49)
print(" %s:%d -> %s:%d" % (ip["src"], tcp["sport"], ip["dst"], tcp["dport"]))
print(" protocol %s" % PROTOCOLS.get(ip["protocol"], "unknown"))
tcp_header = build_tcp_header(50912, 443, 0x9ECF1D2A, 0, flags=0x002) # SYN
ip_header = build_ipv4_header("192.168.1.10", "93.184.216.34", len(tcp_header))
report(ip_header + tcp_header)
PACKET HEADER REPORT
================================================================
Raw bytes (40 total):
0000 4500 0028 1c46 4000 4006 26fd c0a8 010a
0010 5db8 d822 c6e0 01bb 9ecf 1d2a 0000 0000
0020 5002 faf0 0000 0000
-- LAYER 3: IPv4 -----------------------------------------------
Version : 4
IHL : 5 words (20 bytes)
Total length : 40 bytes (payload 20)
Identification : 0x1c46
Flags : DF=1 MF=0
TTL : 64
Protocol : 6 (TCP)
Header checksum : 0x26fd
Checksum verify : VALID (recomputed 0x0000)
Source IP : 192.168.1.10
Destination IP : 93.184.216.34
-- LAYER 4: TCP ------------------------------------------------
Source port : 50912
Destination port : 443
Sequence number : 2664373546 (0x9ecf1d2a)
Ack number : 0
Data offset : 5 words (20 bytes)
Flags : 0x002 -> SYN
FIN 0
SYN 1
RST 0
PSH 0
ACK 0
URG 0
ECE 0
CWR 0
NS 0
Window : 64240
-- THE 4-TUPLE -------------------------------------------------
192.168.1.10:50912 -> 93.184.216.34:443
protocol TCP
5-minute try-it
Rebuild the packet with flags=0x012 (SYN and ACK together) and ack=0x9ECF1D2B, and change the TTL from 64 to 55, then run it again. Both SYN and ACK should show 1 and the checksum should still verify as VALID, because build_ipv4_header recomputes it. Then do the more interesting half: flip a single byte in the finished packet by hand -- change the TTL byte at index 8 without touching the checksum field -- and run the report again. Checksum verify should turn INVALID and the recomputed value should no longer be zero. That is precisely how a router detects a corrupted header.
One important caution
Assuming the IPv4 header is always twenty bytes and ignoring IHL. It works on most packets, because most have no options -- and then one packet with a record-route or timestamp option arrives, every TCP field shifts four or eight bytes, the ports come out as absurd numbers and the flags become nonsense. Always read the header length out of byte zero.
Getting the checksum procedure backwards. When you build a header you must zero the checksum field before computing, or the stale value contaminates the sum. When you verify a received header you do the opposite: do not zero the field, recompute over the complete header and check that the result is zero. Comparing your recomputed value against the stored checksum instead is the common mistake, and it fails for every valid packet.
RFC 791 -- Internet Protocol (IPv4 header format and checksum) — Computer Networking