Fly.io patched and rebooted its whole fleet for Linux kernel CVE-2026-99999 (guest-to-host checksum bug): rotate credentials sent in cleartext 2026-10-15 to 17

View raw markdownEdit on GitHub ↗

TL;DR

Fly.io's post "Well, That Wasn't Fun: October 16 '26" says the Linux kernel bug CVE-2026-99999, which lets a guest VM feed bad checksum-offload metadata to the host's IPv4 stack, bypass the host firewall and read host memory, forced it to patch and reboot every host in its fleet on 2026-10-16, about ten hours of rolling reboots with each host down 15 to 20 minutes. Fly.io says it has no evidence any customer data was exposed and believes exposure is unlikely, but that a malicious app on the same physical host could in theory have read small pieces of other apps' traffic between 2026-10-15 and 2026-10-17, so it asks customers to rotate any credential sent over HTTP or another unencrypted protocol in that window, including over the 6PN private network. The bug was found by pwn.ai during Vercel's sandbox bounty with an AI-assisted harness; the kernel fix merged on 2026-10-14 and the CVE was published 2026-10-15.

What happened

The bug. CVE-2026-99999 (CNA: Linux, published 2026-10-15, CVSS 3.1 9.3 in the CNA record) is "ipv4: validate checksum_start before completing checksum": "If a packet with bad checksum metadata gets into the ipv4 stack, skb_checksum_help can corrupt the network header and cause a bunch of mischief. This was discovered and reported by Paulos, and has been reproduced by others independently since." The record lists the affected range from kernel 2.6.27 and names the fixed versions as 5.10.272, 5.15.223, 6.1.190, 6.6.159, 6.12.113, 6.18.57 and 7.2.11 on the stable branches and 7.3 for mainline. pwn.ai's write-up (Paulos Yibelo, "PwnEscape: Guest-to-Host KVM Escape") describes a guest controlling two bytes of an IPv4 header the host has already trusted, which gives a repeatable 32-byte host-memory disclosure per attempt and, in the researchers' own KVM test setup only, selected 64-byte host-kernel writes and a bounded host-root helper execution. The research ran under Vercel's sandbox bounty (announced 2026-08-18), the root cause went to the kernel security contact on 2026-09-04, Michael S. Tsirkin reproduced it on 2026-09-07, the revised guard was accepted into net.git on 2026-10-08 and merged to mainline on 2026-10-14. The page says the harness was pwn.ai's experimental Omega-Pwn with GPT-5.6 Sol as the primary reasoning engine, that humans led the work, and that it describes no production exploitation. It names Vercel Sandbox as the platform the bounty covered, says Vercel notified other hypervisor and sandbox providers, and names no other provider as affected.

What Fly.io did, per its post. The post is headed "October 16, 2026" and describes events through 2026-10-20; the HTTP page prints no separate publication date, and the post reached Hacker News on 2026-11-06 (item 50397310, "Fly.io Got Breached", 9 points), where commenters objected that the submitter's title asserts a breach the post does not. Fly.io's timeline: 2026-10-15 morning, the kernel CVE team and pwn.ai announce the bug; 2026-10-15 night, Fly.io learns of it and starts mitigations; 2026-10-16 16:22 UTC, an emergency-maintenance notice goes on the status page, described as maintenance without naming the CVE; 2026-10-16, a fleet-wide patch-and-reboot that took about ten hours, each host down 15 to 20 minutes, single-Machine apps down for that period regardless of sequencing, Managed Postgres rebooted in waves so no cluster lost both nodes (17 waves in iad); hosts that got stuck in BIOS, needed reinstalls, or in one gru case lost a power supply stayed down until 2026-10-20, when their Machines were restored. "No Fly Machines or customer data were lost."

What Fly.io says about exposure. The post says Fly.io found no evidence of customer data exposure and "we believe it's unlikely that any of our customers' data was actually exposed", and then states the theoretical case: a malicious app could have read small chunks of other apps' traffic on the same physical host during the window. The ask: "If you ran on Fly.io between October 15 and 17: rotate any credentials you sent over HTTP or another unencrypted protocol in that window", which it says includes unencrypted traffic over 6PN. The post describes Fly's architecture as strong hardware virtualization with an XDP packet filter, and explains that the bug sits below both, in the host kernel's handling of checksum offload metadata that the virtual NIC passes up. The status-page history (read 2026-11-07) lists maintenance entries by weekday without a dated entry this sweep could match to 2026-10-16.

Why it is here. Fly.io is a common deployment target for generated apps, and Managed Postgres is where many of them keep their database. The exposure Fly.io describes is cross-tenant: another customer's app on the same host, not an external attacker. The post's own security advice is the part aimed at this corpus's readers: "Give agents credentials scoped to the job, and make them expire", keep credentials away from agents, and treat content agents read as hostile. The bug also sits under every KVM-based sandbox product in this corpus's agent-sandboxing files; Vercel's statement that it notified other providers is the only cross-provider fact either primary gives, and no other provider's post was found this run.

Am I affected?

  1. Were you on Fly.io between 2026-10-15 and 2026-10-17? If yes, list every credential your app sent in cleartext in that window: HTTP (not HTTPS) calls to other services, plaintext database connections, Redis or other protocols without TLS, and anything over 6PN that you did not encrypt because the network was private. Fly.io's ask covers 6PN explicitly.
  2. Managed Postgres. Connections to Managed Postgres over 6PN without sslmode=require are inside the ask.
  3. Downtime. If a Machine was unreachable for 15 to 20 minutes on 2026-10-16 or stayed down until 2026-10-20, that was the reboot or a host that failed to return, per the post.
  4. Other hosts. If you run your own KVM hosts for agent sandboxes, check the kernel against the CNA record's fixed stable versions (5.10.272, 5.15.223, 6.1.190, 6.6.159, 6.12.113, 6.18.57, 7.2.11, or mainline 7.3); the fix is "ipv4: validate checksum_start before completing checksum".

If you are affected

  • Rotate the credentials identified in step 1; rotating cloud credentials covers provider keys and database passwords. There is nothing to clean up on the Fly.io side; the fleet was patched on 2026-10-16.
  • Move every internal connection to TLS so the next host-level bug has nothing to read. Fly.io's post makes the same point.

Prevention

  • Encrypt inside the private network. 6PN, VPC peering and service meshes are isolation claims, not confidentiality guarantees; a host-kernel bug sits below them. Use TLS to the database and between services.
  • Scope and expire agent credentials. The post's advice matches agent sandboxing and credential hygiene: a credential an agent holds should be the smallest one that does the job, with an expiry.
  • Track the sandbox vendor's kernel line. A KVM sandbox product's security depends on the host kernel it runs; ask the vendor which stable branch it tracks and how fast it reboots.

Sources