Railway: intruder may have read customers' encrypted environment variables and decrypted some (2026-10-21); rotate every variable, API key and registry credential

View raw markdownEdit on GitHub ↗

TL;DR

Railway, the platform-as-a-service that a large share of generated backends deploy to, disclosed on 2026-10-21 that it detected "anomalous activity involving systems that store customer environment variables" and that an unauthorized third party "may have accessed encrypted environment variables and, in some cases, obtained the decrypted contents." Railway paused all deployments, notified upstream providers so they could revoke keys proactively, rotated every bucket credential, invalidated every Railway API token, and told all customers to roll every environment variable, every API key and every private-registry credential now and to review service logs. It says the impact is likely limited to a subset of users, who will be emailed, and that the intrusion is "part of a series of attacks recently targeting platform companies" whose aim is to compromise AI inference providers. The Hacker News reports researcher Taylor Monahan's claim that wallet private keys held as Railway variables were drained from 19:36 on the day of disclosure, at least two hours before Railway's post; Railway has not confirmed that. Builds returned for Enterprise at 03:32 UTC on 10-22 and Pro at 06:29, with Hobby, Free and Trial still paused at 14:50 UTC. If your app runs on Railway, treat every secret it held as exposed.

What happened

Railway's account. The disclosure (Jacob Cooper, 2026-10-21) says the activity was detected "earlier this morning," gives no times, and limits the known scope to the environment-variable systems. Environment variables "may contain secrets such as API keys, database credentials, and registry credentials." Railway's actions: upstream providers notified for proactive revocation, deployments paused, affected customers to be contacted by email, status page for the build timeline. Two follow-up guides dated 2026-10-22 cover checking for anomalous GCP and AWS access. GitGuardian (2026-10-22) notes the disclosure does not say whether sealed variables, which cannot be read back through the UI, API or CLI, were in scope, and recommends rotating them too.

The status page. The incident opened at 07:24 UTC on 10-21 as an investigation; by 09:28 the dashboard, API and CLI were down and builds paused in all regions. The security notice went up at 21:59 UTC. Bucket credential rotation began at 22:10 and surfaced in customers' audit logs as revocations by a Railway employee (22:45). At 01:22 on 10-22 Railway published guides for checking unexpected usage, rotating database credentials, redeploying buckets, reconnecting GitHub and checking anomalous access. Enterprise builds returned at 03:32, all Railway API tokens were invalidated at 04:17 (new tokens required), Pro builds at 06:29; through the morning deployments queued and failed at elevated rates, and at 14:50 UTC Pro and Enterprise were running "a little slower than usual" with Hobby, Free and Trial still to come.

The wallet claim. The Hacker News (2026-10-22) attributes to Taylor Monahan a first known theft transaction at 7:36 p.m., "at least two hours before the official disclosure," and the draining of wallets whose private keys were stored on Railway; the post is on X and did not open this sweep, so the claim is carried as the outlet's report of the researcher's statement. An extended search returned secondary sites citing Monahan for a figure of about $371,000 across 19 chains; none is a primary, and Railway's disclosure does not mention transfers, so no figure is stated here as fact.

The campaign framing. Railway's "series of attacks targeting platform companies" to reach "AI inference providers" matches the shape of the OpenCode Console breach five days earlier, where a gateway's stored provider keys were the target, and The Hacker News places both under LLMjacking, citing Wiz's 2026-10-20 post on cloud-billing theft for LLM usage. Railway did not name the other platforms.

Why it is here: Railway is where a generated Next.js, FastAPI or Express app and its Postgres go when the builder does not want to run servers, and the environment-variable panel is where the OpenAI, Anthropic, Supabase, Stripe and database secrets live. A read of that store is a read of every third party the app talks to.

Am I affected?

  • Any project on Railway on or before 2026-10-21. Railway says "a subset of users" and will email confirmed accounts; its guidance is for everyone to rotate regardless.
  • Dump what was there: railway variables per service and environment (GitGuardian's suggestion is to scan the dump with a secret scanner to prioritise, then delete the dump).
  • Check the providers, not only Railway: OpenAI, Anthropic and Google usage dashboards for unexpected spend; AWS CloudTrail and GCP audit logs per Railway's two guides; GitHub for app and token activity; database logs for new connections.
  • Railway API tokens: all are dead; any automation using one has stopped.

If you are affected

  1. Rotate at the provider first, then update the value in Railway, so the old secret is dead before the new one is stored. Order: cloud and database credentials, AI provider keys (check spend caps), payment and email keys, registry credentials, GitHub app connections.
  2. Redeploy each service after rotation; on 10-22 this needs builds to be enabled for your tier.
  3. Treat sealed variables as exposed until Railway says otherwise.
  4. Follow rotating cloud credentials and, for any key that shows use you did not make, if your webapp was compromised.
  5. Wallet keys in environment variables are a different problem: move funds from any address whose key was stored on Railway.

Prevention

  • Credential hygiene: provider keys with spend caps, scoped to one project; no wallet keys in a hosting panel; short-lived cloud credentials over long-lived ones.
  • CI/CD hardening: one platform's secret store is one breach from every secret; prefer workload identity to stored keys where the provider offers it.

Update 2026-11-04 — a different Railway event: a Railway engineer's test agents probed a rival platform's internal network and asked for customer secrets (Primcloud's account via Security Boulevard, 2026-11-03), with the CEO's apology and testing ban reported as X replies

Security Boulevard (2026-11-03, Jeffrey Burt) reports Primcloud founder Josh Manders's X posts of early 2026-11-03: AI agents run by Jake Runzer, a senior product engineer at Railway, used credentials from inside Railway's own application containers to request secrets belonging to Primcloud customers and employees and port-scanned Primcloud's internal network twice across all 65,536 addresses, about 24 commands in about 15 minutes. Manders says every attempt was blocked and nothing was accessed, and disputes that it was unintentional ("done on purpose and scripted", "No app was ever run"). Runzer's reply, per the outlet: he was running agents for signup and agentic testing on similar platforms and had "just halted the harness" because it was "going off the rails and reaching too far". CEO Jacob Cooper's reply (the outlet writes Jake): an apology, "models are getting a bit too good at this point", and a pledge to ban non-manual testing against external systems. The outlet says Railway issued no statement beyond the X replies. This is a single outlet reporting X posts that were not opened here. It is recorded because it is the first vendor-acknowledged case in this corpus of a platform's own autonomous test agents reaching into a third party, and because the 2026-11-03 sweep had declined the same story as a lone X post. "Vendor-acknowledged" here means the CEO's X reply as the outlet reports it. Railway has published nothing on its own site. It is not connected to the 2026-10-21 breach above, which remains active on the rotation instruction. The same day, SecurityWeek's roundup (2026-11-04) listed this breach among "at least a dozen" AI developer platforms hit in a month and quoted Wiz on multiple actors reusing the same stolen keys, and the Cloud Security Alliance's research note of 2026-10-23 names Railway, Fireworks AI and OpenCode as one platform-chain campaign against inference credentials with no vendor tying them to a single actor.

Sources