Railway breach: rotate secrets

Railway's security disclosure, dated October 21, begins: "Earlier this morning, Railway detected anomalous activity involving systems that store customer environment variables." Its investigation to date indicates that an unauthorized third party may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents. Railway notes that environment variables may contain secrets such as API keys or database and registry credentials. It says it believes this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers, and that it has notified those upstream providers to revoke all keys proactively.

The disclosure says Railway's initial analysis suggests the impact may be limited to a subset of users, but that Railway recommends all users act as a precaution while it completes its review: immediately roll all environment variables stored in Railway, including any API keys, as well as any private registry credentials, and review the logs for those services for unusual activity. As Railway confirms which accounts were affected, it says it will contact those customers directly by email with additional details. At the time of the post, all deployments were paused while Railway investigated, so new versions could not be rolled out until builds were restored.

Railway's status page linked the disclosure at 2:59 PM PT on October 21 and said deployments remained paused, including for Enterprise customers. Later updates on the same page say:

• At 3:10 PM PT on October 21, Railway said it was rotating all bucket credentials, that new credentials would appear automatically in the dashboard, and that services would need a redeploy once builds resumed. At 3:37 PM PT, it said rotation was continuing. At 5:22 PM PT, it said it was still working to restore builds and would publish guides shortly.
• Railway linked its own step-by-step guides for rotating PostgreSQL, MySQL, Redis, and MongoDB credentials, for redeploying buckets and reconnecting GitHub after the credential reset, and for checking OpenAI, Anthropic, OpenRouter, Grok, and Stripe for unexpected usage and GitHub, AWS, and GCP for anomalous access (6:22 PM PT, October 21).
• Builds came back for Enterprise workspaces at 8:32 PM PT and for Pro workspaces at 11:29 PM PT on October 21.
• Railway invalidated all Railway API tokens. A token used with Railway's Public API no longer works, and you need to create a new one (9:17 PM PT, October 21).

Builds came back on every plan on October 22, and on October 23 at 3:06 PM PT Railway marked the incident resolved, saying builds, deployments, the Railway API, and the dashboard were operating normally, with new signups still temporarily paused. The status page has the current state.

Railway's preliminary report, dated October 22 and written by Jacob Cooper and Angelo Saraceno, says its forensic investigation is still ongoing and that a follow up will come as the investigation concludes. Its main points:

• When. Railway believes the attack took place "from October 20th 13:58 UTC to October 22nd 03:32 UTC". That is 6:58 AM PT on October 20 to 8:32 PM PT on October 21. The report says the final timestamp is when Railway confirmed the attacker no longer had access to any internal systems. So the attack started the day before the October 21 disclosure was published.
• What the attacker wanted. The report says the attack was "attempting to obtain AI provider API keys to perform distillation attacks".
• How many workspaces. Railway says "at most 1.85% of all workspaces had some environment variables potentially compromised", which it puts at "about 100,000 workspaces". Its timeline describes an unauthorized query of encrypted environment variables, "of which a subset were decrypted". Railway says it was unable to know which variables specifically were decrypted.
• Who Railway is contacting. The report says Railway has notified the customers that were directly impacted, and is also reaching out to customers it considers high risk, a group larger than the 1.85% of workspaces. On October 23, the status page said Railway had finished notifying users whose workspaces met its criteria for increased risk, and that workspaces outside those groups are not confirmed safe from compromise.
• Vendors. On October 21, Railway notified vendors, and "Vendors were requested to revoke all keys for their service". The report names GitHub, GitLab, Anthropic, OpenAI, OpenRouter, SpaceXSI, and Stripe among the vendors it worked with to revoke the potentially compromised keys. It says the number of users hit by those vendor reset requests was larger than 100,000, and that a revocation notice from a vendor does not by itself mean your key was impacted.
• Accounts and source code. The report says: "The attacker did not access Railway user accounts as part of this incident." On source code, Railway says it is "not ruling out that they accessed user source code" and will confirm in its final forensic investigation report.
• What Railway recommends. Railway recommends that "all customers who signed up prior to October 21 rotate their keys stored on Railway". Its list of recommendations for all customers is to remove public network access to databases, rotate any sensitive variables and keys, check GitHub account audit logs and review all app access, and check audit logs in registries such as Docker Hub. Each one is in the steps under How to fix.

You may be. Railway's preliminary report says it has notified the customers that were directly impacted, and is also reaching out to customers it considers high risk. It also says it was unable to know which variables specifically were decrypted, and it recommends that all customers who signed up prior to October 21 rotate their keys stored on Railway. Treat every secret you kept in a Railway variable as possibly read, and rotate it, whether or not an email arrives.