
Founder @Railway, the peaceful way to ship software. People, Product, Production.
SF via 🇨🇦
Joined June 2012
- Tweets21.4K
- Following1.3K
- Followers34.3K
- Likes36K
Today, we were alerted of malicious activity that compromised Railway's servers
We are sharing this with you broadly and swiftly. We recommend folks rotate their keys out of an abundance of caution.
An urgent notice from the Railway team
blog.railway.com/p/security-…
All deploys have now been restored
Affected customers will finishing receiving emails in the next hour or so
Finally, out of an abundance of caution, we will be posting the post mortem around 9am PST tomorrow
Hobby builds have been re-enabled as of 30 minutes ago
Signups, free builds, and trial builds remain disabled for now. We hope to re-enable the latter two later this evening.
Post mortem remains on track for this evening
Builds have been humming for the last 12 hours on Pro and Enterprise
Due to our stance on broad vs narrow notification, we're also scaling to meet need, and aiming to restore all other tiers over the next few hours into the evening
First pass post mortem also later this evening
As others have pointed out, "humming" isn't exactly correct
Early periods were definitely more stop and start. The last few hours have been more smooth
Hobby now opening back up
General Updates: status.railway.com
Builder Specific: status.railway.com/incident/…
While we believe our definition of impact is quite narrow, we once again aired on the side of notifying quickly and recommending rotation
We understand people are, for a lack of a better word, pissed/disappointed/etc
Working 24/7 in shifts through this. Appreciate the patience
We've begun the process of ramping builds up
We'll continue this into the evening, overnight, and into the morning. Tomorrow we will include a full post mortem
We once again want to apologize for any inconvenience
Builds are temporarily paused
We are currently investigating this issue
Status: Investigating status.railway.com/incident/…
Psychotically good marketing
Ramp remains in a league of its own
Aspirational. We need more media/culture/resonance.
If you can build real life wall hacks
You can build real life single frame aimbot
Much to think about
you don't even have to be in the same room
with the right Wi-Fi setup, someone walking behind brick walls can still leave a measurable trace at the receiver
the graph uses real Wallhack1.8k recordings: orange is walking, blue is an empty space, both on the same scale. the 3D scene illustrates the idea
if your next thought is “i'll just sit still,” watch the quoted post😁
We’re rapidly approaching “There’s no reason to use anything else” because there are no tradeoffs
I benchmarked 5 object storage providers from 6 locations around the world
I included:
- @awscloud S3
- @Cloudflare R2
- @Railway Buckets
- @Backblaze B2
- @neondatabase Storage
It used Files SDK to make it as fair as possible
Here's a summary 👇 (this is also a thread)














