Blog

FB accounts created without Messenger

tfw you accidentally break FB for millions of new users

This was one of my first experience at causing a SEV. I’ve broken production before 🫣 but not bad enough for it to be a SEV. In this situation, millions of new FB accounts were created (over the duration of the outage) without proper messaging inbox thus unable to use Messenger. I’ll dive more into how this happens below. This is part of a series (Project Presentation) where I share stories of my past projects. ...

November 28, 2025

Incident Review Process

Let's make sure it doesn't happen again

In the previous runbook, I mostly focused on things to do during an incident as it happens. This week, we will focus more on what to do after the incident is resolved. This is an opportunity to look back at what went well, what went wrong, and how we can best prepare ourselves or prevent something similar from happening again in the future. This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices. ...

November 14, 2025

Leaked `FakeIncorrectPassword`

When your social media break turns into work

It was Thanksgiving day and I was mostly chilling at home scrolling Threads while watching YouTube. I then came across this post: @justinamphlett Found a pretty horrific dark pattern with Instagram. If you try to log in on mobile web, it fails with ‘incorrect password’. I double check… it’s right, it’s just not working. Gave up, installed the app, password is fine. Tried again today just to see… this time I get the following error message. Looks like a translation key is missing and it’s serving me the backend name for it? FakeIncorrectPassword ?? Are you shitting me? ...

October 31, 2025

The Myth of Technical Complexity

Why being skilled makes you look unskilled

I’ve seen multiple different people fail their promotions due to “lack of technical complexity” in their work. While some of this rings true, more often than not, the underlying technical complexity of their work is not properly understood because - ironically - they cut through it smoothly as strong technical engineers. Unless you were doing code / design reviews for them, you wouldn’t notice it. On the flip side, I’ve had projects listed as my capstone “technical complexity” project of the year that imo wasn’t very complex compared to some other stuff I needed to work on. Why is this happening and how do I make sure my work is properly valued? ...

October 17, 2025

Shipping a project with -2M MAP

Not everything is your problem

There’s a saying in Meta that ‘Nothing is somebody else’s problem’ but sometimes, it really is. At this point in time, I was working on a growth team whose top-line goal was to grow Facebook MAP (monthly active people). So naturally, it’s surprising for me to have shipped something that directly regressed the very metric the team should be growing. This is part of a series (Project Presentation) where I share stories of my past projects. ...

October 3, 2025

Communicating Effectively

Why empathy wins

It’s a common problem at work where you see a problem but you’re having a hard time trying to convince people to support your attempt at fixing it. Worse still, they might not even see it as a problem (or they don’t think it’s something that warrants any attention from anyone). I’m approaching this piece mostly from an “I need my leadership / partners to understand what I do is important” perspective. But the ideology can similarly be used on other occasions like selling an idea to your peers etc. Ideally you start thinking about this as you try to get the green light to pursue your project instead of attempting to justify it after you’ve already invested your time into it with little to show for it. ...

September 19, 2025

GitHub Action Runner Alternatives

Cheaper hosted 3rd party solutions

A few months ago, I wrote about my attempt and reasoning behind managing my own fleet of self-hosted GitHub Action Runners. I’ve since learnt that there are quite a few hosted alternative runners that are both faster and cheaper than GitHub’s default runners. This article is just a quick summary of what I’ve found. Linux Runners Comparison # Provider Free Tier Paid Pricing (2vCPU) Performance Special Features Blacksmith 3,000 minutes/month $0.004/min (50% of GitHub) 2x faster builds, 4x faster cache Unlimited concurrency, gaming CPUs BuildJet $5 one-time credit $0.004/min (50% of GitHub) 2x faster (variable CPU models) 64 AMD + 32 ARM concurrent vCPUs, 20GB cache CarbonRunner ❌ None $0.006/min (25% cheaper than GitHub) Standard performance 90% lower CO2 emissions, 42 regions Cirrus Runners ❌ None $150/month unlimited 2-3x faster performance Fixed pricing, no minute limits, limited concurrency Depot ❌ None $20 for 20,000 minutes, then $0.004/min 30% faster, 10x faster cache Docker-focused, per-second billing, unlimited cache GitHub 2,000 minutes/month $0.008/min Baseline Official support Namespace ❌ None $0.003 Fastest x64 performance 250GB storage, SSH access puzl.cloud 400 vCPU-minutes/month $0.0008/min (€0.000008/vCPU-sec) 2x faster, 5x cheaper Real-time per-second billing, 12vCPU per job Sprinters.sh ❌ None Per-job pricing (10x cheaper) Standard performance Self-hosted in AWS, per-job pricing RunsOn Free for non-commercial €300/year + $0.001-0.003/min (AWS costs) 30% faster, 90% cheaper Self-hosted in AWS, GPUs, Windows Ubicloud 1,250 minutes (~$1 credit) $0.0008/min (10x cheaper) Newer CPUs, good performance Open source, Hetzner hardware WarpBuild ❌ None $0.004/min (50% of GitHub) 2x faster, 50% cheaper Snapshots, unlimited cache macOS Runners Comparison # Provider Pricing Hardware Performance Key Features Cirrus Runners $150/month unlimited M4 Pro (latest) 2x faster performance Fixed pricing, no minute limits, limited concurrency GitHub $0.08/min (10x multiplier) Intel Mac + M2 Pro XL ($0.16/min) Baseline Official support Namespace $0.09 M4 Pro + M2 Pro Fastest available SSH/VNC access, bleeding-edge images WarpBuild $0.04/min (50% cheaper) M2 Pro 25% faster, 50% cheaper Pay-per-use, macOS 13/14 support Widget # I don’t have a better place to put this but I built a very basic widget that takes into account of the “supposed” faster speed (leading to less action minutes) with each of their rates so you can find out which platform fits you best based on your usage (with GitHub’s Action Runner minutes as baseline). ...

September 2, 2025

Hidden Privacy / Security Pitfalls

Sometimes good coding practice isn't good enough

Most software engineers think of this as “someone else’s problem” (usually a privacy engineer or security engineer), but realistically, there’s only so much stuff (bad code lol) they can catch. Especially if you work in some of the more sensitive areas, it’s not realistic to have them do code audits for every change. This means that you as the code author (or reviewer) need to at least pick up some basic instincts about things that can pose risks. ...

August 23, 2025

Sync git submodules with GitHub Action

I needed to share some object code across multiple repos (to ensure the consistency of the RESTful API JSON serialization and deserialization). So I created a dedicated repository where all the shared code lives, then imported it across other repos as a submodule. In this case, I used a library I made - wings - but usually you’d probably use something like Thrift or Protocol Buffers. Another common use case would be to have a repository hosting shared cross-platform (iOS + Android) low-level code. ...

August 8, 2025

Firefighting Heroes

Symptoms of an unhealthy reliability system

We love celebrating heroes in general. So when a hero shows up to save the day, it’s only natural that we make sure to recognize their contribution. While it might feel counterintuitive, we should strive to not need heroes entirely instead of hoping that next time when tragedy strikes, a hero will turn up once again. This is not to diminish the value of heroes. They still play a very important role in firefighting and are doing the right thing in an urgent situation. However, in a perfect world, they shouldn’t be necessary at all and we should at least aspire to work towards building a system with that level of reliability. ...

August 1, 2025