BinHong Lee's Blog

Shifting from Code Writing to Reviewing

binhong@binhong.me (BinHong Lee) — Thu, 02 Apr 2026 00:00:00 -0800

When I built GlobeTrotte last year, vibe-coding had just begun gaining traction but I was the “weird guy” who coded everything by hand. I spent a little over 4 weeks building the backend service (Go), iOS app (SwiftUI), Android app (Jetpack Compose), and web (TanStack) all by myself. I like to think that’s pretty fast for an app of that level of complexity. The tradeoff here however, was that I have almost no time to “handle the business side”.

I wrote about my early skepticism on AI coding back in July last year. TLDR; I tried getting Windsurf’s SWE-1 to refactor navigation on the Android app but ended up watching it spiral into compiler errors after compiler errors as it tried to “code its way out” by piling on even more code. At the time, I concluded that vibe-coding wasn’t ready but one thing I missed was how much the model mattered.

The inflection point #

Building slogx and Git Navigator was almost an entirely opposite experience. slogx was started because I was curious to explore the capabilities of Google’s AI Studio. It definitely impressed me in terms of building good looking UI but its lack of support for anything else was a bottleneck that quickly prompted me to move away (especially to build the SDKs). Git Navigator was started because I wanted to see how well Google’s Antigravity and Claude Sonnet perform. For pretty much an entire month, all my vibe-coding prompts were basically “the graph rendered wrongly in this / that case”. (I probably should’ve intervened earlier but I was lazy and curious if it can eventually figure it out.) Neither could, but Claude Opus (4.5) managed to build an actual working version. I redid the algo anyway eventually with a much simpler / straightforward idea (just DFS lol), but the fact that it worked when others didn’t, makes this an important inflection point to me.

Dev Velocity #

From here on out, most of the code for both slogx and Git Navigator was written by an LLM. I do review every single line of them but that doesn’t always say much especially for areas which I’m not already an expert of. It definitely feels like it’d be faster than if I were to build it myself but I haven’t done actual testing to know for sure. My suspicion is that it’s faster in the first few days, maybe the first week. One month in, I can’t tell if I’m meaningfully ahead of where I’d be otherwise. The LoC isn’t dramatically higher than GlobeTrotte, but because I didn’t write this code, I’m significantly less familiar with my own codebase. That said, I can definitely see the development cycle being significantly faster if I don’t actually do line-by-line code review and instead stick to just vibe-reviewing(?) it whenever I see something kinda sus.

Context switching tax #

Another important shift I noticed is the nature of work itself. Before, you’d be “wired in” while designing then coding where you would do deep focus work while holding the entire problem in your head. Now though, the prompting-and-waiting breaks the chunk where you either scroll social media while waiting or review and prompt a separate session in the meantime. However, this means that you’re constantly context switching between 2 or more project / features and you’re less of a maker but more of a manager making sure your “subordinates” complete their deliverables.

Visualizing the product for LLM #

One of the trickier things that took me too long to figure out was visualization. Like how humans need to look at the end result (print debugging, hot reload etc.) to easily self-correct and understand what went wrong, LLMs need something like this too. If you’re building a webapp, some providers have a Chrome extension or a browser built-in for this purpose. Since I’m building a VSCode extension, I actually didn’t find much tooling around it (ironically lol). When I first made all the different coding agents build the graphing UI for Git Navigator, I spent a lot of time screenshotting and explaining what’s right / wrong with it.

Eventually, I figured out to make it write a script that it can run against a given folder / repo to see how the graph is sorted. Since the rendering code would order the commits (in TypeScript objects) before passing it to be rendered as UI, the script essentially just dumps the order output for easy viewing by the LLM. This cuts me out from needing to repeatedly screenshot it as LLMs can run the script, then read the output and identify if things are ordered correctly or if there’s anything that needs fixing. I think this is probably one of my biggest lessons here, this is like an observability tool except your target audience are LLMs instead of human developers.

Catching shortcuts #

There’s been a few times where I caught the model taking shortcuts that technically worked but were obviously wrong. One time, I asked for a complex git operation (some rebase gymnastic) and Opus 4.5 just used the TypeScript backend to create a temporary bash script and run that, instead of running it programmatically through the TypeScript backend. It worked fine locally on my machine (and any Unix based system I think) but it would break VS Code Remote scenarios where files aren’t local, or on Windows because path separators are wrong. It’s the kind of solution that passes every test you thought to write yet falls apart the moment a real user touches it. This is where review actually matters since LLMs (largely) optimize for “does it work right now” rather than “is this the right solution”. If you’re not catching these, you’re accumulating tech debt at LLM speed.

Catching product gaps #

There are also times where the feature exists and works, but the placement or interaction is off in a “product sense” kinda way. One example I had was asking Codex to implement line and hunk staging for Git Navigator. We had some good planning discussions about the tri-state checkboxes, “include/exclude” wording instead of git jargon, inline diff expansion but, it shipped the entire picker in the side panel. This makes for a weird UX ergonomics because users are looking at the uncommitted changes block when they’re deciding what to commit. So burying the granular staging controls in a separate panel means most people likely wouldn’t even notice that it exists. I had to ask for it to be moved into the uncommitted changes block itself. Here’s another twist, when it did, the LLM’s first instinct was to build a new diff renderer from scratch instead of reusing the existing one used for conflict resolution. I think this is probably the bull case for Product Managers being the beneficiaries of vibe-coding considering that this is likely their strong suit.

Delete everything and try again #

Fortunately, I haven’t had to rely on this strategy too much but it does happen. I think when building one of the stack features (on Git Navigator), Codex completely misunderstood what I meant and kinda just went off the rails despite being given clear specs to follow. I discarded all the changes (thanks git), started another new session, gave it the same spec, then just let it try again. Funnily enough, it worked perfectly on the second run, only needed some minor tweaks before the feature is ready to be committed. I haven’t had to resort to this too much (and I don’t keep good enough git hygiene to always rely on it 🫣) but it’s definitely something to keep in mind if things go weirdly wrong.

Wrap up #

I don’t think I’m going back to handwriting all the code (for now). Not because I’m convinced it’s faster, but because it feels faster. I think it also comes with a level of detachment that makes me feel more comfortable for being ruthless with code I didn’t write while nitpicking for the ideal product experience that I want. It makes it easier to throw things away, cherry-pick only what’s good, and not get precious about any of it. Whether that’s a healthier relationship with code or just a different kind of laziness, you tell me 🫠.

Mentorship

binhong@binhong.me (BinHong Lee) — Fri, 20 Mar 2026 00:00:00 -0800

I’ve done quite a bit of mentorship from short term (one session) to long term (multi-year), from high school kids to university grads or even senior engineers, so I like to think I know a thing or two about mentorship. This piece is specifically geared more towards how one should approach a longer term mentorship as a mentor.

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Be Transparent #

Putting this first because I personally think this is the most important. Be upfront and transparent about yourself and your advice. If you aren’t sure of something, be sure to make it clear to your mentees. Generally, you’re likely a person with authority, which is why you’re the mentor in the first place. Giving out half-baked advice would at best ruin the trust your mentees have toward you, at worst cause catastrophic career derailment to your mentees as they follow your half-baked advice. The easiest way to be transparent is to clearly share your reasoning behind arriving at the given conclusion, allowing them to raise questions or doubts if they have a differing perspective.

Understand their goals #

This should be pretty straightforward. Not everyone tries to optimize for the highest pay or climbing the career ladder; some try to maximize learning opportunities or better work life balance. Knowing this helps you preface the advice you can provide and to help them work towards their goals. It’s also important to explain the tradeoffs of your advice: “this is a great opportunity for you to learn more but there’s always risk in working on something you’re unfamiliar with and not meeting your work expectations”.

Everyone’s a little different #

Tangential to the previous point, not everyone has to be like “you”. Generally, if they bring you a sticky situation, it can be useful to provide your perspective to help them come up with their resolution to it. But otherwise, you shouldn’t just prescribe the exact same thing to all your mentees because everyone’s a little different. The advice they receive should play to their strengths while matching their personal goals. Both of which are likely different from your own—tailor it to their needs.

Guest Lecture #

In university classes, your professor might sometimes invite different industry experts to be a guest lecturer. A similar idea applies here. Sometimes your mentee needs help in an area where you’re not an expert. You likely know someone who’s good at it. If your mentee knows them, recommend they reach out to them; if not, offer to introduce them. (This applies to yourself as well. If you need help in an area you know someone is an expert in, you should reach out to them for advice.)

(Optional) Dig for details #

I labeled this as optional because most people see it as the responsibility of the mentee rather than the mentor. (“I can only help those who want to be helped.”) I can see it both ways so it’s up to you to draw the line. Some mentees aren’t always very direct with their problems. They might even say that everything is going fine. But if you just poke a little, they would start sharing potential issues they foresee could materialize. Generally, this would be a pattern where they would consistently be like this, so you always need to probe a little to find out any underlying problems they might be thinking of.

Wrap up #

Mentorship isn’t about molding someone into a younger version of yourself but rather to help someone become the best version of themselves. The advice that worked for you might not work for them, and that’s okay. Be transparent about what you know and don’t know, understand what they’re actually trying to achieve, and don’t be afraid to admit when someone else would be better suited to help them. At the end of the day, good mentorship is less about having all the answers and more about asking the right questions and connecting people to the right resources. Give them what they need, not what you have.

Securing Logged Out Internal Settings

binhong@binhong.me (BinHong Lee) — Sun, 08 Feb 2026 00:00:00 -0800

I proposed, built, and maintained the only logged out accessible tooling suite in Meta - which I’ve conveniently called “Logged Out Internal Settings” (hereafter as LOIS). This toolset is still used daily today by employees though it’s no longer as well maintained since I left the company earlier last year.

This is part of a series (Project Presentation) where I share stories of my past projects.

Background #

At this point, we were still building out a brand new set of access flow that unifies the experience (for login, account recovery, registration) across Facebook, Instagram, and Messenger on both iOS and Android. Since the whole thing was internal anyway, people were littering internal-only information everywhere for easier debugging and faster development cycle. Between all that, I built a small tool on the landing page that let engineers quickly switch between different dev pods or prod, and another prototyping tool that lets you add any random UI for quick testing. As we got closer to release, I began thinking how to make this “launch safe”.

Problem #

On a basic level, I set up a simple gating where you can only access all the internal-only stuff if you’re already connected to a dev pod. It works fine from a security perspective since there are a bunch of different checks you need to pass before you can be connected to dev pods. But this severely limited the exposure of it since employees not already connected to a dev pod can’t easily utilize the tool to switch into one.

Scenario	Before	After
prod -> dev pod	✅	❌
switch dev pod	✅	✅
dev pod -> prod	✅	✅

I thought this wasn’t an ideal setup because:

prod -> dev pod was one of the most popular use cases
dev pods are generally slower so other tools should ideally still work without it

But the main problem was that all these need to exist before you login to the app so we can’t tell exactly if the user is an employee (or not).

Ideation #

I started with a simple security layer idea like this. Not everything is “super sensitive” and requires the highest level of security check to be accessible. Something like “show debug info” that prints out information about the device / app isn’t exactly sensitive while still extremely valuable for debugging. (Keeping it away from showing to users prevents unnecessary confusion.) This also allows for progressive increments in security access pass to gain more access. For eg. you need to be on corpnet / VPN to access the dev pod connector; once connected, you’re now “promoted” to dev pod access level.

Security layer design

At this point, I started thinking of an alternative to “dev pods” for the highest level access. Some job functions (like QA, design) can benefit from this tool as well but they don’t usually have access to a dev pod. I quietly put together an idea for an access flow similar to how TOTP works. I call this the “Employee Authenticator”. More on this later.

Security #

I found an internal discussion group on general security stuff and asked a few clarifying questions surrounding how each of these checks can / should work and what potential risks it can carry. This helps me validate the idea while also learning about some weird past incidents. Apparently there was once an incident where a public coffee shop Wi-Fi below SF Meta office was falsely flagged as corpnet because some employee gave them the wrong router for setup.

Anyway, after gathering good context and information around it, I filed for a formal security review. The review went smoothly as we received the green light to proceed.

Opportunity #

Our org decided to host its first (rolling?) hackathon where you can form teams and build things (that you wouldn’t otherwise work on) over a given week. I polished up my proposal doc for Employee Authenticator then recruited a few engineers around me to help build it.

Here’s how it works:

Employee Authenticator workflow

While we didn’t win anything from the hackathon, it helped kickstart development of this tooling with the idea that LOIS isn’t just for devs, but also for non-devs. I had to later get it re-reviewed because some roles (like QA) are mostly filled by “Contingent Workers” who can’t be classified as employees legally(?) so I also had it renamed to “Internal Authenticator”.

Note: I never got through with this process entirely so if you’re a Meta employee looking at LOIS Authenticator, it’s likely all over the place since no one really took over ownership of LOIS after I left.

Follow-ups #

As I’ve mentioned at the beginning, some of these internal features already existed but they had no protection and we needed to secure them before we began testing the new flow publicly. One of the minor but important tweaks I made was to add a text label on the menu screen saying “connect to {corpnet / VPN | dev pod} to access more features”. Since engineers who’ve been working on this flow have been using these underlying features day-in-day-out, suddenly seeing them go missing can be a bit confusing. That small text saved countless hours of “Hey BinHong, I don’t see X feature anymore in LOIS.” while quickly nudging them towards the right direction to self-unblock.

Takeaways #

This is an example project where I should have advocated harder for myself to gain better visibility within my org (where my performance review is done). While security teams were happy with our collaborations and many engineers (both within and outside of the org) benefitted greatly from the continued existence of LOIS, I doubt many in the management rank understood the importance of the work and underlying value it provides to engineers. From the users’ (employees’) perspective, I added some layer of friction to access the same tooling; while the value it adds was to ensure we can keep this set of tooling post-public-launch.

The other interesting side effect of the security layer was that it helped build a proper structure around the entire LOIS system. This paved the way for an eventual platformization allowing for each team to build customized toolings into LOIS.

The Peak Bias

binhong@binhong.me (BinHong Lee) — Sun, 25 Jan 2026 00:00:00 -0800

One of the most interesting examples is to look at any average person debating whether Jordan or LeBron is the GOAT. They will inevitably bring up Jordan’s 6-0 finals record as if Jordan only played in the NBA for 6 years. LeBron’s 4-8 record on the other hand is somehow worse than if he never made it to the finals at all (which frankly doesn’t make sense lol). This isn’t really about basketball though. It’s about a fundamental bias in how humans evaluate performance where we disproportionately value peak moments over sustained consistency, even when the consistency represents equal or greater total value.

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

One big project > many small projects #

Similar to “peaks”, humans have a tendency to veer towards one “big, complex” project over a bunch of smaller projects even if they require the same level of effort and expertise. We’ve all seen this happened one way or another. The person who ships one new flashy dev tool get rated as “exceed expectations” while the person who quietly shipped ten incremental performance improvements (that collectively saved more engineering time) gets a “meets all expectations”. A single monolithic launch gets remembered. Ten incremental wins that shipped over 6 months are just blurred together in everyone’s memory by the time performance review season comes around.

The invisible work trap #

This creates some really annoying incentives tbh. Nobody remembers the person who built great monitoring that prevented incidents (the incidents that never was) but everyone remembers the hero who stayed up fixing the 3am production fire (even if better testing infra could’ve prevented it in the first place, but we don’t talk about that lol). The reality is that sustained consistency is harder than occasional peaks. The person who shipped features with good code quality that never breaks in production through consistent diligence gets less credit than the person who firefights a handful of dramatic incidents (even if it might be due to their own lack of care towards shipping). Both require effort, but only one gets remembered as a “peak” moment during performance discussions.

Storytelling as the workaround #

You’ll hear this advice everywhere from performance review packages to design docs to business proposals. Literally any “presentation” type stuff will tell you that storytelling matters. Storytelling is basically the skill of bundling up all your smaller projects into one big narrative to combat this bias. Instead of “shipped 10 performance improvements” you write “led reliability initiative that reduced P95 latency by 40% through systematic optimization.” Same work, just packaged differently.

The uncomfortable truth is that impact without visibility might as well not exist come promo time. The people who are good at this don’t just list what they shipped but create a whole story arc like “Problem → Investigation → Solution → Impact → Future Work.” They connect the dots to show how their smaller projects collectively solved something bigger. This used to feel really inauthentic to me (and honestly sometimes it still does) but it’s a valuable skill to learn because making sure people understand what you did is part of the job. In my opinion, this is also the defining character between a good manager and a great manager.

Hot take #

Here’s the controversial part. I don’t actually think this bias is completely irrational. Peak performance (when it’s genuine) does signal important stuff. An engineer who can architect and ship a complex multi-team migration is demonstrating coordination and ambiguity navigation skills that not everyone has. These are real senior+ skills that are hard to show through smaller projects. The problem isn’t that we value peaks but that we overvalue them relative to consistency. We’re also terrible at telling the difference between genuine peak performance and just being in the right place at the right time (or being good at taking credit for other people’s work lol). Sometimes the person grinding out consistent wins is actually showing way more skill, just in a less visible way.

What to actually do about it #

If you’re someone who tends toward consistent, incremental work (which is totally valid and valuable btw), here’s the reality. You need to adapt your strategy if you want recognition. The most effective move? Take on longer running projects. If you know you can consistently ship quality work, use that as a strength by committing to bigger initiatives that span months instead of weeks. Fortunately, a big long-running project basically counts as a “peak” in terms of how it gets perceived and remembered.

The key is you’re not actually changing your working style since you’re still executing with the same consistency. You’re just changing scope. Instead of 10 small improvements over 6 months, you’re doing a 6-month initiative that involves 10 components. Same work, same consistency, different packaging. Other stuff that helps include actively communicate about your progress (shameless plug for my previous piece on this), bundling your incremental work into themes in perf reviews, and actually quantifying cumulative impact across related projects. You’re not trying to trick people into thinking your work is more valuable than it is but making sure they understand the actual value of work that’s easy to overlook (which sometimes means restructuring how you take on work in the first place).

But I kinda just do whatever #

Some people are “utility players” who pick up whatever project with the highest priority at the time. Usually there isn’t too much “narrative” surrounding “why do x” aside from some vague definition of it being the most urgent work at the time. In this case, you’d rely heavily on documentations and endorsements by the more senior / authoritative voices in the company to make your case. This sometimes also mean that having a great manager is more important (than otherwise) when it comes to building your case for performance review. The safe way is to pick a “theme” and be more intentional about the projects you commit to. Going without is viable but harder, since you rely heavily on others to build your narrative for you.

Wrap up #

The peak bias is real and it affects everything from sports debates to software engineering careers. We remember the dramatic moments and forget sustained excellence where the person who shipped one big project gets promoted over the person who shipped ten valuable smaller ones. You can’t fight human psychology but you can work with it. If you’re the consistent incremental type, consider taking on longer projects that let you leverage that consistency at scale.

You Can Be Both

binhong@binhong.me (BinHong Lee) — Fri, 12 Dec 2025 00:00:00 -0800

I recently saw a post arguing that exceptional people are rare because they’re so focused on their craft that they skip “bureaucratic bullshit” and “often left other skills to rot”. I left a comment explaining that it’s a 50/50 situation. In this piece, I’ll expand my take on this more in depth. Exceptional people comes in different shapes and forms. Pretending that someone can’t be exceptional simply because they learned to deal with the “bureaucratic bullshit” feels like a weird stereotype. (If anything, this reads like the exact type of mentality that gave birth to the “10x engineer” meme lol.)

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Understanding “the game” #

There’s this weird stigma that you’re either a technically strong engineer or good at “playing politics”. These aren’t mutually exclusive. In fact, some of the best engineers I’ve worked with are good at both because they quickly learned that “playing politics” is critical to their success so they get good at it while continue to back themselves up with strong technical depths.

Being good at something doesn’t mean you love doing it. Many of these great engineers would absolutely prefer to just code and solve technical challenges all day rather than spend time in planning meetings, doing stakeholder alignment, or navigating organizational politics. If you gave them the choice, they’d pick the technical work every time. But they recognize that in most environments (especially as you grow in seniority), the non-coding work is the best way to scale your impact so they quickly learned to get good at it.

It’s like pundits complaining about modern NBA playing for the foul calls. These players are still some of the best in the world without it, but they learned that being good at it is critical to their success, so they adapt their game to it. Same thing 20 years ago with the kick-out jumper that Kobe Bryant and Ray Allen did.

What is “the game”? #

A lot of this comes down to understanding what your company / org is biased towards or against in their performance evaluations. Unfortunately, it’s not always what the org needs but mostly what it thinks it wants. Different orgs has different priorities, speaking with a more senior engineer in the org (or observing more carefully) can show. As an example, certain orgs in Meta has very strong preference for engineers with strong technical skills so you need to be strong code reviewer and technical designer to get (a better chance) at promotion. Others prefer more tech leading skills so you need to show relevant ability - roadmap, plan, drive discussions, create scope - for a better shot at earning that promotion. Some places look for a floor raiser to help them build a 0 -> 1 project; others look for someone who can obsessively optimize mature products to extract maximum value. Generally, there are different archetypes that can earn you the promotion but knowing the preferred archetype makes it easier for you to get it.

Play to your strengths #

It’s like if a team signs Messi and makes him play the center back position. I’m sure he can still do fine but the team is not getting the most value out of him. Then when it’s viewed in vacuum (without context), it’d look like he’s just an average player. That’s you during performance review. Your package is generally reviewed in a vacuum without the context of “trying new stuff” or “expanding into new roles” (with the exception of joining a new team). Not playing to your strengths hurt both you and the team because the team didn’t get the most out of you, while you look like a worse engineer than your capability.

If the org’s preferred style is not something that matches your strength, you’d need to consider if putting in the extra work is worth it (maybe you enjoy the people + work enough), or if there are opportunities out there that suit you better. As mentioned before, what the org needs isn’t necessarily what it wants. You can make the case for it but it will take extra effort relative to the alternative. (That said, in many occasions, a successfully made case would allow for better job security as being the only / few people with such specialty.)

Wrap up #

It’s not that you can’t be a good engineer if you don’t fit your company’s performance rubric but rather it’d require extra effort from either you or your manager (or both) to make your case. Some are one-offs (defining a new archetype), while others might be recurring (figuring out the right business reason to justify the time and effort on your projects). At some point, you might want to ask yourself if all these extra work is worth the time and effort.

There’s also another school of thought that view all these as unnecessary bloats in larger tech companies. I’m not here to argue for either (I know, not very opinionated 🫣) but to share what / how to get the most juice out of your work. Most people like this prefer to stick to startups where performance review processes aren’t as rigid. That’s valid too. At the end of the day, that’s just a different game with different rules. The key is knowing which game you’re playing and whether you want to keep playing it.

FB accounts created without Messenger

binhong@binhong.me (BinHong Lee) — Fri, 28 Nov 2025 00:00:00 -0800

This was one of my first experience at causing a SEV. I’ve broken production before 🫣 but not bad enough for it to be a SEV. In this situation, millions of new FB accounts were created (over the duration of the outage) without proper messaging inbox thus unable to use Messenger. I’ll dive more into how this happens below.

This is part of a series (Project Presentation) where I share stories of my past projects.

Background #

When a user tries to create a new account on FB, the system would actually first try to login with the provided credentials to see if it matches any existing account. If it matches, the user would be logged-in to their existing account instead of having a new account created. I added a new feature that make the login / matching system more complex which increases the success rate of it getting matched with an existing account (while maintaining the same low risk profile).

Simple diagram showing the new change

The problem however, was that the account creation process by itself was already a complex and lengthy process that takes a long time. So by adding extra processing to it (before account creation is even triggered), the entire process ended up taking so long that the request was timed-out and account creation process was halted halfway. This means accounts were created with bits-and-pieces missing, most notably it’s messaging inbox.

Discovery #

This was discovered by an engineer in the Messenger org. They started noticing new accounts without messaging inbox and thought it was weird but they didn’t think too much of it. They just manually triggered a job to retroactively create it for those accounts. For the 2 weeks I ran my experiment (a/b test), they just ran the job manually for the subset of new users without a messaging inbox. But when I launch the experience (due to successful a/b test), things got a bit out of hand.

This engineer started seeing significantly (~20x) more new users daily being created without a messaging inbox so they filed a SEV. At some point, someone was able to pinpoint the angle change of their graph with my experiment launch. Seeing as my project directly affects the way account creation process works, it’s very likely to be related, thus I got pulled in.

Verification + Resolution #

I was initially skeptical at their reasoning around why these new accounts are missing the messaging inbox and thought there might be something else. But to prove it either way, I decided to pull the plug and unship the project to see if that solves the issue. It did. I was wrong but now we need to figure out (ideally) if there’s a way to fix this while still shipping the project. We pulled in more help from people who better knows how new account creation works to see if there are other things broken (aside from inbox).

Simple diagram showing the fix

As far as we can tell, only inbox was affected by this. So we essentially moved messaging inbox creation to be triggered earlier (as one of the top-tier jobs to be ran in parallel). We re-tested it and see this fixes the issue for all but one or two interfaces (I think it was FBLite and maybe mWeb that still had issues). I actually kept the SEV in Cleanup state for a bit until we dropped it off as the team (technically just half the team which included me) had a prioritization shift moving away from growth to 0 -> 1 so we just left it as is.

Eventually, the whole account creation flow was replaced anyway as part of the 0 -> 1 effort to launch the new unified experience.

Aftermath #

I was made the SEV owner (being the person who caused it) and filled out the SEV report. I don’t remember too clearly how the review went anymore but I did know that I was really nervous throughout. I learnt a lot from the process and as far as I know, I was not penalized for it in my performance review seeing it wasn’t exactly my failure that caused this (more on No Blame SEV Culture).

Takeaway #

The main takeaway here is that metric monitoring and analysis only goes so far in making sure things work. It generally doesn’t cover things that are affected outside of your immediate team’s scope. Secondly, even if skeptical, don’t assume without proof. I was skeptical that my experiment was the rootcause but I’m glad I committed to verifying it anyway (which proved myself wrong). Stressful at the time, but good learning experience overall.

Incident Review Process

binhong@binhong.me (BinHong Lee) — Fri, 14 Nov 2025 00:00:00 -0800

In the previous runbook, I mostly focused on things to do during an incident as it happens. This week, we will focus more on what to do after the incident is resolved. This is an opportunity to look back at what went well, what went wrong, and how we can best prepare ourselves or prevent something similar from happening again in the future.

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Report #

After the incident, the incident owner should fill out an incident report detailing things like root cause, alerts, remediation, recovery, prevention, etc. Ideally this is made into a template and the author should go into detail about what actually happened vs what should have happened and how it could be improved. The report should be completed at least a day or two ahead of the review to allow for optional pre-reads.

Review #

Usually you’d need someone (or a group of people taking turns) to organize the review. Their job is to pick the set of incidents to review, make sure that incident owners fill out the report adequately, and that everyone relevant is invited and available to attend the review. It’s not really a hard job per se but it requires some async communication and some level of familiarity with the org scope to ensure that all the right people involved in each incident will be there at the review.

Audience #

Aside from incident owners and people directly involved in said incidents, team EMs, senior engineers, and senior org leaders (Director / VP) are critical to have present in the review. EMs are there to ensure visibility and awareness of the importance of follow-up work from the incident review. Senior engineers are there to raise questions and/or provide recommendations especially surrounding detection and prevention strategies. Senior org leaders showing up - at the very least - solidify the importance of the review and enforce that planned follow-ups are actually worked on with allocated time instead of being set aside / deprioritized.

Follow-ups #

After the review, there will likely be a laundry list of follow-up work to be done. Note them down in your task tracker (Jira, Asana etc.) and assign them to the right owners. By default, the incident owner would own it but it’s very common to have it assigned to peers or even external teams, depending on how/what was broken. One of the more common failure modes here is the lack of tracking on these follow-up tasks. Make sure to have a proper process to (at least) have someone - anyone - review these tasks over time to ensure they are being worked on in a timely manner and actually completed or attempted in good faith. It’d be a shame (with a lot of awkward conversation) if there ended up being another similar incident that could have been prevented by said follow-up task.

Pattern #

In some cases, frequent attendees might notice a pattern of incidents and propose wider improvements (instead of specifics to any incident). One example we ran into was how a lot of incidents could have been prevented by having automated tests. This wasn’t because the team didn’t make an effort to write them, but because there was a more underlying issue (outside of the team’s direct purview) causing existing tests to become flaky. We ended up having an entire workstream focused on fixing all the automated test blockers found in the org. (I’ll probably write about my experience working on it someday, under the project presentation series.) Anyway, the point here is that having a consistent set of decision-making audience helps make sure patterns like these can be better discovered instead of overlooked.

Accountability #

I’ve touched on this previously in No Blame SEV (Incident) Culture. Generally, there shouldn’t be any explicit blaming happening in the incident review meeting but as I’ve seen others put it, there could still be “shame”. The way I’ve understood how a lot of this works is that the worse the outage was, the less likely the incident is to be entirely the responsibility of an individual. If you took down the entire production tier of your company, there are likely a lot of things that need to go wrong for it to happen, so it shouldn’t all be on you.

Aside from all the “who’s wrong / who’s to blame” side of accountability, there’s also the part where each responsible team needs to take on follow-up tasks to strengthen their end of reliability to prevent similar incidents from happening again. For example, if an internal framework or component library change breaks a bunch of stuff in the company, the most obvious thing would be for the framework/library team to improve their regression testing. But client teams might also want to consider adding stricter test coverage on critical paths so they can be more defensive and catch potential future breakages earlier.

Wrap up #

The incident review process is a critical part of your team’s long-term reliability. It can be daunting to those presenting their first incidents so a proper report template helps make sure you always have quality content for the review. From there, getting the right people in the room regularly and actually tracking follow-up work helps make sure you don’t fall for the same problem again. Even if you aren’t sure where to start, putting together a rough process can help kickstart it as you re-evaluate and refine it as it goes. The goal isn’t to eliminate all incidents, but to make sure you’re learning from them and getting stronger as a result.

Leaked `FakeIncorrectPassword`

binhong@binhong.me (BinHong Lee) — Fri, 31 Oct 2025 00:00:00 -0800

It was Thanksgiving day and I was mostly chilling at home scrolling Threads while watching YouTube. I then came across this post:

@justinamphlett

Found a pretty horrific dark pattern with Instagram. If you try to log in on mobile web, it fails with ‘incorrect password’. I double check… it’s right, it’s just not working. Gave up, installed the app, password is fine. Tried again today just to see… this time I get the following error message. Looks like a translation key is missing and it’s serving me the backend name for it? FakeIncorrectPassword ?? Are you shitting me?

I immediately recognized that the screenshot is bad. FakeIncorrectPassword is an internal error code used when the integrity system thinks you’re a bad actor (bot, scraper, credential stuffer etc) and so we would show you an Incorrect Password error message regardless of whether the password is correct. But as seen in the screenshot, we show the raw FakeIncorrectPassword string directly instead of the regular Incorrect Password error.

This is part of a series (Project Presentation) where I share stories of my past projects.

Notifying owners #

The user noted that this happens on mobile web which means they happened to land on a test group, since a sister team had just started public testing this new UI / flow on mobile web. I tried ringing up the oncall for their team (who also happened to be the owner of the public test) but I saw that they had actually marked themselves out-of-office for the whole weekend through the upcoming Monday. Well, that’s not good (also weird that they were off while oncall❓).

Comparison between old vs new login screen

Investigation #

Anyway, I had a few hours to kill before I needed to leave to join my friends’ Thanksgiving party. I looked at the time and told myself “I have time” so I decided to start debugging it myself. I’d worked tangentially with this stuff before so I knew a few other things (like 2FA) that also need special handling should be working as intended. My hunch told me that this was caused by the gap between the Python stack (IG server) throwing some error that wasn’t properly consumed by the Hack server stack (that determines the UI). So I searched for FakeIncorrectPassword in the Python stack and found something like TwoFactorError. Then, I looked for TwoFactorError in the Hack stack to see how it works and eventually found the one that didn’t handle the case for FakeIncorrectPassword. Simple enough, I just had to add more stuff (iirc I added a few others as well as the aforementioned FakeIncorrectPassword) to the existing switch case.

Simple diagram to show how the calls relate

Testing #

At first, I tried triggering the FakeIncorrectPassword error on my devserver to test the fix but I don’t really know how most of these things work (having never worked on integrity teams). I eventually figured I could just comment out the integrity system calls and just always force return FakeIncorrectPassword error. Trigger login and take a screenshot for both before and after the fix. Done! I sent the fix out for review and waited for a bit to make sure everything turned green on CI before leaving for the Thanksgiving party (didn’t want to bring my laptop with me since I wasn’t actually oncall lol).

Shipping #

Before heading out, I tagged 2 other oncalls (on top of the OOO oncall) to let them know that I’d written the fix and I’d let them decide if they wanted to ship it or wait till Monday. One of them saw it and immediately approved + shipped my fix. Monday came around and my change got the attention of a few people from the security team. They eventually filed a post-hoc SEV for this and did an in-depth review on what went wrong in the process that led to the leak of this error message.

Takeaway #

The main culprit here is that most Instagram code lives in a different stack (Python) than where the UI is (Hack) and a lot of the Python code just uses string everywhere instead of proper enums for something like this, making it really hard to keep track of all the potential edge cases that could be missed. Having some familiarity with both the Python and the Hack stacks made it easier to trace the problem and spot the gap in the error handling logic. The fix itself was straightforward once I found the right place, but getting there required understanding how errors flow between multiple systems.

The Myth of Technical Complexity

binhong@binhong.me (BinHong Lee) — Fri, 17 Oct 2025 00:00:00 -0800

I’ve seen multiple different people fail their promotions due to “lack of technical complexity” in their work. While some of this rings true, more often than not, the underlying technical complexity of their work is not properly understood because - ironically - they cut through it smoothly as strong technical engineers. Unless you were doing code / design reviews for them, you wouldn’t notice it. On the flip side, I’ve had projects listed as my capstone “technical complexity” project of the year that imo wasn’t very complex compared to some other stuff I needed to work on. Why is this happening and how do I make sure my work is properly valued?

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

The Problem #

During your performance or promotion review, the audience generally includes your manager, their peers, org leaders, and sometimes a couple of very senior ICs in your org. Realistically, none of them have seen your code (though some might have seen your project design docs). So how could they know if something is technically complex? They don’t. Instead, they mostly rely on their intuition (based on what they know of the project, or just the project name alone), or they hinge it on other calibrated agreements (more on this later). If your project name is “Logging for X,” it’s very likely that it will be challenged for its technical complexity level for any senior engineer because “how hard could logging be?” (Turns out, not only can they be hard, but they could also be wrongly designed with long-term negative side effects, especially if your leaders can’t get out of the sunk-cost fallacy. But that’s a story for another day.)

Just a one-line fix #

Sometimes a simple bug is masked by convoluted layers of abstractions, making it extremely hard to discover. We’ve all had bugs like this where everyone was at a loss as to why it was broken, but it turned out to be a simple one-liner fix. If you look at the fix diff alone, it would look straightforward and you’d wonder why no one noticed this earlier. These are technically complex to debug from a given stack trace (or sometimes just screenshots) to the actual root cause. In fact, this is the definition of work for a staff engineer archetype - fixer. Yet, this is also an example of technically complex work that’s commonly overlooked due to how deceptively easy it looks.

Upfront alignment #

The easiest way to prevent later disagreement on the complexity nature of your work would be to align it upfront. Talk to your TL / EM to make sure they agree with your assessment of how complex the work really is. (Excluding PMs here because they generally don’t have strong opinions / say on the technical complexity of your work.) Even if you end up disagreeing, you’d know exactly what you’re getting yourself into ahead of the project and decide accordingly (depending on options available to you). For most junior engineers and some early senior engineers, this is likely something that happened even prior to you knowing about the project. The senior engineer who assigned you the project likely aligned this upfront to make sure the project fits your growth path (meeting expectations / promo) your manager has in mind for you.

Similar past attempts #

Another way you could demonstrate the technical complexity of a project is by citing previous failures at attempting this project. Generally, this requires a project to have been attempted previously (even if there are slight differences in details). It’s important to keep in mind that you should always try to do things only you can do (maybe I’ll write more about it in the future) and delegate everything else. Someone else failing previously is essentially a way to show that ‘you’ were the key ingredient to making the project a success.

If there’s a similar project that was previously executed successfully, that’s a slightly different story. It can still be complex, but the complexity level will likely be reduced by at least 1 level (if not more) depending on how similar they are. The idea is that you already have a good reference that helped scope out a lot of the unknown unknowns, making this simpler.

Borrowing an authority #

Similar to what I previously wrote in The Art of Posting and what Mekka wrote here about The Difficulty Anchor, you can also borrow the authority of someone else more senior in your team or org to endorse your evaluation of the project. This is especially important in areas where you’re not seen as an expert. While it’s ideal to get this on the record early on (as mentioned in the earlier section), this can still be done post-hoc as well. This is especially important when you run into unexpected complexity as you work through the project. Having someone credible validate that “yeah, this turned out to be way messier than we thought” carries much more weight than you making that case yourself.

Wrap up #

The cruel irony of engineering is that doing your job well often makes it invisible. When you smoothly navigate complex technical challenges, stakeholders see the outcome, not the expertise required to get there. Don’t let your competence work against you. The technical complexity of your work needs to be communicated, not just demonstrated. Whether it’s aligning expectations upfront, citing similar past failures, or borrowing authority from senior engineers, make sure the people evaluating you understand what they’re really looking at.

Shipping a project with -2M MAP

binhong@binhong.me (BinHong Lee) — Fri, 03 Oct 2025 00:00:00 -0800

There’s a saying in Meta that ‘Nothing is somebody else’s problem’ but sometimes, it really is. At this point in time, I was working on a growth team whose top-line goal was to grow Facebook MAP (monthly active people). So naturally, it’s surprising for me to have shipped something that directly regressed the very metric the team should be growing.

This is part of a series (Project Presentation) where I share stories of my past projects.

Background #

This was still within the first year of me joining the team. Unfortunately, up to this point, I hadn’t had any successful (+MAP) launches, and I’d been eager to prove myself. The most recent project I worked on before this was to show an educational screen to users right after they logged in (under a specific condition) before they got redirected to newsfeed. Once a user logs an impression on newsfeed, they are considered MAP.

My project failed because for some reason it logged a MAP increase on the very first day - which didn’t make sense because users should be MAP on the first day regardless of whether they’re in the test or control group. If anything, it could have had a slight drop if users dropped off on the interstitial before they reached newsfeed. At this point, my manager suggested that we move on to something else since no one really knew why. I was, however, adamant about figuring out what happened, so I started digging into it anyway. (major mistake)

The Bug #

I dug through the stacktrace step-by-step, the code line-by-line, until I saw a variable that caught my eye: is_platform_login. For some reason, when it was true, we ignored the next param in redirection. I didn’t know what “platform login” was, so it was time to get some help. I asked my mentor / tech lead what “platform login” was and was told that it’s third-party login (think “Login with Facebook” on other sites). It’s something owned by a separate team, and I should go ask this specific person (Kevin) if I had more questions about it. So I decided to test how it worked. I opened up Spotify, clicked “Login with Facebook,” then logged in (through that specific condition), and thereafter I was redirected to FB Newsfeed instead of going back to Spotify to complete my login!

Discovering the scope #

First, I looked up the blame and just asked the person (on a sister team) why they explicitly excluded platform login from this redirection. They told me that’s just how the team tests and ships new features in general. Fair, but I didn’t think it made sense in this case. So I reached out to Kevin and asked if they were aware of the bug. They were surprised and agreed with my assessment that it should be fixed.

I merged the change and started an experiment for the fix. After a while, I pulled the metrics on the experiment and saw that it regressed MAP heavily, around 2M MAP. This wasn’t exactly surprising, as you’ve seen in the image above, since we were redirecting the users back to the third-party platform source, which no longer counted them as MAP. I went back to Kevin and asked what their team’s goal metrics were and if they had a data scientist on their team willing to help with experiment analysis.

Justifying the ship #

We brought the experiment to be reviewed. (If you aren’t familiar, here’s a previous article on how experiment review works.) Most people understood the reasoning behind the ship recommendation, but there were some questions around better understanding whether this was intended user behavior. After some back-and-forth, we agreed to ship this with a long-term holdout to study the potential long-term MAP effect.

A few months later, we pulled the data for the long-term holdout and verified that the long-term MAP regression was closer to 20k. This meant that 90% of the users who were forced to newsfeed (caused by the bug) did not stay MAP for very long and eventually churned one way or another anyway. It’s likely because these users did not intent to actually use Facebook to begin with, thus also unlikely to stick around when they were wrongly redirected.

Takeaway #

For one, I was penalized for essentially spending too much time on work that wasn’t aligned with the team I was on. Realistically, I should have taken my manager’s advice to move on to another project. Even after I’d found the bug, I should have passed this off to Kevin and their team to run the experiment and provided support instead of trying to be the hero. The results from the long-term holdout also didn’t materialize until the next performance review cycle, so that definitely didn’t help.

Finally, I still never actually figured out whether this was the bug that was causing my original project to fail. We suspect it might be related, but I never re-ran it and never got any sort of verification that it solved the issue. It was a lot of learning experience for me overall, but at least I delivered (imo) the ideal user experience and was able to justify the change despite the team’s goal metrics.

Communicating Effectively

binhong@binhong.me (BinHong Lee) — Fri, 19 Sep 2025 00:00:00 -0800

It’s a common problem at work where you see a problem but you’re having a hard time trying to convince people to support your attempt at fixing it. Worse still, they might not even see it as a problem (or they don’t think it’s something that warrants any attention from anyone). I’m approaching this piece mostly from an “I need my leadership / partners to understand what I do is important” perspective. But the ideology can similarly be used on other occasions like selling an idea to your peers etc. Ideally you start thinking about this as you try to get the green light to pursue your project instead of attempting to justify it after you’ve already invested your time into it with little to show for it.

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Understand your audience #

Whoever you’re talking to, they have their own perspective and priorities. They care about certain things. Make sure to understand that and position your proposal from their perspective. One common communication gap is where someone keeps trying to tell their leaders “how hard they / the team has been working” - which is cool and all - but your leaders likely want to know more about “what’s in it for me?”. For example, instead of telling your manager that “I put in 60 hours a week all of last month to make this happen,” it should be more like “I shipped x MAU which is x% of the team / org goal” or “I worked overtime to handle multiple unexpected surprises (have documentation for this) to ensure that we can ship / launch / test {org priority project} in a timely manner.” Changing the framing of your statement (by empathizing with your audience) makes it significantly more appealing to your target audience.

Strategic positioning #

When you pitch a project proposal, you need strong reasoning on why you think it will be worth the time investment. In most cases, you’d be making the case using some sort of metrics or goal that the project is expected to improve. However, that’s not always the case. If you have a goal metric to chase but this project doesn’t directly affect that, think of how it’s affected indirectly. In my previous piece on dev tool valuation, I mentioned engineering cost being the most straightforward measurement available.

On the other hand, if there’s a larger org-priority project, think of how to align your project with it. You might need to make some small changes in the project pitch, but the idea is that you don’t have to build your case from scratch. Instead, you are just piggybacking on an established case by explaining how your project would help with the goal of completing the larger project. This sort of positioning not only helps reduce the amount of foundational work needed to make people understand the why but also ensures that your project will continue to receive attention as a side effect of now being associated with the larger project.

Reprioritization #

Prioritization changes all the time. When there’s a top-down change in priority, make sure to ask clarifying questions to understand the reasoning behind the shift. From there, you have to then recalibrate your (team’s) projects to make sure they’re still aligned with the higher-level goals. On some occasions, this could also mean that you would have to recommend priority shifts on ongoing / planned projects depending on the leadership goal change. For example, you’re in the core infrastructure team and the company decides to pivot to “everything AI”. Make sure to understand “how AI” and potentially shift your prioritization around (like adding better GPU support, direct Ollama support etc.) to fit the leadership goal change.

Goal setting #

Management (and shareholders) love goals. They paint a picture of predictability and allow for an easy way to evaluate how you (or the team) are doing. The higher level you get (with bigger scope and projects), the goal would be closer to company-level goals. The hierarchy probably goes something like this: (company-level goal) make more money -> sell more ads -> get more users -> keep existing users returning (team-level goal) etc. You don’t have to set the goal on how to make the company more money, but you can set a goal on how to keep existing users returning which, by proxy, would have your impact cascaded up.

Wrap up #

Admittedly, this is something even I sometimes forget or overlook when pitching a project. The key insight is that effective communication isn’t about having the best technical solution but rather, it’s about meeting people where they are and speaking their language. When you frame your ideas through your audience’s priorities and align with existing organizational momentum, you’re offering value in return of their support.

GitHub Action Runner Alternatives

binhong@binhong.me (BinHong Lee) — Tue, 02 Sep 2025 00:00:00 -0800

A few months ago, I wrote about my attempt and reasoning behind managing my own fleet of self-hosted GitHub Action Runners. I’ve since learnt that there are quite a few hosted alternative runners that are both faster and cheaper than GitHub’s default runners. This article is just a quick summary of what I’ve found.

Linux Runners Comparison #

Provider	Free Tier	Paid Pricing (2vCPU)	Performance	Special Features
Blacksmith	3,000 minutes/month	$0.004/min (50% of GitHub)	2x faster builds, 4x faster cache	Unlimited concurrency, gaming CPUs
BuildJet	$5 one-time credit	$0.004/min (50% of GitHub)	2x faster (variable CPU models)	64 AMD + 32 ARM concurrent vCPUs, 20GB cache
CarbonRunner	❌ None	$0.006/min (25% cheaper than GitHub)	Standard performance	90% lower CO2 emissions, 42 regions
Cirrus Runners	❌ None	$150/month unlimited	2-3x faster performance	Fixed pricing, no minute limits, limited concurrency
Depot	❌ None	$20 for 20,000 minutes, then $0.004/min	30% faster, 10x faster cache	Docker-focused, per-second billing, unlimited cache
GitHub	2,000 minutes/month	$0.008/min	Baseline	Official support
Namespace	❌ None	$0.003	Fastest x64 performance	250GB storage, SSH access
puzl.cloud	400 vCPU-minutes/month	$0.0008/min (€0.000008/vCPU-sec)	2x faster, 5x cheaper	Real-time per-second billing, 12vCPU per job
Sprinters.sh	❌ None	Per-job pricing (10x cheaper)	Standard performance	Self-hosted in AWS, per-job pricing
RunsOn	Free for non-commercial	€300/year + $0.001-0.003/min (AWS costs)	30% faster, 90% cheaper	Self-hosted in AWS, GPUs, Windows
Ubicloud	1,250 minutes (~$1 credit)	$0.0008/min (10x cheaper)	Newer CPUs, good performance	Open source, Hetzner hardware
WarpBuild	❌ None	$0.004/min (50% of GitHub)	2x faster, 50% cheaper	Snapshots, unlimited cache

macOS Runners Comparison #

Provider	Pricing	Hardware	Performance	Key Features
Cirrus Runners	$150/month unlimited	M4 Pro (latest)	2x faster performance	Fixed pricing, no minute limits, limited concurrency
GitHub	$0.08/min (10x multiplier)	Intel Mac + M2 Pro XL ($0.16/min)	Baseline	Official support
Namespace	$0.09	M4 Pro + M2 Pro	Fastest available	SSH/VNC access, bleeding-edge images
WarpBuild	$0.04/min (50% cheaper)	M2 Pro	25% faster, 50% cheaper	Pay-per-use, macOS 13/14 support

I don’t have a better place to put this but I built a very basic widget that takes into account of the “supposed” faster speed (leading to less action minutes) with each of their rates so you can find out which platform fits you best based on your usage (with GitHub’s Action Runner minutes as baseline).

GitHub Action Minutes

Platform

Ignore Speed

Provider	Speed	Monthly Cost

Note: Sprinters.sh is priced per job instead of per min (like everyone else) so I didn’t include it in the comparison above.

Note2: Cirrus Runners limits on concurrency instead of minutes so the experience might vary (depending on usage).

Note3: “Speed” is loosely defined based on acclaimed performance but not verified in any way, shape, or form.

Wrap up #

Obviously this is a rather simplistic view of the cost comparison (as well as the speed factor being unverified) between different options available in the market. Personally, I opted for Blacksmith for all my linux needs while continue to host my own M4 mac mini at home as the macOS runner (since the electricity cost is still cheaper than any of the above options).

Hidden Privacy / Security Pitfalls

binhong@binhong.me (BinHong Lee) — Sat, 23 Aug 2025 00:00:00 -0800

Most software engineers think of this as “someone else’s problem” (usually a privacy engineer or security engineer), but realistically, there’s only so much stuff (bad code lol) they can catch. Especially if you work in some of the more sensitive areas, it’s not realistic to have them do code audits for every change. This means that you as the code author (or reviewer) need to at least pick up some basic instincts about things that can pose risks.

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Disclaimer: I’m neither a privacy engineer nor a security engineer (but a software engineer) so take this mostly as a way to raise awareness about considering some of these pitfalls while building instead of as an expert opinion on what to do.

Timing attack #

Even if you return the same response every time, you might still be leaking data (or mostly inference) based on the timing it took the server to respond. As an example, a typical email + password login system probably looks like this:

Hash password
Email lookup
Compare hash

Now, considering that password hashing can take quite a bit of time and resources, we could rearrange the order to make it more efficient by making use of “early returns.” Something like this:

Email lookup
Hash password
Compare hash

With this new setup, you’d only do password hashing when an email already exists in your record, thus saving time and processing power from unnecessary password hashing. However, you’re also returning your network request much sooner. This becomes an unintentional leak where a bad actor can use it to tell if an email exists in your database or not by doing some timed response comparison. My favorite solution to this problem (when applicable) is to add a random response delay—not long enough to significantly affect user experience, but just long enough to throw an attacker off.

Scraping #

Generally the easy solution here is to slap a rate limit on it and call it a day. The problem, however, is that you need to realize not all endpoints are created equal. Most evidently, you have an endpoint that returns info about a user when given an ID, while you also have an endpoint that returns info about a list of users when given a list of IDs. Both of these endpoints can’t just share the same rate limiting because scraping the latter endpoint is list_lengthx more efficient than the former.

Credential stuffing #

This is usually the result of password reuse. While you can technically do credential stuffing with just randomly enumerated credentials, the cost is unlikely to be worth the outcome for the attacker (unless there are some specific exceptions). Since people reuse passwords and there have been many previous hacks (unfortunately containing plaintext passwords), your perfectly secure authentication system becomes vulnerable not because of your code, but because of breaches elsewhere. That said, these attacks are usually more obvious, coming from an expected set of IPs (all from the same location or same VPS provider, etc.).

Contactpoint disclosure #

A little different from the rest of the list, this is more of a type of disclosure than a method of disclosure. While many systems are more conservative and opt for the route of never even disclosing if a specific contactpoint (email / phone number) is even associated with the platform, some do allow for it. However, this will require a bit more care in handling such disclosure where we don’t want to disclose other contactpoints being connected to a given contactpoint (even through inference). The main reason being that this risks leaking identities behind pseudonyms where people can connect the dots.

Rate limiting #

Rate limiting is both a valuable tool and also a potential risk of exposure. Most evidently, rate limiting helps reduce (and hopefully prevent) scraping and credential stuffing attacks. However, in some systems, rate limits are set on a per-account basis, making it a risk of exposure as you can do targeted attacks with some reasonable correlation to get information out of it.

Here’s an example:

Condition: email A and phone A are both owned by the same account while email B is not.

try (and fail) login with email A -> no rate limit
try (and fail) login with email B -> no rate limit
try (and fail) login with phone A -> no rate limit
keep trying email A until it’s rate limited
try (and fail) login with email B -> no rate limit (since different account)
try (and fail) login with phone A -> rate limited

There are many reasons (or other situations) where it’s not immediately obvious how introducing an account-based rate limit can be a risk.

Phone recycling #

Many platforms rely on phone number verification to authenticate your ownership of the account, but phone number ownership isn’t exactly immutable. In fact, phone numbers commonly switch hands in most countries for a variety of reasons (especially as people move across countries). Unfortunately, this is a much trickier problem to tackle due to the wide variety of carriers spread across different countries having different processes and regulations around it.

Email recycling #

While email recycling isn’t as common as phone recycling, it still persists (especially those with custom domains). In fact, many years ago one of the largest email service providers actually allowed anyone to register expired email addresses. This became a major attack vector on Facebook since many people back then set their emails to be shown publicly on their Facebook profiles, while also simultaneously stopping using those same email addresses - leaving them to expire. Thankfully, most email providers have learned of such attack vectors and no longer allow deactivated / deleted email addresses to be re-registered ever.

Wrap up #

These pitfalls often emerge from well-intentioned optimizations that look perfectly reasonable in isolation. The key is developing a mental checklist of common attack patterns rather than becoming paranoid about every line of code. Security isn’t just about preventing obvious attacks but instead, it’s to think through the subtle ways your system behavior can be observed and / or exploited.

Sync git submodules with GitHub Action

binhong@binhong.me (BinHong Lee) — Fri, 08 Aug 2025 00:00:00 -0800

I needed to share some object code across multiple repos (to ensure the consistency of the RESTful API JSON serialization and deserialization). So I created a dedicated repository where all the shared code lives, then imported it across other repos as a submodule. In this case, I used a library I made - wings - but usually you’d probably use something like Thrift or Protocol Buffers. Another common use case would be to have a repository hosting shared cross-platform (iOS + Android) low-level code.

It’s a bit of a pain to manually update it across all the different repositories (and check if it causes any regressions) every time I make changes to it. So instead I wrote some tests, set up some GitHub Actions to run them, then have an automated GitHub Action that sends PRs to all the “parent” repos every time it receives a new commit, which will trigger the underlying CI tests so it’d be obvious if it might break anything.

Setup submodules #

First, create the shared git repository. Then in the parent repositories, run git submodule add <path> <git_url>. Commit these changes and now you should see the directory shown as a ref link in GitHub like {dir_name} @ {commit_hash}. It’s important to note that the folder won’t be initialized automatically on clone, but instead you need to either clone with the --recurse-submodules argument or run git submodule update --init --recursive after cloning to get the submodules set up properly.

Granting PATs #

You mostly need 2 PATs here. One for all repositories importing the shared repo to be able to read both the hosting repo and the shared repo so its GitHub Actions can run git submodule init && git submodule update as part of its “checkout” step. (They could be multiple separate PATs each with specific repo access, but in my case I just use the same PAT that allows read access across all repos in the GitHub organization.) The other one is for the shared repo (to be used later) to clone, branch, commit, and create pull requests on all the repos extending it.

Creating GitHub Action for updates #

Here’s something I vibe-coded with Claude for my own project (GlobeTrotte). The main thing that needs updating is the SUBMODULE_PATH and DEFAULT_CONFIG section to fit your use case. It is a bit more on the verbose side of things, but it does work as expected. One tricky callout I’d say is that the PATs can be a little tricky if you need to both read from another repo (as you would to initialize the submodule) and write as you try to push changes to another branch (like “deploy” or “gh-pages” branch). You can instead split them into separate actions with proper dependency and use different PATs depending on the need.

Workflow configuration and triggers #

This sets up the basic workflow structure and defines which repositories should be updated:

name: Update Submodules and Create PR

on: [ push, pull_request ]

env:
  # Static submodule path used across all configurations
  SUBMODULE_PATH: "src/wings"
  
  # Default configuration - can be overridden by workflow input
  DEFAULT_CONFIG: |
    [
      {
        "target_repo": "repo-1",
        "prefix": "./",
        "target_branch": "main",
        "enabled": true
      },
      {
        "target_repo": "repo-2",
        "prefix": "./",
        "target_branch": "main",
        "enabled": true
      },
      {
        "target_repo": "repo-3", 
        "prefix": "app/",
        "target_branch": "main",
        "enabled": true
      }
    ]

The DEFAULT_CONFIG is a JSON array where each object represents a repository that uses your submodule. You can enable/disable repositories individually and specify different target branches per repo.

Setup job: Creating the execution matrix #

This job processes the configuration and creates a matrix for parallel execution across multiple repositories:

jobs:
  setup:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - name: Set up configuration matrix
        id: set-matrix
        run: |
          if [ -n "${{ github.event.inputs.config_override }}" ]; then
            echo "Using override configuration"
            CONFIG='${{ github.event.inputs.config_override }}'
          else
            echo "Using default configuration"
            CONFIG='${{ env.DEFAULT_CONFIG }}'
          fi
          
          # Filter only enabled configurations and format for matrix
          MATRIX=$(echo "$CONFIG" | jq -c '[.[] | select(.enabled == true)]')
          echo "matrix=$MATRIX" >> $GITHUB_OUTPUT
          echo "Configuration matrix:"
          echo "$MATRIX" | jq .

This step filters out disabled repositories and creates a matrix that GitHub Actions can use to run the update job in parallel for each target repository.

Main update job: Repository checkout and setup #

The main job runs for each repository in the matrix. First, it checks out the target repository and sets up git configuration:

  update-submodules:
    needs: setup
    runs-on: ubuntu-latest
    if: ${{ fromJson(needs.setup.outputs.matrix)[0] != null }}
    strategy:
      matrix:
        config: ${{ fromJson(needs.setup.outputs.matrix) }}
      fail-fast: false
    
    steps:
      - name: Display current configuration
        run: |
          echo "🎯 Target Repository: ${{ matrix.config.target_repo }}"
          echo "📂 Submodule Path: ${{ env.SUBMODULE_PATH }}"
          echo "🌿 Target Branch: ${{ matrix.config.target_branch }}"
          
          # Construct full repo path
          FULL_REPO_PATH="{org-name}/${{ matrix.config.target_repo }}"
          echo "FULL_REPO_PATH=$FULL_REPO_PATH" >> $GITHUB_ENV

      - name: Checkout external repository
        uses: actions/checkout@v4
        with:
          repository: ${{ env.FULL_REPO_PATH }}
          token: ${{ secrets.UPDATE_SUBMODULE }}
          ref: ${{ matrix.config.target_branch }}
          fetch-depth: 0
          path: ${{ matrix.config.target_repo }}

      - name: Configure Git
        run: |
          cd ${{ matrix.config.target_repo }}
          git config user.name "GitHub Actions Bot"
          git config user.email "actions@github.com"

Remember to replace {org-name} with your actual GitHub organization name.

Submodule initialization and version checking #

This is the core logic that handles submodule authentication, updates, and determines if changes are needed:

      - name: Initialize and update submodules
        run: |
          cd ${{ matrix.config.target_repo }}
          git submodule init
          
          # Update submodule URLs to use PAT authentication
          git config --list | grep "submodule\." | grep "\.url=" | while read -r line; do
            key=$(echo "$line" | cut -d'=' -f1)
            url=$(echo "$line" | cut -d'=' -f2-)
          
            if echo "$url" | grep -q "github.com"; then
              if echo "$url" | grep -q "https://"; then
                new_url=$(echo "$url" | sed 's|https://github.com/|https://x-access-token:${{ secrets.UPDATE_SUBMODULE }}@github.com/|')
              else
                new_url=$(echo "$url" | sed 's|git@github.com:|https://x-access-token:${{ secrets.UPDATE_SUBMODULE }}@github.com/|')
              fi
              git config "$key" "$new_url"
            fi
          done
          
          # Update submodules
          git submodule update --init --recursive
          
          # Navigate to the specific submodule and get current commit
          cd "${{ env.SUBMODULE_PATH }}"
          CURRENT_COMMIT=$(git rev-parse HEAD)
          echo "Current submodule commit: $CURRENT_COMMIT"
          
          # Fetch latest changes from submodule's remote
          git fetch origin
          
          # Get the latest commit on the default branch
          LATEST_COMMIT=${{ github.sha }}
          echo "Latest submodule commit: $LATEST_COMMIT"
          
          # Check if update is needed
          if [ "$CURRENT_COMMIT" != "$LATEST_COMMIT" ]; then
            echo "UPDATE_NEEDED=true" >> $GITHUB_ENV
            echo "CURRENT_COMMIT=$CURRENT_COMMIT" >> $GITHUB_ENV
            echo "LATEST_COMMIT=$LATEST_COMMIT" >> $GITHUB_ENV
          else
            echo "UPDATE_NEEDED=false" >> $GITHUB_ENV
            echo "✅ Submodule is already up to date"
          fi

This step converts SSH URLs to HTTPS with PAT authentication, then compares the current submodule commit with the latest commit to determine if an update is needed.

Updating the submodule and creating branches #

If an update is needed, these steps handle the actual submodule update and branch creation:

      - name: Update submodule to latest version
        if: env.UPDATE_NEEDED == 'true'
        run: |
          cd ${{ matrix.config.target_repo }}
          cd "${{ env.SUBMODULE_PATH }}"
          
          # Checkout the latest commit
          git checkout ${{ github.sha }}

      - name: Print changes
        if: env.UPDATE_NEEDED == 'true'
        run: |
          cd ${{ matrix.config.target_repo }}
          git diff

      - name: Create feature branch and commit changes
        if: ${{ env.UPDATE_NEEDED == 'true' && github.ref == 'refs/heads/main' }}
        run: |
          cd ${{ matrix.config.target_repo }}
          git add .
          
          # Create a new branch for the update
          SUBMODULE_NAME="${{ env.SUBMODULE_PATH }}"
          BRANCH_NAME="update-${SUBMODULE_NAME}-$(date +%Y%m%d-%H%M%S)"
          echo "BRANCH_NAME=$BRANCH_NAME" >> $GITHUB_ENV
          
          git checkout -b $BRANCH_NAME
          
          # Create detailed commit message
          COMMIT_MSG="Update ${SUBMODULE_NAME} submodule
          
          Updated ${{ env.SUBMODULE_PATH }} from ${{ env.CURRENT_COMMIT }} to ${{ env.LATEST_COMMIT }}
          
          This automated update brings in the latest changes from the submodule.
          
          Repository: ${{ matrix.config.target_repo }}
          Branch: ${{ matrix.config.target_branch }}"
          
          git commit -m "$COMMIT_MSG"
          
          # Push the new branch
          git push origin $BRANCH_NAME

This creates a timestamped branch name, commits the submodule update with a detailed message, and pushes the branch to the target repository.

Creating pull requests #

Finally, this step creates a pull request in the target repository with detailed information about the update:

      - name: Create Pull Request
        if: ${{ env.UPDATE_NEEDED == 'true' && github.ref == 'refs/heads/main' }}
        uses: actions/github-script@v7
        with:
          github-token: ${{ secrets.UPDATE_SUBMODULE }}
          working-directory: ${{ matrix.config.target_repo }}
          script: |
            const submoduleName = '${{ env.SUBMODULE_PATH }}';
            const repoOwner = 'GlobeTrotte-com'; // Replace with actual owner
            const repoName = '${{ matrix.config.target_repo }}';
            
            const { data: pullRequest } = await github.rest.pulls.create({
              owner: repoOwner,
              repo: repoName,
              title: `Update ${submoduleName} submodule`,
              head: process.env.BRANCH_NAME,
              base: '${{ matrix.config.target_branch }}',
              body: `## Submodule Update
            
              This PR updates the \`${{ env.SUBMODULE_PATH }}\` submodule to the latest version.
            
              ### Changes
              - **From Commit:** \`${{ env.CURRENT_COMMIT }}\`
              - **To Commit:** \`${{ env.LATEST_COMMIT }}\`
            
              ### Details
              This automated update was triggered by a push to the main branch and brings in the latest changes from the submodule repository.
            
              Please review the changes in the submodule repository before merging this PR.
            
              ---
              *This PR was created automatically by GitHub Actions*`,
              draft: false
            });
            
            console.log(`Pull request created: ${pullRequest.html_url}`);
            
            // Add labels if desired
            try {
              await github.rest.issues.addLabels({
                owner: repoOwner,
                repo: repoName,
                issue_number: pullRequest.number,
                labels: ['dependencies', 'submodule-update', 'automated']
              });
            } catch (error) {
              console.log('Note: Could not add labels (labels may not exist in target repo)');
            }

Summary and completion #

These final steps provide logging and status reporting:

      - name: Summary
        run: |
          if [ "${{ env.UPDATE_NEEDED }}" == "true" ]; then
            echo "✅ Submodule updated and PR created successfully!"
            echo "Repository: ${{ matrix.config.target_repo }}"
            echo "Path: ${{ env.SUBMODULE_PATH }}"
            echo "Branch: ${{ env.BRANCH_NAME }}"
          else
            echo "ℹ️ Submodule ${{ env.SUBMODULE_PATH }} in ${{ matrix.config.target_repo }} is already up to date."
          fi

  final-summary:
    needs: [ setup, update-submodules ]
    runs-on: ubuntu-latest
    if: always()
    steps:
      - name: Workflow Summary
        run: |
          echo "🎉 Submodule update workflow completed!"
          echo "Processed repositories based on configuration matrix."
          
          if [ "${{ needs.update-submodules.result }}" == "success" ]; then
            echo "✅ All updates completed successfully"
          elif [ "${{ needs.update-submodules.result }}" == "failure" ]; then
            echo "❌ Some updates failed - check individual job results"
          else
            echo "ℹ️ Workflow completed with status: ${{ needs.update-submodules.result }}"
          fi

[Bonus] Auto-accept + merge action #

I’m leaving this as an open recommendation (and homework lol) for you to implement. The assumption here is that the change is generally safe and so it should be auto-accepted + merged into the target repositories once it passes all the tests.

Wrapping Up #

This automated submodule update workflow can save significant time and reduce the risk of human error when managing shared code across multiple repositories. While the initial setup requires some configuration and PAT management, the long-term benefits of having consistent, up-to-date shared code across your entire project ecosystem are substantial. The workflow provides clear visibility into what’s being updated through detailed pull requests and commit messages, while still maintaining the safety net of requiring CI tests to pass before any changes are merged. Consider implementing the bonus auto-merge functionality only after you’ve gained confidence in your test coverage and are comfortable with the automation’s reliability.

Firefighting Heroes

binhong@binhong.me (BinHong Lee) — Fri, 01 Aug 2025 00:00:00 -0800

We love celebrating heroes in general. So when a hero shows up to save the day, it’s only natural that we make sure to recognize their contribution. While it might feel counterintuitive, we should strive to not need heroes entirely instead of hoping that next time when tragedy strikes, a hero will turn up once again. This is not to diminish the value of heroes. They still play a very important role in firefighting and are doing the right thing in an urgent situation. However, in a perfect world, they shouldn’t be necessary at all and we should at least aspire to work towards building a system with that level of reliability.

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Bian Que and his brothers #

If you’ve never heard of the story about Bian Que and his brothers, here’s a Meta AI summary of it with all the relevant details:

Bian Que was a renowned Chinese physician who lived during the Warring States period (475-221 BCE). According to legend, Bian Que and his two older brothers were all skilled in medicine.

The story goes that the eldest brother treated illnesses before they became apparent, the second brother treated illnesses when they were still subtle, and Bian Que treated illnesses when they were severe and obvious.

When people saw Bian Que’s dramatic cures, they credited him with great skill. However, Bian Que said that his brothers were actually more skilled because they treated problems before they became serious.

This story highlights the importance of preventive medicine and early intervention in Chinese medical philosophy.

I mention this because one time when I was complaining about all the fires happening in our org and how I keep getting looped into them to help put them out, my then tech lead cited this story to me (which is a little offensive 😡 but she’s not wrong lol). I think in general, this is an accurate depiction of the misplaced prioritization and recognition where we consistently reward “heroes” solving issues rather than those who worked to prevent fires from happening to begin with simply due to visibility and measurability.

Automate whenever possible #

I remember once reading a Site Reliability Engineer write (verbatim), “my job is to automate myself out of my own job” which I thought was an interesting way to look at things but also an obvious starting point. To do away with the reliance on firefighting heroes, automate the things they do whenever possible. As an example, when something breaks and you find an automated test for that specific breakage, have a system that automatically runs bisect against said test to find the blame commit causing the breakage. Taking it a step further, the system can run all these automated tests every time before release and if anything is broken, bisect to find the blame commit and revert it, then run everything again until there’s a good stable version that’s ready for release.

Prevention > Cure #

As the idiom says, “prevention is better than cure”. In a previous piece about Process over People, I’ve talked about the focus on building process as a preventative measure. This is a similar point where we should ideally focus more on preventing outages from even happening instead of just putting them out after they’re already on fire. The hard part however is - yet again - that the visibility of good prevention work is never as prominent as good firefighting work. There is a two-fold solution to this (where ideally both need to happen). Firstly, if you (or you know someone) are doing important prevention work, ensure people are aware of their work. Secondly, leadership needs to prominently acknowledge the value of such work and reflect it as such in whichever-way-they-usually-do-recognition and in performance reviews.

Accountability on reliability #

One of the common pitfalls here is that, once something is shipped, the responsibility to keep it running falls on the team (whoever is on-call at the time of incident) instead of the individual who shipped an incomplete product. This is something I’ve previously touched on here. If someone is taking credit for the successful launch (and all its underlying impact), they should also be held accountable if the launch causes reliability issues (be it during or shortly after the launch). As to how accountability looks, it can range anywhere from taking ownership of resolving the issue long term, to being penalized in their performance review cycle for an incomplete launch. There’s quite a bit of nuance here depending on factors like the scale of outages, any visible warning signs, how preventable the issue was, etc.

Wrap up #

This is as much (if not more) of a culture problem as it is a technical problem. Building reliable systems isn’t just about writing better code or implementing more monitoring. It requires fundamentally shifting how we think about and reward engineering work. We need to move away from the hero worship that celebrates last-minute saves and instead build cultures that value the unglamorous work of prevention, automation, and long-term system health. Heroes will always be needed in truly exceptional circumstances, but if your organization consistently relies on them to keep the lights on, it’s a sign that your systems, processes, and culture need serious attention. The goal isn’t to eliminate heroes entirely but to build systems so robust that heroic interventions become rare exceptions rather than regular occurrences.

Major Incident Runbook

binhong@binhong.me (BinHong Lee) — Fri, 25 Jul 2025 00:00:00 -0800

I wrote a similar version of this internally at Meta a few years ago for my org after finding myself in the middle of a few SEV1s in a row – and being consulted / asked for support in other similar situations. I thought this might be something useful to share (as a public version) as well. This won’t be perfectly fitting for all use cases, but having a runbook works as an anchor in the midst of chaos, helping to get you unstuck from “what’s next?”. Admittedly, this is an incomplete runbook that serves more as a template for your team or company to complete with more specific tooling guides (using which tool to achieve what, etc.).

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Stop the Bleed! #

The highest priority is to stop the bleeding immediately. Flip a killswitch, roll back changes, apply server-side fixes, apply client-side fixes - in that order. A killswitch generally propagates faster, thus is preferred over code changes, but even then, prioritize rolling back changes instead of forward fixing. Forward fixing adds more unknown factors into the mix (because that’s more new code which could now cause new / different problems); rolling back, on the other hand, is more predictable. Server-side fixes over client-side fixes should be pretty obvious since you can guarantee the server version (as the service provider), but you can’t always force a client to update (native apps), or there could be some caching involved (web).

Multiple Workstreams #

During a major incident, there are generally multiple things that can or need to happen in parallel. Break them down into clear workstreams and delegate a domain expert to run each of them. If more things are discovered down the line (or the situation changes), switch up the workstream and get different people (domain experts) involved to run different things. Since you’re dealing with an incident (which are usually time sensitive), never hesitate to call people. This needs to be said a lot because people constantly hesitate about false positives or getting on others’ bad side for inaccurately calling them up. But you won’t know what you don’t know without getting the domain expert to show up and verify what you’re seeing.

Chat Management #

Alongside having separate workstreams, you should also set up separate chat threads for each workstream to keep the main chat low on noise. That said, make sure to announce the establishment and / or major milestones of each new workstream clearly within the main chat so everyone relevant is correctly included. You might be in a lot of different chats and things will be chaotic, so you will need to take extra care in ensuring that the right people are in the right chat to allow them to get their different tasks going. Everyone should still be in the main chat, but most discussion should happen in the workstream chat.

Sync Meetings #

There are generally 2 types of sync meetings. First is the “everything just happened so we started a meeting to info dump and get everyone up to speed.” This meeting will always be chaotic with a lot of people joining to figure out what’s happening and what to do next. (There might also be people who only joined out of curiosity, but as long as they aren’t interrupting, they are the least of your problems - unless Zoom is not scaling lol.) Set out a clear goal, a set of tasks, and an owner for each of those tasks. Finally, set a follow-up date and time (usually in a few hours) for everyone to regroup with new findings and progress to determine next steps.

The second type is the follow-up and / or recurring meeting. This will feel more like your regular team standup meeting except with a lot more urgency. It’s important to note that if there’s any major breakthrough, people shouldn’t wait until the next planned meeting time to report (because, well, this is an outage lol) but should instead share it with everyone immediately. This goes back to the previous point about chat management where you (as the incident manager) will need to monitor each workstream chat to catch something like this. After resolving the core issue, there might still be important cleanup work needed to be handled with a similar level of urgency to prevent the same outage from happening again in a very short time. In these long-running incident cleanups, you might opt to have daily (or even twice-a-day) sync meetings until the cleanup is complete.

Wrap up #

As a whole, you will largely be more like a PM / TPM instead of a regular engineer when handling a lot of this alignment work. If you think someone else is better suited to handle this (another more senior engineer or your manager, etc.), ask for help while you focus on the thing that you do best (likely as the subject matter expert on investigation or remediation). Remember that incident management is a skill that improves with practice - don’t expect to be perfect on your first few incidents, even experienced engineers can feel overwhelmed when everything is on fire. Don’t forget to acknowledge the team’s efforts since major incidents are stressful for everyone involved, and use this runbook as a starting point while adapting it based on your team’s specific needs and continuously improving your processes with each incident.

Understanding the Value of Dev Tools

binhong@binhong.me (BinHong Lee) — Fri, 18 Jul 2025 00:00:00 -0800

I’m largely approaching this from the perspective of building an internal dev tool (since this is mostly where my personal experiences are coming from), especially if you’re someone who wants to build an internal tool but is having trouble framing it in a way where its value can be understood by the decision makers. I’m also excluding situations where you “have to” build certain tools to track certain information in order to comply with a legal requirement, since I don’t think that would be controversial to comply with. I’d imagine the framework for using an external dev tool, while similar, also poses extra complexity on paying vs building.

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Cost of engineers #

Generally, the simplest way would be to calculate how much time is saved for an employee, multiplied by the cost of hiring the employee. This directly shows how much money the organization is saving by just having this tool around. Pretty straightforward, not a lot of controversy.

Value generated #

In organizations where engineering is considered a profit center, and not a cost center, it might make more sense to calculate it with the value generated by the employee against the time saved. That said, this is usually much harder to measure compared to the above, thus more uncommon unless the tooling itself is a direct value generator.

Urgency (Disaster Recovery) #

In a disaster recovery situation, the time is significantly more valuable than the average “engineering hours” since it’s directly affecting the bottom line of company profit or other top-line metrics (or even consumer trust of the product). Admittedly I’ve never figured out the perfect framing around this, but I’d reckon phrasing it as “used to resolve x, y, z incidents” (and if available, attach the severity level for each of the incidents to it) would suffice in clearly articulating this.

Employee satisfaction #

This is one of the more subjective measurements. While you can attempt to quantify it through things like survey responses, it’s important to keep in mind that the improvement will automatically become the new baseline. For example, an 8/10 last year will become a 5/10 this year, so you need to improve it even more to get that same 8/10 again this year. On some occasions, additional improvement will hit the point of diminishing returns and have very little investment value. It’s important to take notice and begin pivoting effort into better sustainability.

One-off use case #

Depending on the level of effort, one-off use cases can be a bit of a dangerous misdirection / mismanagement of valuable resources (time). It’s sometimes hard to notice, but if it takes you longer to build the tool than to just do the work without it, or the tool provides very minimal value compared to the amount of work needed to build (and maintain) the tool, building them makes no sense. Generally, this is usually a rare problem and only exists for top-down asks because leadership might not understand the day-to-day workflow for impacted employees. Instead, we generally see the opposite where even the most valuable / impactful tools take a lot of work to convince leadership that they’re worth the investment.

Not-Invented-Here Syndrome #

Another common trap in building internal tools is the Not-Invented-Here Syndrome. This is especially true for mission-critical tools (alert paging, traffic monitoring, authentication etc.) where it would make more sense (for most companies / orgs) both financially and expertise-wise to outsource them to dedicated product experts in the market, making “buying” a better option than “building”. “Buying” here doesn’t necessarily mean monetary purchase either. For example, if your team begins scaling your code base and needs a large-scale cross-stack build tool, it would make more sense to adopt something like Bazel or Buck instead of building your own new build tool from scratch.

Wrap up #

Understanding the value of dev tools requires matching your measurement approach to what resonates with your organization’s priorities. The goal isn’t just to justify building tools, but to build the right ones which can sometimes be as simple as buying an existing solution or accepting a manual process.

Internal Tooling Ideas

binhong@binhong.me (BinHong Lee) — Fri, 11 Jul 2025 00:00:00 -0800

For years, I built and maintained the only logged-out accessible dev tool set / platform at Meta. That earned me some reputation (in a certain circle) of being “the idea guy on internal tools”. Whenever I’m asked about how I keep coming up with good ideas for valuable tools to build, my go-to answer has been “I build tools when I get annoyed while doing my job”. I’ll try to expand that into more details below.

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Repetitive Typing #

The biggest value of having logged-out accessible dev tools is that you no longer have to log in to access dev tools. Login, while generally not too big of a friction for one-off situations, can be very bothersome if you have to type your credentials over and over again (for dev purposes) as you login (update config) → logout (test) → login (update config) → logout (test). If bypassing or removing the typing aspect is not possible, consider if it can be replaced with multiple choice button selection or just a single copy-paste friendly input.

Unnecessary Waiting / Processing #

There are generally 2 types of waiting problems. The first one is where the wait takes a really long time and can be done asynchronously. Either move the wait into an async manner and integrate a messaging system to notify the author when it’s completed, or pre-process them based on known patterns. For example, companies with large native app (iOS / Android) code bases have long build times. You can integrate build tools like Bazel with remote caching so clients don’t have to rebuild everything from scratch. Since an engineer will likely want to pull from remote HEAD (or stable) daily, you can have a cron job that pulls and builds daily in the morning before the engineer starts working, thus turning their wait block time into an invisible operation done automagically while they are having their breakfast or morning coffee. The second type is where the wait is one part of the process and can’t be (deterministically) pre-processed ahead of time / asynchronously. In this case, consider if bypassing is an option (e.g., skips in dev but runs in prod). If not, you can also try making it non-blocking (move it off the main thread, batch processing, backfill etc.) as the user proceeds through the process. If all else fails, you can always turn that part of the process into an async step with an integration to a messaging system notifying the author once it’s ready to proceed to the next step.

Device Config > Manual Hard-code #

In many of these cases, you can probably do manual hard-coding to get the intended behavior. However, doing so increases the risk of accidentally committing such hard-coded changes into prod (if not caught during code review). On top of that, when working on a complex code base, the engineer might not be familiar with the code pointer for where the hard-code needs to go. For instance, if you want to bypass the rate-limit of your feature but the rate-limit logic is owned by a separate team in an unfamiliar code base. As an added bonus, this also allows non-technical members of your team (designers, PMs etc.) to do product audits, dogfooding, and in-depth testing independently without needing dedicated engineering support.

Non-deterministic Behaviors (ML override) #

This is somewhat similar to the manual hardcode situation mentioned above where you might want to override an ML decision to manually test out different decision combinations by the model. ML models (and AI) are relatively non-deterministic by design, but when building / testing a product, you want to make sure that you covered all the different possible scenarios. Having an override to switch between different potential responses allows for better coverage on both manual and automated testing, ensuring that your change behaves as intended.

Chore #

This is more about employee satisfaction when working on a repetitive task may be boring or frustrating. Having tools to automate such work allows them to explore more interesting / challenging work, thus improving employee satisfaction, even if it doesn’t necessarily save a significant amount of time. The best “realistic” way to measure this would be a rating system where you count the number of employees using it and ask for their feedback (both positive and negative), then compile to show the value it provides.

Wrap up #

Building internal tools is about removing friction from your team’s daily workflow. The best tools emerge from genuine pain points you experience while doing your job. Start small and focus on the annoyances that happen most frequently. The key is to stay observant of your own frustrations and act on them. Every great internal tool started with someone saying “there has to be a better way to do this”.

Early takes on vibe-coding

binhong@binhong.me (BinHong Lee) — Thu, 03 Jul 2025 00:00:00 -0800

I keep hearing about vibe-coding and I’ve always written the majority of code myself. While at Meta, I got a chance to try out CodeCompose. It worked really well as an autocomplete but when it tried to do anything more than 5 lines at a time, it would - on many occasions - commit bugs that aren’t immediately obvious at first sight. Generally, I’ve caught them by looking at the generated code and wondering “huh this isn’t how I’d do this, why?”. That said, it definitely helped me code and ship faster especially on mundane tasks. Vibe-coding though, seems like taking it to a whole new level (using even less supervision and care on the code being committed).

Perfect for small, isolated problems #

I started my attempt by making Claude code out a GitHub Action workflow file. I have a submodule setup (where a repo is shared and imported across multiple other repos) and wanted to have an automated way to tell how its changes will affect code on other repos while also creating PRs to keep them updated. Seems like a perfectly fine isolated problem to try this out on. I did run out of tokens a few times (being on a free plan) so I had to get creative but it largely worked. I’d say it behaved like a normal engineer writing a first version (which isn’t perfect) but can understand and work its way through debugging and resolving the issue slowly when given clear information on what went wrong.

Not for complex changes in an intern-size project #

Note: Using the phrase “intern-size” here because back then, there was a weird rumor that interns were expected to ship 10k LoC as part of their internship to get return offers in FAANG lol. I don’t think it was ever true but definitely a standard people worked towards.

Now that I’ve got it working on an isolated problem, I wanted to see how it might handle a complex change in a pre-existing project. I have an Android app codebase (for GlobeTrotte) with around 8k+ LoC so I decided to try it on there (using SWE-1 from Windsurf). This is the instruction I provided (admittedly a complex one):

add new navhost to edittripactivity and make each of edit day and edit place a separate screen instead (so it push-and-pop for each small edit)

PS: edittripactivity is a file name (technically EditTripAcitivity.kt but I think the LLM understood it), navhost is a concept of how screen navigation works in Jetpack Compose.

The LLM took 20+ minutes before running out of time which required me to make a continue call not just once but twice before telling me it was done. It’s all chaos from here on out. It tells me that there are a bunch of errors so it tries to write more code (?) leading to more errors, so more code, then more errors etc. At some point, I mentioned that there were 88 errors and it figured to try compiling and reading the compiler error (instead of looking for them itself) but that barely cut down the number of errors. I just kept telling it that there were more errors and it just kept trying to code itself out of the mess by adding more code and thus more errors. I eventually gave up and ran git checkout . to clean everything up.

Losing track of signatures #

At some point, it started making up stuff that either existed with a different name, or something that it thought should exist but didn’t (or it forgot to add the implementation for it, I can’t tell). The first example is that it keeps calling PlaceItem() even though there’s no object with that name (and all the please fix error prompts never saw it touching them). There is however, an object called Place() which I’m assuming is what it was referring to. The second example is where it called updateDay(delete = true) despite the fact that updateDay() has a bunch of other required params while it also doesn’t have delete as a param. I can only assume that it just inferred the functionality of the function without actually understanding if it worked as intended.

Ask clarifying questions #

The prompt I provided is a bit vague to be honest. It’s asking to make a UX change without actually providing any design example but rather just describing it with words as if the other person would easily understand it. The LLM went to work immediately with that prompt without asking for more clarifying questions like how the screens get triggered, how the layout should work, how the UI should look etc. I think if LLMs can learn to ask clarifying questions, it can be invaluable for situations like this where the ask might be a little too vague to work off of.

Phenomenal auto-complete machine #

I’d be remissed if I didn’t mention the auto-complete capabilities of AI coding assistants. In short, they are consistently phenomenal especially when it comes to boilerplate code needing minor tweaks here and there. The AI would make the necessary tweaks automatically making it a breeze when going through the more mind-numbing part of the code base. This is a consistent experience both when I was at Meta (using CodeCompose) and now using Windsurf for my personal project.

Is it a mid-level engineer yet? #

Short answer, no. Long answer, it depends. In terms of raw coding ability in an isolated environment, I think it’s meeting the mid-level engineer mark just fine (maybe even better due to its breadth generally uncommon among “humans” lol) but it’s the everything else part that’s an issue. For starters, I expect a mid-level engineer to ask for help instead of mindlessly trying to commit code (or send out PRs) over and over again that isn’t compiling. I also don’t (usually) have to nudge them that their code isn’t compiling or failing tests. They can see it themselves and would go work on debugging and fixing them proactively. This is on top of all the issues mentioned above when working in a not-even-that-large of a codebase.

For now though, it seems like it’s still not good enough to take over even just the coding part of my job so I guess I’m going back to implementing the new navhost for my Android app by myself.

Experiment Review Process

binhong@binhong.me (BinHong Lee) — Fri, 27 Jun 2025 00:00:00 -0800

Mature growth teams would organize a centralized experiment review meeting as a way to share learnings to a wider audience, consult for feedback / next step recommendations, while also holding engineers accountable for the changes they are attempting to ship. The review sessions should be open to anyone to sign up (presenting their experiments) or to participate in general. However, key decision makers (like senior growth engineers, product managers, designers, data scientists) should be required to attend so decisions to ship / not ship / iterate will be made with everyone’s concerns addressed.

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Understanding the metric shifts #

When presenting (and reviewing) an experiment, not only do we look at metric shifts but we need to also make sense of them. Understanding them helps inform making better future changes and iterations while ensuring that this change is working as intended instead of some potential regressions on metric blind spots. This usually means that metric analysis for experiments would go beyond just top-line goal / guardrail metrics but also onto regular impression / click loggings to make sure any underlying funnel shifts make sense.

As an example, when you add stories to the top of your app, you’d see an increase in stories traffic. Make sure to also verify that users are actually clicking on the newly added stories component. At the same time, you might be sacrificing screen time on other parts of your app (especially the feature that used to occupy that screen real estate) so you should also see some drop on that end.

Atomic changes #

This is a bit of an extension from the previous point. It’s not uncommon for someone inexperienced to test an overly complex change at one go instead of breaking it into a list of smaller incremental changes. In order to understand which change moved which metric, you need to properly isolate your changes into individual groups (or mix-and-match them with multiple test group setups). This allows you to not only understand what works and what doesn’t, but also how each of these different changes might have affected one another. If someone brings in an experiment that’s overly complex, you should feel comfortable asking “why can’t it be smaller (or be run with more test groups)?”.

Conflict of interest #

Last week in Growth Engineer we talked a little bit about how sometimes engineers would fall into the trap of looking solely at metrics, over-optimizing them. This is generally good for the engineers themselves (being able to boast larger numbers on their launch) but bad for the business (since they are unlikely to be sustainable long term and will probably regress over time). The role of an experiment review process is to ensure that underlying risks or concerns like this get called out and taken into consideration for ship decisions. In the next few paragraphs, we will explore a few examples of such conflict of interest.

p hacking #

Usually intentional but sometimes oblivious to the underlying logic, an engineer might participate in p hacking that makes the experiment look better than it really is. One of the most obvious examples would be when someone picks a weirdly specific date range to pull their metrics instead of the more logical (like 1 / 2 week average) standard range. While most experimentation tools would do the statistical analysis and show you results with 95% / 99% / 99.9% confidence levels, there could still be variance (especially on the absolute value derived from its average). On a large change (or accumulation of a lot of smaller changes), this can amount to a rather significant amount.

Long term holdout #

On some occasions, the effect of a change (especially negative effects) can only be observed over a longer period of time thus making it important to set up a long-term holdout to better understand the longer-term impact of the change. This generally requires some level of product sense and experience to notice thus we would heavily rely on experienced members in the org to call this out. One common example would be “user intent” (which was previously discussed more in-depth here) where long-term behavior of users will better reflect a user’s intent as they adopt new habits towards the product change. The other important part of this (often overlooked) process is to follow up on how the long-term holdout performed. It’s not uncommon for teams to launch with a holdout then completely forget about the existence of a holdout (especially if the team or product space ended up going through reorgs) leading to a group of poor users still stuck with the sub-optimal experience.

Wrap up #

A strong experiment review process acts as both a learning forum and a quality gate, ensuring experiments are properly analyzed and potential conflicts of interest are surfaced before they impact users. The goal isn’t to slow down shipping, but to ship smarter by using the collective experience in the room to catch blind spots that individuals might miss.

Growth Engineer

binhong@binhong.me (BinHong Lee) — Fri, 20 Jun 2025 00:00:00 -0800

While I’ve shipped a lot of growth wins (literally the first line on my resume), I’m actually very far from a prototypical growth engineer. That said, in this piece, I want to explore a bit more into what it’s like being a growth engineer and what makes you good at being one. Growth engineers are generally 1 -> 100 experts instead of 0 -> 1. They fine-tune every little detail by running a lot of experiments with marginal changes to understand the user problem and drive growth impact (line goes up).

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

A/B Test Everything #

When I say everything, I mean everything. Every button change, content change, margin change, slap an experiment on it. You need to be numbers obsessed and understand how anything moves the topline metric (the metric you / your team cares the most about). In that same sense, you might want to run experiments in chunks of small changes instead of a big chunk to better understand how each small change affects the user behavior. This would later help you in better understanding what is a good growth lever vs what isn’t. Alternatively (depending on the type of changes), you can also have multiple variants with different mixes to achieve something similar.

For more on A/B testing, check out this other post I wrote a few weeks ago here.

Understanding Scale #

Scale decides necessity and value. If you have no scale, then there’s nothing at the top of your funnel to begin with, which makes optimizing the rest of the funnel a common “preemptive optimization” behavior. Don’t add complexity in places you don’t need just because “it’ll be useful in the future”. You can build them in the future when needed, then actually measure the value of such a feature to validate your hypothesis (more on this later about A/B testing).

Identifying opportunities #

I wrote a separate complementary piece on this a few weeks ago here mainly because it ended up being so long that it probably deserves to be its own separate thing. This will be a core part of your job and in my opinion the biggest separation factor for a “pure bred” (lol) growth engineer. Many engineers get stuck or move away from growth over time (especially around IC5 -> IC6 level) as they realize that they can no longer come up with new ideas and directions to continue growing the product sustainably.

Cost analysis #

On some occasions, there will be cost tradeoffs. Most commonly, by spending more money you can get more users (promotions, SMS cost, etc.) which makes it very important to define and understand how much spending is reasonable to earn how many users (cost per user). From there, you should only target to ship projects where it shows that the cost per user is lower than the given guideline. Important to note that cost here isn’t always directly money, but also valuable real estate (prominent position in the app) or attention (push notification, email, messages) which some could also have a hard-cap (on top of the aforementioned cost per user).

User intent #

You can’t force someone to love you.

When someone (or a team) is metric chasing, the actual user (and their intent) can sometimes be neglected. It’s always important to keep them in mind and practice some empathy to better understand what the user actually wants and build around that. Even if you were able to trick users into something they do not intend to do, the long-term effect of it is likely unsustainable.

I had a personal counter-example here where I once fixed a 3rd party login issue (after login success, instead of redirecting users back to the 3rd party, we would send them to news feed) resulting in a 2M MAU loss from our test. However, it made sense only because we count every person who sees feed as MAU even when (in this case) they did not intend to do so. We ran a separate long-term experiment and saw that the actual loss was significantly lower than that (like maybe only 5% of the original number) because these people were really just trying to login to a 3rd party app (Spotify, Tinder etc.) instead of wanting to browse news feed.

Failed tests #

When you take a bet and it didn’t materialize, your instinct might be to lay low and quietly do away with the project entirely. That would be a massive mistake. Instead, you should focus on what learnings or takeaways you have that can be shared with everyone else especially for someone who might want to try this again in the future. Generally, there are some key factors on failure root-cause. Make sure to clearly articulate them as you announce your experiment result for the purpose of having clear future reference. If / when someone were to try this again, they can look back at your attempt and see if the landscape had changed enough (based on your learnings) that it might warrant a new attempt.

Long term impact #

On one hand, everyone hated the idea of a long-term holdout (where a certain group of users will just have missing features for an extended period of time); on the other hand, you need a way to measure long-term impact of your changes. That said, due to lack of commonality in this practice, the accuracy of such measurement may not be as high as regular shorter-term tests.

Long term impact is a myth - some E6+ Growth Engineer

The timing makes things worse. By the time you see results from a long-term experiment, you’re usually in a different performance review cycle than when you built it. This makes it nearly impossible to properly reward thoughtful long-term thinking or penalize short-sighted decisions. What should be a performance measurement tool becomes just a way to look back and say ‘oh, that’s interesting’ after it’s too late to matter.

Wrap up #

Growth engineering is fundamentally about disciplined curiosity. You’re constantly asking “what if we change this?” and then actually finding out through experiments rather than debates. It’s the difference between having opinions about user behavior and having data about user behavior. Always remember that growth isn’t just about running A/B tests. It’s important to also understand how / why users behave a certain way. The challenges are often less technically complex but more about measurement accuracy, experiment design, and building user-friendly interfaces that even your grandma can easily understand.

Product Growth Opportunities

binhong@binhong.me (BinHong Lee) — Fri, 13 Jun 2025 00:00:00 -0800

It’s never easy to come up with new ideas that help with growth, but identifying the problem makes it easier. You’ll notice that for the most part in this piece, I’ll talk about “where” the opportunities are instead of “what” because that’s usually very domain specific and highly depends on the type of problem you ended up needing to solve.

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Funnel analysis #

The first thing you learn about identifying opportunities in growth is funnel analysis. Build a funnel logging to understand the user journey and go through it to see where you have the biggest drop. From there, you can take a deeper dive into the flow to understand why it is the way it is and see if it can be improved (or if there’s any bugs to fix). Similarly, you can go through bug reports to find patterns on how users are being stuck on a specific part of your flow that might have contributed to the funnel losses.

Product Experience #

Alternatively, you can go straight to the bug reports to see if there are any obvious blocking issues. From there, you can test the fix with an experiment and eventually ship them. Even when it doesn’t yield material impact, the changes are generally safe to ship (unless it caused unexpected regressions) since they are largely considered “bug fixes”. Aside from that, there are other common issues like app performance, screen load time that have a diminishing return curve in user impact.

My favorite example here was when Uber was working on an iOS rewrite, while everyone was debating about how cellular download limit matters (or not), a data scientist pulled together an experiment to show its material impact (full story here - which is one of my favorite software war stories). Similarly, your team should design and run experiments to understand how certain product experiences can have material user impact (instead of just theorizing over them).

“We’ve tried that before” #

Often times, new people will join and suggest old ideas as new. When that happens, it’s easy to dismiss the idea by citing that something similar has been tried previously and didn’t yield the expected result. The hard (and valuable) thing here, however, would be to understand why it failed previously and if anything has materially changed since then that might now allow this to be a viable idea to be re-attempted. You might still fail, but you will likely learn something new instead of learning the same old lesson again (which would be a waste of everyone’s time and resources).

Studying your competitors #

Sometimes you might find a specific step that has problems (where you’re losing a lot more conversion than anywhere else) but you can’t figure out how to fix it. Studying how your competition does it can help you see if there’s a better way to do that. This doesn’t mean their solution would perfectly fit your problem, but it might be worth testing to see if that solution is similarly applicable for you. Alternatively, they might also be struggling with the same issue and what you’re both doing is (at least for now) the best solution that anyone has thought of so far. I listed this a lot lower intentionally because it’s generally not been that valuable of an option compared to the other ways of identifying new opportunities above.

Survivorship Bias #

I wrote this section title with the “WWII plane with red dots” image in mind. In case you aren’t familiar with it, here’s Claude describing it. In a similar sense, it can be very easy to fall into the trap of focusing on the demand of your existing users instead of building for new users you have yet to (and want to) acquire. Not to say that existing user feedback is unimportant (which I’ve literally just said the opposite previously), but rather that it’s important to understand and distinguish which feedback helps keep your existing users happy while which feedback is preventing more users from adopting your product.

Wrap up #

Finding growth opportunities isn’t about hunting for silver bullets, but more about understanding where your product is bleeding users and why. Don’t let growth pursuits distract from building something people actually want. The best growth “hack” is still making a good product that gets people what they want.

A/B Testing

binhong@binhong.me (BinHong Lee) — Fri, 06 Jun 2025 00:00:00 -0800

This is a basic introduction on how to run a good A/B test. A/B testing is a method where your user pool is segmented into multiple groups, allowing you to test different product interactions and understand how these changes affect user behavior. For any metric / data driven team, A/B testing serves as a critical tool in measuring success.

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Tooling #

If your employer does not already have a working tool for experimentation, I’d recommend adopting one of the existing tools / platforms (like Statsig, GrowthBook, LaunchDarkly etc.) specifically built with this in mind. On a basic level, they should provide a toggle that splits users into 2 (or more) groups for testing purposes. Their downstream metrics (after first exposure) will then be used to measure if / how these changes might have affected the users. While it’s possible to build your own A/B testing tool in-house, the build and maintenance cost is likely not worth it compared to just adopting ready-made ones.

Goal + Guardrail metrics #

“When a measure becomes a target, it ceases to be a good measure” - Goodhart’s Law

Goal metrics serve as a target, the specific thing you want to change or improve. While I largely agree with Goodhart’s Law, I also believe that a sufficiently comprehensive set of metrics will help mitigate the downside (or at least minimize it) thus making it still relatively “a good measure”. This is where guardrail metrics come in to ensure we aren’t just “sacrificing x to boost y” especially in an unsustainable manner.

As an example, if you want to increase usage on your app, you can provide generous discounts (or even pay users to do so) but that’s obviously very unsustainable so you should set a clear budget on how much you can spend on these promotions and / or cost per new active user acquired you allow for such a project to ship. Here your goal metric would be “new active users acquired” while your guardrail would be cost (lost revenue). They need to both look good from the experiment before you decide on shipping the product change.

Size and Duration #

Generally, size and duration are the 2 main levers to help get your experiment sufficient exposure that allows you to make (statistically) informed decisions. If you don’t have a large user base then many of your tests would need a large test group with long test duration to show any statistically significant movement. This unfortunately means that your new feature release cycle will be much slower as your experiments will be queued. In this scenario, you will need to smartly pair up different features that complement one another into the same experiment.

Holdout #

In order to accurately measure your team’s impact over a certain period of time, you can create a team-wide holdout that keeps a small group of users without the new enhancements. From here, you can see the value of all your team’s projects over that period of time (quarter / half) collectively. When you run experiments, you are only testing the impact of specific changes while here you are testing them collectively. Some changes conflict with each other while others complement each other. Ideally you catch them early and run variations of different combinations but you don’t always catch them all so having a team wide holdout helps with tracking that.

Novelty effect #

When a new restaurant first opens in a popular area, it will be filled with people lining up to try it out. These people will then attract more people who also decide to check out the restaurant out of curiosity. However, the crowd will slowly wind down over time as it loses its novelty status. If it’s any good, it would still remain popular but unlikely to be as popular as it first opened. This is called novelty effect. When you add a new button to the screen, the user’s likelihood to click on it increases due to their curiosity on what it does. You don’t want to measure that. Instead, you want to measure the actual traffic (and effect) of it after the novelty wears down. Just like how a good restaurant will still be popular long after its initial launch, you want your feature to be valuable / impactful even long after it’s been launched and users clearly understand what it does.

Security #

When making a security related change (fix or enhancement), a lot of these concepts become a bit trickier especially for critical issues. Mainly because bad actors can figure out that you are doing an A/B test based on their account ID / device ID / IP address and do targeted exploits on the unprotected test group. This makes it rather a futile effort to understand whatever topline metric movement you want to study of this change on regular benign users but instead, becomes more like opening a new window for exploits.

Wrap up #

Running good A/B tests isn’t just about having the right tools—it’s about building the discipline to ask the right questions and measure what actually matters. The fundamentals covered here will prevent most common pitfalls that lead to misleading results or shipping changes that hurt your product. A/B testing is a means to an end. The best growth teams quickly kill bad ideas and double down on what’s working. Start simple, get comfortable with the process, and don’t let perfect be the enemy of good—a simple, well-run test is infinitely better than a complex experiment that sits in planning forever.

No Blame SEV (Incident) Culture

binhong@binhong.me (BinHong Lee) — Fri, 30 May 2025 00:00:00 -0800

Every time there’s a major outage at Meta, the first question I get from friends and family is usually “did they fire the person who caused it?” which is where I have to explain this concept of No Blame SEV Culture. Especially for an outage so big that a significant number of users are affected, the individual causing it likely does not have ill intent and there are likely multiple different processes and systems that failed along the way to get us here in the first place.

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Process over People #

When something goes wrong (especially something really catastrophic), it’s usually a combination of both process and people problems. The difference here is that process is more deterministic compared to people. People have off-days, get tired, make mistakes etc. so it’s important to have a process (or automated systems) in place to prevent that. This can mean anything from adding more test coverage, lint rules against bad code patterns, and / or more alerts. It is however important to note that they need to maintain a certain level of quality bar. As mentioned in the previous article, flaky / broken tests are tech debt, same goes for noisy lint rules and alerts. Too many noisy lint rules and alerts would lead to engineers disregarding them or adopting a “wait-and-see” mentality which is not ideal in preventing future outages.

Expensive Lesson #

One of the more interesting quotes I’ve read repeatedly (both within and outside of Meta) about people who caused outages is that they just learned an expensive lesson through that specific outage. Firing them (or letting them go) would mean that your company just paid that expensive price of such a lesson for an employee without actually benefiting from it. This employee will then bring this lesson with them to their next employer who would then benefit from such experience.

Fear #

One of the more significant downsides of blame, is that you now instill fear in making any sort of production changes (even calculated ones). Instead, it’s important to keep in mind that as your product / infra grows, so should your process. Having strong fear in taking any responsibility for even attempting to improve or make fundamental changes breeds complacency. This can be fine in certain organizations and products (like government software, health tech etc.) where there’s almost no tolerance for any sort of outages. That said, this is where you should have good chaos engineering and fail-safe practices to ensure the resiliency of your system.

No Blame ≠ No Responsibility #

This is a bit of an exception or outlier effect (and likely the most controversial part of this whole piece). Usually when you cause a really major outage (or multiple for that matter), it’s really not your fault (or shouldn’t be). But sometimes, smaller outages are understandably less “well protected” because we expect people to still care about the things they work on. If you continuously cause outages due to recklessness (“lack of care”) especially within a short period of time, you should still be held accountable for it. It’s especially common when someone chases the topline metrics movement against a tight timeline (end of a performance review cycle). This does not mean that people should be finger-pointing during the incident review, as before, that should be used to focus on what could’ve been better instead. However, it should be brought up separately as part of the performance conversation. It’s important to note that this is a scenario where a quantitative change leads to a qualitative change since an increase in quantity (of incidents) leads to a change in narrative thus should not be used to penalize those who’ve only caused one (or maybe two) incidents in a given period of time.

Aside from that, if there isn’t any runbook or recovery plan prepared ahead of time (especially for predictable issues - *subjective*), it demonstrates a lack of good planning and foresight into the feature. This is - frankly - a lack of competence and the project owner should take responsibility for the rather incomplete launch. However, the reality is that many individuals would launch buggy projects, claim credit for all the good it brings, while oncalls (spread across the team) pay for the lack of implementation quality. This is especially true when they immediately switch teams after project launches and no longer have to maintain or deal with the aftermath of their uninspiring launch.

Wrap up #

No blame culture means that you aren’t fully responsible just because you accidentally touched the house of cards causing it to collapse. Instead, we need better protection around it - like building a fence around it, using LEGO blocks instead of cards, etc - to make sure it doesn’t break down easily again after someone accidentally touches it, or just prevent people from accidentally touching it altogether. This means we hold those who are responsible for ensuring the protection accountable instead of those who inevitably discovered the problem. It’s like how we don’t blame white hat hackers for discovering an exploit; we pay them bounties as a way to thank them for discovering them. We should thank those who found holes in our system’s reliability instead of penalizing them for finding it.

Code Review Culture

binhong@binhong.me (BinHong Lee) — Fri, 23 May 2025 00:00:00 -0800

Ask a programmer to review 10 lines of code, he’ll find 10 issues. Ask him to do 500 lines and he’ll say it looks good. - @girayozil

Code review is a really subjective thing where each team or even individual runs things very differently. However, a bad code review process can lead to bad code smells and unnecessary tech debt (just ask all the vibe coders out there 🫣). I will try my best to share my rather opinionated takes while explaining the reasoning behind each of them.

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Overly prescriptive formatters #

I’m highly supportive of having a preset of an overly prescriptive formatter. It doesn’t matter if it’s spaces vs tabs, 2 spaces vs 4 spaces, curly braces on the same line or newline; as long as you have a formatter that keeps it consistent. This cuts down on all the unnecessary time reviewing and fixing code formatting while still being easy to read and scan through. Make them as strict as possible, ideally 100% reproducible where every line of code only has one way it can be formatted. Again, it doesn’t matter which style you pick but just pick something and stick with it, it’s all about consistency here.

Bias against newbies #

There was some research that finds that when someone is new to the stack / team / industry, reviewers tend to be stricter and more nitpicky against their code changes relative to a long-timer or a more senior engineer. Personally, I see this as a feature instead of a bug. A long-timer has earned their reputation (assuming that’s how they are a long-timer and not just a bad engineer successfully avoiding accountability at every turn) and can be afforded to be treated with more leniency in their code. A newbie however benefits from stricter code review feedback teaching them the style and tradeoffs the team values when it comes to coding patterns, thus learning good practices along the way.

In an ideal world, teams would have detailed style guides on how everything is written and every new person will be given ample time to read and understand them as part of onboarding. But the reality is that this rarely happens between the outdated style guides and the lack of proper onboarding process, so code review frequently becomes the first time a new person has the opportunity to learn about a team’s style guide (aside from linters and formatters). Let’s make sure to get it right so they don’t develop the wrong habits.

That said, it’s important to note that senior engineers should still hold themselves to a higher level in producing quality code as a way to lead by example. It will be rather unconvincing to a newbie if their reviewer writes subpar code while holding others to a higher quality bar.

Ask for further details #

Sometimes the change request itself is lacking details about the project or the reason behind such changes. Always ask for further details. In many occasions, it’s just the author overlooking it or making an assumption that their reviewers share the same level of context the author has about the project. This is a bit of an extension from the previous section but if you are accepting without understanding why, you are in a sense “blindly accepting” the change (even if in this case, it’s just partially blind lol).

Similarly, if the change is too complex (or if it’s bundling too many changes at once), ask for it to be broken down into atomic changes for better / easier review. Having smaller changes also allows for each change to be better reviewed by different domain experts instead of everyone crowding on the same change request. (I understand this is not always feasible so there will be exceptions but ideally, this is done whenever possible.)

Never blindly accept / accept to unblock #

These are generally two separate scenarios but their result is generally the same. A change that is otherwise not meeting the code review bar is getting accepted and pushed to production. Blind acceptance generally reflects more of a bad team culture where people either don’t take code review seriously or they don’t feel comfortable pushing back in code review, both of which are bad. “Accept to unblock” generally comes with good intention but doing so essentially voids the role of a reviewer. The most common reason for its use is due to reviewers not feeling comfortable about blocking a change but not wishing to take responsibility if that ends up being the bad commit causing an outage.

The only exception here would be when it’s a time sensitive change where “it can’t be worse than it is now” and that the author is the domain expert of the change. The most common example being a code change attempting to mitigate an ongoing production issue. (I’ll write up an incident runbook at some point which will most likely include this.)

Fix now not later #

On some occasions, you might run into a response in your review that they will “fix it in a future change”. (Generally, it’s more common in scenarios where your work adopts the idea of “stacked diffs”.) This is a bad pattern that should be avoided at large (unless absolutely necessary) mainly because this change now hinges on a separate future change to be correct, and that the other change could be independently reverted - for any external reasons - without this change, thus causing this bad pattern to now persist in your codebase.

Involve the Subject Matter Expert #

If you’re tagged / asked to review a change but you think someone else should take a look too, tag them. Most people are very open to reviewing changes when approached directly (especially if they are the SME, the change likely affects them / their team directly too). I’ve lost count of the number of times where I wrote some code, pinged the SME, then learned that whatever I’m doing is an anti-pattern which “works fine now” but has no guarantee that it will continue working that way. Funnily, I’ve even had an experience where the framework team pulled me in to review changes because I was considered the SME for the use case another person was attempting. In general, it’s better to have these discussions before a change is committed instead of after (especially if it ends up causing unexpected incidents later on).

Prototyping #

Code changes that are meant to be prototypes (especially RFCs) should clearly indicate that in their title (or some obvious tags / flags). There are 2 types of prototype code, one which is more of an exploration that is never meant to be shipped; while the other is a recommendation for adoption. In both scenarios, there should be clear indication of which type of prototype code this is and what the intention behind the prototype is (what are you trying to achieve?). For the former - exploration that is never meant to be shipped, the change should also clearly mark that it should never be shipped (hidden / unpublished etc.) with little comments all over explaining each of the “mocks” or “overrides” done to achieve the exploration. For the latter - recommendation for adoption, the change should be reviewed as strictly as any regular changes (potentially stricter since it’s likely coming from a newbie as defined above).

Long review time #

This is more of a polarizing problem where your team either never heard of this problem, or it’s something your team is struggling immensely with. In some teams, the lack of proper “incentive” for doing code review has greatly reduced team velocity in shipping due to time taken for a code change sitting around waiting for it to be reviewed. While I’d usually chalk this up as an “incentive” problem, I personally believe this as a “lead by example” cultural shift. Usually teams / orgs better at this have more technically driven high level ICs who do a lot of (strict?) code reviews themselves, thus fostering a good code review culture throughout their team / org. So in that sense, “be the change you want to see”.

Wrap up #

The culture of code review is something very intentional and will take effort to both cultivate and maintain. The value of good culture however, is usually not something directly measurable (will write a separate piece about intangibles in the future). Having bad culture generally isn’t something immediately obvious nor is it something that can be turned around overnight, but rather something that’s simmered over a long period of time until it’s boiling, by then it’s too late and people are leaving in droves.

Building a Simple Newsletter Service

binhong@binhong.me (BinHong Lee) — Wed, 21 May 2025 00:00:00 -0800

When I had the idea on the series about The Opinionated Engineer, I figured maybe that’s something worth setting up a newsletter service over. That said, after some consideration, I just thought that I didn’t need anything too complex and decided to roll my own because “how hard could it be?”. As far as I can tell, I think it’s working fine so far(?) and took me around one whole day to put the whole thing together from my couch. You can find all of it’s source code here on GitHub.

Aside from Substack, pretty much all other services provide it with a “freemium” model where you are charged based on the number of subscribers (regardless if they are paid). Admittedly, I doubt my newsletter would get to that point (I try to not take myself too seriously lol) but I also didn’t need most of the things those services are providing. Not to take anything away from them, they all seem really useful if someone is doing that for a living, but I mostly write for fun and as a way to do some knowledge preservation (future self-reference) with the sharing part mostly being a bonus.

As for Substack, I didn’t look too deep into things and while they don’t have the best of reputation, their product seems to fit my use case best where I won’t have to pay anything ever unless I started having paid subscribers. That said, I thought I didn’t need something as fancy but just something simple that goes with my personal blog.

Design #

It’s a pretty straightforward design with a DB storing the email, a key (used in place of the email as an identifier), and it’s confirmation status. When someone subscribes, they get sent an email to confirm their subscription, which they can pick to either “Confirm” (where we update the confirmation status) or “Unsubscribe” (where the entry is removed from the DB). First reason to use a key is so that we know the confirmation or unsubscribe click is coming from the users’ email instead of just people manually typing the link with someone else’s email as the param. Secondly, it prevents unintentional leak of the user’s email in address bar / browser history. Not necessarily a big deal but just not the ideal privacy design. (You could argue that exposing the key that way isn’t great either but I’d imagine being unintentionally unsubscribed from a my newsletter is likely a non-issue for most people lol.)

Python core library only #

The whole thing is written in Python (with some HTML templates) without using any third party library. I can’t say this was fully intentional but I generally like to avoid external libraries unless absolutely necessary which in this case, it’s not. As for “why Python”, it was an arbitrary decision as I’ve been doing interviews with Python and sometimes feel like I probably should brush up on it’s syntaxes lol. (Kept having to Google random things like “split string in Python”, “define object in Python” is uhh, not exactly ideal while in the middle of a coding interview session lol.) I did regret on this slightly because this can’t be “compiled into a binary” where it can just be dumped somewhere to run by itself forever with some config files and HTML files. Instead, it needs the server to have Python installed and run the actual Python code. I’d prefer keeping and running only the binary on server but I digress.

As for it being HTML / CSS only, it’s mostly an extension from my personal site and blog being that way which I’ve previously discussed about here. My personal opinion is that our growing reliance on JavaScript to do things had made us forget how powerful raw HTML and CSS could be in general. But also, I’ve been pleasantly surprised at how maintainable this has made my site as a whole. No worries about vulnerabilities or chasing and rebuilding it on the next / new shiny framework.

Email HTML quirks #

For the most part, email HTML behaves mostly the same with regular page HTML but there’s a few things that didn’t work which I’d like to point out:

<button type="submit" /> within a <form method="post" />
var() CSS params
html {}, :root {}, body {} selector in CSS
:hover - I read that some email provider supports them but most don’t
<style /> block (Gmail client specific) - Despite what Google says, only inline CSS is rendered on Gmail’s own clients*
JavaScript - expected to prevent XSS lol

I also tried to include everything in the email itself and didn’t make any references to external files for CSS which I assume would likely be stripped off as well.

*The content of the email do remain so if you use a 3rd party client (like Mail on iOS) it’d render just fine but Gmail apps or web client won’t render anything in <style /> tag.

Credential stuffing #

Shortly after I deployed it to the open internet (like within 48 hours), I started seeing weird traffic with random emails signing up but never confirming their sign-ups. Shubham mentioned that this is a “stuffing attack”. I did some more digging and find that this is a common thing where bad actors would just randomly try (usually old hacked) credentials on your site to see if they work. In my case, my param is set as email so it’s likely super obvious that it takes an email. To be quite honest, I haven’t figure this all out but I added some secret key and some extra logging to track bad ips. It’s not perfect but it seems to be doing it’s job so far (along the confirmation check) in blocking majority of the spam traffic from actually signing up. As to why this is bad, I woke up to multiple emails mentioning that my send attempt to certain emails were bounced 🙃, probably a bad reputation thing if my email continues to be bounced.

Wrap up #

This is a fun little experience and mostly a good opportunity to brush up my Python skill better. (Ironically, I ended up doing my next interview in TypeScript lol.) I’d imagine if I started having more, different, thematic series in the future, it might make sense to allow for segmented subscription / un-subscription but that’s work for another day. For now, I hope you enjoy reading this as much as I had fun building and writing this. Please subscribe below (or don’t lol).

The Art of Posting

binhong@binhong.me (BinHong Lee) — Fri, 16 May 2025 00:00:00 -0800

This piece is specifically more about Meta’s unique culture of “posting” about things but I imagine it can be a useful reference on communication in general. Many people loathe this process and complain about how it’s just a lot of “self-promotion” and while there’s definitely some truth to it, it’s also a valuable communication avenue that you can leverage to your advantage. As we will explore further below, posting by itself is more of a bonus as the post will still require actual substance (which is your work lol).

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

Proposing an idea #

A common theme of complaints you might see in a work environment is how someone else steals your idea. Making a post serves as a paper-trail on this being your idea as the original author. It’s also a useful cross-referencing tool when you run into discussions where your (previously suggested) idea could be a solution, you can share a link instead of going through your idea all over again. By itself, it also serves as a valuable RFC where others are free to comment with their concerns and / or how they feel about things in general. From there, you could gather feedback or even retool your idea a little to eventually become a valuable proposal and contribution towards your team goal.

Tracking progress #

When working on a long running project (especially one involving many different people in different stages), posts serve as a good way to track and announce progress, signaling to others if / when they need to start getting involved or otherwise. It’s also a valuable communication tool where others can keep track of your project if it ended up affecting their own project (intentionally or otherwise). One of the more common “behavioral” gap issue for an E4 -> E5 promo is where an E4 can quietly work through the project and launch everything themselves while an E5 would communicate this clearly (outwardly) before even starting to ensure that their project does not end up conflicting with the timeline of other partner teams unintentionally.

Wrapping up a project #

Often times (especially on a long running complex project) by the time it comes to finally closing out the project, everyone is exhausted and you’re just glad that “it’s done” and you can finally move on to the next thing. This is frequently a missed opportunity to nicely wrap up the project and show your appreciation towards everyone who helped make it happen. It’s like you went through all the trouble picking the perfect gift, understanding which variant fits them best, finding where to get one, but didn’t bother to wrap the gift in a wrapper or a nice little gift bag. Nothing wrong with it, and the value of the gift is still good but having a nice wrapper or even just a little ribbon would’ve been a cherry on top of the perfect gift. Similarly, making a post wrapping up the launch and sending thanks to everyone who supported the launch goes a long way in terms of both visibility and relationship building.

Not all projects are straight up successful launches and / or without leftover cleanup work to do. In fact most projects have some level of cleanup work remaining after it’s launched. The wrap up post is the best time to clearly lay them out and at least ensure that they are tracked. Similarly, you might have learnings from the project (especially for failed launches) which can be useful to share for future references if someone were to run into similar issues or to re-attempt this project again in the future.

Calling out potential issues #

Honestly this section kinda feels like just spelling out “regular social interactions” but sometimes it just needs to be said. Both as an author or as a reader of a post, you should take the opportunity to raise any potential issue you might be concern about in the comment of the post. As an author, this allows you to essentially crowdsource potential solution to your concern (maybe someone else has a “solution looking for a problem” lol). As a reader, you might have different perspective that sees potential risks that the author might have missed, or they figured it out but made an assumption that others would know, your question allows them to clarify their thought process or solution to your question.

There are some exception where if you think something is not suitable to be discussed in such an open setting but that’s largely up to your own discretion.

Pre-read from Subject Matter Experts #

Sometimes you have an idea on stuff you might not be super familiar with (or at least not something you’re seen as an expert) and that’s fine. But it can be valuable to leverage people around you for their expert opinion. It’s also a great way to build work relationship as people like to be seen as an expert. I’ve largely found most people to be helpful and open to help look at whatever you’ve written as long as you have some level of relationship with them (be it working on a previous project together or trauma bonding through SEVs). Having a SME review your post provides credibility to the things you’re writing / proposing and helps get your stuff attention from the correct stakeholders.

Wrap up #

Admittedly, this is less of an opinion piece and more of a “how to” (or “how I do x”) piece. I don’t expect this to change minds entirely on your take about how this is just a lot of self-promotion theatrics but I hope to at least provide a framework on how to use this to your own advantage.

Push Fearlessly with Automated Testing

binhong@binhong.me (BinHong Lee) — Sun, 04 May 2025 00:00:00 -0800

Conventional wisdom is that writing tests slows you down, because the alternative is that you don’t need to write tests. In fact, Meta (then Facebook) was famously lacking of automated test inline with it’s famous “move fast and break things” mantra. However, from what I’ve seen, I beg to differ.

This is part of a series (The Opinionated Engineer) where I share my strong opinions on engineering practices.

When to write tests? #

If you love something, put a ~~ring~~ test on it.

I find this quote as a good general guiding principle. You should definitely have tests for all the “core” features. For all the additional features (or edge case handling), you’d need to make a decision / tradeoff based on how critical it is for something to work, and how frequent you might be changing it.

Fearlessly #

The main goal of writing tests is to prevent regressions. Without automated tests, you’d have to rely on other (less reliable) ways to avoid regression. Having (non-flaky) tests makes it easy to consistently verify when / how a regression is introduced. With the right CI system setup, you can make sure that you don’t release new versions / feature that would accidentally break previously released features in an unexpected way.

Assumptions / Prerequisite #

The expectation here is that you already have a working CI/CD pipeline setup which comprehensively runs all your tests for each commit merges and deployments. Alternatively, if you have too big of a test suite to run (or too costly), you could instead only run relevant tests on each individual commits while running all tests for each deployment. If there’s a test that failed before deployment, you could still bisect to find the blame commit that broke the test.

What about manual testing? #

The main issue with manual testing is that you don’t know what you don’t know. You know your code change affects x and y so you test them both and they work as intended but what you didn’t realize was that it also breaks z entirely which you did not test because that was unexpected. Having an automated test means you’d just always run the test suite (within your CI/CD process) which would be able to catch the issue.

What about QA testing? #

QA testing generally happens on a weekly or bi-weekly (either definition) cadence. At best, you can do them on a daily basis but depending on the scale or speed at which your team is executing, it might still be hard to narrow down the specific commit that is causing the regression. With automated tests, you can bisect through your commit history without having to worry about setup and cleanup steps (assuming your tests are properly written).

Why not just write test for everything? #

In an ideal world where the cost to write test is 0, this would be a prescribed solution and everyone will be adopting TDD (Test Driven Development) but instead, we live in a real world where writing tests cost time and energy so it’s important to pick and choose things worth investing on vs things that aren’t. Such tradeoff is subjective to the criticality of said system and how often the system might be changed / updated.

Broken Tests #

Broken tests are tech debts. Either prioritize fixing them or delete them if they are no longer relevant. Keeping them around gives false impression on the state of the test coverage while potentially adds unnecessary operational cost in running these tests. Same applies to flaky tests. In fact, I’d go a step further by saying that flaky tests are worse than broken tests because at least broken tests are deterministic (which is one important characteristic of a quality test).

Wrap up #

When you have working quality automated tests, you can push new changes fearlessly knowing that it won’t cause regressions on features protected by these tests. This allows you to move faster both in terms of thinking about and testing all the possible regressions your change might cause, while also saving time on root causing a regression (if / when it does happen).

Self-Hosted GitHub Actions Runners

binhong@binhong.me (BinHong Lee) — Fri, 25 Apr 2025 00:00:00 -0800

This post will focus more on the “why” and some of the quirks around doing so instead of the “how” since GitHub already have a very comprehensive guide around it.

Just wanted to take a bit of time to acknowledge that it’s been almost 3 years since I last wrote a blog post. It’s been a busy 3 years for sure but hopefully I’ll be writing more consistently moving forward. (I do have quite a few half-written drafts that I’ve never published which I can’t tell if it’s still relavant at this point.)

What’s a GitHub Actions Runner? #

Basically instead of running GitHub Actions (GitHub’s own CI/CD) on GitHub’s machines, you can run them on your own devices. Running GitHub Actions on your self-hosted GARs would not count against your monthly Action minutes (since you are not using their machines) and can be an effective cost saving alternative.

Why do I even need to self-host it? #

The main use-cases is when you have private GitHub repositories so you only have 2000 included Actions minutes. While it might make sense for you to pay for GitHub Pro (which comes with more GitHub minutes), I didn’t find myself needing all other features that comes with it. On top of that, I happen to have a few old computers lying around so I figured I’d use them as ever-on GitHub runners. (So far, the electricity bill seems to be cheaper than paying for extra minutes or paying for any external equivalent CI service.)

Why not Gitea Runners (or other self-hosted CI services)? #

Personally, I already have GitHub Actions setup so migrating to a different CI service will take additional effort while still having similar limitations (for free plans). That said, Gitea actions mostly allows you to reuse GitHub action yamls (with a few minor exceptions like artifact upload / downloads). I stuck with using GitHub instead of fully migrating to self-hosted Gitea mainly due to the way dependabot and / or renovate-bot is already easily integrated without needing yet another action to trigger.

Linux (with WSL) #

I have a somewhat old Windows laptop (almost 6 years old, but broken battery and built-in keyboard) which is still pretty powerful aside from the fact that it can no longer be used as a laptop. It really hasn’t been used in any meaningful way for a while now so I decided to do some cleanup then setup multiple instances of WSL (with the same distro for convenience sake). The one issue I did ran into was when we needed postgres running in the background. Generally, you’d use a docker image of it on GH action but you actually can’t have multiple copies of postgres running on the same device because it’d complain about port 5432 being in use. (I tried blocking port sharing between instances but it didn’t seem to resolve the issue.) Eventually, I just install postgres on every instance but they all just uses the service from the first instance (which occupies the 5432 port). So far, it has mostly been working as expected.

macOS (with UTM) #

For the past few years, the only macOS device I own was a 2014 MacBook Air (which surprisingly still holds up… to some extend). So when I decided to start building an iOS app for GlobeTrotte, I knew I needed to buy a new Mac device. I ended up getting a BNIB M4 Mac Mini from some dude on FB Marketplace (for like $500). That said, I slowly learnt the pain of an immobile dev device so I copped for a used M2 MacBook Air (also for around $500). With that, my Mac Mini is officially freed up to do whatever so I install UTM on it to replicate the WSL setup of having multiple VM running on a device. However, I quickly ran into storage limitation issue since I need at least ~50GB storage per instance in order to run iOS simulators with XCode on it. The main instance only have 256GB storage where more than half is already occupied with all sorts of things other stuff. So instead, I setup a HDD dock and tried running UTM images on them. For what it’s worth, it works but the experience wasn’t exactly great. The whole OS moves really slowly / laggy but I guess if it works, it works(?). I am contemplating on getting an external m.2 SSD to see if it’d do better. Will update this section of the post if I ended up doing so.

Pros #

Aside from the obvious advantages of not having to pay for CI fees and repurposing your old tech devices, it’s also valuable to have a consistent device setup especially across dev and CI environment (so you know test failures aren’t due to some missing setups). That said, I don’t think anything else particularly stood out aside from the cost factor and the ease of use (and it being quasi-industry-standard).

Cons #

In my case, these are being ran on my personal devices at home so if my internet goes down, essentially my entire CI service goes down with it (which isn’t ideal). As mentioned above, I’ve also explored the option to host them on VPS but the cost (for a machine that can consistently run it) is really high. Lower tier VPS (like those available for $5/mo) generally have too little RAM causing it to freeze up a lot when running heavier operations (like e2e tests). Maybe I can bring an old computer to a friend’s / family’s house and run it on there but then you’d incur electricity bills at their place. (As far as I can tell, it’s not much but that’s generally not the way to treat people who trust you.)

Wrap up #

I did this mainly to save money and hassle to figure out which plan I need or what CI service fits me best. I used to run my own CI service with drone.io many years ago (which you can read more about here) but ran into the exact issue on VPS having rather limited RAM. Hopefully this setup can stay around for a bit longer and facilitate GlobeTrotte on moving fast.

Automated Link Preview Image Card

binhong@binhong.me (BinHong Lee) — Tue, 31 May 2022 00:00:00 -0800

A while ago, I saw this tweet thread from Simon Willison about how he added social media preview cards to his TILs. He detailed how he did this through a combination of Puppeteer, Vercel, SQLite, and some other stuff I didn’t understand 😅.

At that time, I was manually taking these screenshots by hand to be included as part of the commit, which is to say it’s not very efficient. Anyway, this stayed on my backlog of “things to explore” for a really long time until recently during this long weekend (where I also took additional PTOs), I had some free time and decided to look into this. The goal is simple, automate creating the preview images for a list of given / defined urls.

Playwright #

I know I just mentioned above that Simon used Puppeteer for his implementation but I just recently picked up Playwright for another project (GlobeTrotte) and figured that it seems like the right tool for this. Also, Playwright has better TypeScript support in general so I decided to opt for Playwright instead. (If you didn’t know, Playwright is built by the same team of engineers who built Puppeteer but has somehow all moved over from Google to Microsoft to build Playwright. There were some uhh interesting discussions about this when Playwright was first announced publicly.)

The minimum code needed for a working version is pretty short. It looks something like this:

import { chromium } from "playwright-core";

async function genRun(): Promise<void> {
  const browser = await chromium.launch():
  const env = await browser.newContext({
    baseURL: "http://localhost:1313/blog",
    viewport: {
      width: 1200,
      height: 600,
    },
  });
  const page = await env.newPage();
  await page.goto("/2022-05-31-automated-link-preview-image");
  await page.waitForLoadState("networkidle");
  await page.screenshot({
    animations: "disabled",
    fullPage: false,
    type: "jpeg",
    path: "preview/automated_link_preview.jpg",
  });
  
  await browser.close();
}

@globetrotte/altimeter #

Since I might want to use this in multiple places, I figured it’s probably a good idea to create an npm package for this so I can reuse it everywhere (with just some slight config change).

The package itself is really just a CLI tool for now. First, create a json config file based on this doc (depending on what you want from these screenshots). Then, run npm i -g @globetrotte/altimeter && altimeter config.json and you should see the screenshots show up once the run is complete.

Note: Unfortunately straight up calling npx @globetrotte/altimeter config.json doesn’t work as of now (unless you already have playwright installed before which can’t be guaranteed) since it won’t trigger the postinstall script to install playwright.

Alternatively, it can also be generated by including the package as a dependency, then import AltimeterConfig from the package, set the variables, and write the output to a JSON file. Something like this:

import { AltimeterConfig, AltimeterDestination } from "@globetrotte/altimeter";

async function genConfig(urls: string) {
  const config = new AltimeterConfig();
  config.baseURL = "http://localhost:1313/blog";
  
  config.destURLs = urls.map((url) => {
    // You can also do some modification here to set a better "name"
    return new AltimeterDestination(url, url);
  });

  // This line is to add the index page itself (omit if not needed)
  config.destURLs.push(new AltimeterDestination("index", "/"));

  config.dir = "dist/preview";
  writeFileSync(path.join(__dirname, "config.json"), JSON.stringify(config, null, 2));
}

Then you can run it with altimeter config.json and it will start taking the screenshots and saving them into the specified folder.

`<meta />` tags #

To actually make the images show up when your web page is linked, you need to add the appropriate meta tags depending on which social media you want the image to be fetched and displayed.

Open Graph Protocol #

Open Graph Protocol is used on many web services (including Facebook, LinkedIn, Mastodon) to fetch the preview image so you need to add the followings onto your site within the <head> element. You can also (optionally) specify the image width and height. From my personal experience, it doesn’t matter if they are absolute or relative path as they seem to work either way.

<head>
  ...
  <meta property="og:image" content="preview/automated_link_preview.jpg" />
  <meta property="og:image:height" content="600" />
  <meta property="og:image:width" content="1200" />
  ...
</head>

Twitter #

Twitter has its own documentation for the display card. At the time of writing, it seems like they only take absolute path, relative paths don’t seem to work as intended.

<head>
  ...
  <meta name="twitter:card" content="summary_large_image">
  <meta name="twitter:image" content="https://binhong.me/blog/preview/automated_link_preview.jpg" />
  ...
</head>

You can use this card validator built by Twitter to try if your newly set <meta /> tags are getting picked up. Unlike just tweeting it out on your profile and see how it work, this would always force fetch from the given link and overwrite whatever cache they already stored of the link.

Google #

Honestly, I couldn’t find a good documentation on how to go about setting the image properly. I only found this about setting the max-image-preview but nowhere does it say how to tell Google which image to use. I looked into how some of the news websites does it instead. NYTimes uses the image tag as below. The Verge seems to use some sort of Schema.org setup but within a <script /> tag (instead of <meta />) and label it as application/ld+json type. I also checked GitHub (since they actually do one of these screenshot as preview image stuff really well) but I didn’t see any <meta /> tag that stood out.

<head>
  ...
  <meta name="robots" content="max-image-preview:large">
  <meta name="image" content="https://binhong.me/threads/preview/index.jpg" />
  ...
</head>

GitHub Action #

I currently use the above package as a CLI on my own blog (specifically this one that you’re reading right now). I didn’t want to have to run a bunch of things manually everytime I write a new post before I deploy so instead, I “delegated” this part of the job to GitHub Action.

Add the generated directory to .gitignore
Use Node.js in the GitHub Action (- uses: actions/setup-node@v1)
Make sure the devserver site is up and running (in my case, hugo serve -D)
Run npm install
Run npx altimeter config.json
Run build if needed (hugo --buildFuture)
Deploy! 🎉

Note: If you are generating this for static content, you might need to setup a server to serve up those static content on a localhost address. I recommend following this guide to set up a server (instead of using expressjs).

Wrap up #

This was a fun ride. Honestly, I ran into a few problems here and there (mainly, setting up monorepo in GlobeTrotte and creating CLI with TypeScript) but it’s relatively straightforward otherwise. The actual coding part took less than a day while writing up the documentations (and this blog post) probably took close to double the time.

Rebuilding Personal Site from Scratch

binhong@binhong.me (BinHong Lee) — Mon, 31 Jan 2022 00:00:00 -0800

As time goes on in my journey of experimenting with different technologies on building a web interface, I figured it’s time for my personal site to return to the fundamentals. So I decided to build it with only HTML + CSS with one of the goals being that it should work perfectly fine with JavaScript disabled.

Before this, it’s built with Vue. You can still find the source here.

Design #

Inspiration #

Skip this next paragraph if you’re not interested in my old site’s origin story.

My old site is structured more like a resume than a personal self intro. It’s rather serious by nature since it was built at a time when I was looking my first job out of college. I stole the design from the personal site of a great friend of mine, Pat Pataranutaporn (just one out of the plethora of pages) and dulled it down a little. He rebuilt his entire personal site shortly after but I stick with it since I built it in a way that’s rather easy to maintain (as in adding and removing stuff overtime). It served me well during those times but at this point a redesign is rather long overdue.

I can’t really do design (unless I’m given a rigid component library then maybe 😅) so I looked up how some of the seasoned software engineers design their site and pick out the parts I like about them. Here are some of the websites I took inspiration from:

Ideology #

One of the most obvious common traits among them is how minimal they are while still showing a lot of personality. The other being that 2 / 3 have a dark mode toggle while Jane’s site is dark mode only. I stick with these ideas and decided to be more playful with my text colors especially since I can’t draw.

One other takeaway I had was that I really liked the monospaced font used on Cassidy’s site. I tried Victor Mono initially since I really like it (and use it day-to-day on all my text editors) but I find it’s readability isn’t great for longer paragraphs. I decided to settle for Fira Code instead (first font that introduced me to ligatures for programming).

Note: Monospaced fonts in general aren’t meant for regular text reading anyway so this is something completely within expectation.

Implementation #

Dark mode #

Since I’m not using JavaScript, this is going to be tricky. That said, major credit to Shubham for pointing out the basic concept of its implementation in this tweet. On top of that, there’s just a lot of CSS gymnastics where I learn about the different selectors and what they each does in CSS (+, >, ~, []).

One important thing of note is that, the toggle “button” (the detail component) has to be at the same level as the top level of the rest of the components on the screen. I keep my toggle on the right which is sufficient to prevent colliding / blocked by other stuff on screen.

Smaller Screens #

I tried my best to make everything as modular as possible so it should properly reorganize itself dynamically based on the given screen size. However, there are always exceptions and this is where @media screen and (max-width: 1000px) comes in handy for addressing these edge cases.

As I add a few more sections to my side, I started to realize that the link menu might not fit on certain display sizes. In general, it just follows this w3c guide. I originally did one implementation that used the similar idea of the dark mode toggle above but I opted for :hover instead of [open] due to its interaction where I think it makes more sense to be dismissed when I tap on random places on the screen.

In order to make it more accessible, it is also important to add the appropriate aria labels. This page has a rather comprehensive guide on the proper labels to add for each fields.

Truncation (somewhat unrelated) #

This is more for the blog part of the site where I find the default truncation being too lenient thus generating some of the overly long (and not uniformed) summary text.

.description {
  display: -webkit-box; 
  overflow: hidden; 
  word-break: break-word;
  -webkit-line-clamp: 3; 
  -webkit-box-orient: vertical; 
}

Takeaway #

Overall it’s been a great experience to relearn and understand what we can do even without any of the modern JavaScript frameworks. It also serves as a great reminder that web existed for quite a while before JavaScript came around. While they may not be as pretty and reactive as it is today, they still works and have a lot to offer (especially as HTML and CSS standards continues to evolve as well).

Upgrading Vue TS Project (2.x -> 3.x)

binhong@binhong.me (BinHong Lee) — Sun, 12 Dec 2021 00:00:00 -0800

While Vue 3 has been out of beta for quite a while, (as noted on their website) a migration build is still a work in progress so the current advice is to stay with Vue 2 if it’s a non-trivial project. I procrastinated on this article for so long that a migration build has since been released. Feel free to check it out here.

That said, as Parcel v1 is no longer maintained (and with v2 supporting only Vue 3), I figured it’s time to upgrade.

Why not Parcel v2? #

This is actually quite straightforward. I’ve been wanting to try out vite for a while since I first saw it mentioned on an esbuild issue (asking for vue SFC support). I did consider snowpack for a while but I still decided to stick with vite seeing how its also created by Evan You and it seems to have mildly better performance.

I did initially try to upgrade the project to Parcel v2 seeing as how simple it was described as. However, I was wrong and instead of sinking anymore time into it, I decided to move onto something else.

Moving onto vite #

The one thing that really sealed it for me (with vite) was how simple the move turned out to be. I created an empty project with their template, clone the relatively minimal default config over, and it’s “mostly” working. (Well, “mostly” because my project was still on Vue 2 instead of Vue 3.) Funnily enough, this was the same reason that led me to opting with Parcel initially since it needed very minimal configuration to get started.

`vue-router` #

According to the official Vue 3 guide, the way to call the router is to import and call it the VueRouter you defined. This is slightly different from the previous version where you call this.router to use it. Well technically you can still do exactly that, as long as you define router in your global properties like below.

import { createApp } from "vue";
import App from "./App.vue";
import router from "./router";

const app = createApp(App); 
app.config.globalProperties.$router = router;
app.use(router);

At this point, router navigation should work just fine but you might see tsc (or vue-tsc) throwing type errors saying that this.router is of type unknown. We’ll fix this later when discussing about vue-tsc.

`vue-tsc` #

One of the main upgrades that Vue 3 provides is vue-tsc. On top of the regular type checking already available with tsc, vue-tsc also do type checking for inline TypeScript / Vue code in template (like v-for, v-if, :props…).

Volar (vscode) #

If you do your Vue project development on vscode, you probably have Vetur installed. While Vetur works with Vue 3 projects, Volar is an upgrade to that comparatively as it uses vue-tsc and displays those new type errors inline. They do not complement / work with each other though so make sure you remove one before you install the other.

Shims #

So if you’re not already familiar with shims, it’s essentially a type declaration file to tell your IDE what are the types of the vue file exports (and tsx files). Here’s a stackoverflow question on why it’s needed.

`shims-vue.d.ts` #

This is the generic one that’d be generated when you create a new Vue project.

declare module "*.vue" {
  import { DefineComponent } from "vue";
  const component: DefineComponent;
  export default component;
}

`shims-vue-run-time.d.ts` #

If you make router calls in your code with this.$router directly, you might find vue-tsc complaining about missing type declaration for it even if you already declared app.use(router); in your main.ts file. It seems like that doesn’t inherit the types imported automatically so you need to declare it separately for vue-tsc to know where to look.

import VueRouter from "vue-router";

declare module "@vue/runtime-core" {
  interface ComponentCustomProperties {
    $router: VueRouter;
  }
}

Of course, if you also call other stuffs similarly. As an example, let’s say you call this.$notify with ElementPlus. You’d also include that in this type declaration file like this:

import { ElNotification } from "element-plus";
import VueRouter from "vue-router";

declare module "@vue/runtime-core" {
  interface ComponentCustomProperties {
    $notify: ElNotification;
    $router: VueRouter;
  }
}

As for why I can’t just put these lines of codes in the same type declaration file as above, it threw me some error when I tried that. Separating them somehow solved the issue for me.

index.html #

Usually this should just be in default state so the only change that needs to be made was to add this additional param of type="module" to the script tag calling the entrypoint.

`1`	`<script type="module" src="/src/main.ts"></script>`

Pug #

For pug in Vue 2, you can have your whole block indented (kinda like the regular html block). But in Vue 3, it’s no longer supported so your outer most div can no longer be indented.

Wrap up #

Overall, there are some bumps here and there but the process is relatively easy to understand especially with the help of new tools like volar (and faster compilation of vite).

PostgreSQL on Azure Pipeline with OSX

binhong@binhong.me (BinHong Lee) — Sun, 18 Jul 2021 00:00:00 -0800

While the defacto way of running postgres on Azure Pipeline according to this guide seems to be using the docker image, the MacOS image on Azure Pipeline does not have docker support due to some licensing issue. So instead, this has to be setup and handled manually. We’d use brew here since it comes installed in the OS image and is the easiest way to deal with it as far as I’m aware.

Install (optional) #

By default postgres should already be installed in the included MacOS image but if you prefer to run it on a specific version you’d have to force reinstall the version you want just to be sure.

steps:
...
- script: brew install postgresql@13.1
  displayName: Install PostgreSQL
...

Multiple Versions (optional) #

If you’d like to test that it works with multiple versions of PostgreSQL, you can setup a version matrix for it.

strategy:
  matrix:
    postgresql-11-10:
      psqlVersion: '11.10'
    postgresql-13-1:
      psqlVersion: '13.1'

Then amend the installation part to make use of the value from the matrix.

steps:
...
- script: brew install postgresql@$PSQL_VERSION
  displayName: Install PostgreSQL
  env:
    PSQL_VERSION: $(psqlVersion)
...

Start Service #

Despite being installed by default, it doesn’t run by default so you’ll need to add something like this to get the service started.

steps:
...
- script: brew services start postgresql
  displayName: Start Postgres Service
...

Set up DB #

Now, let’s create a database called db_name owned by the user called username with password as password. (You can replace each of these with whatever that works best for you / your process.)

steps:
...
- script: |
    psql postgres -w -c "CREATE DATABASE db_name;"
    psql postgres -w -c "CREATE ROLE username WITH SUPERUSER CREATEDB LOGIN ENCRYPTED PASSWORD 'password';"
    psql postgres -w -c "GRANT ALL PRIVILEGES ON DATABASE db_name TO username;"
  displayName: Setup Database
...

Brew Optimization (optional) #

At this point, postgres should already be working as intended but it’s not exactly ideal considering the first brew call actually takes a long time since it’s also trying to run update and cleanup which can take a while. Considering that running brew update and brew cleanup probably has minimal benefit on a CI, we can add the following environment variables so brew wouldn’t do that.

1
2
3

variables:
  HOMEBREW_NO_INSTALL_CLEANUP: 1
  HOMEBREW_NO_AUTO_UPDATE: 1

OS Condition (optional) #

If you’re including this as part of a multi OS testing config, you can add an additional condition for each line of the commands here to make sure it doesn’t run on other OS-es causing unexpected failures. Similarly, you can add its equivalent for commands specific to other OS-es.

steps:
...
- script: brew services start postgresql
  displayName: Start Postgres Service
  condition: eq(variables['Agent.OS'], 'Darwin')
...

Full config #

Finally when put together, your config likely looks something like this.

variables:
  HOMEBREW_NO_INSTALL_CLEANUP: 1
  HOMEBREW_NO_AUTO_UPDATE: 1

pool:
  vmImage: macos-latest

strategy:
  matrix:
    postgresql-11-10:
      psqlVersion: '11.10'
    postgresql-13-1:
      psqlVersion: '13.1'

steps:
- script: brew install postgresql@$(psqlVersion)
  displayName: Install PostgreSQL
  condition: eq(variables['Agent.OS'], 'Darwin')

- script: brew services start postgresql
  displayName: Start Postgres Service
  condition: eq(variables['Agent.OS'], 'Darwin')

- script: |
    psql postgres -w -c "CREATE DATABASE db_name;"
    psql postgres -w -c "CREATE ROLE username WITH SUPERUSER CREATEDB LOGIN ENCRYPTED PASSWORD 'password';"
    psql postgres -w -c "GRANT ALL PRIVILEGES ON DATABASE db_name TO username;"
  displayName: Setup Database
  condition: eq(variables['Agent.OS'], 'Darwin')

Hopefully you find this article helpful!

Ava with Vue (2.x) + TypeScript (and NYC)

binhong@binhong.me (BinHong Lee) — Sat, 22 May 2021 00:00:00 -0800

When working on a project with Vue (TypeScript), I faced a lot of roadblocks trying to get code coverage to work. The issue is largely covered in this issue thread. Most solutions / workarounds for that seems to revolve around changing the bundling flow with webpack. Considering my lack of familiarity with webpack (on top of my use of scss and pug in Vue files), putting together a webpack config that works for me turns out to be extremely challenging. That somewhat prompted me to move the build and test process away from vue-cli while starting to experiment with different tools that could work. One thing to note is that, I’ll be using babel to handle TypeScript compiling / transpiling instead of ts-node as recommended from Ava’s guide.

Babel #

We use babel to help Ava properly transpile the Vue and TypeScript files. This is especially useful if you would like to get accurate code coverage for your tests.

Here’s how the .babelrc should look like:

{
  "presets": [
    "@babel/preset-env",
    "@babel/preset-typescript",
    "babel-preset-typescript-vue"
  ],
  "plugins": [
    "@babel/plugin-transform-runtime",
    "@babel/plugin-transform-typescript",
    [
      "module-resolver",
      {
        "alias": {
          "@": "./src"
        }
      }
    ]
  ],
  "env": {
    "test": {
      "plugins": ["istanbul"]
    }
  }
}

A few things to note here:

Alias set is optional. You only need it if you actually use the alias. You can also set multiple aliases or point it to a different directory depending on how your project is setup. Make sure you sync it with your tsconfig.json to prevent confusion / unwanted errors
istanbul plugin is also optional. This is only needed if you want to also set up code coverage with nyc (more on this later).

Setup File #

Setup file is there to set the test environment considering that these Vue component files are supposed to be ran inside a Vue instance with some preset, this file serve as a similar purpose but limited only to testing use.

Here’s a sample _setup.js where you can build on top of depending on what you need / use. See original reference here.

var html;
var options = { url: "http://localhost/" };
require("jsdom-global")(html, options);

// Fix the Date object, see <https://github.com/vuejs/vue-test-utils/issues/936#issuecomment-415386167>.
window.Date = Date;

// Setup browser environment
const hooks = require("require-extension-hooks");
const Vue = require("vue");

Vue.config.productionTip = false;
Vue.config.devtools = false;

hooks("vue").plugin("vue").push();
hooks(["vue", "ts"])
  .exclude(({ filename }) =>
    filename.match(/\/node_modules\//),
  )
  .plugin("babel")
  .push();

if (!window.localStorage) {
  window.localStorage = {
    getItem() {
      return "{}";
    },
    setItem() {},
    clear() {},
  };
}

localStorage = window.localStorage;

One of the changes I made was to set localStorage so you can also use it for the test.

You’ll also need to include this file in your ava.config.js like below:

export default {
  extensions: ["ts", "vue"],
  require: ["./src/tests/_setup.js"],
  babel: true,
  files: [
    "src/tests/**/*.spec.ts",
  ] 
};

At this point you should be able to run your Ava tests as intended and it would work just fine. However, if you want to have code coverage for the project, read on on how to setup your NYC config.

NYC (optional) #

One of the main reasons I started exploring for this option is to get good code coverage report. This setup allows us to let NYC generate proper coverage reports.

{
  "instrument": true,
  "sourceMaps": false,
  "all": true,
  "check-coverage": false,
  "per-file": true,
  "include": [
    "src/**/*.{ts,vue}"
  ],
  "exclude": [
    "**/*.{spec,test}.{js,ts}",
    "src/scripts/**/*",
    "src/tests/**/*",
    "*.js"
  ],
  "extension": [
    ".ts",
    ".vue"
  ]
}

Conclusion #

Personally, I wrote this piece based on my work on GlobeTrotte. I’ve tried my best to cleanup the code for it to be as reusable as I can. Feel free to let me know if there’s any issue with it. Hopefully you find this helpful.

Cross Language Object Parsing

binhong@binhong.me (BinHong Lee) — Sun, 09 Aug 2020 00:00:00 -0800

I was working on a side project where I had the backend and frontend setup to run on different programming languages. As I was working on it, it started to bug me a little that I have to write the same object / struct and enum for each language in use everytime I introduce a new enum or struct. It can also be confusing / dangerous if I updated one side and forget to update the other.

At first, I just wrote some shell script to automate this but it quickly become unfeasible. (Clarification: it’s still possible to do this, but without any sort of object oriented support, the code gets quite messy quickly and I didn’t want to maintain that.) Now, at this point I should let you know that there are actually multiple existing solutions (Apache Thrift or Protobuf) that are way more matured, robust, and designed for use in a much larger scale. If you’re working on an enterprise level solution (or looking to learn something that’s more transferable into your future work), you might want to take look into them instead. I personally wanted something simpler and more flexible so I began working on wings.

Installation #

If you have nimble (comes with nim) installed, you can install wings with nimble install wings. As of now, the only alternative to that is to get the binary and place it in a folder included by the path (like /usr/bin/) which unfortunately means that it won’t automatically update when a newer version is released. I’m still working on (and learning how to) deploy it to more package managers for easier installation process.

Usage #

You write a wings file like the example below then run wings <FILENAME>.wings.

go-filepath examples/go/classroom
kt-filepath examples/kt
nim-filepath examples/nim
py-filepath examples/py
ts-filepath examples/ts

py-import examples.output.py.people
ts-import { IWingsStruct }:wings-ts-util
import examples/input/emotion.wings
import examples/input/homework.wings

py-implement People
ts-implement IWingsStruct

# Any person who is studying in a class

struct Student {
  id          int       -1
  name        str
  cur_class   str
  feeling     Emotion   Emotion.Meh
  is_active   bool
  year        date
  graduation  date
  homeworks   []Homework
  something   Map<str,str>
}

ts-func(
  public addHomework(hw: Homework): void {
    this.Homeworks.push(hw);
  }
)

As you can see, not only does it support generating files based on given specific parameters, it also supports importing other wings files directly which will also be generated by wings (unless specified otherwise in your config file).

Library #

Since it’s very beginning, wings has been published on nimble as a library for nim. With this, you will be able to make further customizations by importing it as a library especially if you only want to use a specific part of it. From there, you can even use build tools to incorporate it as part of your workflow if supported. (Shameless plug, I wrote the PR for please’s basic nim and nimble support through pleasings. Though admittedly it isn’t quite as complete as you can see here I use my own version of the config in the repo.)

I’ve also created stones, a separate library that contains some generic functions and tools used in wings.

Generated Files #

Struct #

Unlike the afromentioned existing alternatives, it doesn’t change the way the data was stored and transferred. You’ll still be converting them from and to JSON and communicate through JSON (so they will still be human readable). Therefore, comparatively, it will be “slower” than the alternatives as it will inherit the same “slowness” of JSON (depending on how it is serialized and deserialized in each individual languages) and bulkiness (it’s a pretty well established fact that JSON string takes more space in memory than something like a serialized thrift object).

Enum #

Nothing much about them. They’re just normal enums that starts from 0 and increment by 1 for each element in it. Custom type and initialization is in the works and might change up the syntax a little when support is added.

Util Classes #

While the generated files (generally) work out of the box, they by themselves are still lacking of certain features that can be applied across all Struct or Enum classes. This is where the util classes comes in. They are written in each of their respective targeted languages as a package / library. Currently, there’s only a Util class for TypeScript since golang already have all the features in the util class baked in with its JSON support.

Config #

While the use of a config file is optional, its also where the core value “customizable” comes from. Config files are accpeted in JSON format and should be added as an argument when calling wings with -c:<CONFIG.json>. Some of the more simple / straightforward configurable value includes things like logging which controls the verbosity of the output message in the terminal, skipImport which lets you dictate if you’d want to also generate imported files or skip doing so, and header which lets you specify your customized the header comment for all the generated files. On top of that, you can also configure how you want the output files to be formatted by specifying your own templates in langConfig (more on this below).

Template #

As mentioned above, you can also write your own or import someone else’s templates (with remoteLangConfigs) to configure the output files in your own format or even to output to languages that are not officially supported. This allows wings to be used in a more dynamic way both in terms of range of supported languages and style format instead of being limited by what’s officially supported (like in Thrift and protobuf though they do already have a wide range of support).

Takeaways #

Overall, I think one of the main takeaway is that if there’s already an actively maintained existing solution, it’s probably better to use that than to create your own. By now you might have realizes that this project is slowly morphing into something that looks similar to the existing alternatives but in a much simpler form.

Admittedly, I’ve only started learning about how a programming language is written only after a few months into the project so the design and structure is not ideal. However, since this is also much simpler compared to a full blown programming language, I can still get away with it on this. That said, I might start to incorporate some of these principles into wings as I continue to work on it (including writing a proper lexer and parser for it).

Of course, this is just a little side project spawn off of another side project. There’s also the long time dilemma of “should I just write them manually or do I need to write codes (or use an external library) to write them” (the NIH syndrome). In this case the later took way more time than the former, but this is a side project so it’s more about learning than meeting non-existent delivery dates anyway. I’ve also find it fun to have been working on the project so far and I hope you find this helpful. If you’d like to learn more about it (or use it yourself), check out the project website at wings.sh. If you like what you’ve read, I’d appreciate a star on the wings repository.

Multi-language Support Build Tool

binhong@binhong.me (BinHong Lee) — Wed, 07 Aug 2019 00:00:00 -0800

I recently started learning about Go to find out what’s all the hype about. So I figured what’s a better way than to build a project out of it? As I was setting up my machine and going through tutorials step-by-step, I was slightly annoyed by the fact that the development of Go projects are limited to inside the $GOPATH.

Personally, I have all my side projects stored at the top level of the computer. It might not be the best way to do it but I’m liking that I can see all my projects at one glance. I guess one could argue that you can soft link the folder into the folder inside $GOPATH and I would pretty much achieve the same effect. But it didn’t quite feel right to me, sounds like a really hacky workaround that might not worth the maintenance cost as I switch to a separate machine only to have to do everything all over again.

*Note: Yes, I did eventually found out that you can now do development without being in $GOPATH with go mod.

Instead, I started looking into build tools that can do that. Admittedly, I intentionally picked something that’s not so mainstream (Buck, Bazel, Pants) and found Please. (The other part of me liked the idea of naming my run alias as plz work because it’s mildly funny.)

Build Tools #

So what is a build tool and why do we need them? In most occasion of a side project, you build and run each part of your project separately in a package of its own with only one (or two) programming languages each. In which you never needed to cross compile or compile multiple projects at once in parallel. Even when you do, there’s nothing that needs more than a couple lines of shell scripts to string them together since the performance gain through parallel compiling is not that significant if at all.

However, this does not hold the same for companies that adopt the concept of monorepo. Instead of multiple smaller repositories, they host all of their code (which can be a mix of many different languages and requires many different build flows) in just a single repositories. In this case, different projects can also share the same library that is already imported and in use by the other project thus making sure that the imported library version is always consistent across all the projects.

Here’s a quick summary of why and why not. I’m not here to debate if you should or should not use a build tool or monorepo. I’ll leave them to the experts. I’m just here to share about the possibility of doing so.

Advantages:

Multi-language compatibility
Not limited to any specific path
Consistently work across supported devices
Smoother e2e workflow (if set up correctly)

Disadvantages:

Slightly steeper learning curve
Can be complicated to set them up

Installation #

There are a few pretty straightforward way to install it according to their documentation though admittedly, you probably shouldn’t just run some random script from some random website online (but then again, compiling from source can be a bit of a pain) so that’s really up to you.

*Note: You mostly only need this installed to initialize the project. Others wanting to build / develop the project does not necessarily have to have please installed locally for them. Similar to how Gradle works with ./gradlew on machines without Gradle installed, Please generates a ./pleasew file that can be ran as an alternative if one does not have Please installed in the machine.

Setup #

BUILD files #

Honestly the BUILD file syntax for Please looks very much alike (if not identical) to some of the other more popular build tools like Bazel or Buck. I don’t know of the exact name for it but it is very Python-ish. In fact, GitHub linguist’s language.yml actually explicitly classifies them as Python. I recommend checking out the documentation as they are pretty good in detailing the available fields and their purpose for it.

External Dependencies #

This is probably the most tedious part of the whole process. It makes you realize how much have you’ve taken tools like dep or even npm and yarn for granted. Unlike those other tools I’ve mentioned, you’ll have to manually list each and every one of the external dependencies (and their dependencies) properly so Please knows which to prioritize and how to build and manage them in the most efficient parallel way. Fortunately, this should be a one time thing as even if you consistently increase dependencies, chances are you already have the dependencies of the new dependencies listed.

Code Structure #

You have a lot of freedom here. While you might not want to go crazy and have your code all over the place (making them referencing each other a nightmare), you can afford to structure them slightly different from the conventional way since the build tool will handle the hierarchical setup for you. For instance, you can have your main go file 2 levels (or more) inside some folder unlike the usual top level requirements. Similarly, if you need some folders of multiple different programming languages to overlap in between each other, that’s pretty possible (though I would question the need for it and if there’s a rational reasoning behind it).

pleasings #

Aside from the built-in supported languages and rules, they also have an additional repository that is home to the rules that are not actively maintained / updated thus not being part of the core functionality / rule. These includes support for things like Android, Rust, Scala, Kotlin and even Nim (okay, I wrote the one for Nim so I’m just shamelessly sneaking it in here).

genrule() / gentest() #

Even with pleasings, there’s a possibility where there’s something you work on (or need for your project) that is not supported. In my project, I’m also compiling Vue.js code for the web. In this case, I used a genrule() to build it with pnpm (similar to npm and yarn) and a gentest() to run its tests. While it’s pretty straightforward to do it this way, I’ve also effectively reduced the benefits of using a build tool since I’m mostly waiting on this one rule (and rely on pnpm’s build efficiency instead of Please’s) when building my entire project. Here is an example how I did mine:

genrule(
    name = 'pnpm',
    visibility = ['PUBLIC'],
    outs = ['node_modules'],
    cmd = ' && '.join([
        "top_level=$(pwd | awk -F'plz-out' '{print $1}')",
        "pnpm i",
        "ln -s \"$top_level\"\"node_modules\" \"node_modules\"",
    ]),
)

genrule(
    name = 'build',
    outs = ['dist'],
    cmd = ' && '.join([
        "current=$(pwd)",
        "cd $(pwd | awk -F'plz-out' '{print $1}')",
        "pnpm run build",
        "mv \"dist\" \"$current\""
    ]),
    deps = [':pnpm'],
)

*Note: Please do have some support for JavaScript in the pleasings repository in which you can use their yarn_library() and js_binary(). However due to the nature of node.js dependency hell nowadays, handwriting the yarn_library() rules for all the libraries you want + all the libraries those libraries depends on and so forth will probably take forever. I also did not look too deep into this to tell if this would work any differently from just running yarn install just by itself.

CI Testing #

One of the other benefit of having a build tool properly set up is that it would be slightly easier (and more consistent) to set up your CI testing both in terms of testing environment and testing outcome. Of course, there are still some external dependencies like language version installed etc but for the most part, build tools would always pull fresh version of the new build regardless if there is a cached version on the machine (that might mess with the consistency of the test result).

VSCode #

For most code editors, their linter will rely on the env value of $GOPATH to look for third party libraries used in your go codes. Since those libraries don’t live in there when building with Please (it is in the plz-out folder inside your project folder), you will want to have a settings.json file that points the linter to the right location consisting of the compiled libraries. Something like this:

{
    "go.inferGopath": false,
    "go.gopath": "{workplaceFolder}/plz-out/go:/{workplaceFolder}/plz-out/go/src",
    "files.watcherExclude": {
        "**/plz-out/**": true
    }
}

*Note: you will want to replace {workplaceFolder} with an absolute path to wherever your project lives since the linter seems to complain about using a relative path as $GOPATH.

Hopefully this has been helpful in opening you up to trying and mixing different programming languages into your project. Why? To me, side projects are just experimental grounds for fun. If it sounds interesting enough, I’ll try and make it happen (or in this case, see if someone has done it and use their ready build tool).

If you are interested, this is the side project I’ve been working on. Thus far, its still not functional. It will most likely take another few months or so (depending on my progress and how much free time I have committed to this outside of work) before I can deploy a minimally working version of it.

Functional and flexible shell scripting tricks

binhong@binhong.me (BinHong Lee) — Tue, 07 May 2019 00:00:00 -0800

Its 2019 now, who writes shell scripts anymore? Am I right? Well, apparently I do. ¯\_(ツ)_/¯

Shell scripts vs python or perl #

There are some good arguments for that here and here which mainly revolve around 2 things:

Shell exists in all Unix systems and makes use of system default features.
Shell is an “interactive command function” designed to get user inputs during the process of running them.

Also, here’s an additional relevant reading about the differences between sh and bash.

Arguments #

In some occasions, you will need to pass an argument (or expect one) into the script like how you might pass a param into a function. In that case, you will use something like $1 for the first argument, $2 for the second. Here’s an example of how it would look like:

`1`	`echo "The input message was $1."`

1
2

./run_this.sh userInput
The input message was userInput.

Note: The params are separated by spaces so if you want to input a string as a param that contains a space, it might need to be something like ./run_this.sh "user input" just so "user input" would be counted as $1 entirely.

In the occasion where you are not sure how long the user input might be and you want to capture it all, you would use $@ instead. In the following example, I took in the entire string and print them out word by word after breaking them into a string array according to the spaces in between.

userInputs=($@)
for i in "${userInputs[@]}";; do
  echo "$i"
done

./run_this.sh who knows how long this can go
who
knows
how
long
this
can
go

Functions #

If you have done any sort of programming, you should be familiar with the concept of functions. It’s basically a set of commands / operations that you will be repeating over and over again. Instead of repeating it multiple times in your code, you can put them into a function. Then just call the function which effectively reduces the lines of code that need to be written.

Side note: If you don’t know already, LOC is a horrible metric for any sort of measurement in terms of programming. Don’t take this from me, take this from Bill Gates:

“Measuring programming progress by lines of code is like measuring aircraft building progress by weight.”

Here’s how a normal function looks like:

# Declaring the function
doSomething() {

}

# Calling the function
doSomething

Pretty straightforward and easy to understand. Now, here’s a few differences between functions in shell scripts and a normal programming language.

Parameters #

If you were to pass a parameter / use a parameter into a function in Java, you have to declare them in the function declaration. They look something like this.

public static void main(String[] args) {
    doSomething("random String");
}

private static void doSomething (String words) {
    System.out.println(words);
}

In the shell, however, they do not require a declaration of types or names at all. Each of them is like a separate script that lives in the script itself. If you were to use a param, just pass it in and call it like how you would do it if you were taking in input for this script at the top level. Something like this:

doSomething() {
    echo $1
}

doSomething "random String"

Similar to above, if you want to take in everything, you will use $@ instead of $1 since $1 would only use the first input (and $2 for the second etc.).
Functions need to be declared ahead of where they are being called. (Usually beginning of the file before any main operations.)

Return #

Let’s say we create a script like below named run_this.sh:

doSomething() {
    echo "magic"
    return 0
}

output=`doSomething`
echo $output

Now let’s run it and see what is being assigned to the output variable.

1
2

$ ./run_this.sh
magic

Note that instead of 0, it shows magic instead. This is because when you do output=`doSomething` , it assigns the output message to output instead of the return value since output message is how you communicate almost anything in shell script.

So when does it make sense to use the return call then? When you are using it as part of an if statement. Something like this:

doSomething() {
    echo "magic"
    return 0
}

if doSomething; then
    echo "Its true!"
fi

1
2

$ ./run_this.sh
Its true!

In this case, return 0 means true while return 1 meant false in a traditional boolean sense.

Multi-line echo #

There are times when you need to print a multi-line message. There are a few ways to go around this. The easiest way is to use echo multiple times like this:

1
2
3

echo "line1"
echo "line2"
echo "line3"

It works but probably not the most elegant way to get around this. Instead, you can use cat << EOF instead. Something like this:

cat << EOF
line1
line2
line3
EOF

Note that there should not be anything (including spaces or tabs) before EOF. If you want to do it in an if statement, it should look something like this.

if [ "a" == "a" ]; then
  cat << EOF
line1
line2
line3
EOF
fi

Realize that even the messages itself are aligned to the left. This is because if you leave them tabbed, the output message shown in the command line will also be tabbed. Also, if EOF is tabbed, shell would complain about it and usually ends the script there.

Flags / Options #

You’ve probably seen some of the scripts or commands that comes with an ability to add flags (and sometimes arguments for the specific flag). Something like git commit -a -m "Some commit message".

Here’s a quick example of how it looks like (I’ve tried to be as comprehensive as possible with the example.)

while getopts ac: opt; do
    case $opt in
        a)
            echo "\"a\" was executed."
            ;;
        c)
            echo "\"c\" was executed with parameter \"$OPTARG\"."
            ;;
        \?)
            echo "Invalid option: -$opt"
            exit 1
            ;;
        :)
            echo "option -$opt requires an argument."
            exit 1
            ;;
    esac
done

$ ./run_this.sh


$ ./run_this.sh -a
"a" was executed.

$ ./run_this.sh -c
option -c requires an argument.

$ ./run_this.sh -c abcd
"c" was executed with parameter "abcd".

$ ./run_this.sh -a -c abc
"a" was executed.
"c" was executed with parameter "abc".

$ ./run_this.sh -x
Invalid option: -x

In the above example, the differences between option -a and -c is that in the getopts line, c has a colon (:) following it after therefore telling the program to expect a parameter for the option. Another thing to keep in mind is that the options need to be declared in an alphabetical way. If you declare something like acb, the b declaration would be ignored and using the -b flag would lead to the error message instead of the b case in the switch condition.

Thanks for reading!

References #

This article was originally published on freeCodeCamp.

Test Driven Development with Alexa SDK

binhong@binhong.me (BinHong Lee) — Mon, 29 Oct 2018 00:00:00 -0800

What is Test Driven Development? #

According to Wikipedia, it means “Requirements are turned into very specific test cases, then the software is improved to pass the new tests, only.” Basically, you would first have to write the test of the software/feature you are developing before you start working on the developing the software/feature itself.

Here are a few more articles on the benefits and why you should apply TDD:

Test Driven Development by Jan Olbrich
Isn’t TDD twice the work? Why should you care? by Navdeep Singh

Preparation #

Here are a few thing you might want to have set up or installed before getting started:

Amazon Web Service account (a free one will work just fine)
node.js (personally though, I prefer installing and maintaining it through nvm)
Text editor of your choice

Setting up #

Create a new project folder (mkdir project-name)
Navigate into that new folder (cd project-name)
Initialize npm (npm init)
Follow through everything as you prefer (if you don’t know what it is, just press enter and it’ll keep the default) except for test command:, put in node test/testflow.js instead. The command line should look something like the snippet below.

This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.
See `npm help json` for definitive documentation on these fields
and exactly what they do.
Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.
Press ^C at any time to quit.
package name: (emptyproject) 
version: (1.0.0) 
description: 
entry point: (index.js) 
test command: node test/testflow.js
git repository: 
keywords: 
author: 
license: (ISC)

Now you have node.js setup for the project.

Unit testing Shout out to robmccauley for maintaining testflow that provides unit testing for Alexa skills. Here’s what the following commands will do :

Clone testflow repository into your project at the ‘test’ folder. (If this project is also a git project itself, I recommend doing git submodule add instead of git clone.)
Create a folder named ‘dialogs’
Create a file named ‘default.txt’ in the ‘dialog’ folder
Edit the ‘SourceCodeFile’ line in the cloned ‘testflow’ repository to point to the ‘index.js’ in the base level of the folder (feel free to edit it to your personal preference).

git clone git@github.com:robm26/testflow.git test
mkdir dialogs
touch dialogs/default.txt
sed -i '9c const SourceCodeFile = "../index.js"' test/testflow.js

Inside the ‘default.txt’ file in the ‘dialogs’ folder, you should have the intents you are trying to test. The default in the testflow repository consist of the followings but you should definitely add some more customized ones to it depending on what you are trying to do in your skill.

1
2
3

LaunchRequest
AMAZON.HelpIntent
AMAZON.StopIntent

In case where you are taking in variables through utterances, you can do something like this :

`1`	`DoSomething Input1=userInput1 Input2=userInput2`

To get a deeper understanding into using testflow, I recommend looking into its tutorial page and the blog post by the author himself.

Developing the skill #

To be honest, I don’t have too much to talk about on this topic. Here is some sample code of how a sample skill can look like. (Also remember to do npm install --save alexa-sdk.)

var Alexa = require('alexa-sdk')

exports.handler = function (event, context, callback) {
  var alexa = Alexa.handler(event, context)
  alexa.registerHandlers(handlers)
  alexa.execute()
}

var handlers = {
  'LaunchRequest': function () {
    this.emit(':ask', 'Welcome to the sample skill. Say help for help.')
  },
  
  'Intent1': function () {
    this.emit(':ask', 'This is the first intent.')
  },
  
  'Intent2': function () {
    var input1 = this.event.request.intent.slots.InputOne.value.toString()
    var input2 = this.event.request.intent.slots.InputTwo.value.toString()
    var response = doSomething(input1, input2)
    this.emit(':ask', response)
  },

  'AMAZON.HelpIntent': function () {
    this.emit(':ask', 'Insert some help message here.')
  },

  'AMAZON.StopIntent': function () {
    this.emit(':tell', 'Have a good rest of your day!')
  }
}

function doSomething (input1, input2) {
  if (input1.localeCompare(input2)) {
    return "They are the same string!"
  } else {
    return "They are different!"
  }
}

I recommend looking into the official Alexa GitHub repository that provides all kinds of sample skills that you can work on top of.

Deployment #

Alexa Skill Kit #

Go to this page and select ‘Create Skill’. In there you should see options to ‘add intents’ and ‘custom slot types’. You should add the appropriate slot types and intents accordingly from above. After you are done with that, select ‘Endpoint’ from the side bar and select ‘AWS Lambda ARN’ as the skill’s service endpoint. Make a copy of the Skill ID as you will need it later.

AWS Lambda #

Create an AWS Lambda function selecting ‘from scratch’. After that, you should have an option to ‘add trigger’ where you would select ‘Alexa Skills Kit’. Now in the configuration dialog, you should put in the Skill ID you took a copy of previously then select ‘Add’.

After this, select back to the box that labeled your project name. There should be a ‘Function code’ dialog box appeared below. While you can definitely copy and paste your code onto the online text editor, there might be chance where the packages you have installed and used in your code is included by default. It is also possible that you have more than 1 JavaScript files for the skill.

In this case, you would select ‘Upload a .ZIP file’ under ‘Code entry type’. Locally, zip up all the JavaScript files along with the node_modules folder then upload the resulting zip file.

Submission #

When you are done, you can go back to the Alexa Skill Kit page, select the ‘Distribution’ tab and fill out the appropriate information before submitting your skill for review. Hopefully everything goes well and your skill gets published at the Alexa Skill Store!

Credits #

This article was originally published on Hackernoon.

How to host multiple domain names and projects on one server

binhong@binhong.me (BinHong Lee) — Wed, 29 Aug 2018 00:00:00 -0800

I own multiple domain names, and each one hosts a different side project. For the longest time, everything that required ‘hosting’ was hosted on Heroku. But their free tier can be quite limited, it can also get costly quickly if you are paying for each separate project. So instead, I decided to explore putting all of them together using NGINX (recommended to me by Jane Manchun Wong).

Required Resources #

Virtual Private Server (VPS) #

You’ll need a virtual server such as DigitalOcean or EC2 by AWS. Personally I uses Vultr (here’s the non-referral link) which costs me about $2.50 / month.

Domain Names #

You will need to register a few domain names. Assuming that you probably already have them, make sure your domain names are pointing at the name servers of your VPS. There should be a DNS section in your domain name service dashboard where you can select “custom DNS” or something similar. If you are not sure what the nameservers of your VPS are, you should be able to find that info easily through a simple search of “nameserver” + VPS service name.

Setting up NGINX #

Installation and basic setup #

Reference from How To Install Nginx on Ubuntu 16.04

Run the following commands through SSH-ing into the VPS. It will install NGINX, set firewall rules allowing it, and set NGINX to autostart on boot.

sudo apt-get update
sudo apt-get install nginx
sudo ufw allow 'Nginx HTTP'
sudo systemctl enable nginx

Configuration setup #

Reference from Host Multiple Domains on One Server/IP with Apache or nginx

The default virtual.conf location should be at /etc/nginx/conf.d/virtual.conf. I recommend backing up the default file before making any changes. (If it doesn’t exist, you can just create it.) Edit the file to look something like the following:

server {
    listen 80;
    root /opt/htdocs/binhong;
    index index.html index.htm;
    server_name binhong.me;
    location / {
        try_files $uri $uri/ =404;
    }
}

server {
    listen 80;
    root /opt/htdocs/breakups;
    index index.html index.htm;
    server_name breakups.life;
    location / {
        try_files $uri $uri/ =404;
    }
}

server {
    listen 80;
    root /opt/htdocs/breakupsAPI;
    index index.html index.htm;
    server_name api.breakups.life;
    location / {
        proxy_redirect http://127.0.0.1:8080/;
    }
}

Here are a few things to look at:

server block — Each of these should represent each different domain or subdomain in use.
root — This is the location where the (HTML) files are loaded from.
server_name — (sub)domain name(s) that should load these specific files.
proxy_redirect — in cases where you are redirecting a specific subdomain to an active server, you will want to add this and put the IP location after it. (For local servers, either http://127.0.0.1:port or http://localhost:port should work as intended.)

`1`	`sudo systemctl restart nginx`

After you are done, restart the server so the new configurations will be loaded and applied.

Cloning and linking #

Now remember, since you have your directory pointing at /opt/htdocs/websiteName, your initial thought might be to clone your projects into these folders. This can work, but it’s not ideal since many operations in these folders require root access to really do anything.

Instead, you can clone them into your user folder or anywhere else like you normally would, and then create a soft link to connect the path to your repository folder. Something like this:

1
2

git clone git@github.com:binhonglee/binhonglee.github.io ~/website
sudo ln -s ~/website /opt/htdocs/binhong

Of course, when you are cloning a Node.js static site folder (ReactJS, Angular or Vue.js), you will want to install (npm install) and build (npm run-script build) them. Then link the ./build folder instead of the base level of the cloned repository. (Similarly for Jekyll sites, but use the ./_output folder instead.) As for active servers, just make sure your server is running on the same port as it is listed in the configuration file.

Set up HTTPS with certbot #

Thanks to Let’s Encrypt, you can now get free and easy HTTPS certificates. With the introduction of certbot, everything just got even easier!

Reference from How To Secure Nginx with Let’s Encrypt on Ubuntu 16.04

sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python-certbot-nginx
sudo certbot --nginx -d binhong.me -d www.binhong.me -d api.breakups.life -d breakups.life -d www.breakups.life

Just run the above for all your domain and subdomain names and certbot will take care of everything. If you were to renew the certs, you can run the following so the certbot will help you renew your SSL certificate.

`1`	`sudo certbot renew --dry-run`

Updating everything #

Now that you have everything up and running, you might be thinking, well there seems to be an awful lot to remember if/when I need to update something. Unfortunately, that’s kinda true, but we can always make it easier by adding a script that does it for us.

Here is how one would look:

# Update SSL certificates
sudo certbot renew

killall screen

# normal html websites
cd websiteSource
git pull
cd ../

# node.js static site
cd nodeProjectFolder
git pull
npm install
npm run-script build
cd ../

# ruby jekyll site
cd jekyllFolder
git pull
bundle install
bundle exec jekyll build
cd ../

# node.js server
# make sure to stop the existing session before going ahead to start a new one
cd nodeServer
git pull
screen -dmS nodeServer
npm install
screen -S nodeServer -X stuff "cd ~/nodeServer
"
screen -S nodeServer -X stuff "npm start
"
cd ../

Thanks for reading! Let me know if you have any questions in the comments below.

Other References #

This article was originally published on freeCodeCamp Medium.

Hosting your own Git server with Gitea

binhong@binhong.me (BinHong Lee) — Mon, 20 Aug 2018 00:00:00 -0800

Most people (including myself) host their personal projects on a third-party free Git hosting websites like GitHub, GitLab, Bitbucket etc. While that is sufficient for most people, it is also pretty fun to have your own Git service hosted on your own domain name of choice. While I trust my fellow engineers at these companies, I am using this as a personally backup to those services if anything goes wrong (like GitLab. Though to their credit, they were able to recover most of it).

Requirements #

There really isn’t much requirements except for a working machine and a proper working network. Preferably, you would be hosting this on a Virtual Private Server (DigitalOcean, Vultr, Linode etc.). However, you can also mess around with it on your own machine (it takes and requires very little resources by itself) or even host it at home on a Raspberry Pi!

Installation #

Referenced from Gitea’s own documentation

While there is a Gitea package in Debian’s contrib (which most popular Linux OS such as Ubuntu and Mint based itself on), it doesn’t seem it was maintained by the Gitea folks themselves so I wouldn’t advice using it. Instead, the instructions below will provide an easy way to just download and run it with binary.

Database #

You’ll need a database to maintain all the information of user accounts and such. Gitea comes with support for SQLite, MySQL and PostgreSQL out of the box so you can pick either of those that you are familiar with to go with. Personally I picked PostgreSQL as I’ve always wanted to learn how to mess with one. In your choice of database, you should create a user (preferably only specifically for the use of Gitea) and a database for Gitea. You will need to fill out these credentials later when setting up Gitea.

Gitea #

As Gitea calls itself ‘a painless self-hosted Git service’, the setup is indeed very straightforward. Simply download and run it depending on they way you want to do it.

1
2

wget -O gitea https://dl.gitea.io/gitea/1.5.0/gitea-1.5.0-linux-amd64
chmod +x gitea

Optionally, you can also verify the GPG signature of the downloaded file with Gitea’s GPG key for security purposes before running it.

1
2

gpg --keyserver pgp.mit.edu --recv 0x2D9AE806EC1592E2
gpg --verify gitea-1.5.0-linux-amd64.asc gitea-1.5.0-linux-amd64

To run Gitea, it is as simple as the following…

`1`	`./gitea web`

By default, it will be running on port::3000 of the device IP (http://localhost:3000/). In case you prefer running Gitea from source, they also have specific documentations for that. I had a lot of troubles trying to set it up myself so YMMV.

Load Balancer (Optional) #

Referenced from Gitea’s own documentation

If you are running this on a remote server (especially if you are also hosting a bunch of other stuffs), you will need to use a load balance (like Apache HTTPD or NGINX) to forward a specific domain to the port. Append your configuration file with the following as you see fit.

Apache HTTPD #

<VirtualHost *:80>
    ProxyPreserveHost On
    ProxyRequests off
    ProxyPass / http://localhost:3000/
    ProxyPassReverse / http://localhost:3000/
</VirtualHost>

NGINX #

server {
    listen 80;
    server_name git.example.com;

    location / {
        proxy_pass http://localhost:3000;
    }
}

Configuration #

Open your browser and navigate to the wherever Gitea was hosted. In there, you will be filling out the database settings as you have configured above (for PostgreSQL, it should be on 127.0.0.1:5432, not sure about the others).

Few things to look out for:

Domain / Application URL - This is important to be set up properly as they are used for SSH and HTTPS connection when it comes to cloning and pushing.
Disable Self-registration - Considering that you are probably setting this up only for your own use, you should select this option so random people wouldn’t just sign up and host projects on your server.
Admin username and password - Since you have disabled self registration, you should then set this up in here and use it as your primary account. If you are concerned about security, you can also use the admin account to setup a day-to-day use account so in case the day-to-day account is compromised, you still retain full control of the server through your admin account.

Usage #

For the most part, I’m using this as a backup to my projects hosted on GitHub and GitLab (project pages is an amazing feature) so ideally, I am mirroring most (if not all) of my projects. As part of being a mirrored repository, Gitea will periodically check the source repository for updates so you don’t have to manually sync them. That said, I have quite a few projects that I moved away from mirroring and I’ll talk about why below.

Maintenance #

In case where your VPS (or Raspberry Pi) might restart constantly or so, it might be a chore to always manually start the service every time it is restarted. You should then consider configuring Gitea to run as a service (Ubuntu, Windows).

Drone.io #

When talking about the topic of Git, its hard to stay away from CI. It is such a great idea of automation that saves so much pain and problem especially if it is a huge project that takes a lot of resources and hard to run the tests locally. Here, I’ll talk about drone.io which can also be self-hosted along with your Gitea server.

Requirements #

You will need to have docker installed in your machine as drone as a whole relies quite a bit on docker to work. The machine where drone is hosted will also require quite a bit of memory and processing power as running CI can be quite a chore for servers.

Installation #

Referenced from drone’s own documentation

First, get the docker image for drone.

`1`	`docker pull drone/drone:0.8`

Next, create a file named docker-compose.yaml and fill in the following.

version: '2'

services:
  drone-server:
    image: drone/drone:0.8

    ports:
      - 8000:8000
      - 9000:9000
    volumes:
      - /var/lib/drone:/var/lib/drone/
    restart: always
    environment:
      - DRONE_OPEN=true
      - DRONE_HOST=http://drone.example.com
      - DRONE_GITEA=true
      - DRONE_GITEA_URL=http://git.example.com
      - DRONE_SECRET=${DRONE_SECRET}

  drone-agent:
    image: drone/agent:0.8

    command: agent
    restart: always
    depends_on:
      - drone-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DRONE_SERVER=drone-server:9000
      - DRONE_SECRET=${DRONE_SECRET}
      - DRONE_DATABASE_DRIVER:postgres
      - DRONE_DATABASE_DATASOURCE:postgres://root:password@1.2.3.4:5432/postgres?sslmode=disable

Few things to keep in mind in the file above:

DRONE_SECRET - This is a secret string of your choice used for authentication purpose.
DRONE_HOST - URL where this drone server is hosted.
DRONE_GITEA_URL - Link to your Gitea server location (it can also be http://localhost:3000/ link)
DRONE_DATABASE_DRIVER - Database setup is optional. In their documentation, they mention the setup process of MySQL and PostgreSQL. Though do note that their documentation doesn’t seems to be accurate at least from my experience setting it up.

In the same directory run the following and drone will be running on http://localhost:8000/.

`1`	`docker-compose up`

Load Balancer (Optional) #

Similarly, if this is hosted on a remote server, you will have to add the following with the load balancer of your choice.

Apache HTTPD #

<VirtualHost *:80>
    ProxyPreserveHost On

    RequestHeader set X-Forwarded-Proto "https"
    ProxyPass / http://127.0.0.1:8000/
    ProxyPassReverse / http://127.0.0.1:8000/
</VirtualHost>

NGINX #

server {
    listen 80;
    server_name drone.example.com;
    location / {
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;

        proxy_pass http://127.0.0.1:8000;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_buffering off;
        chunked_transfer_encoding off;
    }
}

Usage #

As mentioned, drone relies on docker not only for its installation but also running the CI itself. Each repository that you want to configure for drone should have to have a .drone.yml file like below at the top level of the repository.

pipeline:
  build:
    image: node:${NODE_VERSION}
    commands:
      - npm install
      - npm run test

matrix:
  NODE_VERSION:
    - latest
    - "7"
    - "6"
    - "4"

Few things to pay attention to in the sample file:

image - Drone supports any docker images available for download. You can look for available docker images at the Docker Hub.
matrix - Like the scenario above, you want to test the project in multiple environments but doesn’t want to retype most of the stuffs, you can use matrix as a variable to hold all the different versions you would like to run tests on.
commands - This is the list of commands to run for the test to take place. You should also include installation of project dependent packages (such as mvn install, npm install, bundle install etc).

Personal Notes #

Mirrored Repositories #

Currently from my own experimentation, adding new commits to mirrored repositories (which are read-only) that are mainly hosted elsewhere would not trigger drone to run CI. Therefore, if you were to run drone on your own server, you can either enable it directly with the origin of the repository mirror or push twice to 2 separate servers for every commit. I’m still looking for a solution for this so hopefully there’s a better way than that.

Memory issues #

Some of these instances can take quite a bit of memory spaces to run. Personally, I am hosting this on a 256MB VPS squeezing it with a bunch of other stuffs I hosts. This actually causes an issue where the tests cannot be run properly. As seen here, at the end of the build log, it was just abruptly ‘Killed’ and ended there and then with no further information. I am still working on investigating this but it is very likely due to the lack of remaining unused memory space on the VPS.

Static IP #

In case you are hosting this on a Raspberry Pi at home for personal use, you would want to set a static IP direction on your router. This would help make sure that when in your network, you can always connect to the device through the same IP.

NGINX #

I did not go deep into either of the load balancers in here as I am currently working on a separate piece that will focus more onto the configuration and use of NGINX to hosting multiple domain names and ‘servers’. Keep a look out for that!

You can read more about NGINX on my next post here.

References #

This article was originally published on codeburst.io.

Building a side project over the weekend

binhong@binhong.me (BinHong Lee) — Sat, 02 Jun 2018 00:00:00 -0800

Few months ago during my job search, I decided that I want to add some new skills to my arsenal (resume) so I went ahead to build a bill splitting webapp (and the month before, an Alexa skill). Granted, the backend was built weeks before during a hackathon (shoutout to HackISU), I’m just focusing on building the UI that makes post requests to the backend. In the process, I also ended up fixing some inherent flaws I left behind in the backend.

Idea #

Be realistic. You don’t want to end your weekend with an unfinished product that would cause you to lose focus of work / study during the week because you “really need to go back and get it done with”. On the other hand, you also don’t want to just straight away give up in your first few hours because “it’s impossible”.

In cases where the thing you really wanted to build is huge and complicated, try breaking it down into smaller pieces and maybe span it over multiple weekends. The point being to have a realistic idea of what to expect at the end of the weekend and achieve it while not mounting yourself additional pressure (and distractions) to ‘complete’ the other parts of it during your work / study week that would jeopardize what you are supposed to be doing during the day.

Preparation #

Plan #

I would be lying if I said I have a detailed plan of how long I’ll take to get each part done. I just draft up the most minimal version that I think covers the basic need and leave it at that. Don’t over dictate yourself that you don’t have flexibility in case something happened to be harder than you thought, which is almost always the case.

Pick something you are not already familiar with. The best way to improve is to get out of your comfort zone. It’s also only a weekend long so even if that turn out to be something you dislike, at least you now have a solid opinion backed with experience on that.

Food & beverages #

I think this is a major one. Figure out how you’re going to get food supplies over the weekend that will reduce the amount of time you spend not coding. If you usually cook for yourself, just make sure to look for meals that takes little time to prepare and prepare ahead of time too.

Also do you even Soylent? (Credit to Jane Manchun Wong)

Process #

Hold yourself accountable. I’m not saying to reject every single invitation from your friends to hang out but be you have to be aware and be flexible of your schedule. If you are in the train of thoughts or ‘in the zone’ to finish up one major part of it, let your friends or whoever you’re gonna hang with know slightly in advance. I’m sure they wouldn’t mind the slightly additional wait while you have a peace of mind out partying or doing whatever instead of subconsciously constantly thinking about what you would be implementing.

Take it slow. The faster you go in the beginning, the faster you burn out. This happens a lot and everywhere. It can be overwhelming in the beginning so you slowly take your time to walk through things and watch some tutorials (YouTube is amazing) to get help on learning to build whatever you wanted to (especially with new technologies you never tried).

Take frequent breaks. I can’t stress how important it is to relax in between. You don’t get maximum efficiency by rushing things, you get them by being in between too comfortable and too overwhelmed. Personally, I would take a break and go out for a walk and fresh air every 30 mins or so. This highly differs among individuals so you have to figure out what works best for you.

Wrap up #

Don’t get too hung up on the project. I’m not saying you shouldn’t be proud of it but the skills you obtained in the process will very likely to be much more valuable than that. As mentioned, you don’t want to feel obligated to come back and ‘complete’ the project in the future when you’re procrastinating while you actually have other priorities like project deadlines, homeworks etc. so its important to wrap up your project properly in a way that its a minimal product presentable but also something you understand that you can always add onto whenever you feel like it.

Of course, in an off chance that it blows up, congrats! For starters, maybe you can start thinking about how to monetize it? I don’t know (this has not happened to me). Maybe some guys at I can handle the business side can help you with that. 😉😂 /j

This article was originally published on my personal Medium publication.

Getting started with Travis-CI

binhong@binhong.me (BinHong Lee) — Sun, 22 Apr 2018 00:00:00 -0800

What is CI and why should I care about it? #

CI stands for Continuous Integration which as the name suggests, is an integrated tests that runs continuously on every time when someone pushes a commit (or mostly when opening a Pull Request).

As for why is it important, it automates the whole testing modules. Ideally, you would still want to test your changes locally before pushing them to remote but in some occasion where you forgot or the program requires a lot more resources than your current machine can handle, CI comes in very handy. That aside, if your project happen to have people wanting to contribute code, a properly written test with CI will help you in reviewing the changes and ensuring that the contribution would not break the build.

Why Travis-CI? #

There are plenty of CI tools out there (cricleCI, AppVeyor etc.) so why Travis-CI? The straightforward answer is that its the easiest one to get started with especially if you are already on GitHub. This is actually also one of its major drawback which is how tightly tied Travis-CI is onto GitHub. As far as I know, it is not possible to use Travis-CI outside of GitHub. That aside, if you host your projects on GitHub, that should be the least of your concern.

Setup #

First, sign up at their website, link your GitHub account and enable it for the repository you would like to implement this. Do note that this is only for public repositories. If you would like to do this for a private repository instead, you will need to sign up for Travis-CI Pro.

Create a file named .travis.yml and add the first line with the programming language you used (unless its Android in which you would then write Android instead of Java).

1
2

language: java
# (or cpp, android, node_js etc.)

Now, there’s also a possibility that running the test might require sudo access to the testing environment in which you would then add the following line above the language line.

1
2

sudo: required
language: cpp

Testing environment #

Ideally, you will only be testing it in one environment which means this will be a one liner of this.

`1`	`os: linux`

However, in most occasion, we will be testing our software with a few different kinds of environment (at the time of writing, Travis-CI seems to only Ubuntu and OSX as seen here). Sometimes, 2 lines is enough.

1
2
3

os:
  - linux
  - osx

But more time than not, you need complicated configuration for each one of them (especially when it comes to C++ projects).

Here is how the environment configuration for one of my projects looks like (shortened):

matrix:
  include:
    - os: linux
      addons:
        apt:
          sources:
            - ubuntu-toolchain-r-test
          packages:
            - g++-7
      env:
        - MATRIX_EVAL="CC=gcc-7 && CXX=g++-7"

    - os: osx
      osx_image: xcode8
      env:
        - MATRIX_EVAL="brew update && brew install gcc && CC=gcc-7 && CXX=g++-7"

Language specific settings #

For some languages, it is a lot more complicated than the others due to the way their versioning works.

Android #

This is probably the most complicated one I’ve come across. It took a lot of tries and failure for me to figure out how it works. (The documentation has also since updated which helps greatly.)

android:
  components:
    - tools
    - platform-tools
    - build-tools-26.0.2
    - android-26
    - platform-tools
    - extra-android-support
    - extra-google-google_play_services
    - extra-google-m2repository
    - extra-android-m2repository
    - addon-google_apis-google-26
    - sys-img-armeabi-v7a-android-26

C++ #

There isn’t specific custom configuration available for C++ like for Android above but due to the nature of its complicated OS configuration, you will need to import the matrix from before into the system before installing anything that might be needed to run the test itself. Therefore you will need to add the following.

1
2

before_install:
  - eval "${MATRIX_EVAL}"

Java #

For Java, due to the differences between OpenJDK, OracleJDK and also differences from versions to versions itself (shoutout to OracleJDK 9 & 10), you might want to test your software with every version available.

jdk:
  - oraclejdk9
  - oraclejdk8
  - openjdk8
  - openjdk7

Python #

Similarly, you can test your software or package against different versions of Python including the development versions.

python:
  - "2.7"
  - "3.6"
  - "3.6-dev"
  - "pypy2.7"
  - "pypy3.8"

*Note: Due to recent changes, seems like 3.7-dev, 3.8-dev, and nightly would not work as of now.

Node.js #

Again, you might need to test your software with multiple versions or node.js except the differences between each version is not as drastic but there are a lot more versions comparatively.

node_js:
  - "node"
  - "9"
  - "8"
  - "7"
  - "6"
  - "5"
  - "4"
  - "lts/*"

By doing this, you will be testing your software towards possibly 8 versions of node.js.

Install and run test #

If you look into the specific documentations about the configurations for each languages, you will find that many of them has default install and test scripts but you can always overwrite them with your own scripts or commands like the following.

before_install:
  - ./beforeInstall.sh
install:
  - ./install.sh
before_script:
  - ./preTest.sh
script:
  - ./test.sh

*Note: if you testing in macOS environment and is installing any dependencies through Homebrew (including language dependencies in C++), you will have to do a brew update before running anything brew related to prevent an error.

Deployment #

If you host your project externally while using your GitHub repository as your code control, Travis-CI can also help automating the deployment process. Depending on the service provider you use, their configuration can vary greatly from one another. Below, I included some of those that I think is the more popular ones. As you might have guessed, you can also deploy to multiple developers at once. I recommend checking out the documentation page before implementing any of these.

before_deploy:
  - ./before_deployment
deploy:
  - provider: heroku
    api_key: "YOUR_API_KEY"
    skip_cleanup: true
    on: production
    run:
      - "./deploy"
  - provider: pages
    github_token: $GITHUB_TOKEN
    skip_cleanup: true
    on: production
    run:
      - "./deploy"
  - provider: npm
    api_key: "YOUR_API_KEY"
    email: "email@domain.com"
    on: production
  - provider: elasticbeanstalk
    access_key_id: <access-key-id>
    secret_access_key:
      secure: "Encypted <secret-access-key>"
    region: "us-east-1"
    app: "your-app-name"
    env: "your-app-environment"
  - provider: pypi
    user: "username"
    password: "password"
after_deploy:
  - ./post_deployment

Code coverage #

Along with testing, I would also recommend using a tool to analyze your code coverage. It will show you how much coverage your test has over your codes (and usually, use cases). Personally, I use the tool called codecov.

Add the badge! #

If you are like me, the reason I was hooked up by this is because long time ago when I first got on GitHub, I saw plenty of repositories have the badge that shows if their build is passing or failing. I thought that was really fancy and wanted one of those so I started getting myself to learn what CI as a whole is about and how can I make use of it. Turns out, its a lot more useful than just being pretty!

This article was originally published on codeburst.io.

RESTful API with Spark Kotlin

binhong@binhong.me (BinHong Lee) — Wed, 21 Mar 2018 00:00:00 -0800

When speaking of building a backend API, the most common tools is either Spring Boot for Java or ExpressJS for JavaScript. Even for Kotlin, Spring Boot and JetBrain’s own ktor is the usual option to go with. But today I’ll explore into the less popular option of spark-kotlin.

Disclaimer : I am not in anyway affiliated nor do I work at the project. I just simply stumbled upon it and started using it when working on a side project. (coming soon™)

Understanding REST #

If you already know or built a RESTful API, you can skip this part. There are already quite a few guides out there that goes into the specifics of what REST is and all the amazing things about it. So instead, I’ll just cover the basics (more like, that’s all I know about it). It’s the standard way of communication on the internet through HTTP. In the world of internet, the frontend web interface needs a way to communicate with the backend. While the frontend can say all kinds of things, they still need to declare a set of standard ‘phrases’ between them so the backend knows what the frontend is asking for.

Setting up #

Here are a few thing you might want to have installed before getting started:

Maven / Gradle
Kotlin
Java? (not too sure but you’ll probably need it)
(optional) IntelliJ

Getting started #

According to the official GitHub repo of spark-kotlin, here are the following things to include to use it for either Maven or Gradle. Personally, I’m using Gradle for this simply because I’ve never used it outside of Android Development. (Also, its the new sexy thing. Kinda?)

<dependency>
    <groupId>com.sparkjava</groupId>
    <artifactId>spark-kotlin</artifactId>
    <version>1.0.0-alpha</version>
</dependency>

dependencies {
    ...
    compile "com.sparkjava:spark-kotlin:1.0.0-alpha"
    ...
}

The basics #

This is what attracted me to use it over spring-boot. It just looks so much easier and straightforward comparatively. Admittedly, I’ve never used spring-boot for server development so I could be totally wrong especially if/when it comes down to a large scale application.

GET request #

import spark.kotlin.*

fun main(args:Array<String>) {
    get("/") {
        "Welcome to website abc."
    }
}

That’s it! I could’ve just ended the article here and you can pretty much figure out the rest on your own. But yea, let’s explore into a few more interesting features.

POST request #

import spark.kotlin.*

fun main(args:Array<String>) {
    post("/checkExist") {
        checkItem(request.body())
    }
}

fun checkItem(input:String): String {
    return if(exists) "Exists." else "Does not exist."
}

fun exist(input:String): Boolean {
    //Some check to make sure given string exist
}

For POST request, they work as easily too! All you need to know is that the request body can be retrieved with request.body().

More fun stuff #

redirect(), params() #

Now, what if you want to have the same output for separate links? Well, you can just do redirect("/path"). You can also get information from the URL through params(":path") tag. Something like this…

import spark.kotlin.*

fun main(args:Array<String>) {
    post("/checkExist") {
        redirect("/checkExist/" + request.body())
    }

    get("/checkExist/:id") {
        checkItem(params(":id"))
    }
}

fun checkItem(input:String): String {
    if (exist(input)) {
        return "Exist."
    } else {
        return "Does not exist."
    }
}

fun exist(input:String): Boolean {
    //Some check to make sure given string exist
}

Instance API #

You can also have a separate instance API that is running on a different port or with a different setting than the static API simply by declaring it as a separate variable.

import spark.kotlin.*

fun main(args:Array<String>) {
    val http = ignite {
        port = 4567
        ipAddress = "0.0.0.0"
    }

    http.get("/") {
        "Welcome to website abc."
    }

    http.get("/404") {
        status(404)
        "Page does not exist."
    }
}

Of course, there are a lot of more advanced features (such as session, uri, etc.) that I did not cover. Feel free to explore them on your own following the README of the repo.

From here on out, I’m including some relevant but not exactly spark-kotlin stuff on it.

Dealing with JSON #

In many (or most) occasions, both requests and responses will be in JSON format. Here’s where Google’s Gson library comes in handy. It is really easy to use and had a wide range of support. I put together a simple example below to show how it is and can be used.

import com.google.gson.GsonBuilder

fun main(args:Array<String>) {
    // Some codes
    val json: String = "{\"id\": 32151986,\"info\": \"someWords\"}"

    // Creating the item from JSON
    val newItem: Item = GsonBuilder().create().fromJson(json, Item::class.java)

    // Creating JSON string from item
    print(GsonBuilder().setPrettyPrinting().create().toJson(newItem))
}

data class Item (val id: Long, val info: String)

Unit testing #

It took me a while to find a testing library that works as easily as I wished. This is a test library made specifically for the spark framework by Despegar called spark-test.

import com.despegar.http.client.*
import com.despegar.sparkjava.test.SparkServer
import spark.servlet.SparkApplication

class YourTest: SparkApplication {
    override fun init() {
        main(arrayOf())
    }
}
var testServer: SparkServer<YourTest> = SparkServer(YourTest::class.java, 4567)

class Tests {
    @Test
    fun getTest() {
        var get: GetMethod = testServer.get("/", false)
        var httpResponse = testServer.execute(get)
        assertEquals(httpResponse.code(), 200)
        assertEquals(String(httpResponse.body()), "Welcome to website abc.")
        get = testServer.get("/404", false)
        httpResponse = testServer.execute(get)
        assertEquals(httpResponse.code(), 404)
    }

    @Test
    fun checkExistTest() {
        val post: PostMethod = testServer.post("/checkExist", "someInput", true)
        val httpResponse = testServer.execute(post)
        assertEquals(httpResponse.code(), 200)
        assertEquals(String(httpResponse.body()), "Exist.") // or "Does not exist."
    }
}

A few things to keep in mind though:

This is a Java library made for spark-java instead of spark-kotlin
The server needs to be running on the background (in my case, on port 4567) for this to work
Unable to attach a request body for DELETE request

Note: As mentioned at the time of writing, the DELETE request is unable to attach any kind of request body to it making it hard to test your DELETE request. Personally, I use different URL each feature so I just redirect a POST request from the same url to the DELETE request and test it through the POST request instead. Not ideal, but it works for now.

References #

Kotlin WebApp Tutorial by Wayne Ellis
Build REST Service with Kotlin, Spark Java and Requery by Vaclav Souhrada

This article was originally published on Hacker Noon.

Being lucky

binhong@binhong.me (BinHong Lee) — Sat, 10 Feb 2018 00:00:00 -0800

Ah, you’ve graduated. Now what? Oh right, still no job offer! *panics*

As an international student, I have come across hundreds of guides on “how to land an interview”, “how to get a job”, “how to solve DSA questions” etc but almost none of them were specifically written with an international student in mind. I’m not here to dismiss the quality of such guides but rather, to be an add-on (or plugin if you will), to help international students understand their challenges better.

You’ve got a lot more to prove #

Let’s face it, with all the hassle for potential visa sponsorship or even losing you due to the visa lottery program, you have got a lot more to prove of being worthy of that extra investment. If they can hire someone to do the same job without all that hassle for similar or even slightly more expensive prices, they would hop on that right away. It just doesn’t make sense financially to invest in you when you have a higher chance of deciding not to or not being able to continue working for them 3 years later (after training you from a raw talent to a solid engineer).

I have people around me who constantly complain about this. I do too sometimes but then I thought, I’m not ‘top of the class’ smart or ‘winner of multiple hackathons’ smart. Also, complaining about it won’t help your case. A job won’t fall from the sky randomly just because you complained hard enough. If you are just an average graduate (like myself), there is really very little incentive to think you are irreplaceable (technologically speaking) though that doesn’t mean you won’t be in 3 years time.

Going for the big four #

The big four in tech usually refers to Google, Facebook, Amazon and Apple (or Microsoft). Of course, anything in this paragraph also applies to most of the larger tech firms. These companies usually don’t take ‘the need of visa’ too much into consideration (if at all) as part of their hiring process but they also have higher barriers of entry than the smaller or more traditional companies.

This means that you don’t need to worry so much about your application getting thrown out immediately after you check the ‘will need visa sponsorship in the future’ box. That said, you will need to be better than an average graduate. Spend some time on HackerRank, build some side projects, diversify your skill sets and most importantly, building connections. In the ocean of resumes that companies like these receive every year, it is really hard to stand out. One of the easiest way to get to the front of the line is through referrals.

Go on LinkedIn, GitHub, local hackathons or your local developer meetups to chat and get connected with some engineers or even product managers. In many occasions, they would be more than happy to provide you with a referral since it would also net them some sweet bonuses. Before asking for referrals, make sure to do your own research through the company job site and shortlist about 3 positions you are interested in. Depending on the company and its referral system, this will probably make the job a lot easier for the person referring you.

My 3 months of OPT unemployment is almost up! #

The most popular way of extending this is to look for a professor you enjoy working with to help with his/her research. Alternatively, you can also look for a local super early stage registered startup or open source organization that might interest you and volunteer to provide free (or paid with equity) work.

While the general consensus is to never work for free, you should be very clear with what you are getting back in return. For example, you might be interested in helping the cause of what the organization is working towards or that you really want to work with the CTO and learn from him directly. Things like this might become an asset (or as Cal Newport calls it, career capital) in the future.

Remember, the requirement is to work at least 20 hours a week. Since you’re working for free in many occasions, it’s definitely possible to negotiate your hours and reserve the rest of the time devoting it towards your job searching effort.

The first job is always the hardest #

Don’t take this from me. Take this from almost every other person in the industry I’ve ever met. Get an internship whenever you can! Even if this means you might have to go back to your home country or work on some shitty project. If you are like me, only realizing the importance of this in your senior year, start working on side projects or open source projects. Develop niche, specific skills and people skills. Few months ago, I wouldn’t have known that my experience in developing an Alexa skill (because why not?) would helped me that much in my interview. If you know you won’t be able to impress recruiters with your experience, grades or anything, you’ll need to make it up with something else. Remember, you’ve got a lot more to prove, more than your peers with no visa issues.

Being lucky #

I was lucky. I got a phone call from a recruiter asking me if I’m interested in this position. After taking forever to schedule an interview, it turns out the team needed a few more engineers (who knows about ML and speaks Malay ‘natively’) so I was pretty much hired right away. The job offer fell from the sky. It came from somewhere I least expected.

This is not just me. Almost every time I come across an international student getting an offer for a position at a tech giant for the first time, its always about ‘I was lucky’ somewhere down the story. When people hear that, they rule out that entire story as anecdotal but I think this is a consistent pattern worth exploring. Everyone gets lucky once in a while, but are you ready when your luck strikes? If you aren’t, no matter how many final round interviews magically fall onto your lap, you will fail all of them.

One could argue that if you are good enough, luck is irrelevant. That’s probably true. But then if you are that good, you are probably just sitting at home debating which offer to pick up instead of sitting here reading this piece of article from some random guy on the internet. It’s not just about being lucky, its also about being ready enough to fully utilize that moment of luck.

This article was originally published on my personal Medium publication.

I could become you tomorrow and no one will know

binhong@binhong.me (BinHong Lee) — Mon, 20 Nov 2017 00:00:00 -0800

Backstory #

I lied, but maybe someone else could. If you didn’t know already, Malaysia just suffered possibly its largest data breach ever with personal data of millions of Malaysians (including your full name, IC number, phone number and home address) up for grabs as long as the price is right. If you are not freaked out yet, you should be and I’ll tell you why. I don’t think many people understand how dangerous these data can be. I’m also honestly mildly infuriated about this. No, not about the creator of sayakenahack.com (hereafter referred as Keith) but rather how this whole thing panned out. I’ve mentioned before in private conversations with friends about my concerns on cybersecurity in Malaysia but never really gave it too much thoughts. Not until I discovered this whole fiasco.

Let’s face it, with the amount of telecommunication companies being breached, it’s fair to assume that anyone who registered a phone number with their personal info on or before 2014 have their personal information leaked. (This was also the general consensus of anyone who had seen the raw data, I didn’t.) By the time you are reading this article, the site should have already been taken down due to MCMC blocking it. (This does not however, means that the data is gone. Plenty of shady individual everywhere still have a copy of all these dangerous piece of information.) I waited for it to be taken down before sharing this as I will cover an exploit that can possibly end up producing some potentially damaging information with some very basic scripting skills (less than 1 hour of Googling).

The true risk #

When I brought this up to some of my friends, their reactions were rather “Meh, I don’t have anything of worth anyway. Why would they wanna hack me?”. This is where you’re wrong. They don’t have to hack you, you’re ALREADY HACKED. As to what can they do with it? Identity theft, basically THEY CAN BECOME YOU. With that, I mean they can apply for loans from banks on your behalf, register phone numbers (to run shady business) on your behalf, essentially enjoy all the benefits of being you without any of the drawbacks.

When I first read about the breach, I thought, “it’s been 3 years now, as long as you’ve moved and changed phone number, they pretty much “only” have your IC no. It’s bad, but maybe not as severe as I initially might imagined”. Could I have been any more wrong. If you are a registered voter and your IC no is fallen into the wrong hands, whoever this is can easily pull up your full profile (full name, home address) from the SPR website. And while doing so, every single device that this set of data goes through (router, ISP, some surveillance services, or even a MIM attack) gets to see all of these information as well since the site is not encrypted. (It’s 2017 and TLS is free!)

Responses #

While I don’t think Keith’s approach is perfect (minor things like captcha, API dealing with non-existent query etc), I still think that is the closest solution available which is both informative (educational) and maintain minimal harm to the victims.

With MCMC shutting it down, I really hope that they are actually aware of the risk and provide a much better solution (considering that’s what their job is). To all the people out there calling this as a potential phishing, the website ask for numbers where you may or may not put in an actual existing NRIC and does not ask for anything else. If this person were to phish for NRIC, he might as well brute force the SPR website which comes with 100% accuracy and all your personal information.

And then there is Lowyat.net, no, the data is not in any way manipulated as it was simply masked for the safety of the victims because as you said it yourself, “the sheer amount of information on the site could subject it to abuse”. I think you mean raw data not the masked data. Now that’s just a cheap shot at the choice of words, I read through some of the threads in the Lowyat forum and it seems like they are working closely with MCMC to ‘fix’ this. Hopefully, we will see something soon. As for now, I still stand by Keith on the ‘rights to know’ for the victims.

Technical considerations #

Feel free to skip this if you are not into the technical exploitation

That said, it has come to my attention* that the API allows easy data scraping. Remember I mentioned an exploit in the beginning? This is it. By knowing the birthday and birth state of an individual, it is possible to do a targeted background search through scraping with the API. While this approach might not be efficient if your intention was to do a mass scrape of all the data as mentioned by Keith, the limitation in place is more than enough for such targeted digging. I do not want to go into details but if you string all these information I mentioned about throughout the article together, you’ll easily figure out what I am talking about.

Admittedly, this is less of a threat considering that if you are able to circumvent the captcha used at the SPR site with some sort of image recognition, its easily way worse than any kind of vulnerabilities Keith’s site can be.

*I didn’t figure this out, credits to this random person who wanted to stay anonymous

After thoughts #

Honestly, I’m widely upset by the of public’s reaction or lack thereof. Anyone who got their hands onto such information can just show up and ‘become you’ figuratively tomorrow but you couldn’t care less? I guess I’m going a little too far with my fear-mongering approach but I really think the public awareness of such danger is at little to none.

Much like the whole Equifax fiasco in the US, I sincerely think this is an important educational opportunity for both the public and the government officials to understand the importance of cybersecurity. I’m in no way an expert (not even a beginner so to say) in this realm but the ignorance over such topic is hurting the well-being of not just any individual but the trust in the country. With no punishment in place for organizations mishandling such damaging information (that allows impersonation), how am I ever going to trust anyone, any company, with any of my personal details anymore?

This article was originally published on my personal Medium publication.

Use Terminal in Windows with Style

binhong@binhong.me (BinHong Lee) — Tue, 15 Aug 2017 00:00:00 -0800

When the Linux Shell in Windows was first released, I used it and didn’t quite like it. Fast forward a year later, I heard it is a lot better now (also the fact that my familiarity with terminal commands improved helped) so I decided to give it another try. It is a really fun experience so far. In case you haven’t install the Windows Subsystem for Linux (WSL), follow this step-by-step guide by Microsoft to get started. Then, install Hyper for Windows from here. If you didn’t already, you will also need node.js for the package management for plugins of Hyper.

Opening the config file #

At the Hyper window, press Ctrl + ,. It should open the config file “.hyper.js” with your default text editor. In case it throws an error, it is probably because Windows is trying to execute the js file instead of opening it for editing. You have 2 ways to work around this.

(Not recommended) If you have an editor that can be opened through Powershell, then you can go ahead and do atom .hyper.js (or whatever text editor yours is).
(Recommended) Navigate to the user folder with your File Explorer (“C:\Users%YourUserName%\”). Right click “.hyper.js”, select “Properties”, change the “Opens with: ” to a text editor like ‘Notepad’ etc.

After this, you should be able to pull up the config file easily with either pressing Ctrl + , or through the Hamburger Menu into ‘Edit’ → ‘Preferences’.

Pointing Hyper to the Linux Shell #

From the config file, look for the shell: '', option as seen from the screenshot above, change it to shell: 'C:\\Windows\\System32\\bash.exe\', as suggested by the comment. In my personal experience so far, neither keeping nor removing ‘shellArgs’ make any changes to the bash shell. While you are at it, feel free to add any kind of plugins you want to it. Personally, I added 'hyperterm-cobalt2-theme' because I like the theme.

You should now be able to use Hyper just like a normal Linux Shell.

General development environment and tools #

Since you will most likely be using this for development, you should definitely run the following commands to set yourself up.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential
sudo apt-get install python-pip
pip install --upgrade pip

Among all of the above, installation of python-pip is crucial to the next step in installing Powerline for Hyper (or more like bash).

Installing Powerline #

This section is referenced from SETTING UP POWERLINE SHELL ON WINDOWS SUBSYSTEM FOR LINUX #

Run the following to install Powerline.

cd ~
git clone https://github.com/banga/powerline-shell.git
cd powerline-shell
cat config.py.dist > config.py
./install.py

WARNING: you should not at anytime remove powerline-shell folder (or attempt to) without reverting the following changes that you will make to the .bashrc.

1
2

cd ~
nano .bashrc

Feel free to use any other command line editor if you prefer (I use vim personally). Add the following lines to the file.

function _update_ps1() {
    PS1="$(~/powerline-shell/powerline-shell.py $? 2> /dev/null)"
}

if [ "$TERM" != "linux" ]; then
    PROMPT_COMMAND="_update_ps1; $PROMPT_COMMAND"
fi

*Note: If you happen use nano (like the given example), do note that Ctrl+X will not exit the program. You can do F2 instead.

Save and restart Hyper (or refresh) and it should look something like this. Realize the missing characters which are supposed to be the arrows.

Fonts for Powerline #

For the character to show, you will need Powerline fonts installed and set it in your config file Ctrl + , for Hyper.

Go to https://github.com/powerline/fonts, pick a font you like (I use monofur) or clone the repo and install all the fonts. Remember to install it as Windows, meaning you will be running the ps1 file (PowerShell) instead of the sh file. If it throws an error saying something along the lines of “running scripts is disabled”, follow the instructions here to enable it and run it again.

When you are done with the installation, remember to change the font through the config file, then reload Hyper. At this point, it should look something like the following screenshot.

(Optional) Installing npm #

This section is referenced from Installing nvm, nodejs and npm on Ubuntu by Randika Rathugamage #

The obvious way to do this is by installing it through the default package manager. However, the npm installed through apt-get is 3.5.2 version without node. I personally prefer installing it with nvm starting by cloning the nvm repository and adding it to .bashrc.

1
2
3

cd ~
git clone https://github.com/creationix/nvm.git
echo "source ~/nvm/nvm.sh" >> .bashrc

From here, you can either reset Hyper or do a source .bashrc to get nvm working for you. After that, you can do nvm ls-remote to look at all the versions available before deciding which to install or you can just do nvm install node and call it a day.

(Optional) Configuring aliases #

It is important to note that the Linux file system is separated from the Windows file system (located in C:/User/your-name/AppData/Local/lxss). Windows went as far as to not show the folder in File Explorer even if you select to show hidden files just so you don’t mess with it. In one scenario where I made a change to one of the files in there from File Explorer through my text editor in Windows, the WSL no longer recognizes the existence of such file in the directory immediately.* This is why you should keep all your files in your Windows side of file system and access them through something like cd /mnt/e/git/. Retyping the file path over and over again can be annoying so instead, you can set an alias by adding the command below but changing the location to the directory where you store your development files.

*Workaround: I ended up creating a new file with vim in bash and copy pasting the entire existing file from Windows. I also renamed the files beforehand to prevent file conflict.

`1`	`echo 'alias gitfiles="cd /mnt/e/git/"' >> ~/.bashrc`

From here on out, you can just do gitfiles in Hyper (or bash) and you will go directly to your development folder.

Reset #

This section is referenced from How to remove/reset Windows Subsystem for Linux on Windows Insider Build 14316 #

If at any point, you seems to have screwed up your Linux subsystem, you can always reset it by going to your command prompt cmd.exe with admin privilege (Run as administrator) and input the following commands to reset the Linux subsystem.

1
2

lxrun /uninstall /full
lxrun /install

Afterthoughts #

Initially, I did this with zsh (since I personally uses Powerlevel9k on my Mac and I love it) but zsh doesn’t seems to play well with Hyper (specifically with the plugins) causing a lot of weird configuration behaviors. Even without Hyper, the alignments for Powerlevel9k specifically goes way off causing the input line to be on the next line instead. I speculate this is due to zsh not being built with Windows in mind and does not have such compatibility.

This article was originally published on my personal Medium publication.