How many source-code repositories should a startup have

Recently, this question came up during the discussion. “How many source-code repositories should a startup have?”

There are two extreme answers, a single monorepo for all the code or repository for each library/microservice. Uber, for example, had 8000 git repositories with only 200 engineers!

I think both extremes are wrong. Too many repositories make it hard to find code and one single repository makes it harder to do simple things like testing, bisecting (to find buggy commit), deciding repository owners.

Here’s what I have seen works best.

  1. Backend code – ideally, one single repository.
  2. Frontend (web) code – one single repository. This can be separately tested and even outside contractors can access this. The web code is anyways shipped to the client, so, leaking it isn’t that big of a worry.
  3.  Mobile code – one repository per platform (Android, iOS, etc.) if there is no code dependency. Or one single repository if a common codebase like React Native or Flutter is being used. Again, those who access the mobile code won’t need access to the backend code. And given that this code in compiled form is sent to the users, there is less worry around leaking it.
  4. Open-source code – One repository per open-source package.

As opposed to a monorepo setup, this setup makes it harder to make simultaneous changes to backend and frontend, or backend and mobile apps. And that should be the right behavior anyways. Since backend services and frontend can be, or eventually will be, deployed independently of each other. Mobile apps are deployed independently anyways, so, the backend has to provide backward-incompatibility for that.

The two-step approach to big code modifications

We all have to make significant code changes from time to time. Most of these code changes are large. Consider the scenario that you merged one such significant change, and then other team members made a few more changes on top. Then a major bug is detected. You desperately make the fix. It makes it in. You declare a victory, and a few hours later, your colleague notices another bug/crash/performance regression. Your commit cannot be reverted. It isn’t just about you. Many others have built on top of the change you made—the code sloths along in this broken state for a few days before you eventually fix it. Everyone has faced this issue at some point or the other.

If the code change is small, this is a non-issue, you can revert it and fix it at your own pace. Therefore, when making significant code changes, always try to do it in two different commits (pull requests). In the commit, you add the new code, add a switch (command-line flag or a constant) to turn it on, and keep that switch off by default. In the second commit, turn the switch on. Now, if there is a problem, you immediately turn the switch off and start working on a fix. No one else will deal with the broken code. It would be even better if the switch is a command-line flag since you can turn on the flag for 10% of the machines (or users) and see the behavior for a few days before rolling it out to 100%. It big teams, it is usually good to add a comment to switch mentioning when it should expire or else you will end with Uber scale problem.

There are a few cases where this cannot be done, for example, during big code refactoring. I think big code refactoring touching code written by multiple teams is almost rarely justified in a single commit.

Examples of some cases where this is useful

  1. Switching over from consuming data via v1 of some API to v2
  2. Switching over from returning computed results to stored results (for faster response time but possibly inaccurate outcome)
  3. Switching over from JSON to Protocol Buffer/Thrift
  4. Switching over from VMs to Kubernetes (don’t delete the old code yet, you might regret it)

Incremental testing: save time and money on CI for monorepo

To use monorepo or not is an eternal debate. Each has its pros and cons. Let’s say you decide to go with monorepo, one major issue you will face over time is slow testing. Imagine a monorepo, consisting of an Android app, an iOS app, some backend code, some web frontend code. In only very few occasions will someone modify more than one of those simultaneously.

Further, add to the fact that most of these projects confined to their directories would be using different build systems as well, for example, gradle for Android, yarn/npm for Javascript, go/rust/java/npm for the backend. The total build time, as well as test time, will only grow over time. It annoys developers making small modifications to their part of the codebase. And it slows down the development velocity drastically.

Read More

How to deploy side projects as web services for free

In 2020, the web is still the most accessible permission-less platform. For the past few months, I have been playing and building side-projects to simplify my life. I started with a Calendar Bot for scheduling events, DeckSaver for downloading decks from Docsend, AutoSnoozer for email management, and StayInTouch for maintaining follow-ups.

When I started on this journey, I had the following in my mind.

  1. Cost of domain ~ 12$ a year or 1$ a month
  2. Cost of a VM ~ 10$ a month

Read More

Docker 101: A basic web-server displaying hello world

A basic webserver

Docker containers are small OS images in themselves which one can deploy and run without worrying about dependencies or interoperability. All the dependencies are packed in the same container file. And the docker runtime takes care of the interoperability. You are not tied to using a single language or framework. You can write code in Python, Go, Java, Node.js, or any of your favorite languages and pack it in a container.

Consider a simple example of a Go-based webserver

Read More

The first two statements of your BASH script should be…

The first statement is a Mac, GNU/Linux, and BSD portable way of finding the location of the bash interpreter. The second statement combines

    1. “set -e” which ensures that your script stops on first command failure. By default, when a command fails, BASH executes the next command. Looking at the logs, you might feel that the script executed successfully while some commands might have failed. Caveat: Be careful about applying it to existing scripts.
    2. “set -u” which ensures that your script exits on the first unset variable encountered. Otherwise, bash replaces the unset variables with empty default values.
    3. “set -o pipefail” which ensures that if any command in a set of piped commands failed, the overall exit status is the status of the failed command. Otherwise, the exit status is the status of the last command.

References:

  1. Unofficial Bash strict mode
  2. ExplainShell

Keep your dotfiles bug-free with Continuous Integration

Update: As of April 2020, I have switched over to GitHub Actions. Travis CI has become buggy and flaky over time and I got tired of trying to keep the builds green. My GitHub action scripts can be seen here.

Just like many software engineers, I maintain my config files for GNU/Linux and Mac OS in a git repository. Given that, I wrote a fair bit of them in interpreted code, notably, Bash, it is a bit hard to ensure that it is bug-free. The other problem I face is that packages on homebrew, the Mac OS package manager becomes obsolete and gets deleted from time to time.

I added CI testing on Travis CI to prevent these breakages and to ensure that my dotfiles are always in good shape for installation. The great thing about Travis CI is that it is entirely free for open-source repositories even for testing on Mac OS containers.

Read More

Stanford CS251: Cryptocurrencies, blockchains, and smart contracts

Lectures

  1. Introduction
  2. Creating a Digital currency
  3. Bitcoin Overview
  4. Bitcoin Blockchain
  5. Bitcoin Mining
  6. Bitcoin Miner interactions and Game Theory
  7. Cryptocurrencies: Community, Economics, and Politics
  8. Alternative Consensus
  9. Wallet & Anonymity
  10. Anonymity on Blockchain
  11. Altcoins
  12. Ethereum
  13. Ethereum
  14. Ethereum Governance
  15. Bitcoin Side-chains (guest talk)
  16. Bitcoin Payment channel
  17. Guest talk on Legal by Ben Lawsky  – does not seem worthy of transcribing
  18. Advanced Topics – Quantum Computing, Threshold Signatures, and storing secret state on public chains
  19. Advanced Topics – Smart property, publicly verifiable randomness, and prediction markets
  20. Guest talk by Adam Ludwin (CEO, chain.com) – does not seem worthy of transcribing

The notes are based on the 2016 version of the course CS251

Android: Fragment related pitfalls and how to avoid them

  1. Don’t use platform fragments (android.app.Fragment), they have been deprecated and can trigger version-specific bugs. Use the support library fragments (android.support.v4.app.Fragment) instead.
  2. A Fragment is created explicitly via your code or recreated implicitly by the FragmentManager. The FragmentManager can only recreate a Fragment if it’s a public non-anonymous class. To test for this, rotate your screen while the Fragment is visible.
  3. FragmentTransaction#commit can fail if the activity has been destroyed.
    “java.lang.IllegalStateException: Activity has been destroyed”
    Why – This can happen in the wild where say right before FragmentTransaction#commit() executes, the user gets a phone call and your activity is backgrounded and destroyed.
    How to trigger manually – The easy way to manually test this is to add a call to Activity#finish() right before FragmentTransaction#commit.
    Fix – Before doing FragmentTransaction#commit(), check that the activity has not been destroyed –Activity#isDestroyed() should return false.
  4. FragmentTransaction#commit can fail if onSaveInstanceState has been called.
    “java.lang.IllegalStateException: Can not perform this action after onSaveInstanceState”
    Why – This can happen in the wild where say right before FragmentTransaction#commit() executes, the user gets a phone call and your activity is backgrounded and paused.
    How to trigger manually – The easiest way to manually trigger this behavior is to call Activity#onSaveInstanceState in your uncommitted code right before the call to FragmentTransaction#commit
    Fix 1 – call FragmentTransaction#commitAllowingStateLoss but that implies that your fragment would be in a different state then the user expects it to be.
    Fix 2 – The better way is to ensure that the code path L which leads to FragmentTransaction#commit is not invoked once Activity’s onSaveInstanceState has been called but that’s not always easy to do.
  5. FragmentManager is null after Activity is destroyed.
    “java.lang.NullPointerException: … at getSupportFragmentManager().beginTransaction()”
    Why – This can happen when the activity has been destroyed before getSupportFragmentManager() is invoked. The common cause of this is when a new fragment has to be added in response to a user action and the user immediately backgrounds the app, again, say due to a phone call, after clicking the button before getSupportFragmentManager() is invoked. Another common case is where an AsyncTask which will call getSupportFragmentManager() in onPostExecute and while the task is engaged in the background processing (doInBackground), the activity is destroyed.
    How to trigger manually – call Activity#finish() before getSupportFragmentManager().beginTransaction()
    Fix – If getSupportFragmentManager() is being invoked in the Activity, check if it’s null. If it is being invoked inside a Fragment check if isAdded() of the Fragment returns true before calling this.
  6. Avoid UI modifications which are not related to a FragmentTransaction with FragmentTransaction committed using commitAllowStateLoss
    Why – Any UI modifications like modifications of the text in a TextView are synchronous while the execution of a FragmentTransaction via FragmentTransaction#commitAllowStateLoss() is asynchronous. If the activity’s onSaveInstanceState is invoked after the UI changes have been made but before commitAllowStateLoss is called then the user can end up seeing a UI state which you never expected them to see.
    Fix – use commitNow() or hook into FragmentManager.FragmentLifecycleCallbacks#onFragmentAttached(). I will admit this I haven’t found a simpler fix for this. And this issue is definitely an edge case.
  7. Saving Fragment State
    As mentioned earlier, a Fragment is re-created on activity recreation by FragmentManager which will invoke it’s default no-parameter constructor. If you have no such constructor then on Fragment recreation, the app will crash with “java.lang.InstantiationException: MyFragment has no zero argument constructor”. If you try to fix this by adding a no argument constructor then the app will not crash but on activity recreation say due to screen rotation, the Fragment will lose its state. The right way to serialize a Fragment’s state is to pass arguments in a Bundle via setArguments.

    The Fragment code should then use getArguments() method to fetch the arguments. In fact, I would recommend a Builder pattern to hide all this complexity.

    Consider this complete example,

  8. Inside your Fragment code, if you want to decide whether it is safe to execute a UI code or not, rely on isAdded(), if it returns true, it is safe to perform UI modifications, if it returns false, then your Fragment has been detached from the activity either because it has been removed or because the host (Fragment/Activity) is being destroyed.
  9. Callbacks
    To callback into the parent activity/fragment in case of action inside your Fragment, say, a user click, provide an interface (say, MyFragmentListener) which the holding activity/Fragment should implement. In Fragment#onCreateView() get the host via getHost(), cast it to MyFragmentListener, and store it in the instance variable of your Fragment class. Set that instance variable to null in Fragment#onDestroyView(). Now, you can invoke callbacks on this MyFragmentListener instance variable.
  10. Backstack
    Backstack is nuanced and my grasp of it is still limited. What I do understand is that if you want your Fragment to react to the back key press then you should call FragmentTransaction#addToBackStack(backStackStateName) while adding the Fragment via FragmentTransaction and remove it while removing it. Removal from the back stack is a bit more nuanced. Note that, manual removal of a fragment from the back stack is not required in Activity#onBackPressed() as long as your Activity inherits from FragmentActivity.

 

Subscribe For Latest Updates

Signup for our newsletter and get member-only articles