The two-step approach to big code modifications

We all have to make significant code changes from time to time. Most of these code changes are large. Consider the scenario that you merged one such significant change, and then other team members made a few more changes on top. Then a major bug is detected. You desperately make the fix. It makes it in. You declare a victory, and a few hours later, your colleague notices another bug/crash/performance regression. Your commit cannot be reverted. It isn’t just about you. Many others have built on top of the change you made—the code sloths along in this broken state for a few days before you eventually fix it. Everyone has faced this issue at some point or the other.

If the code change is small, this is a non-issue, you can revert it and fix it at your own pace. Therefore, when making significant code changes, always try to do it in two different commits (pull requests). In the commit, you add the new code, add a switch (command-line flag or a constant) to turn it on, and keep that switch off by default. In the second commit, turn the switch on. Now, if there is a problem, you immediately turn the switch off and start working on a fix. No one else will deal with the broken code. It would be even better if the switch is a command-line flag since you can turn on the flag for 10% of the machines (or users) and see the behavior for a few days before rolling it out to 100%. It big teams, it is usually good to add a comment to switch mentioning when it should expire or else you will end with Uber scale problem.

There are a few cases where this cannot be done, for example, during big code refactoring. I think big code refactoring touching code written by multiple teams is almost rarely justified in a single commit.

Examples of some cases where this is useful

  1. Switching over from consuming data via v1 of some API to v2
  2. Switching over from returning computed results to stored results (for faster response time but possibly inaccurate outcome)
  3. Switching over from JSON to Protocol Buffer/Thrift
  4. Switching over from VMs to Kubernetes (don’t delete the old code yet, you might regret it)

Programmable Money and value capture

Money serves three purpose – unit of accounting, a medium of exchange, and a store of value. Cryptocurrencies have been compared to Programmable Money. Anything programmable requires an experimentation platform for iterations and improvement. Bitcoin seems to have won the “store of value” battle. Ethereum has the developer mindshare and is the preferred experimentation platform. Multiple cryptocurrencies are still fighting the battle to be the medium of exchange.

BTC dominance chart from CoinMarketCap

BTC dominance chart from CoinMarketCap

The amusing part is that every cryptocurrency startup envies Ethereum’s developer ecosystem and is trying to attract developers. But there isn’t any real value capture being the experimentation platform. A successful product has a high chance of leaving Ethereum and migrating users to its chain. The real battle, I believe, remains in becoming the medium of exchange, being the programmable Visa & Mastercard equivalent.

Closeum – closed-source in disguise

Earlier, the Software world was rigidly divided between closed and open-source software. Microsoft Windows is closed-source, GNU/Linux is open-source. Microsoft Office and Lotus Notes are closed-source, LibreOffice is open-source. Turbo C++ is closed-source, and GCC is open-source.

But now, a new class of software products has emerged whose core is open-source, but still, the open-source software is of limited use. One model is to offer some critical and useful functionality in a closed-source layer via a managed service in AWS/GCP/Azure, for example, Redis is open-source, but useful modules on top of it are not. Another model is to use licensing gimmickry, for example, MongoDB is licensed under SSPL which requires that if anyone offers MongoDB as a service, then the source code of the full service must be published under this license. The third approach is to make the core software open-source but make it dependent on closed-source cloud services. For example, the node package manager (npm) is open-source, but a closed source company owns the default npm registry. Android is open-source, but most day-to-day application ranging from Google Maps to Google Music are closed-source. Now onwards, rather than calling such software open-source, we should call them closeum.

Startup founders: How not to write an email

Consider this email,

And now consider this one,

Hi Ashish,

You signed up for the Orchard beta not too long ago, and we’re excited to finally send you an invite!

(Just to jog your memory, Orchard helps you make the most of your relationships, keeping you up to date on where you’re spending your time and who you need to catch up with. It’s somewhere between a personal CRM and a todo list for your connections.) [Emphasis mine]

You should receive an invite from Apple’s Testflight service in the next few minutes. It will contain a code or link you need to install the app. You’ll also need to have the TestFlight app installed on your phone. If you need us to send the invite to a different email address, just reply to this message and let us know.

Once you’ve checked out the app, please send us your feedback — this is a beta and we need your help to improve! Be honest, frank, and opinionated: you won’t hurt our feelings. Bugs, ideas, concerns, etc all fair game. You can reply to me or send to team@orchard.ai.

Thank you!

Brian and the entire Orchard team

brian@orchard.ai

Which one do you think your early adopters have a higher likelihood of understanding and responding to?
Don’t forget while you might live and breathe your startup, your early adopters have probably signed up to try several such services, and unless you remind them what your product is about, they might as well delete the email and move on.

The “key” problem in cryptocurrency

All cryptocurrencies are eventually tied to a “private” key. You lose this key, and the funds are gone, forever. Millions worth of bitcoins have disappeared from the circulation due to lost keys. You can memorize the key by mapping it into passphrase consisting of memorizable words but if you forget that, like many others, the coins are unrecoverable. An alternative is to trust a centralized service like Coinbase, but then all the benefits of investing in a decentralized currency are gone. Lastly, one can use a hardware wallet, but again, if you lose the wallet, the key is lost. If you keep the key on your device, then a malware might target and try to steal it someday. Thus, even if you are bullish on cryptocurrencies, there are no good decentralized ways of holding a significant chunk of your net worth in cryptocurrencies.

The problems are compounded further by the fact that to a great extent, the key access is all or none. If you lose your key, you lose all the funds. If I got hold of your key, I could transfer 100 % of your funds, irrevocably. Compare this to your ATM debit card, if I get hold of it, along with your pin, there is only so much damage I can do before the ATM limit hits. Further, your credit card used on a malicious website can hit you with a limited amount of fraud before the fraud warnings block the card to prevent further loss of funds.

An ideal approach, hopefully, will consist of decentralized smart contracts comprised of multiple keys tied to the same passphrase with a different set of transfer limits. So that, if one forgets one word in the passphrase and cannot access 100% of the funds now but can access at most 1% funds every day and slowly drain out their account to a newer account. Some of these unlocking codes will be held at different places, some centralized services for custody and ease of use; some kept on the personal device for ease of use where even if they get stolen the damage is contained. A further addition could be a beneficiary account where the funds would be transferred to in case the account lays dormant for a certain period.

Server vs mobile development: Where the code runs matter

When the code runs on your servers, you have much more control over the “context” in which it runs. On the mobile devices, the device OS and the user control the context. This difference leads to some subtle implications.

One significant set of differences comes from the lack of control of the platform. For server-side code, one can choose from a wide array of languages. For the mobile code, however, the best choice would almost always be the one dictated by the platform – Java/Kotlin on Android and  Objective-C/Swift on iOS. Further, for the server-side where one can stick to a particular version of the language. In the case of mobile, the platform controls the language version. Same goes regarding the hardware choices – one can choose to use different types of server machines specialized in handling those jobs, eg. GPUs for math-intensive computes. While for the mobile-code, one had to write a good enough fallback to support a wide-enough set the devices. Similarly, the server-side has to rarely worry about the server killing a running process while it is normal for mobile OSes to kill backgrounded processes eventually.

The other set of differences comes from the temporary changes to the platform introduced by the carrier and the user. A network request running on mobile has to be robust enough to deal with intermittent broken connectivity to outright unavailability of a data connection, e.g., due to the user switching to the airplane mode. On mobile, network type matters as well. A network request on cellular would usually cost more than a network request on Wi-Fi to the user and a network request on roaming even more. On mobile, the code has to be aware of not doing unnecessary work when the user is low on battery. Server-side code is rarely subjected to such constraints.

Lastly, it is possible to parallelize and speed up the server-side code by adding more resources like RAM or better CPUs. If a certain number of servers are not enough, you can add more. There isn’t usually a way to offload compute-intensive or memory-intensive work off of the mobile devices without trading it off for network latency and sometimes, user’s privacy as well. While it might be preferable to go with a multi-processing approach in the server code to avoid concurrency issues, on the mobile, however, multi-threading being more straightforward and less resource-intensive is almost always the choice.

Apple vs Google: Naming of flagship Android vs iPhone

iPhone

  1. iPhone
  2. iPhone 3G -> iPhone 3GS
  3. iPhone 4 -> iPhone 4S
  4. iPhone 5 -> iPhone 5S
  5. iPhone 6 -> iPhone 6S (and plus sizes)
  6. iPhone 7 (and plus sizes)

Android

  1. Nexus One
  2. Nexus S
  3. Galaxy Nexus
  4. Nexus 4
  5. Nexus 5
  6. Nexus 6
  7. Nexus 5X & Nexus 6P
  8. Pixel & Pixel XL

While iPhone is recognized as a global name while erstwhile Nexus and now, Pixel has almost no branding outside of the Android fanboys.

Further on Google’s naming snafu:

  1. Nexus 7, Nexus 10, and Nexus 9 are tablets. 7 & 10 were launched with 4 and 9 was launched later.
  2. Pixel brand was originally used for Chromebook Pixel.
  3. Nexus 5X and Nexus 6P were new versions of Nexus 5 and Nexus 6, respectively. It seems like the two teams couldn’t agree on a single suffix letter.

The Internet and the City Advantage

Cities have been central to the human civilization. Their dense population provides a platform for the serendipitous interactions and cross-pollination of ideas from different domains, their abandoned portions provide cost-effective real estate to struggling artists and entrepreneurs, their riches provides jobs, sometimes, side-jobs for the innovators to experiment. No wonder innovation in a city grows super-linearly (~(size)4/3) with its size.

But the Internet was supposed to destroy all advantages a city has over the rural areas. The Internet was supposed to convert all of us into a global community. It did. But the cities have emerged even stronger, everywhere. One way to rationalize this is to realize that the Internet provided a platform to the cities a rent seeking ability which was extremely limited in the pre-Internet era. When a family in Kansas books an Airbnb in Thailand using Mastercard credit card, books the flight with Expedia, and uses an Uber in Thailand then these intermediaries take a cut. Similarly, when a person in Nebraska buys West Texas Intermediate from New York stock exchange, then New York stock exchange takes a cut. Now, what happens when these companies take these cuts? A big chunk of that is used for salaries, expenses,  or charity donations which usually are highly localized activities and benefit the cities where these companies are headquartered. Most of these rent-seeking activities were either non-existent or of limited leverage in the pre-Internet era.

Note:

  1. The first paragraph of this blog post is heavily inspired by “Where good ideas come from” book.
  2. The city term I am using is akin to an urban area. In the US, a suburb is seen as a separate entity, which I am treating as a city in this parlance.

Consumer Internet: why audio can’t be as big as text, photos, and videos

Bandwidth of different senses

The bandwidth of different senses

 

Our brain loves distractions, and multi-tasking gets bored quickly. When we read text or watch a photo, it engages us visually, a video (with audio) engages us even more. The bandwidth of eyes is much larger than the bandwidth of our ears. When we are watching something, it utilizes more bandwidth and hence occupies more of our attention span. Also, given the way our eyes work, we can focus more on the exciting aspect of the visual feed. Compared to that, audio underutilizes our brain’s bandwidth. Further, the unidimensional flow of audio data at a linear speed does not mimic our ability to process it. Contrast forced direct listening with how non-linearly humans read.

How eyes jump forward and backward while reading

How eyes jump forward and backward while reading

And that’s why video games are even more engaging than video. They utilize the bandwidth even further by forcing us to think and act in the game.

Since audio underutilizes our brain’s bandwidth, it leaves spare bandwidth for distractions, including eating food, driving, and exercising. No wonder most audio consumption is passive and happens as a secondary activity as opposed to being a mainstream activity like reading or watching movies.

90% vs 99%

Consider two systems, the first one is 90% reliable and the second one is 99%. The wrong to compare them is to compare the reliability and conclude that the second one is 9% (or 10% if you take 90% as the base) better. The right way to compare them is to compare the unreliability and conclude that the first system fails in 10% of the cases while the second one fails only in 1% and hence, is 10X more error-prone than the second. The reliability comparison is a vanity matrix while the unreliability comparison not only demonstrates the user perception (“The user saw then crashes in past one hour” vs “The user saw one crash in past one hour”) drastically but also shows the effort that goes into making the system more reliable.

Note: While I am taking reliability as an example of the metrics, the argument can be made for similar metrics.