To use monorepo or not is an eternal debate. Each has its pros and cons. Let’s say you decide to go with monorepo, one major issue you will face over time is slow testing. Imagine a monorepo, consisting of an Android app, an iOS app, some backend code, some web frontend code. On only very few occasions will someone modify more than one of those simultaneously.

Further, most of these projects confined to their directories would be using different build systems as well, for example, gradle for Android, yarn/npm for Javascript, go/rust/java/npm for the backend. The total build time and test time will only grow over time. It annoys developers making small modifications to their part of the codebase. And it slows down the development velocity drastically.

At my previous job, I ended up writing an elementary version of the incremental testing system. We had a set of directories, most unrelated to each and some dependent on each other. As long as we come up with a good directory-dependency tree, a script can check if any files in the pull request modified any of those dependencies; if yes, the test continues, if not, it is skipped. In this way, we cut down a lot of extraneous builds. The goal was not to eliminate all extraneous builds but to minimize the biggest offenders. And that alone made the testing much more manageable. We saved money by reducing the number of Circle CI instances we reserved, and we increased our developer velocity. In a monorepo like ours, you can expect a 5-10x faster testing time.

The code is open-source, so I can easily reference it here. We used Circle CI, but the approach is generic enough and can be used on any other build system as well.

Consider, for example, protocol-test

Bash
1
2
3
4

# Dependencies
FILES_TO_CHECK="${PWD}/packages/protocol,${PWD}/packages/utils,${PWD}/.circleci/config.yml"
# Incremental testing script
./scripts/ci_check_if_test_should_run_v2.sh ${FILES_TO_CHECK}

The wrapper script

Bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Usage: ci_check_if_test_should_run_v2.sh <comma separate list of paths to check>
DIRS_TO_CHECK=${1}
CHANGED=$(node -r ts-node/register ../../scripts/check_if_test_should_run_v2.ts --dirs ${DIRS_TO_CHECK})
cd -
if [ $CHANGED = false ] ; then
  echo "No changes in ${1} - skipping  testing"
  # https://discuss.circleci.com/t/ability-to-return-successfully-from-a-job-before-completing-all-the-next-steps/12969/6
  circleci step halt
  exit 0
fi
echo "Something $CHANGED, tests should not be skipped"

And the full script which finds out the patch of this pull request over the base branch can be seen here. The most crucial method being,

Typescript
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

async function checkIfTestShouldRun() {
  const currentBranch: string = await getCurrentBranch()
  const isCriticalBranch: boolean =
    isStagingBranch(currentBranch) ||
    isProductionBranch(currentBranch) ||
    isMasterBranch(currentBranch)
  if (isCriticalBranch) {
    logMessage('We are on staging or production branch')
    console.info('true')
    return
  }

  const branchCommits: string[] = await getBranchCommits()
  if (branchCommits.length === 0) {
    logMessage('No commits found; this is most likely a bug in the checking script')
    process.exit(1)
  }
  for (const commit of branchCommits) {
    logMessage(`\nChecking commit ${commit}...`)
    const paths: string[] = dirs.concat(['../../yarn.lock'])
    const anyPathsChanged: boolean = await checkIfAnyPathsChangedInCommit(commit, paths)
    if (anyPathsChanged) {
      console.info('true')
      return
    }
  }
  console.info('false')
}