{
  "commands": [
    {
      "commandline": "yarn test --json  "
    }
  ],
  "preSyncCommands": [],
  "buildCommands": [
    {
      "commandline": "nvm install 14.18.0"
    },
    {
      "commandline": "nvm use 14.18.0"

    },
    {
    "commandline": "nvm alias default 14.18.0" 
    },
    {
      "commandline": "yarn "
    }
  ],
  "remoteDirectory": "/tmp/remote_dir/",

  "environment": {},      
  "excludedFromSync": ["log/", ".git/", "node_modules"],
  "excludedFromWatch": ["log/", ".git/","log", ".git", "node_modules"],
  "projectToken": "********",
  "framework": "Jest"
}

Rails Rspec Dev.to Codebase

The Dev.to website is released as an open source project called forem. We are using it here as an example Rails with Rspec project.

It is a real world project that is actively maintained and has a large community of contributors. It is a great example of a large Rails project that has grown over the years. Like most real world Rails apps functionality has increased and so has test size and speed as development has progressed. The test suite is extensive and includes browser tests using Capybara and Selenium.

Running the entire test suite locally takes a long time. I stopped at 60 minutes because my laptop was starting to overheat and was becoming unresponsive. This is pretty common for large Rails projects. Developers often cannot run their tests locally anymore - which is the problem Brisk is here to fix.

In the recording we run the entire test suite in 1 mintue 30 seconds. This includes all of the Rspec tests including browser tests. This is at least a 40X speedup, from over 60 minutes to 1 minute 30 seconds and makes it feasible for developers to run their tests pre commit again.

With Brisk CI, it means that test run times are no longer a bottleneck for CI. Merging and deploying code is faster and more reliable.

Rails Rspec Making Things Even Faster

The big drag on test speed in this test suite is the browser tests. We spin up a chrome instance with Selenium to test actual interactions with the browser. Unfortnately this can be quite flakey and so it is common to use rspec retry to retry the tests if they hit timeouts. This doesn't solve the actual flakiness but does retry it. With such a large number of tests we do hit some retries which result in slowing down the entire test suite (because we are waiting on the slowest test which may have been retried a number of times). I would estimate that if some attention was paid to the flakiness of the browser tests we could get the test suite down to under 1 minute perhaps less than 40 seconds. If we also split out the longer tests into multiple shorter tests with fewer test cases per file we could speed things up even further.

⚠️ Some of the following demos are outdated and show older versions of brisk but I'll leave them here until I update them ⚠️

Rails Minitest Activerecord Test Suite

In this demo we use the Rails codebase (cloned from https://github.com/rails/rails) as our source folder. We are running a subset of the tests, the ones from the activerecord directory.

Rails Minitest brisk.json

{
  "listTestCommand": "./listTests.sh",
  "commands": [
    {
      "commandline": "cd activerecord  bin/test ",
      "args": []
    }
  ],
  "preSyncCommands": [],
  "buildCommands": [
    {
      "commandline": "bundle install",
      "args": []
    }
  ],

  "environment": {},

  "excludedFromSync": ["log/", ".git/", "node_modules", ".rvm"],
  "excludedFromWatch": ["log/", ".git/", "log", ".git", "node_modules"],

  "projectToken": "********",
  "framework": "Rails",
  "image": "rails"
}

Rails Minitest Example Usage

Below is a video of the Demo showing running the tests using brisk. This is a subset of the tests, running only the activerecord tests. Running the tests on my machine locally a Macbook Pro 2020 takes over 2 minutes 25 seconds. Running in Brisk it completes in less than 19 seconds.

Rails Minitest Making Things Even Faster

Running on Brisk each test run takes less than 20 seconds. At this point we are limited by the slowest test, it's actually trivial to increase parralizability for most tests. Most people don't because there is normally no reason to, but now there is.

{
  "listTestCommand" : "/tmp/remote_dir/listTest.sh",
  "commands": [
    
    {
      "commandline": "  ./node_modules/.bin/cypress run ",
      "workDirectory": "/tmp/remote_dir/",
      "args": [],
      "background": false
    }
  ],
  "preSyncCommands": [],                         
  "buildCommands": [
    {
      "commandline": "nvm install 14.18.0",
      "workDirectory": "/tmp/remote_dir/",
      "args": []

    },
    {
      "commandline": "nvm use 14.18.0",
      "workDirectory": "/tmp/remote_dir/",
      "args": []

    },
    {
    "commandline": "nvm alias default 14.18.0" ,
    "args": [],
    "workDirectory": "/tmp/remote_dir/"
    },
    {
      "commandline": "npm ci ",
      "workDirectory": "/tmp/remote_dir/",
      "args": []
    }
    ,
    {
      "commandline": "npm run start:ci  ",
      "workDirectory": "/tmp/remote_dir/",
      "args": [],
      "background": true
    }
    
  ],

  "environment": {

  },

  "excludedFromSync": ["log/", ".git/", "node_modules"],
  "excludedFromWatch": ["log/", ".git/","log", ".git", "node_modules"],
  "projectToken": "********",
  "framework": "Cypress",
  "image": "node-lts"
}