As a CLI From Your Terminal

Brisk was originally designed - and still works - as a CLI tool. You can use it to run all of your tests from your development environment in the cloud.

The following examples show how to run tests using Brisk from your terminal. The setup will also work with your CI/CD pipeline if you want to run tests automatically.

Jest React Test Suite

A side-by-side run of Brisk versus running Jest locally.

Jest Description

In this demo we use the React codebase (cloned from https://github.com/facebook/react ) as our source folder. Without brisk it takes 3 minutes 12 seconds to run the test suite on my machine. With Brisk we can run it completely in 12 seconds. Furthermore with Brisk this 12 seconds is the time of the slowest test. If we worked on optimizing that test we should see an even faster run. Also, adding additional tests shouldn't make our test runs slower, provided we have adequeate concurrency and they themselves are under 12 seconds.

Jest is running in the top pane and the same test suite running through Brisk is on the bottom. With Brisk we can run the entire test suite before Jest has finished running a single test. I kick off a jest run manually then switch panes to brisk and hit "r" to initiate a run. All of the tests run through brisk in the time it takes for Jest to load locally.

This demo was run on my 2020 MacBook Pro with 16 GB RAM.

Jest brisk.json

{
  "commands": [
    {
      "commandline": "yarn test --json  "
    }
  ],
  "preSyncCommands": [],
  "buildCommands": [
    {
      "commandline": "nvm install 14.18.0"
    },
    {
      "commandline": "nvm use 14.18.0"

    },
    {
    "commandline": "nvm alias default 14.18.0" 
    },
    {
      "commandline": "yarn "
    }
  ],
  "remoteDirectory": "/tmp/remote_dir/",

  "environment": {},      
  "excludedFromSync": ["log/", ".git/", "node_modules"],
  "excludedFromWatch": ["log/", ".git/","log", ".git", "node_modules"],
  "projectToken": "********",
  "framework": "Jest"
}

Rails Rspec Dev.to Codebase

The Dev.to website is released as an open source project called forem. We are using it here as an example Rails with Rspec project.

It is a real world project that is actively maintained and has a large community of contributors. It is a great example of a large Rails project that has grown over the years. Like most real world Rails apps functionality has increased and so has test size and speed as development has progressed. The test suite is extensive and includes browser tests using Capybara and Selenium.

Running the entire test suite locally takes a long time. I stopped at 60 minutes because my laptop was starting to overheat and was becoming unresponsive. This is pretty common for large Rails projects. Developers often cannot run their tests locally anymore - which is the problem Brisk is here to fix.

Brisk running the entire Forem / Dev.to test suite in 1 minute 30 seconds instead of > 60 minutes.

In the recording we run the entire test suite in 1 mintue 30 seconds. This includes all of the Rspec tests including browser tests. This is at least a 40X speedup, from over 60 minutes to 1 minute 30 seconds and makes it feasible for developers to run their tests pre commit again.

With Brisk CI, it means that test run times are no longer a bottleneck for CI. Merging and deploying code is faster and more reliable.

Rails Rspec Making Things Even Faster

The big drag on test speed in this test suite is the browser tests. We spin up a chrome instance with Selenium to test actual interactions with the browser. Unfortnately this can be quite flakey and so it is common to use rspec retry to retry the tests if they hit timeouts. This doesn't solve the actual flakiness but does retry it. With such a large number of tests we do hit some retries which result in slowing down the entire test suite (because we are waiting on the slowest test which may have been retried a number of times). I would estimate that if some attention was paid to the flakiness of the browser tests we could get the test suite down to under 1 minute perhaps less than 40 seconds. If we also split out the longer tests into multiple shorter tests with fewer test cases per file we could speed things up even further.

⚠️ Some of the following demos are outdated and show older versions of brisk but I'll leave them here until I update them ⚠️

Rails Minitest Activerecord Test Suite

In this demo we use the Rails codebase (cloned from https://github.com/rails/rails) as our source folder. We are running a subset of the tests, the ones from the activerecord directory.

Rails Minitest brisk.json

{
  "listTestCommand": "./listTests.sh",
  "commands": [
    {
      "commandline": "cd activerecord  bin/test ",
      "args": []
    }
  ],
  "preSyncCommands": [],
  "buildCommands": [
    {
      "commandline": "bundle install",
      "args": []
    }
  ],

  "environment": {},

  "excludedFromSync": ["log/", ".git/", "node_modules", ".rvm"],
  "excludedFromWatch": ["log/", ".git/", "log", ".git", "node_modules"],

  "projectToken": "********",
  "framework": "Rails",
  "image": "rails"
}

Rails Minitest Example Usage

Below is a video of the Demo showing running the tests using brisk. This is a subset of the tests, running only the activerecord tests. Running the tests on my machine locally a Macbook Pro 2020 takes over 2 minutes 25 seconds. Running in Brisk it completes in less than 19 seconds.

Rails Minitest Making Things Even Faster

Running on Brisk each test run takes less than 20 seconds. At this point we are limited by the slowest test, it's actually trivial to increase parralizability for most tests. Most people don't because there is normally no reason to, but now there is.

Cypress Kitchensink codebase

In this demo we use the Cypress Kitchensink codebase (cloned from https://github.com/cypress-io/cypress-example-kitchensink) as our source folder.

Cypress brisk.json

{
  "listTestCommand" : "/tmp/remote_dir/listTest.sh",
  "commands": [
    
    {
      "commandline": "  ./node_modules/.bin/cypress run ",
      "workDirectory": "/tmp/remote_dir/",
      "args": [],
      "background": false
    }
  ],
  "preSyncCommands": [],                         
  "buildCommands": [
    {
      "commandline": "nvm install 14.18.0",
      "workDirectory": "/tmp/remote_dir/",
      "args": []

    },
    {
      "commandline": "nvm use 14.18.0",
      "workDirectory": "/tmp/remote_dir/",
      "args": []

    },
    {
    "commandline": "nvm alias default 14.18.0" ,
    "args": [],
    "workDirectory": "/tmp/remote_dir/"
    },
    {
      "commandline": "npm ci ",
      "workDirectory": "/tmp/remote_dir/",
      "args": []
    }
    ,
    {
      "commandline": "npm run start:ci  ",
      "workDirectory": "/tmp/remote_dir/",
      "args": [],
      "background": true
    }
    
  ],

  "environment": {

  },

  "excludedFromSync": ["log/", ".git/", "node_modules"],
  "excludedFromWatch": ["log/", ".git/","log", ".git", "node_modules"],
  "projectToken": "********",
  "framework": "Cypress",
  "image": "node-lts"
}

Running the tests on my machine locally a Macbook Pro 2019 takes over 2 minutes 44 seconds. Running in Brisk it completes in less than 41 seconds.

Cypress Making Things Even Faster

Cypress is interesting because it has a lot of setup. I think we could make it faster but this is also close to our max time, adding more tests won't make it slower (provided we add more workers).