DevOps Voyage

How to: Execute RSpec in parallel locally

2018-10-22T00:00:00+00:00

Last entry in series about stable and faster test suite. Previous posts were focusing on parallel execution when running the test suite on CI nodes. This time we will see how we can run it locally as well. You can find previous articles here

How to: Road to fast and stable test suite
How to: Get most of the database cleaner
How to: Execute RSpec in parallel locally (You are here)

ParallelTests gem

So far we used Knapsack gem for dividing our tests in order to evenly run them on CI, but we can’t really use it for running them all in parallel locally. This is where ParallelTests gem will come in handy.

ParallelTests splits tests into even groups (by the number of lines or runtime) and runs each group in a single process with its own database.

So in short, each process spawned by ParallelTests should have its resources isolated from one another, so they don’t interfere with each other.

Basic setup

Add parallel tests to development and test groups in Gemfile.

gem 'parallel_tests', group: %i[development test]

Next, let us decide on the number of processors You want to use. By default parallel_tests will set it to number of CPUs available (i.e. 4 cores with hyperthreading, will count as 8 cores). If You would like to override it, append [<no of processors to run>] to tasks below or better, export environment variable to have it the same for all the tasks, i.e.

export PARALLEL_TEST_PROCESSORS=8

Just remember to put it in something like .bashrc or .env file, in order to have it always loaded between sessions.

Resources configuration

1. Database

I assume You use a database, otherwise, you wouldn’t have issues with slow running tests in the first place. For this to work, we need to update our config/database.yml with an extended database name.

test:
  database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>

This will append the processor number to your database name, whenever you will run tests with more than 1 process, i.e.

First process yourproject_test
Second process yourproject_test1
etc

Now let’s create all databases.

rake parallel:create

And load the schema.rb or structure.sql to all DBs created above.

rake parallel:prepare

2. Capybara

Capybara servers should run on separate ports, i.e.

Capybara.configure do |config|
  config.server_port = 9887 + ENV['TEST_ENV_NUMBER'].to_i
end

3. Other resources

You should do the same for any other kind of resource You use, those should be completely separated from each other, i.e.:

files, i.e. for rails/sprockets cache should lay in different directories
redis, use different DBs per process
sphinx, run a separate instance per each processor
etc

Checkout extensive wiki for details.

Lets run this!

rake parallel:spec
# => 8 processes for 500 specs, ~ 62 specs per process

Full throttle!

Awesome! Now we can run our test suite locally as fast as on CI.

Speedup process boot

Now when spawning all those processes, each of them will take some time, based on the size of your application. We can speed this up with spring, which comes by default with rails installations these days. To make it work we will have to do a small patch for it, otherwise it won’t work with parallel_tests.

This is due to the fact that with spring, when it boot up the server process, the configuration will be already set. This means DB name will always equal to yourproject_test in each spawned process based on the server one.

In order to mitigate this, we will pick up the correct DB configuration after forking the server process. You can find it in parallel_tests Wiki.

Create new file under config/spring.rb - it should get picked up by spring automatically.

require 'spring/application'

class Spring::Application
  alias connect_database_orig connect_database

  # Disconnect & reconfigure to pickup DB name with
  # TEST_ENV_NUMBER suffix
  def connect_database
    disconnect_database
    reconfigure_database
    connect_database_orig
  end

  # Here we simply replace existing AR from main spring process
  def reconfigure_database
    if active_record_configured?
      ActiveRecord::Base.configurations =
        Rails.application.config.database_configuration
    end
  end
end

Then you can either prepend DISABLE_SPRING=0 to commands, export it, or put in .bashrc/.env file.

With this, bin/rake parallel:spec will boot up way faster by making use of spring preloader. You should notice this by extra lines in output like

Running via Spring preloader in process 20005
Running via Spring preloader in process 20012
Running via Spring preloader in process 20015
... etc

Tests distribution

Another thing to consider is, how to evenly distribute tests across processes. parallel_tests has similar functionality as knapsack, it can log tests runtime in a JSON file and then use it to spread them evenly across all processes. To do so add .rspec_parallel file in project root directory, so on next run it will create the report, and use it in consecutive executions.

--format progress
--format ParallelTests::RSpec::RuntimeLogger --out tmp/parallel_runtime_rspec.log

NOTE: Remember to put any significat config from .rspec file to .rspec_parallel, i.e. --require spec_helper - as parallel tests will use the later only, what can lead to issues with tests.

Now, this is good for having local runtime as low as possible, but what if we would like to use the knapsack report, which we already have? Sadly parallel_tests doesn’t have any integration for it, but we can play around and add it by ourselves - because we can :)

Let’s create a wrapper task we will use to run it

# lib/tasks/knapsack.rake
namespace :knapsack do
  task local: :environment do |_, _|
    ENV['CI_NODE_TOTAL'] = ENV['PARALLEL_TEST_PROCESSORS']
    ENV['RAILS_ENV'] = 'test'

    require 'knapsack'
    require_relative '../../config/boot'
    require_relative 'parallel_tests_patch'

    ParallelTests::CLI.new.run(['--type', 'rspec', 'spec'])
  end
end

Now a small monkey patch to use knapsack allocator in parallel tests

# lib/tasks/parallel_tests_patch.rb
require 'parallel_tests/rspec/runner'

class ParallelTests::RSpec::Runner
  def self.tests_in_groups(tests, num_groups, options = {})
    puts 'ParallelTests with Knapsack runtime report :woohoo:'

    (0...num_groups).map do |index|
      ENV['CI_NODE_INDEX'] = index.to_s
      Knapsack::AllocatorBuilder
        .new(Knapsack::Adapters::RSpecAdapter)
        .allocator
        .node_tests
    end
  end
end

Let’s test it!

rake knapsack:local
# ParallelTests with Knapsack runtime report :woohoo:
# 8 processes for 500 specs, ~ 62 specs per process

NOTE For local execution I would still use parallel_tests allocator instead, as it will be generated based on our machine performance, whereas knapsack is supposed to be based on the CI node.

Runtime comparison

Processes	Spring?	Runtime log?	Runtime
8	yes	yes	`8m 22.928s`
7	yes	yes	`8m 55.850s`
8	no	yes	`8m 57.309s`
6	yes	yes	`10m 03.678s`
9	yes	yes	`10m 09.448s`
8	yes	no	`12m 59.819s`
1	no	no	`42m 00.501s`

As you can see above, our suite runtime without any parallelization takes quite a while, around ~42 minutes.

When we parallelize it with 8 processes result differs based on extra switches.

Without the runtime log to evenly distribute tests, take the longest (even with spring support).

We can also see that having more than 8 processes is also degrading runtime.

In our case the best results are achieved when:

running against 8 processes on 8 available cores
run together with runtime logs for tests distribution (-4'30")
processes are preloaded by spring (-30")

This can be due to the fact that we have a lot of IO in tests, in tests and appications that do heavy computing, less processes can actually give better results. As usuall measure, compare and take the most performant option ;-)

Summary and what’s next

From now on it should be easy to run Your test suite in parallel, both locally and on CI with help of knapsack and parallel_tests gems.

The test suite itself should be also faster and more stable thanks to better usage of DatabaseCleaner.

Thankfully Rails 6 should bring us built-in support for

running tests in parallel
better transactions handling
and multi-database connection support

So we won’t have to hack our way through in new apps, for old ones running on Rails <= 5 we are already covered.

Stay tuned and happy hacking!

How to: Get most of the database cleaner

2018-09-26T00:00:00+00:00

In the previous post, we saw how we can divide and speed up test suite by using Knapsack gem from ~1 hour to 11 minutes. We also cached our dependencies on CI and started looking out for randomly failing tests. This time we will take a look at how to setup DatabaseCleaner in order to make use of different strategies for wiping out data between test runs in order to squeeze in even more from the suite.

DatabaseCleaner gem

DatabaseCleaner is a set of strategies for cleaning your database in Ruby. It’s a simple yet powerful tool that You most probably already use. Even though it’s simple, its configuration can get tricky at times. Messing up configuration can cause extra failures related to database pollution. So let us see how we can set it up in order to have a stable test suite.

Basic setup

Add database cleaner to test group in Gemfile (if it’s not yet there).

group :test do
  gem 'database_cleaner'
end

Make sure you load rspec/support helpers, it’s loaded by default with RSpec setup in spec/rails_helper.

# depending on age of the project look into rails or spec helper file
Dir[Rails.root.join('spec', 'support', '**', '*.rb')].each { |f| require f }

Create spec/support/database_cleaner.rb with

RSpec.configure do |config|

  config.before(:suite) do
    DatabaseCleaner.clean_with(:deletion)
  end

end

This is a good starting point, it will make sure the database is clean before the whole suite starts in case there are any leftovers from previous run.

Transactions - fastest all-rounder

Transactions should be used by default, when you need to use database calls it’s the fastest strategy available. DB Cleaner will open new transaction on each test case and roll it back when it’s finished. This way it doesn’t need to truncate all the tables, even though You used just one.

RSpec.configure do |config|

  # ...

  config.before(:each) do
    DatabaseCleaner.strategy = :transaction
    DatabaseCleaner.start
  end

  config.after(:each) do
    DatabaseCleaner.clean
  end

end

Deletion - rescue for after commit callbacks

Sometimes transactions are not enough, especially when you use after commit callbacks in ActiveRecord. As the name implies those are executed after committing the transaction. This can’t work with the previous strategy as transaction would still be open on the DB Cleaner level.

For this kind of unit tests where you touch DB and need access to data after committing the transaction, you can use deletion strategy. Which won’t be significantly slower than the transaction for small data sets.

To fix this up without switching all the tests to deletion, we can use RSpec examples metadata, in order to pick correct strategy on demand.

  # ...

  config.before(:each) do
    DatabaseCleaner.strategy = example.metadata[:strategy] || :transaction
    DatabaseCleaner.start
  end

  # ...

With this whenever we need to use the deletion strategy, we will just add strategy: :deletion, like:

# for whole block
context "executes after commit callbacks", strategy: :deletion do
  # ...
end

# or a single example
it "creates thumbnails on save", strategy: :deletion do
  # ...
end

Truncation - safe acceptance examples

Truncation is the slowest but also the safest and most stable strategy. There are no compromises when it comes to clean state here. It’s the best strategy for acceptance tests when using capybara with javascript driver.

Again we can make use of metadata, and use it by default for all js related tests.

  # ...

  config.before(:each) do
    DatabaseCleaner.strategy =
      if example.metadata[:js]
        :truncation
      else
        example.metadata[:strategy] || :transaction
      end

    DatabaseCleaner.start
  end

  # ...

Truncation or Deletion?

Rule of thumb should be:

For big setups before the tests, when you create a lot of objects with associations that use foreign keys, etc - use truncation, as it’s a fixed time regardless of the amount of data. This also means the more tables you have the more time it will take, as it takes the same amount of time for empty tables as well. You can think about it as drop plus create table, indexes, etc.

On the other hand, when you have a simple setup, with unit alike tests - use deletion as it’s faster for small datasets, it won’t recreate tables or indexes.

If you are interested in details, then here’s a great answer for PostgreSQL on StackOverflow explaining it thoroughly.

Bonus no.1: Cleaning on demand

Life is hard, and usually, you will find yourself with a really big setup of objects. This can happen especially when you try to test some kind of filtering or querying classes, which do read-only queries against data, without any side effects. In such cases, it would be great if we could disable DB Cleaner per example, and run it only once in before/after all blocks. Again this is easy to achieve with examples metadata, lets us see.

  # before :suite ...

  config.before(:all, :cleaner_for_context) do
    DatabaseCleaner.strategy = :truncation
    DatabaseCleaner.start
  end

  config.before(:each) do |example|
    next if example.metadata[:cleaner_for_context]

    DatabaseCleaner.strategy = # ...
    DatabaseCleaner.start
  end

  config.after(:each) do |example|
    next if example.metadata[:cleaner_for_context]

    DatabaseCleaner.clean
  end

  config.after(:all, :cleaner_for_context) do
    DatabaseCleaner.clean  
  end

  # ...

Lets go into detail what do we do here:

before :all, :cleaner_for_context block, will run once for whole context/describe block. We used truncation, as it will usually be faster in such examples.
We skip before and after :each blocks whenever we set :cleaner_for_context metadata.
We clean database once for whole context/describe block with after :all

Then in our test, we can do

RSpec.describe CoreDashboardQuery, :cleaner_for_context do

  before :all do
    @data = # ...
  end

  let(:data) { @data }

  # it ...
end

Note: Remember that whenever you use before/after all blocks, those are run once for the given context/describe block, so:

don’t do any data updates or additions within those blocks, unless you want to shoot yourself in the foot with a canon
whenever you use this technique, create the setup on top of the file, don’t hide it somewhere nested on the bottom

Bonus no.2: Catch data pollution issues

Getting back to our test suite example, which is still dark and full of terrors. We had random failures that were related to test leftovers in the database. Those were caused in examples that were running with :transaction strategy, were before :all block was lurking in the depths of nested context blocks.

The easiest way to find those is to crash loudly whenever the database is not clean in after :all block. To know what to count, look out for errors like “Couldn’t create Project because of unique key validation” or similar. To catch those drop this line in our setup.

# still in our spec/support/database_cleaner.rb file
RSpec.configure do |config|

  class DirtyDatabaseError < RuntimeError
    def initialize(meta)
      super "#{meta[:full_description]}\n\t#{meta[:location]}"
    end
  end

  # ...

  config.after(:each) do |example|
    next if example.metadata[:cleaner_for_context]
    DatabaseCleaner.clean
    raise DirtyDatabaseError.new(example.metadata) if Project.count > 0
  end
end

This will crash as soon as there are records still in DB, and will print example which caused it:

Failures:

  1) ProjectsQuery#build_query sorts projects by title
     Failure/Error: raise DirtyDatabaseError.new(example.metadata) if Project.count > 0

     DirtyDatabaseError:
        ProjectsQuery#build_query sorts projects by title
         ./spec/queries/projects_query_spec.rb:278

Of course, you don’t want to run it every time, so just drop it in whenever You see some suspicious errors.

Summary

Our final DatabaseCleaner configuration for RSpec.

class DirtyDatabaseError < RuntimeError
  def initialize(meta)
    super "#{meta[:full_description]}\n\t#{meta[:location]}"
  end
end

RSpec.configure do |config|

  config.before(:suite) do
    DatabaseCleaner.clean_with(:deletion)
  end

  config.before(:all, :cleaner_for_context) do
    DatabaseCleaner.strategy = :truncation
    DatabaseCleaner.start
  end

  config.before(:each) do |example|
    next if example.metadata[:cleaner_for_context]

    DatabaseCleaner.strategy =
      if example.metadata[:js]
        :truncation
      else
        example.metadata[:strategy] || :transaction
      end

    DatabaseCleaner.start
  end

  config.after(:each) do |example|
    next if example.metadata[:cleaner_for_context]

    DatabaseCleaner.clean

    # raise DirtyDatabaseError.new(example.metadata) if Record.count > 0
  end

  config.after(:all, :cleaner_for_context) do
    DatabaseCleaner.clean  
  end
end

In our case, we were using :truncation for everything. In order to speed things up, we added above test configuration to use :transaction by default for everything, and we left :truncation for JS tests only.

We also used :deletion instead of :truncation for unit alike tests, that used after commit callbacks, i.e. attachment objects that are processing files when it’s saved. In those cases, the deletion was taking a fraction of the time of truncation.

On the way, we found out which tests were polluting database, and we were able to fix it easily. When we finished the move, our suite runtime dropped by another 2 minutes and now stays stable at ~9 minutes per job.

Next time we will try making use of knapsack to run the test suite in parallel locally, so stay tuned and happy hacking!

How to: Road to fast and stable test suite

2018-09-12T00:00:00+00:00

It’s an extension for my presentation from Ruby User Group Berlin meetup in September, 2018, You can find slides here.

A beginning of the story on how we speed up test suite in one of the projects at Akelius from one hour to less than 10 minutes.

TL;DR: Divide and Conquer

start with making tests run in parallel in the easiest possible manner
start tracking tests that fail in each of the subsets of tests
reproduce and fix issues locally

Why?

I don’t think it needs to be said out loud but here it goes - slow and flaky test suites are bad for you! If Your suite runs for over 10 minutes you are hurting yourself and your team. But let us start from the beginning. When I started in Akelius, application test suite was running for an hour. Yes, you read it correctly - an HOUR. It also had order dependant test failures, so 1 in 4 builds could randomly fail.

Oh no, not again!

By using Github checks, that CI needs to pass in order to merge a Pull Request, this would basically block the deployment to next stages. Due to a random failure, the build would be usually restarted in order to move forward - which doesn’t mean it would be green on the next run.

Finally, some time later, after 1-3 restarts.

This resulted in:

very long feedback loops
lots of distractions due to context switching
frustration
and a useless test suite, which was more of a drag than a help
cost us lots of time

So how did we handle this? We had to speed it up first, in order to easier track down random test failures. How to do it with least effort in the ruby world?

Enter: knapsack gem

Knapsack splits tests across CI nodes and makes sure that tests will run comparable time on each node.

Knapsack doesn’t require any extra setup for additional services you are using, like database or chrome. Instead, it relies completely on the isolation provided by CI nodes. The only logic involved is how to evenly divide test files across nodes, so nodes run time is balanced, i.e. all nodes finish in ~5 minutes, instead of one finishes in 2 minutes and another one in 8.

Installation is quite easy and documentation covers a variety of ruby test runners.

Parallelize tests with the knapsack

Add gem in Gemfile.

group :test, :development do
  gem "knapsack"
end

Add tasks in Rakefile.

Knapsack.load_tasks if defined?(Knapsack)

Bind knapsack in top of spec helper file.

require "knapsack"
Knapsack::Adapters::RSpecAdapter.bind

Generate report, so knapsack will know runtime of the specific test files and will be able to evenly split them across nodes. It’s best to run it on actual CI node, so results will be closest to real ones.

# locally
$> KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec

# or on CI, i.e. edit .travis.yml file
script:
  - "KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec"

When the build completes, it will print out the new JSON report which will look something like:

{
  "spec/models/supply_spec.rb": 0.7876174449920654,
  "spec/models/tax_spec.rb": 0.22003436088562012,
  "spec/models/text_document_spec.rb": 3.3623762130737305,
  "spec/models/user_spec.rb": 318.7685122489929,
  ...
}

Commit report to the repository in knapsack_rspec_report.json file.

The last step is to actually configure CI setup. In our case it was TravisCI, but you can find docs for CircleCI and others as well, on knapsack documentation page. Here we say that we run our tests across 8 workers.

env:
  global:
    - RAILS_ENV=test
    - CI_NODE_TOTAL=8 # total number of workers

  matrix:
    - CI_NODE_INDEX=0
    - CI_NODE_INDEX=1
    # ... 5 more vars
    - CI_NODE_INDEX=7

script:
  - "bundle exec rake knapsack:rspec"

Now lets watch them fly! 🚀

First runtime improvement, from ~60 to ~12 minutes.

Bonus #1: Outsource linters

In our case we were using linters during CI run. By moving to CodeClimate, we shaved another ~30 seconds per build.

Afterwards we could let go 4 gems from Gemfile, which reduced total number of dependencies by 11.

Bonus #2: Use headless chrome and cache dependencies

Always try to prefer dependencies, that come preloaded on CI machines and make use of them. This will usually result in extra bonus seconds, that can be spent running suite, instead of installing dependencies by hand.

One of such dependencies is chrome, which we use for acceptance tests.

# .travis.yml
# remove
before_install:
  - sudo apt-get install chromium-browser

# add
addons:
  chrome: stable

And if you are using headless chrome, instead of xvfb, you need to download the chromedriver for it to work. We make use of builtin caching mechanism, to download it, only if it’s missing in the cached directory. With this we saved extra ~20s per build. And we can easily install newer version if needed.

# .travis.yml
cache:
  bundler: true
  directories:
    - ~/bin

before_install:
  - ./bin/ci_install_chromedriver

In bin/ci_install_chromedriver we specify the script for checking and installing chromedriver. We test against specific version, which will help invalidate the cache when we update. You can use something simillar for other dependencies as well.

#!/bin/bash
set -Eeuo pipefail

VERSION="2.41"
NAME="chromedriver_${VERSION}"
STORAGE=https://chromedriver.storage.googleapis.com

install_chromedriver() {
  rm -f ~/bin/chromedriver*

  wget ${STORAGE}/${VERSION}/chromedriver_linux64.zip

  unzip chromedriver_linux64.zip
  rm chromedriver_linux64.zip
  mv -f chromedriver ~/bin/${NAME}
  chmod +x ~/bin/${NAME}

  ln -s ~/bin/${NAME} ~/bin/chromedriver
}

[ -e ~/bin/${NAME} ] || install_chromedriver

Track and fix random failures

Since we now have fast running test suite, we don’t only fail fast, we also reproduce the problem fast as well - as we now know the subset of tests that’s causing the failure. In order to reproduce those locally, we first need to find the job that failed, its CI_NODE_INDEX, and rspec order seed. First one decides which files to run, second dictates in which order those are run.

We now can run a subset of tests, which takes only ~8 minutes instead of 60.

After finding the job, you should be able to locate the seed, at the bottom of the failed test output.

**4049** is what we are looking for.

Now, we can easily reproduce it locally, by running following command.

$> CI_NODE_TOTAL=8 \
   CI_NODE_INDEX=5 \
   bundle exec rake "knapsack:rspec[--seed 4049]"

You can also try using RSpec --bisect flag, to create a minimal repro case for the ordering dependency. It will usually also find other issues on the way and take some time depending on the test suite. Usually it’s worth the wait, and if it’s taking too long, you can always get results so far by hitting CTRL+C.

$> CI_NODE_TOTAL=8 \
   CI_NODE_INDEX=5 \
   bundle exec rake "knapsack:rspec[--seed 4049 --bisect]"

# =>

Report specs:
spec/models/user_spec.rb
# and a load of other tests scoped by knapsack

Leftover specs:

Running via Spring preloader in process 17947
Bisect started using options: "--seed 4049 --default-path spec -- spec/models/user_spec.rb and a load of other tests scoped by knapsack"
Running suite to find failures... (4 minutes 23.9 seconds)
Starting bisect with 6 failing examples and 921 non-failing examples.
Checking that failure(s) are order-dependent... failure(s) do not require any non-failures to run first

Bisect complete! Reduced necessary non-failing examples from 921 to 0 in 12.51 seconds.

The minimal reproduction command is:
  rspec ./spec/features/attachment_spec.rb[1:1:1,1:2,1:3] ./spec/features/project_spec.rb[1:1] ... --seed 4049 --default-path spec --

We had one example failing in the job, and we have found 6 in total by running it with --bisect flag.

Fix the issue and repeat

This is subject for whole new post, or most probably, even a book. It will mostly depend on the test suite, but in general look out for:

if using Rails, autoloading issues, especially when you have deeply nested structures like ActiveRecord models and Single Table Inheritance
using sleep statements, commonly found in acceptance tests
database pollution, i.e. some deeply nested before :all blocks, which run once for the whole file and not in the scope of nested describe/context blocks
make sure that the order of the elements doesn’t play a role in tests that don’t demand it, i.e. results in arrays

If You can’t fix them, as they happen, don’t blindly restart the build. Instead note down the CI_NODE_INDEX and seed. You can use some simple TODO list, but from my experience it’s better to plan it properly. For example, create maintenance ticket in JIRA (or in project management tool you are using), keep track of it, write down other failed examples. So when the time comes, You can fix them in one go or assign as subtasks within the team.

Profit & Sanity

After the first improvements, tests were running around 49 minutes faster per build, which resulted in:

faster iterations on the tasks at hand
running the whole suite more often without the fear of failure
fewer distractions and context switching due to waiting for the results
team happiness and time to delivery :)

There was one small drawback though - by using 8 workers, we block most of our available CI capacity. You can probably already imagine, pushing one commit after another and blocking CI for everyone. This is something to look out for, also maybe there are ways to cancel previous builds on the same branch. We will investigate it in future when it becomes a bottleneck.

Next steps

First improvements resulted in a build time of around 11 minutes, in next article we will go over how we shaved another 2 minutes, by using a mix of different strategies for cleaning up a database. We will also explore Jenkins, as a CI setup with kubernetes support and using docker images for build environments.

Stay tuned and happy hacking!