Exploring Minitest Concurrency

Among all the other morsels of sunshine and goodness it offers, Minitest gives you the ability to denote that some or all of your test cases are able to run concurrently, and as pointed out in the source comments, that can only mean one thing: that you rule and your tests are awesome.

I wanted to better understand how and when to use parallelization in my own code, so I dove into the source and ran some simple bench tests to come up with some guidelines for how to put it to work.

The basics

There are two ways to switch on parallelization in your tests. You can apply it to all test cases in your suite by adding the following lines to your test_helper.rb file:

1
2
3
4
5
6
7
# test/test_helper.rb

require 'minitest/hell'

class Minitest::Test
  parallelize_me!
end

The inclusion of Minitest::Hell ensures that the tests in all cases will be executed in random order by overriding Minitest::Test.test_order to return :parallel. (Random test ordering also happens to be the default behavior, but this can be overridden per-test case.) The call to parallelize_me! causes all test cases to queue tests across a pool of concurrent worker threads via the Minitest::Parallel::Executor helper class. The implementation is worth checking out because it’s super simple and does its job nicely - a really great example of restrained, focus code.

It’s also perfectly reasonable, though presumably somewhat less awesome, to call parallelize_me! only on certain individual test cases when some tests require execution in a strict order. It’s also possible to override global parallelization settings on individual test cases to require serial execution by calling i_suck_and_my_tests_are_order_dependent! within the test class.

One additional feature that’s not documented anywhere that I could find except in the source code: you can specify the number of worker threads started and run by Minitest::Parallel::Executor by providing an environment variable on the command line when you run your tests like so: N=4 rake test. (Minitest defaults to N=2 if no other value is given.)

Ruby concurrency vs. Ruby parallelism

There’s an important difference between these two concepts that needs to be made clear before moving forward: concurrency indicates an ability to manage several tasks, while parallelism implies that I can actually do multiple things simultaneously. (Ilya Grigorik summed this up better than I ever could in a talk he gave on the subject.) As an example, I might be able to juggle lots of chainsaws (excellent concurrency), but I probably can’t hold more than one of them at any given time (no parallelism).

In Ruby, whether you achieve parallelism or only concurrency depends on the thread model implemented by the interpreter. This is a decision that you the developer make. Minitest can’t influence it either way. Concretely, MRI’s thread model maps Ruby Thread objects onto system-level threads, but the Global Interpreter Lock (GIL) ensures thread safety by ensuring that only one is ever allowed to execute at any given time, so parallelism within a single Ruby process is impossible. Other Ruby implementations - JRuby and Rubinius, as examples - don’t have the GIL, and so true parallel execution is possible in both these environments.

Ryan Davis, the author of Minitest, makes no secret about what Minitest::Parallel was written for:

I don’t care in the slightest about trying to make your tests run faster. You deserve your pain if you write slow tests. What I do care about greatly is making your tests hurt and this will do that.

- Ryan Davis

And he’s right, of course. Randomizing the order in which tests are executed gives you a certain measure of confidence that you’ve managed to write independent tests, and that’s the default behavior Minitest provides. But by then executing tests across multiple threads that don’t share state between, you’re adding an additional level of isolation insurance. But why shouldn’t we have all of that and the fastest possible test execution? Are there any trade-offs?

The setup

I set up a base project that consisted of an empty Ruby gem with a basic Rakefile and code to generate 1000 tests spread over ten more or less identical test cases which all followed the same pattern:

1
2
3
4
5
6
7
8
9
10
11
12
# test/one_fake_test.rb

require "test_helper"

class OneFakeTest < Minitest::Test
  (1..100).each do |i|
    define_method("test_#{ i }") do
      (1..8000).reduce(:*)
      assert(true)
    end
  end
end

My goal was to demonstrate test suite performance for three options:

• Baseline - serial execution only
• Minimally parallel - concurrent with two workers
• Maximally parallel - concurrent with eight workers (= number of CPU threads on my development machine)

In order to see how Minitest behaved across different interpreters, I ran the same test using MRI Ruby 2.1.1, JRuby 1.7.9, and Rubinius 2.2.10.

For each implementation and each concurrency level, I executed three test runs and calculated the averages values for suite execution and number of assertions per second as reported by Minitest as well as the total and CPU times reported by the Unix time command. The tables below show the averages along with the relative differences over the baseline (serial) case in parentheses.

Round 1: MRI

The GIL optimizes for single-threaded execution, so while the additional concurrency might be giving us better assurances that each test is isolated and independent, it’s not free. It’s not too painful in this test application, but if I were running a suite for a large, well-covered Rails application where some slower tests are usually unavoidable, I might feel like the added insurance is too expensive.

Round 2: JRuby

My expectations for JRuby were higher because it uses a threading model that allows for fully parallel execution in multicore environments, and I was not disappointed. Despite being kind of a slow starter, JRuby was able utilize a lot more available CPU when I increased the concurrency, and I was able to run the test suite in about one-third the time.

Round 3: Rubinius

Rubinius was the quickest interpreter I tested by a mile. Even though the performance gains from each additional thread weren’t as substantial as I saw in JRuby and tapered off more quickly, the fact is that Rubinius was really nimble, did a good job utilizing multiple cores, and needed about half the startup time of JRuby. I haven’t used Rubinius before, but this experience has convinced me to take a closer look at it in the future.

The long and short of it

In the end, I learned as much about Ruby and Ruby implementations as I did about Minitest, and so even though this all took some time to think through and set up, it was time well spent.

• Concurrent execution in Minitest isn’t just (or even primarily) about speed. Even with no improvement in performance, the increased test isolation is a worthwhile benefit.
• When using a Ruby implementation that allows for true parallel execution, enable parallel execution with concurrency equal to or approaching the number of available CPU cores on your test system.
• When using MRI Ruby, you’ll see the best performance when running in a single thread, but taking a slight performance hit to run concurrently in two threads would probably be a good compromise.

Your mileage may vary when it comes to implementing (or not) concurrent execution within your own real-world tests, but hopefully this will give you a better understanding of how it works in Minitest, what you gain from it, and how to make it work for you.

Additional resources

• Minitest Parallelization and You by Ryan Davis
• No Callbacks, No Threads & Ruby 1.9 by Ilya Grigorik
• Concurrency in Ruby Explained by Matt Aimonetti
• The GIL and MRI by Jesse Storimer