nondeterministic incorrect counting of tests

[pushcx]

I created a new test with a single assert Module to confirm a module was correctly loaded.

To my surprise, I saw Passed: 3. I re-ran the script and saw Passed: 1. I ran it a few times in a row, with no file edits in between:

pushcx@tweed ~/code/hotcake:main % $ be qt test/hotcake_optimist.test.rb                                                                                            
██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ 100% 
                                                                                                                                                                    
Passed: 1 | Failed: 0 | Errors: 0                                                                                                                                   
1s pushcx@tweed ~/code/hotcake:main % $ be qt test/hotcake_optimist.test.rb                                                                                         
██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ 100% 
                                                                                                                                                                    
Passed: 3 | Failed: 0 | Errors: 0                                                                                                                                   
1s pushcx@tweed ~/code/hotcake:main % $ be qt test/hotcake_optimist.test.rb                                                                                         
██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ 100% 
                                                                                                                                                                    
Passed: 1 | Failed: 0 | Errors: 0

So there's some kind of inconsistent overcounting bug.

[pushcx]

Probably the same bug, I caught be qc -w test/registry.test running a test twice (first and last errors). This file has 3 tests.

test/registry.test.rb:12
  deleting a task decreases the number of tasks in the registry
    unexpected NoMethodError
      undefined method 'registry=' for an instance of Task
        /home/pushcx/code/hotcake/lib/hotcake.rb:45:in 'Registry#[]'

test/registry.test.rb:6
  creating a task increases the number of tasks in the registry
    unexpected NoMethodError
      undefined method 'registry=' for an instance of Task
        /home/pushcx/code/hotcake/lib/hotcake.rb:45:in 'Registry#[]'

test/registry.test.rb:12
  deleting a task decreases the number of tasks in the registry
    unexpected NoMethodError
      undefined method 'registry=' for an instance of Task
        /home/pushcx/code/hotcake/lib/hotcake.rb:45:in 'Registry#[]'

Passed: 1 | Failed: 0 | Errors: 3

(running a few times more, I got it to error that same test 3 times in a run)

[pushcx]

I minimized a test case. The tests are definitely not being run multiple times. The results are being counted multiple times.

In a checkout of the quickdraw source, create raises.test.rb:

# frozen_string_literal: true

test "raise" do
  e = assert_raises(StandardError) { raise StandardError }
  assert_equal StandardError, e.class
end

Run:

for i in {0..10}; do bundle exec exe/qt raises.test.rb | grep Passed ; done                               
Passed: 100 | Failed: 0 | Errors: 0                                                                                                                                 
Passed: 68 | Failed: 0 | Errors: 0                                                                                                                                  
Passed: 108 | Failed: 0 | Errors: 0                                                                                                                                 
Passed: 68 | Failed: 0 | Errors: 0                                                                                                                                  
Passed: 104 | Failed: 0 | Errors: 0                                                                                                                                 
Passed: 66 | Failed: 0 | Errors: 0                                                                                                                                  
Passed: 62 | Failed: 0 | Errors: 0                                                                                                                                  
Passed: 84 | Failed: 0 | Errors: 0                                                                                                                                  
Passed: 62 | Failed: 0 | Errors: 0                                                                                                                                  
Passed: 70 | Failed: 0 | Errors: 0                                                                                                                                  
Passed: 64 | Failed: 0 | Errors: 0

The overcount is higher the faster the test runs, hence the grep.

The overcount is related to multiple processes. Adding -p 1 eliminates it. Adding -t 1 doesn't change the behavior.

[joeldrapper]

Thanks @pushcx! I have an idea where this might be coming from. I’m pretty sure it is 100% of the time running all the tests, it’s just counting wrong.

[pushcx]

It is always running the tests exactly once.

Errors and failures are also duplicated:

# frozen_string_literal: true

test "raise" do
  assert_raises(ArgumentError) { raise StandardError }
end

$ be exe/qt raises.test.rb | cat
[elided 33 repetitions]
(raises.test.rb)
  raises.test.rb:3
    raise
      raises.test.rb:4

expected block to raise ArgumentError but StandardError was raised instead

(raises.test.rb)
  raises.test.rb:3
    raise
      raises.test.rb:4

expected block to raise ArgumentError but StandardError was raised instead

Passed: 0 | Failed: 35 | Errors: 0