• Start the longest tests first. If you have for example 2 tests, a short one and a long one, the long one will be the sequential bottleneck. If you have many additional short tests, they will run in parallel to the longer one. If you have 2 workers, 1 will be dedicated to the long task while the other will process many of the short tests. If the total execution time of the short tasks is greater than the longer task then the total execution time will be at least the duration of the long task.
  • If we were not to schedule the longest tests first, we run the risk of running many short tests on a worker and then running the longest test, which may be suboptimal.
  • As you accumulate many short tests this issue is mitigated. However you have to be careful of distributing your longest tasks on separate workers as much as possible.
• If you know the exact duration of a test (or an approximation of it, given a desired quantum), determine if you can solve the scheduling problem in a reasonable amount of time such that solving the scheduling and running the scheduled tests takes less time than running the tests.
• From the standpoint of execution, we will prefer to have the best response time possible (i.e., run the shortest tasks first to get most of the tests executed as soon as possible).
• As you have many short and long tests, it may make sense to run all the short tests first and keep all the long tests for the end, ignoring the heuristics that suggests running the longest tests first.
• Other important consideration when running tests are shared environments/resources between tests. If those take a non-neglectable amount of time to setup/teardown and could be reused, it may be beneficial to group such tests on the same worker in order to avoid paying the cost of setup/teardown.
  • Setup/teardown duration should be recorded and be associated to each test.
  • Some system have different types of setup/teardown: global, per class/module/scope. Being able to be aware of whether the test system will need to re-run a set of setup/teardown functions may help with scheduling the tests.
• While we may be collecting test duration data, it is important to be aware that this data may end up being tainted if used in a continuous integration system. What this means is that we may sometimes record durations that are out of the ordinary due to a bug in the code, or because the tests execute on different hardware.
  • To mitigate this issue we may use different approaches such as using the median of the recorded values (which is more robust to outliers than the mean), use a percentile (such as 95%) to determine the expected duration, record the duration of successful tests separately from failed ones, record tests in separate buckets defined by the user (e.g., defined on the CPU spec used to run the test), etc.
• If the number/list of tests hasn't changed since the last run, it would also be beneficial to store the computed schedule so that it is only computed once and reused many times, which is a common use case. The only time you may not want to do this is if computing the schedule is very cheap and represent a neglectable amount of time in the overall testing process.

Example use case - pytest-xdist

• By default pytest does not store any prior test duration so we would have to estimate that all tests are of equal duration.
  • As we run tests, we may start to collect information about the duration of parameterized tests. This may serve us to determine whether the parameterized test has the same duration over different set of parameters and serve as a base to start providing estimates for the remaining parameters combination of this test.
  • A system similar to the cache provided by pytest may be used to record the duration of tests

Concepts

• Scheduler: responsible for taking a set of tests with meta data and scheduling them (i.e., determine in which order they will be executed).
• test: the unit of code to execute.
• test duration: the duration of a test. Test duration may be represented as a distribution since tests generally do not complete in an exact and fixed amount of time.
• test unit: a collection of tests that we can assume as being the same test (e.g., parameterized tests).

References

• https://en.wikipedia.org/wiki/Scheduling_(computing)
• https://en.wikipedia.org/wiki/Queueing_theory
• https://en.wikipedia.org/wiki/Queuing_Rule_of_Thumb
• https://docs.pytest.org/en/latest/cache.html

Tesseract (an open source OCR engine) supports a TSV format as output. I looked online for some documentation about the columns but couldn't find anything, so I looked at the source code.

Here is a summary description of each column, what they represent, and the range of valid values they can have.

• level: hierarchical layout (a word is in a line, which is in a paragraph, which is in a block, which is in a page), a value from 1 to 5
  • 1: page
  • 2: block
  • 3: paragraph
  • 4: line
  • 5: word
• page_num: when provided with a list of images, indicates the number of the file, when provided with a multi-pages document, indicates the page number, starting from 1
• block_num: block number within the page, starting from 0
• par_num: paragraph number within the block, starting from 0
• line_num: line number within the paragraph, starting from 0
• word_num: word number within the line, starting from 0
• left: x coordinate in pixels of the text bounding box top left corner, starting from the left of the image
• top: y coordinate in pixels of the text bounding box top left corner, starting from the top of the image
• width: width of the text bounding box in pixels
• height: height of the text bounding box in pixels
• conf: confidence value, from 0 (no confidence) to 100 (maximum confidence), -1 for all level except 5
• text: detected text, empty for all levels except 5

Here is an example of the TSV format output, for reference.

References

• https://tesseract-ocr.github.io/tessdoc/Command-Line-Usage.html

If you've been having some problems with your connection or especially with firefox, check if you have FasterFox. If you indeed do, please uninstall this crap. I was on dnsstuff.com and got blocked by one of their script telling me I had FasterFox (which in that case was true, even though I generally don't install that thing). I was having a hard time loading pages for no reason but I had good internet download speed using FlashGet.

In the end, I removed FasterFox, restarted FireFox, everything is back fast and furious :).