Clarify (misleading?) bit about parallelism and credit usage in Test Splitting Tutorial

[tony-pizza]

Location

• https://circleci.com/docs/test-splitting-tutorial/
• https://github.com/circleci/circleci-docs/blob/881833257eb31528899955b537070d5a1cf63c08/jekyll/_cci2/test-splitting-tutorial.adoc

Explanation

This is based on the following assumption:

• When using parallelism, CircleCI bills for sum total compute time (e.g. 2 parallel runs at 5min counts as 10min of credit usage)

If this assumption is incorrect, please let me know!

Tutorial states you will:

See how test splitting can help decrease pipeline run times and optimize credit usage.

However, I don't feel that the Test Splitting Tutorial really explains how test splitting optimizes credit usage. And, in fact, I believe test splitting would typically increase credit usage, as each parallel run would spend some time doing some test setup or initialization.

Am I missing something here?

[rosieyohannan]

Hello! Thanks for pointing this out @tony-pizza . Agreed, it is misleading and not correct for all. This feature is primarily a speed optimization. I will update the tutorial. My understanding from talking to engineers here is that you can see a reduction in credit usage if a job that is taking a very long time, e.g. and hour . And it gets reduced using parallelism and test splitting down to, say 5 mins.

[KieranP]

Yes, I think this confuses people and it needs to be made very clear on the docs. Splitting tests does not reduce the total cumulative time, but infact almost always adds to it. Splitting a 60 minutes into 10 nodes does not mean you will have 10 nodes of 6 minutes (totaling 60 minutes). With the setup overhead, it will end up being more. If the setup overhead for that original 60 minutes is 5 minutes, then splitting it into 10 nodes will result in ten 11 minute runs, or a cumulative 110 minutes, nearly double original 60 minutes. So while your start to finish time goes from 60 minutes to 11 minutes, you end up paying for double the time in CI credits.