feat(anta): Limit concurrency #680

carl-baillargeon · 2024-05-16T05:04:18Z

Description

This PR improves the test runner by introducing a generator-based approach for managing test coroutines and setting a configurable limit on the number of concurrent tests.

Instead of loading all test coroutines into a list, the runner now uses a generator to yield tests. This approach prevents memory overload and improves performance when dealing with a large number of tests.
A limit on the number of concurrent tests is introduced to avoid overwhelming the runner. This limit is configurable with an environement variable (hidden).

Implementation:
The generator yields test coroutines, ensuring that only a limited number of tests are scheduled and run concurrently.
Upon reaching the concurrency limit, the runner waits for some tests to complete before scheduling new ones from the generator.

Fixes: #713, #832

Checklist:

Update FAQ to reference Scaling ANTA documentation
Update Scaling ANTA documentation
Add section on JSON catalogs vs YAML (faster)
Need to retest on digital twin @dlobato
Support None timeouts
Confirm if PoolTimeout default should be set to None
Update benchmark + unit tests
ci: add codspeed to benchmark ANTA #826
Confirm that the number of open file descriptors never exceed device * max_connections (100 by default)

sonarqubecloud · 2024-06-12T16:43:30Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-07-04T14:03:49Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

anta/runner.py

github-actions · 2024-08-28T18:50:12Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

sonarqubecloud · 2024-08-28T19:07:52Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
84.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-09-11T13:47:41Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2024-10-02T15:05:25Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

codspeed-hq · 2024-10-02T15:15:13Z

CodSpeed Performance Report

Merging #680 will not alter performance

_{Comparing carl-baillargeon:refactor/runner_limit (610d2a2) with main (995943a)}

Summary

✅ 22 untouched benchmarks
🆕 2 new benchmarks

Benchmarks breakdown

	Benchmark	`main`	`carl-baillargeon:refactor/runner_limit`	Change
🆕	`test_setup_tests[1-device]`	N/A	14.3 ms	N/A
🆕	`test_setup_tests[2-devices]`	N/A	14.3 ms	N/A

github-actions · 2024-10-10T18:49:29Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2024-11-04T21:15:42Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

for more information, see https://pre-commit.ci

anta/runner.py

mtache · 2025-01-21T10:36:09Z

anta/settings.py

+
+    Parameters
+    ----------
+    default_timeout : float | None


Default values are defined twice here.
Once in this file, the other definition is in the timeout CLI option.
I would suggest to keep the default values in this file, remove the default value in Click and point to the documentation.

Agreed with removing the default value in the CLI.

mtache · 2025-01-21T10:49:24Z

anta/cli/utils.py

@@ -163,6 +190,7 @@ def core_options(f: Callable[..., Any]) -> Callable[..., Any]:
        show_envvar=True,
        envvar="ANTA_TIMEOUT",
        show_default=True,
+        type=FLOAT_OR_NONE,


I would suggest to remove the default value here since it is defined in the anta.settings module already.
Then you can also get rid of the FLOAT_OR_NONE type :)

Same as above, agreed with removing the default value here but we do need to keep FLOAT_OR_NONE to support None (no timeout) at the CLI. We do not support None today:

(.venv) ~/git_projects/anta (main ✔) anta nrfu --timeout None table Usage: anta nrfu [OPTIONS] COMMAND [ARGS]... Try 'anta nrfu --help' for help. Error: Invalid value for '--timeout': 'None' is not a valid float.

mtache · 2025-01-21T10:50:50Z

mkdocs.yml

    - Developing ANTA tests: advanced_usages/custom-tests.md
    - ANTA as a Python Library: advanced_usages/as-python-lib.md
+    - Caching in ANTA: advanced_usages/caching.md
+    - Scaling ANTA: advanced_usages/scaling.md


Can we add a Settings section that render the BaseSettings models in anta.settings?

mtache · 2025-01-21T11:14:04Z

anta/settings.py

+
+    On POSIX systems, this value is used to set the soft limit for the current process.
+    The value cannot exceed the system's hard limit.
+    """


Missing ANTA_NOFILE attribute in the doctsring.

mtache · 2025-01-21T11:14:32Z

anta/settings.py

+
+
+class MaxConcurrencySettings(BaseSettings):
+    """Environment variable for configuring the maximum number of concurrent tests in the event loop."""


Missing ANTA_MAX_CONCURRENCY attribute in the docstring

mtache · 2025-01-21T11:15:10Z

anta/settings.py

+class HttpxResourceLimitsSettings(BaseSettings):
+    """Environment variables for configuring the underlying HTTPX client resource limits.
+
+    The limits are set using the following environment variables:


Use attribute section to have a table in mkdocs?

mtache · 2025-01-21T11:28:59Z

anta/_runner.py

+        if self.max_concurrency <= 0:
+            msg = "Concurrency limit must be greater than 0."
+            raise RuntimeError(msg)
+
+        # NOTE: The `aiter` built-in function is not available in Python 3.9
+        tests = generator.__aiter__()  # pylint: disable=unnecessary-dunder-call
+        tests_ended = False
+        tests_pending: set[Task[TestResult]] = set()
+
+        while tests_pending or not tests_ended:
+            # Add tests to the pending set until the limit is reached or no more tests are available
+            while len(tests_pending) < self.max_concurrency and not tests_ended:
+                try:
+                    # NOTE: The `anext` built-in function is not available in Python 3.9
+                    test = await tests.__anext__()  # pylint: disable=unnecessary-dunder-call
+                except StopAsyncIteration:  # noqa: PERF203
+                    tests_ended = True
+                    logger.debug("All tests have been added to the pending set.")
+                else:
+                    # Ensure the coroutine is scheduled to run and add it to the pending set
+                    tests_pending.add(asyncio.create_task(test))
+                    logger.debug("Added a test to the pending set: %s", test)
+
+            if len(tests_pending) >= self.max_concurrency:
+                logger.debug("Concurrency limit reached: %s tests running. Waiting for tests to complete.", self.max_concurrency)
+
+            if not tests_pending:
+                logger.debug("No pending tests and all tests have been processed. Exiting.")
+                return


Suggested change

if self.max_concurrency <= 0:

msg = "Concurrency limit must be greater than 0."

raise RuntimeError(msg)

# NOTE: The `aiter` built-in function is not available in Python 3.9

tests = generator.__aiter__() # pylint: disable=unnecessary-dunder-call

tests_ended = False

tests_pending: set[Task[TestResult]] = set()

while tests_pending or not tests_ended:

# Add tests to the pending set until the limit is reached or no more tests are available

while len(tests_pending) < self.max_concurrency and not tests_ended:

try:

# NOTE: The `anext` built-in function is not available in Python 3.9

test = await tests.__anext__() # pylint: disable=unnecessary-dunder-call

except StopAsyncIteration: # noqa: PERF203

tests_ended = True

logger.debug("All tests have been added to the pending set.")

else:

# Ensure the coroutine is scheduled to run and add it to the pending set

tests_pending.add(asyncio.create_task(test))

logger.debug("Added a test to the pending set: %s", test)

if len(tests_pending) >= self.max_concurrency:

logger.debug("Concurrency limit reached: %s tests running. Waiting for tests to complete.", self.max_concurrency)

if not tests_pending:

logger.debug("No pending tests and all tests have been processed. Exiting.")

return

if self.max_concurrency <= 0:

msg = "Concurrency limit must be greater than 0."

raise RuntimeError(msg)

tests_ended = False

tests_pending: set[Task[TestResult]] = set()

while tests_pending or not tests_ended:

async for test in generator:

# Add tests to the pending set until the limit is reached or no more tests are available

if len(tests_pending) < self.max_concurrency:

# Ensure the coroutine is scheduled to run and add it to the pending set

tests_pending.add(asyncio.create_task(test))

logger.debug("Added a test to the pending set: %s", test)

else:

logger.debug("Concurrency limit reached: %s tests running. Waiting for tests to complete.", self.max_concurrency)

break

if len(tests_pending) < self.max_concurrency and not tests_ended:

tests_ended = True

logger.debug("All tests have been added to the pending set.")

if not tests_pending:

logger.debug("No pending tests and all tests have been processed. Exiting.")

return

It looks equivalent right?

mtache · 2025-01-21T15:06:58Z

anta/device.py

+            session_settings["limits"] = httpx_limits
+            self.max_connections = httpx_limits.max_connections
+        else:
+            self.max_connections = DEFAULT_HTTPX_MAX_CONNECTIONS


More robust logic to instantiate the asynceapi.Device object then retrieve self._session.limits.max_connections.
Because of https://github.com/encode/httpx/blob/10b7295922741b91a15751029e6ad3e8e5efb9f3/httpx/_client.py#L655, httpx default values will be used.
Use a read property?

I believe you are suggesting to not store max_connections as a separate attribute, but rather create a read-only property that retrieves it directly from HTTPX? Limits is injected down the line to create the httpcore connection pool. We could potentially retrieve it by doing self._session._transport._pool._max_connections but since these are all privates it's too risky IMO.

mtache · 2025-01-21T15:10:10Z

anta/device.py

        timeout
-            Timeout value in seconds for outgoing API calls.
-        insecure
-            Disable SSH Host Key validation.
+            Global timeout value in seconds for outgoing eAPI calls.
+        httpx_timeout
+            Optional HTTPX Timeout object for fine-grained timeout configuration.
+            If provided, it will override the global timeout.


timeout defined twice.
Use timeout: httpx.Timeout object instead and remove timeout: float.
Use a read property to get the Timeout instance of this AyncEOSDevice instance.

Ah yes I didn't know HTTPX supported different types. Type would be timeout: Timeout | float | None to avoid breaking change. Should we also support the tuple type (below) like HTTPX?

TimeoutTypes = Union[ Optional[float], Tuple[Optional[float], Optional[float], Optional[float], Optional[float]], "Timeout", ]

I think we should also replace httpx_limits with just limits to be consistent with timeout then.

mtache · 2025-01-21T15:10:43Z

anta/device.py

-            host=host, port=ssh_port, username=username, password=password, client_keys=CLIENT_KEYS, **ssh_params
-        )
+            ssh_settings["known_hosts"] = None
+        self._ssh_opts: SSHClientConnectionOptions = SSHClientConnectionOptions(**ssh_settings)

    def __rich_repr__(self) -> Iterator[tuple[str, Any]]:


implement timeout and limits ?

mtache · 2025-01-21T15:13:21Z

anta/settings.py

+# Default values for HTTPX resource limits
+DEFAULT_HTTPX_MAX_CONNECTIONS = 100
+DEFAULT_HTTPX_MAX_KEEPALIVE_CONNECTIONS = 20
+DEFAULT_HTTPX_KEEPALIVE_EXPIRY = 5.0
+
+# Default values for HTTPX timeouts
+DEFAULT_HTTPX_CONNECT_TIMEOUT = 5.0
+DEFAULT_HTTPX_READ_TIMEOUT = 5.0
+DEFAULT_HTTPX_WRITE_TIMEOUT = 5.0
+DEFAULT_HTTPX_POOL_TIMEOUT = 5.0


Can we use https://github.com/encode/httpx/blob/10b7295922741b91a15751029e6ad3e8e5efb9f3/httpx/_config.py#L246 instead of copying the default value?
Not sure if the value will render in mkdocs.

if it changes upstream it may make us sad one day

We could put our own default values here instead with the current 30s default we have. From our testing we have a pretty good idea what works best:

# Default values for HTTPX resource limits DEFAULT_HTTPX_MAX_CONNECTIONS = 10 DEFAULT_HTTPX_MAX_KEEPALIVE_CONNECTIONS = 10 DEFAULT_HTTPX_KEEPALIVE_EXPIRY = 5.0 # Default values for HTTPX timeouts DEFAULT_HTTPX_CONNECT_TIMEOUT = 30.0 DEFAULT_HTTPX_READ_TIMEOUT = 30.0 DEFAULT_HTTPX_WRITE_TIMEOUT = 30.0 DEFAULT_HTTPX_POOL_TIMEOUT = None

mtache · 2025-01-21T15:15:22Z

tests/benchmark/test_anta.py

@@ -37,6 +37,7 @@ def test_anta_dry_run(

    results = session_results[request.node.callspec.id]

+    # TODO: Use AntaRunner in ANTA v2.0.0


Is this needed?
I think the anta.runner.main function uses the AntaRunner class already.

We can consider removing main() altogether and use AntaRunner everywhere in v2.0.0.

mtache · 2025-01-21T15:15:29Z

tests/benchmark/test_anta.py

@@ -69,6 +70,7 @@ def test_anta(

    results = session_results[request.node.callspec.id]

+    # TODO: Use AntaRunner in ANTA v2.0.0


I put this TODO everywhere we are using main().

mtache

Default value code could be simplified/cleaned.
Good idea to have this anta.settings module, we should use it for ANTA_DEBUG, in another PR ;)

sonarqubecloud · 2025-01-21T15:27:07Z

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
95.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions bot assigned carl-baillargeon May 16, 2024

carl-baillargeon force-pushed the refactor/runner_limit branch from bf88158 to d7de654 Compare June 12, 2024 16:43

github-actions bot added the state: conflict label Jul 4, 2024

gmuloc reviewed Aug 1, 2024

View reviewed changes

anta/runner.py Outdated Show resolved Hide resolved

gmuloc reviewed Aug 1, 2024

View reviewed changes

anta/runner.py Outdated Show resolved Hide resolved

carl-baillargeon force-pushed the refactor/runner_limit branch from b623e90 to 4879120 Compare August 28, 2024 18:49

github-actions bot removed the state: conflict label Aug 28, 2024

github-actions bot added the state: conflict label Sep 11, 2024

mtache changed the title ~~refactor(anta): Refactor runner to use a generator with a limit~~ feat(anta): Limit concurrency Oct 2, 2024

feat(anta): limit the number of tests run concurrently

83206e8

mtache force-pushed the refactor/runner_limit branch from 68e259e to 83206e8 Compare October 2, 2024 15:05

github-actions bot removed the state: conflict label Oct 2, 2024

gmuloc and others added 2 commits October 7, 2024 09:53

Merge branch 'main' into refactor/runner_limit

d86f85c

Merge branch 'main' into refactor/runner_limit

6d59aa9

github-actions bot added the state: conflict label Oct 10, 2024

Merge branch 'main' into refactor/runner_limit

d2aa7c7

github-actions bot removed the state: conflict label Nov 4, 2024

gmuloc and others added 2 commits November 4, 2024 23:30

Refactor: Fix github merge conflict resolution

7ac0358

[pre-commit.ci] auto fixes from pre-commit.com hooks

96da6ef

for more information, see https://pre-commit.ci

gmuloc reviewed Nov 4, 2024

View reviewed changes

anta/runner.py Outdated Show resolved Hide resolved

test: Temporarily disable one benchmark

f2f3c6a

github-actions bot removed the state: conflict label Jan 17, 2025

carl-baillargeon added 9 commits January 18, 2025 14:01

Added AntaRunner

ca75b60

AntaRunner to BaseModel

95f32ff

AntaRunner to BaseModel

ae97095

Fix

c566d2f

Fix unit tests

54af7e6

Fix manager

e72842b

Update guide

f2def95

Added deprecated decorator

4737b1f

Remove codspeed artifacts

a40c5c7

carl-baillargeon marked this pull request as ready for review January 19, 2025 19:20

carl-baillargeon requested review from gmuloc, mtache and dlobato January 19, 2025 19:20

mtache reviewed Jan 21, 2025

View reviewed changes

mtache requested changes Jan 21, 2025

View reviewed changes

Merge branch 'main' into refactor/runner_limit

610d2a2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(anta): Limit concurrency #680

feat(anta): Limit concurrency #680

carl-baillargeon commented May 16, 2024 •

edited

Loading

sonarqubecloud bot commented Jun 12, 2024

github-actions bot commented Jul 4, 2024

github-actions bot commented Aug 28, 2024

sonarqubecloud bot commented Aug 28, 2024

github-actions bot commented Sep 11, 2024

github-actions bot commented Oct 2, 2024

codspeed-hq bot commented Oct 2, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024

github-actions bot commented Nov 4, 2024

mtache Jan 21, 2025

carl-baillargeon Jan 22, 2025

mtache Jan 21, 2025

carl-baillargeon Jan 22, 2025

mtache Jan 21, 2025

mtache Jan 21, 2025

mtache Jan 21, 2025

mtache Jan 21, 2025

mtache Jan 21, 2025

carl-baillargeon Jan 22, 2025

mtache Jan 21, 2025 •

edited

Loading

carl-baillargeon Jan 22, 2025

mtache Jan 21, 2025

carl-baillargeon Jan 22, 2025 •

edited

Loading

carl-baillargeon Jan 22, 2025

mtache Jan 21, 2025

mtache Jan 21, 2025

gmuloc Jan 21, 2025

carl-baillargeon Jan 22, 2025

mtache Jan 21, 2025

carl-baillargeon Jan 22, 2025

mtache Jan 21, 2025

carl-baillargeon Jan 22, 2025

mtache left a comment

sonarqubecloud bot commented Jan 21, 2025



		class MaxConcurrencySettings(BaseSettings):
		"""Environment variable for configuring the maximum number of concurrent tests in the event loop."""

		@@ -37,6 +37,7 @@ def test_anta_dry_run(

		results = session_results[request.node.callspec.id]

		# TODO: Use AntaRunner in ANTA v2.0.0

		@@ -69,6 +70,7 @@ def test_anta(

		results = session_results[request.node.callspec.id]

		# TODO: Use AntaRunner in ANTA v2.0.0

feat(anta): Limit concurrency #680

Are you sure you want to change the base?

feat(anta): Limit concurrency #680

Conversation

carl-baillargeon commented May 16, 2024 • edited Loading

Description

Checklist:

sonarqubecloud bot commented Jun 12, 2024

Quality Gate passed

github-actions bot commented Jul 4, 2024

github-actions bot commented Aug 28, 2024

sonarqubecloud bot commented Aug 28, 2024

Quality Gate passed

github-actions bot commented Sep 11, 2024

github-actions bot commented Oct 2, 2024

codspeed-hq bot commented Oct 2, 2024 • edited Loading

Merging #680 will not alter performance

Summary

Benchmarks breakdown

github-actions bot commented Oct 10, 2024

github-actions bot commented Nov 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtache Jan 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carl-baillargeon Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtache left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Jan 21, 2025

Quality Gate passed

carl-baillargeon commented May 16, 2024 •

edited

Loading

codspeed-hq bot commented Oct 2, 2024 •

edited

Loading

mtache Jan 21, 2025 •

edited

Loading

carl-baillargeon Jan 22, 2025 •

edited

Loading