Fix for redis memory check failure after link flap and also sometimes cpu usage high failure #15732

wumiaont · 2024-11-25T15:15:18Z

Description of PR

Redis memory check result is not a stable value using "redis-cli info memory | grep used_memory_human". It's found on a stable system (BGO converged, no port flapping etc), the above check could have memory usage difference by more than 0.2M.

Followings are CLI output from 202405 and 202205.

202405:
admin@ixre-egl-board30: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/'
2.64
admin@ixre-egl-board30: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/'
2.74
admin@ixre-egl-board30: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/'
2.52

202205:
admin@ixre-egl-board64: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/'
6.02
admin@ixre-egl-board64: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/'
6.26
admin@ixre-egl-board64: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/'
6.14

We can see that 202405 has some memory optimization for redis and it's not using as much memory as 202205. 0.2M memory usage difference could easily reach the memory usage threshold of 5% in 202405.

Solution is to get the average redis memory usage before and after link flap. using 5 seconds interval and 5 times query and then get the average memory usage for redis. Also make the threshold to 10% from 5%. With this fix it's found that the redis memory check will not fail for 2405 after link flap.

This commit also provide a fix for sometimes CPU utilization check failed for orchagent after link flap. The reason is in scaling setup (34k routes) orchagent takes more time to calm down.

Summary:
Fixes # (issue) #15733

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

Approach

What is the motivation for this PR?

Fix test failures

How did you verify/test it?

OC tests run with the fix. Did not see the test failed.

… cpu usage high failure

mssonicbld · 2024-11-25T15:31:40Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/platform_tests/link_flap/link_flap_utils.py:134:1: E302 expected 2 blank lines, found 1 tests/platform_tests/link_flap/test_link_flap.py:7:86: E231 missing whitespace after ',' flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

arista-nwolfe

Change looks good to me, thanks!

tests/platform_tests/link_flap/test_cont_link_flap.py

tests/platform_tests/link_flap/test_link_flap.py

mssonicbld · 2024-12-04T16:33:54Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Failed - hook id: end-of-file-fixer - exit code: 1 - files were modified by this hook Fixing tests/platform_tests/link_flap/link_flap_utils.py Fixing tests/platform_tests/link_flap/test_cont_link_flap.py check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/platform_tests/link_flap/test_link_flap.py:79:23: E127 continuation line over-indented for visual indent flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2024-12-04T16:41:03Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Failed - hook id: end-of-file-fixer - exit code: 1 - files were modified by this hook Fixing tests/platform_tests/link_flap/link_flap_utils.py Fixing tests/platform_tests/link_flap/test_cont_link_flap.py check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Passed flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2024-12-09T06:49:41Z

@wumiaont PR conflicts with 202405 branch

… cpu usage high failure (sonic-net#15732) Description of PR Redis memory check result is not a stable value using "redis-cli info memory | grep used_memory_human". It's found on a stable system (BGO converged, no port flapping etc), the above check could have memory usage difference by more than 0.2M. Followings are CLI output from 202405 and 202205. 202405: admin@ixre-egl-board30: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/' 2.64 admin@ixre-egl-board30: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/' 2.74 admin@ixre-egl-board30: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/' 2.52 202205: admin@ixre-egl-board64: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/' 6.02 admin@ixre-egl-board64: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/' 6.26 admin@ixre-egl-board64: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/' 6.14 We can see that 202405 has some memory optimization for redis and it's not using as much memory as 202205. 0.2M memory usage difference could easily reach the memory usage threshold of 5% in 202405. Solution is to get the average redis memory usage before and after link flap. using 5 seconds interval and 5 times query and then get the average memory usage for redis. Also make the threshold to 10% from 5%. With this fix it's found that the redis memory check will not fail for 2405 after link flap. This commit also provide a fix for sometimes CPU utilization check failed for orchagent after link flap. The reason is in scaling setup (34k routes) orchagent takes more time to calm down. Summary: Fixes # (issue) sonic-net#15733 Approach What is the motivation for this PR? Fix test failures How did you verify/test it? OC tests run with the fix. Did not see the test failed. co-authorized by: jianquanye@microsoft.com

#15955) * Fix for redis memory check failure after link flap and also sometimes cpu usage high failure (#15732) Description of PR Redis memory check result is not a stable value using "redis-cli info memory | grep used_memory_human". It's found on a stable system (BGO converged, no port flapping etc), the above check could have memory usage difference by more than 0.2M. Followings are CLI output from 202405 and 202205. 202405: admin@ixre-egl-board30: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/' 2.64 admin@ixre-egl-board30: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/' 2.74 admin@ixre-egl-board30: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/' 2.52 202205: admin@ixre-egl-board64: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/' 6.02 admin@ixre-egl-board64: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/' 6.26 admin@ixre-egl-board64: redis-cli info memory | grep used_memory_human | sed -e 's/.:(.)M/\1/' 6.14 We can see that 202405 has some memory optimization for redis and it's not using as much memory as 202205. 0.2M memory usage difference could easily reach the memory usage threshold of 5% in 202405. Solution is to get the average redis memory usage before and after link flap. using 5 seconds interval and 5 times query and then get the average memory usage for redis. Also make the threshold to 10% from 5%. With this fix it's found that the redis memory check will not fail for 2405 after link flap. This commit also provide a fix for sometimes CPU utilization check failed for orchagent after link flap. The reason is in scaling setup (34k routes) orchagent takes more time to calm down. Summary: Fixes # (issue) #15733 Approach What is the motivation for this PR? Fix test failures How did you verify/test it? OC tests run with the fix. Did not see the test failed. co-authorized by: jianquanye@microsoft.com * Fix syntax issue --------- Co-authored-by: wumiao_nokia <wu.miao@nokia.com>

Fix for redis memory check failure after link flap and also sometimes…

7545de4

… cpu usage high failure

wumiaont requested a review from prgeor as a code owner November 25, 2024 15:15

Fix pre commit

066db1c

rlhui added the Request for 202405 branch label Nov 27, 2024

rlhui requested review from abdosi and arista-nwolfe November 27, 2024 18:54

rlhui assigned wumiaont Nov 27, 2024

rlhui added the P0 label Nov 27, 2024

arista-nwolfe approved these changes Nov 28, 2024

View reviewed changes

tests/platform_tests/link_flap/test_cont_link_flap.py Outdated Show resolved Hide resolved

arlakshm requested a review from liamkearney-msft December 4, 2024 00:43

liamkearney-msft reviewed Dec 4, 2024

View reviewed changes

tests/platform_tests/link_flap/test_cont_link_flap.py Outdated Show resolved Hide resolved

tests/platform_tests/link_flap/test_link_flap.py Outdated Show resolved Hide resolved

Code change based on comments

4ba5262

Fix

1fcdb52

Fix

6656970

liamkearney-msft approved these changes Dec 5, 2024

View reviewed changes

yejianquan merged commit 700c02b into sonic-net:master Dec 9, 2024
16 checks passed

yejianquan added the Approved for 202405 branch label Dec 9, 2024

mssonicbld added the Cherry Pick Conflict_202405 label Dec 9, 2024

yejianquan mentioned this pull request Dec 9, 2024

Fix for redis memory check failure after link flap and also sometimes… #15955

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix for redis memory check failure after link flap and also sometimes cpu usage high failure #15732

Fix for redis memory check failure after link flap and also sometimes cpu usage high failure #15732

wumiaont commented Nov 25, 2024 •

edited

Loading

mssonicbld commented Nov 25, 2024

arista-nwolfe left a comment

mssonicbld commented Dec 4, 2024

mssonicbld commented Dec 4, 2024

mssonicbld commented Dec 9, 2024

Fix for redis memory check failure after link flap and also sometimes cpu usage high failure #15732

Fix for redis memory check failure after link flap and also sometimes cpu usage high failure #15732

Conversation

wumiaont commented Nov 25, 2024 • edited Loading

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you verify/test it?

mssonicbld commented Nov 25, 2024

arista-nwolfe left a comment

Choose a reason for hiding this comment

mssonicbld commented Dec 4, 2024

mssonicbld commented Dec 4, 2024

mssonicbld commented Dec 9, 2024

wumiaont commented Nov 25, 2024 •

edited

Loading