Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed: Speed up Automated Tests by sharing apks. #3747

Closed
wants to merge 16 commits into from
Closed

Conversation

MohitMaliFtechiz
Copy link
Collaborator

@MohitMaliFtechiz MohitMaliFtechiz commented Mar 12, 2024

Fixes #2096

  • Using uiDevice.waitForIdle() method instead of BaristaSleepInteractions.sleep in our test cases. Because BaristaSleepInteractions.sleep runs everytime even the view is displaying on the screen which takes the extra time to run the test case, but now we are using uiDevice.waitForIdle() method which perform the further operation if desired view is already displaying otherwise wait for that view.
  • Improved the deleting of zim files in localibrary, since we already pause for a second to check if the delete dialog is displayed or not, and again we are waiting before clicking on it. The same issue for while clicking on the PlayStoreRestrictionDialogTest where we are first waiting for the dialog to visible and then waiting for clicking on it. So we have removed the second wait since this dialog is displaying in both test cases.
  • Improved the TopLevelDestinationTest. Because we were testing the drawer options fragments (Settings, Bookmarks etc) from the download screen and when we pressed the back button it came to the download screen and refreshed its data due to this our next functionality had to wait to finish this. Now we are testing all the navigation screens from the LocalLibraryScreen which makes this test case faster.
  • Improved ObjectBoxToLibkiwixMigratorTest for clearing the previously saved bookmarks.
  • Refreshing the data before checking the loaded data in DownloadTest.
  • Increasing the Retry count for loading data and downloading start to properly work with a low internet connection.
  • Added Notification permission in our test cases for API level 33.
  • Improved our ZimHostFragmentTest test to boost the test case speed. It was running in 2 minutes, but after this change, it now completes in 27 seconds.
  • Increased retry count to 5 for flaky test cases on API level 33, where failures occurred multiple times due to the heavy google_apis. so to address this we have increased it to 5 to properly run the test cases.
  • We are removing the ZimHostFragment test from the API level 33 because, most of the time, the emulator does not have WiFi service. Running this test on this emulator is not worthwhile, as we are testing this code on WiFi see https://github.com/kiwix/kiwix-android/actions/runs/8265444407/job/22611097563?pr=3747#step:6:1980.
  • Improved the ObjectBoxToLibkiwixMigratorTest test case, as it occasionally crashes during testing on API level 33 due to emulator lag on google_apis. The extensive data migration exacerbates the emulator's performance issues, leading to crashes during subsequent test processes. To address this issue, we have reduced the migration volume from 10K to 1K.
  • Removed the enabling wifi for adb as it is unused now because sometimes API level 33 emulator does not have the wifi, and we have introduced this for API level 33 emulator .
  • Increased the disk size of the API level 33 emulator since the emulator is crashing on this API level while performing the test cases so it might require more resources to run smoothly.
  • Removed the clearing app data after the first test case failed because it did not receive test cases after the first test failure https://github.com/kiwix/kiwix-android/actions/runs/8295560144/job/22702831374?pr=3747#step:6:3057.
  • Improved system path in our test cases since sometimes the emulator treats app-specific directory as READ_ONLY_FILESYSTEM see https://github.com/kiwix/kiwix-android/actions/runs/8308488756/job/22738768095?pr=3747#step:5:12547.
  • Added timeout for waiting for the idle state of emulator https://github.com/kiwix/kiwix-android/actions/runs/8308488756/job/22738768095?pr=3747#step:5:23824 since by default it has 10 second but the API level 33 emulator is heavy and takes more time to become idle so we have increased the timeout to properly come in the idle state.
  • Added the 'testDirectory' to our scanning process, as on API level 33, the app-specific path is occasionally treated as READ_ONLY. Additionally, for certain test cases where we need to load the zim file in the LocalLibraryScreen, we've included the test directory in our scanning process, as it will be automatically handled by the OS.

Before test cases were completed on API level 33(Android 13) in 18 Minutes and 57 seconds and by doing the above changes now it is completed in 14 minutes on the same API level so almost 5 minutes have been reduced.

@MohitMaliFtechiz MohitMaliFtechiz marked this pull request as draft March 12, 2024 11:00
Copy link

codecov bot commented Mar 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 53.16%. Comparing base (f7a34c1) to head (e1c69aa).
Report is 2 commits behind head on main.

❗ Current head e1c69aa differs from pull request most recent head 29287df. Consider uploading reports for the commit 29287df to get more accurate results

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #3747      +/-   ##
============================================
+ Coverage     52.08%   53.16%   +1.08%     
- Complexity     1268     1291      +23     
============================================
  Files           292      292              
  Lines         10983    10983              
  Branches       1455     1455              
============================================
+ Hits           5720     5839     +119     
+ Misses         4306     4188     -118     
+ Partials        957      956       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@MohitMaliFtechiz MohitMaliFtechiz force-pushed the Fix#2096 branch 2 times, most recently from f6e1b55 to 0526cbb Compare March 14, 2024 05:56
@MohitMaliFtechiz MohitMaliFtechiz marked this pull request as ready for review March 14, 2024 08:43
@MohitMaliFtechiz
Copy link
Collaborator Author

* Improved our CI to generate the test apk, we are using this in our all test cases.
* Added apk to `testInstrumentationRunnerArguments` option which takes the test apk when running the test cases.
…tions.sleep` in our test cases. Because `BaristaSleepInteractions.sleep` runs everytime even the view is displaying on the screen which takes the extra time to run the test case, but now we are using `uiDevice.waitForIdle()` method which perform the further operation if desired view is already displaying otherwise wait for that view.

* Improved the deleting of zim files in localibrary, since we already pause for a second to check if the delete dialog is displayed or not, and again we are waiting before clicking on it. The same issue for while clicking on the PlayStoreRestrictionDialogTest where we are first waiting for the dialog to visible and then waiting for clicking on it. So we have removed the second wait since this dialog is displaying in both test cases.
* Improved the TopLevelDestinationTest. Because we were testing the drawer options fragments (Settings, Bookmarks etc) from the download screen and when we pressed the back button it came to the download screen and refreshed its data due to this our next functionality had to wait to finish this. Now we are testing all the navigation screens from the LocalLibraryScreen which makes this test case faster.
…saved bookmarks.

* Refresh the data before checking the loaded data in DownloadTest.
* Increasing the Retry count for loading data and downloading start to properly work with low internet connection.
* Improved our `ZimHostFragmentTest` test to boost the test case speed. It was running in 2 minutes, but after this change, it now completes in 27 seconds.
…e failures occurred multiple times due to the heavy `google_apis`. so to address this we have increased it to 5 to properly run the test cases.

* We are removing the `ZimHostFragment` test from the API level 33 because, most of the time, the emulator does not have WiFi service. Running this test on this emulator is not worthwhile, as we are testing this code on WiFi.
@MohitMaliFtechiz MohitMaliFtechiz force-pushed the Fix#2096 branch 3 times, most recently from 89657df to 16e8cad Compare March 15, 2024 11:35
@MohitMaliFtechiz MohitMaliFtechiz marked this pull request as draft March 15, 2024 11:38
@MohitMaliFtechiz MohitMaliFtechiz force-pushed the Fix#2096 branch 3 times, most recently from 34e4667 to 026dcac Compare March 15, 2024 17:22
…nally crashes during testing on API level 33 due to emulator lag on google_apis. The extensive data migration exacerbates the emulator's performance issues, leading to crashes during subsequent test processes. To address this issue, we have reduced the migration volume from 10K to 1K.

* Increased the disk size of API level 33 emulator since emulator is crashing on this api level while performing the test cases so it might require more resource to run smoothly.
* Clearing the previous installed application data when launching the test cases for the first time to properly run the test cases.
* Removed the enabling wifi for adb as it is unused now because sometimes API level 33 emulator does not have the wifi, and we have introduced this for API level 33 emulator.
* Removed the APK sharing in test cases since if a text case fails it crashed all the test cases even we re-run that test case in second run.
@kelson42
Copy link
Collaborator

@MohitMaliDeveloper So we can remove the draft?

…uery` test cases on API level 33 since numerous security updates have been included, preventing us from modifying the default behavior of ContentResolver. So running the test case on this APi level leads to test failure.
@MohitMaliFtechiz
Copy link
Collaborator Author

@kelson42 Not now, CI is not stable right now, the last run was unsuccessful(Before the last run CI was green), I have made some new changes in CI. Hope this will stabilize the CI.

… sometimes the emulator treats app-specific directory as `READ_ONLY_FILESYSTEM`.
…ault it has 10 second but the API level 33 emulator is heavy and takes more time to become idle so we have increased the timeout to properly come in the idle state.
* Improved the functionality of closing dialog where any system dialog is visible on the window.
…, the app-specific path is occasionally treated as READ_ONLY. Additionally, for certain test cases where we need to load the zim file in the LocalLibraryScreen, we've included the test directory in our scanning process, as it will be automatically handled by the OS.
@MohitMaliFtechiz
Copy link
Collaborator Author

Duplicate of #3758.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Speed up Automated Tests by sharing apks
3 participants