Multithreading & Caching Optimizations for RefreshIG and ExtractMatBundleOperation #481
Conversation
…now includes fhir server entry in arguments.
…ientUtils to collect POST calls and execute them at the end.
…ses for thread safety. Declaring certain variables outside of loops for stability.
…into refresh-ig-caching
…ty reasons (ddos false flag)
…needed due to the nature of modifications of each by the process.
…etter performance.
…usted IOUtils.copyFile to return false if the file wasn't copied, true if it was, to accurately track how many files get copied during TestCaseProcessor.bundleTestCaseFiles
…actMatBundleOperation with multithreading, updated matching test accordingly
… file count due to old approach to eliminating duplicate resource processing
…y issues in tests. IGBundleProcessor checking fhirUri for null before reference. RefreshIGOperationTest using wiremock to monitor post calls to mock fhir server. pom.xml updated for wiremock. Will possibly try different libraries for similar approach.
…LibraryDependenciesTasks. Adjusted error handling for clear information in console when shouldPersist finds an error.
…evious approach to IOUtils.getFilePaths (caching retained) because readability exceeds any performance benefit from Files.walk
… to reduce duplication of code
There was a problem hiding this comment.
pom was updated to include wiremock which lets me prevent actual calls to urls during refreshIG and test the POST calls
There was a problem hiding this comment.
This gives a really simple way to take in a list of tasks, send them off on their own thread, then wait for each to finish.
There was a problem hiding this comment.
updated bundleLibraryDependencies to be multithreaded for performance enhancement
There was a problem hiding this comment.
MeasureProcessor updated to use multithreading. This class extended BaseProcessor for other functions. Possible refactor should include moving those functions to their own class and let it extend BaseProcessor instead of the Abstract parent class extending BaseProcessor.
There was a problem hiding this comment.
ExtractMatBundleOperation modified to use multithreading as well as enhanced the directory processing so all child folders are processed recursively. If only X levels of child folders should be looked at, I can make adjustments.
There was a problem hiding this comment.
bundleTestCases enhanced with caching for performance improvements and modified to return informational messages about the number of files copied. IOUtils.copyFile is a boolean method and uses its own caching, where if the file is not copied it returns false.
There was a problem hiding this comment.
An Abstract class housing the bundle method, and persistBundle method that QuestionnaireProcessor, PlanDefinitionProcessor, and MeasureProcessor all once had. The similarities between the methods in each class were such that a common abstract class could be formed.
There was a problem hiding this comment.
Modified bundleValueSets so that if an exception is hit, it's immediately thrown so that failure on this method can have more detailed information for the user during the bundle process. A list of failed items alongside their reason are displayed at the end of the process.
There was a problem hiding this comment.
Modified for the ExtractMatBundle process to avoid duplicate resources from being processed (previous approach was unreliable.)
There was a problem hiding this comment.
This class was heavily modified so that as the refreshIG process proceeds, any post calls made in the persist methods are actually stored in a list of tasks to be processed at the very end.
There was a problem hiding this comment.
copyFile modified to be a boolean method, indicating when the copy was successful or not. A basic counter was added so the user can be informed of the exact number of files copied at the end of the process. copyFile also now uses caching based off a map where the input and output paths are used as a unique key. This improves performance substantially.
methods such as getResourceFileName and getPaths have had caching introduced for performance improvements.
There was a problem hiding this comment.
info method modified to simply output the message using System.out.println. This brings the information back to the console for the user.
There was a problem hiding this comment.
Modified for ExtratMatBundle to avoid processing duplicate resources
There was a problem hiding this comment.
This test was failing in Windows. I modified it so it runs the same way in windows as it does in macos, considering the file separator. Now building locally on windows machines works.
There was a problem hiding this comment.
With the persist modifications, added wiremock to avoid actual calls to the test server and instead count the post calls to ensure they're accurate.
There was a problem hiding this comment.
Utilizing new method name
There was a problem hiding this comment.
Fixed up test based on existing RefreshIG test.
|
Closing this in favor of #486 |
Applied multithreading to various processes to enhance performance, and implemented some caching optimizations to resolve duplicate processing.
RefreshIG also now persists "tests-" files along with measures when a fhir server url is provided.
ExtratMatBundleOperation's -dir argument can be used against a single folder and all subfolders
By creating this PR you acknowledge that your contribution will be licensed under Apache 2.0