Improve edx-platform requirements file structure

[iamsobanjaved]

edx-platform's requirements files pretty closely follow the recommendations in OEP-18. But as Kyle McCormick pointed out recently, the pip-tools README has since been updated to recommend something a little different. Try making the following changes:

1. Benchmark the performance of make upgrade and make requirements before making any changes.
2. Instead of basing each .in file on the previous fully pinned output file via -r, use -c instead. This should result in a requirements file that has just the dependencies needed for the packages listed in the file currently being processed, but at versions compatible with the previous layer of dependencies.
3. Rename the output files generated from an input file that builds on top of a previous layer with a _layer suffix, like testing_layer.txt.
4. Introduce new files with the original output names that include all the dependencies the previous ones did; for example, the new testing.txt would consist of the lines -r base.txt, -r coverage.txt, and -r testing_layer.txt.
5. Benchmark the performance of make upgrade and make requirements after the changes.

What we hope to gain from this change are the following:

• Faster make upgrade performance by not having to process as many dependencies for each layer
• Better ease of reviewing dependency updates, since most dependency versions will only appear in a single requirements file, whereas many currently appear in 3 or more.

In the process, we will lose having a single file (development.txt) that lists essentially all of the packages used by edx-platform. But we rarely use this, and we could easily reproduce it by running pip-compile on the new development.txt.

If the results are good, write new tickets for the following:

• An update to OEP-18 to recommend this new structure for requirements files
• Updating all repos owned by arch-bom to comply with the new structure (these could be either BOM or ARCHBOM tickets)
• Communicating the change to the rest of the organization

Copied description from this private JIRA issue: https://2u-internal.atlassian.net/browse/BOM-2483

[iamsobanjaved]

There is a WIP PR that went stale (re-open the PR and make it ready):

• feat: improve requirements file structure #31085

There is a suggestion on the PR about how to continue the work: #31085 (comment)