Upgrade checkpoints during CI testing

[aarmejach]

Currently, if a PR modifies the checkpoint format CI testing will fail until the checkpoint is manually upgraded.

It would be great that if a PR also includes a cpt_upgrader patch, all checkpoints used for CI testing apply the cpt_upgrader before running the test, enabling seamless CI testing for such PRs.

[powerjg]

I think this is a great idea. However, I don't have any ideas on how to implement it. Do you have any ideas for pointers to get started on this?

If you're feeling adventurous, a draft PR to update the github workflow would be much appreciated!

[powerjg]

Sorry, I guess I didn't use my brain before I responded... Thinking about this a bit more...

Do you think it would work to do the following

1. Never update the checkpoint resources that are stored in our google bucket between releases
2. For every checkpoint test, download the checkpoint resource and then run the checkpoint upgrader before running the test

• Would it work so that the checkpoint upgrader would do nothing if the checkpoint is on the right "version"?
• Can we update a checkpoint across multiple checkpoint upgraders at once?
• Will this take a significant amount of CPU time?

3. When release time comes, run the checkpoint upgraders and upload new checkpoints that are compatible with the new release.

I am not very familiar with the checkpoint upgrader, so I hope that someone with more knowledge can step in and check to see if this makes sense.

[aarmejach]

Do you think it would work to do the following

1. Never update the checkpoint resources that are stored in our google bucket between releases
2. For every checkpoint test, download the checkpoint resource and then run the checkpoint upgrader before running the test
3. When release time comes, run the checkpoint upgraders and upload new checkpoints that are compatible with the new release.

Yes, this is exactly what i had in mind. Sorry my previous description was a bit vage.

• Would it work so that the checkpoint upgrader would do nothing if the checkpoint is on the right "version"?

yes

• Can we update a checkpoint across multiple checkpoint upgraders at once?

yes, it runs all pending upgrades. A bit of testing on this would be good, but that is how it is meant to work.

• Will this take a significant amount of CPU time?

I don't think so, should be marginal wrt any of the tests using a checkpoint.

[BobbyRBruce]

Better idea (unless i'm missing something): Why don't we just embed the upgrader in gem5 and if gem5 is presented with an old checkpoint will parse upgrade it prior to loading. That we we can use old checkpoints indefinitely.

This way if a checkpointing test fails, you only need to update the internal upgrader.

[aarmejach]

I think that is OK, but it is a bit more intrusive.

I am a bit paranoid, but I'd rather keep my checkpoints untouched and get an error, than getting checkpoints upgraded automatically (without me explicitly choosing to do so). Even if a message is printed saying checkpoints have been upgraded, it can be easily missed on batch runs, and maybe results are not reproducible for whatever reason anymore.

[BobbyRBruce]

I think that is OK, but it is a bit more intrusive.

I am a bit paranoid, but I'd rather keep my checkpoints untouched and get an error, than getting checkpoints upgraded automatically (without me explicitly choosing to do so). Even if a message is printed saying checkpoints have been upgraded, it can be easily missed on batch runs, and maybe results are not reproducible for whatever reason anymore.

Is this really a valid thing to be paranoid about? It seems to be a moot point being paranoid about checkpoints changing if you're using checkpoints generated from version X of gem5 and use them in Version >X. Across gem5 versions literally anything can change.

[aarmejach]

Been thinking about situations where this could be problematic, and you are probably right, i.e.,cChanging versions usually means updating checkpoints.