UpdateTime in Spanner Writing is Slightly in the Future

[renjiezh]

Describe the bug
This is found in rpc recordBlobRequests. The UpdateTime in the spanner writing operation is slightly after the database current time, thus this writing fails. The clock of the server is not strictly consistent with the database.

It happened 6 times within 1 minute and recovered.

Steps to reproduce
Run stress test and there is a chance to reproduce.

Component(s) affected
Duchy

Version
v0.5.7-rc4

Environment
QA env

Additional context
Error message example:

Cannot write timestamps in the future 2024-08-09T10:19:56.098900141Z > 2024-08-09T10:19:56.098625Z (current time) because the allow_commit_timestamp column option is set to true for column Computations.UpdateTime, or for a corresponding shared key column in this table's interleaved table hierarchy.

global computation id example: BOxRTMtdgo8

[SanjayVas]

The Spanner error is just an indication of a broader bug: update time is not monotonically increasing. Note that this must not just be monotonically increasing for a single resource, but also across resources of the same type.

The following pattern should be possible:

1. List all resources of some type updated since time X
2. Record the latest seen update time from (1) as Y
3. List all resources of the same type updated since time Y

The result of (3) must not miss any resources, and Y >= X must hold. Therefore, no write with update time <= Y can be committed after a write with update time Y.

I believe this bug affects both Spanner and Postgres Duchy implementations.