Correctness QA failed: Anchor Not Detected

[stephhuynh18]

Context: https://discord.com/channels/484729862368526356/1168910129278750740/1295665827965567006

[stephhuynh18]

2024-10-15T08:15:11.1271315Z   console.log
2024-10-15T08:15:11.1271814Z     Creating documents
2024-10-15T08:15:11.1272067Z 
2024-10-15T08:15:11.1272650Z       at Object.<anonymous> (src/__tests__/correctness/slow/ceramic-cas-integration.test.ts:69:13)
2024-10-15T08:15:11.1273216Z 
2024-10-15T08:15:11.2752917Z   console.log
2024-10-15T08:15:11.2753384Z     Waiting for anchor of genesis records
2024-10-15T08:15:11.2753729Z 
2024-10-15T08:15:11.2754711Z       at Object.<anonymous> (src/__tests__/correctness/slow/ceramic-cas-integration.test.ts:80:13)
2024-10-15T08:15:11.2755304Z 
2024-10-15T08:15:11.2780740Z   console.debug
2024-10-15T08:15:11.2784440Z     Waiting for stream kjzl6kcym7w8y8wg16fsp5a2z5pos9id9wxr66mz07zu0ka92xn2a2mwj4qnjz4 to be anchored. Current time: 2024-10-15T08:15:11.275Z. Current stream state: ***"type":3,"content":***"step":0***,"metadata":***"controllers":["did:key:z6MknUPMwE7vNNUkZvpTGEBa2dEH91kjZfteBdFJYAaAnW7B"],"model":"kjzl6hvfrbw6c6saoihupac29mc2casslccf1arjf2vwyxthwaxxbfgdmb7s2u3","unique":"usHYXEw5JqVICmDj"***,"signature":2,"anchorStatus":"PENDING","log":[***"cid":"bagcqceraue47dz2dt35mdecfzqdpj7c66v4vclpsqeyg6p5irjtp5gqj33qa","type":0***],"doctype":"MID"***
2024-10-15T08:15:11.2787339Z 
2024-10-15T08:15:11.2787702Z       at src/utils/ceramicHelpers.ts:107:17
2024-10-15T08:15:11.2788135Z 
2024-10-15T08:30:11.1692035Z FAIL src/__tests__/correctness/slow/ceramic-cas-integration.test.ts (1362.402 s)
2024-10-15T08:30:11.1695465Z   Ceramic<->CAS basic integration
2024-10-15T08:30:11.1696542Z     ✓ basic crud is anchored properly, single update per anchor batch (460964 ms)
2024-10-15T08:30:11.1697705Z     ✕ multiple documents are anchored properly, multiple updates per anchor batch (900001 ms)
2024-10-15T08:30:11.1698271Z 
2024-10-15T08:30:11.1698908Z   ● Ceramic<->CAS basic integration › multiple documents are anchored properly, multiple updates per anchor batch
2024-10-15T08:30:11.1699582Z 
2024-10-15T08:30:11.1699821Z     thrown: "Exceeded timeout of 900000 ms for a test.
2024-10-15T08:30:11.1700990Z     Add a timeout value to this test to increase the timeout, if this is a long-running test. See https://jestjs.io/docs/api#testname-fn-timeout."
2024-10-15T08:30:11.1701800Z 
2024-10-15T08:30:11.1702010Z       58 |   ***)
2024-10-15T08:30:11.1702363Z       59 |
2024-10-15T08:30:11.1703107Z     > 60 |   test('multiple documents are anchored properly, multiple updates per anchor batch', async () => ***
2024-10-15T08:30:11.1703877Z          |   ^
2024-10-15T08:30:11.1704276Z       61 |     const content0 = *** step: 0 ***
2024-10-15T08:30:11.1704786Z       62 |     const content1 = *** step: 1 ***
2024-10-15T08:30:11.1705280Z       63 |     const content2 = *** step: 2 ***
2024-10-15T08:30:11.1705616Z 
2024-10-15T08:30:11.1706014Z       at src/__tests__/correctness/slow/ceramic-cas-integration.test.ts:60:3
2024-10-15T08:30:11.1706859Z       at src/__tests__/correctness/slow/ceramic-cas-integration.test.ts:16:1

aws logs:

[2024-10-15T08:19:06.149Z] INFO: 'Created anchor commit with CID bafyreigfoi5bs37ccln7ixgotrbzk77egdvts4qtgycilsipq6ldfugjda for commit bagcqceraue47dz2dt35mdecfzqdpj7c66v4vclpsqeyg6p5irjtp5gqj33qa of stream kjzl6kcym7w8y8wg16fsp5a2z5pos9id9wxr66mz07zu0ka92xn2a2mwj4qnjz4'

[2024-10-15T08:19:10.599Z] INFO: 'Found request for bagcqceraue47dz2dt35mdecfzqdpj7c66v4vclpsqeyg6p5irjtp5gqj33qa of kjzl6kcym7w8y8wg16fsp5a2z5pos9id9wxr66mz07zu0ka92xn2a2mwj4qnjz4 created at 2024-10-15'

So successfully anchored by 8:19 and found at 8:19 but node didn't see it

[stephhuynh18]

Delegated anchoring code might be some kind of bug where the tree didn't get sent timely (could be a misconfiguration)