Code

Write JSON state#997

Code

Merged

KaspariK merged 22 commits intomasterYelp/Tron:masterfrom u/kkasp/TRON-2237-write-json-stateYelp/Tron:u/kkasp/TRON-2237-write-json-stateCopy head branch name to clipboardOct 29, 2024

+555-268

Commits

Commits on Oct 29, 2024

• Add types to DynamoDB restore flow. Add Persistable abstract class and to_json methods for classes in DynamoDB restore flow. Write an additional attribute to DynamoDB to capture non-pickled state_d…

  Show description for b4a7e16
  KaspariK

  committed
  b4a7e16

  Copy full SHA for b4a7e16

• Remove old comments. Delete mcp state metadata.

  KaspariK

  committed
  c081772

  Copy full SHA for c081772

• Return these

  KaspariK

  committed
  88de418

  Copy full SHA for 88de418

• Try itest validation without mcp_state

  KaspariK

  committed
  909e9b2

  Copy full SHA for 909e9b2

• Fix isinstance check for Action to_json

  KaspariK

  committed
  1a5238b

  Copy full SHA for 1a5238b

• Comment cleanup. Add cleanup TODOs for TRON-2293. Rename val in dynamodb_state_store to something a little more explanatory now that we have 2 versions

  KaspariK

  committed
  c18cce4

  Copy full SHA for c18cce4

• Remove shelve key test since we no longer save mcp_state or state_metadata

  KaspariK

  committed
  7fb271d

  Copy full SHA for 7fb271d

• Remove deprecated itest. Rename some variables to make things a bit clearer.

  KaspariK

  committed
  677d6e3

  Copy full SHA for 677d6e3

• Remove unnecessary list call

  KaspariK

  committed
  e262cbc

  Copy full SHA for e262cbc

• Replace ignore with proper types for sorted_groups

  KaspariK

  committed
  83d7c65

  Copy full SHA for 83d7c65

• Remove skip_validation since removing MCP StateMetadata

  KaspariK

  committed
  002f935

  Copy full SHA for 002f935

• Add more types and a few TODO tickets

  KaspariK

  committed
  f5ab20c

  Copy full SHA for f5ab20c

• Remove debug logs, add constant for transact_write_items, update transaction limit, and change setitem signature for take value tuple

  KaspariK

  committed
  00a50a9

  Copy full SHA for 00a50a9

• Check that rendered_command and docker_image are not None before creating task

  KaspariK

  committed
  0eca03f

  Copy full SHA for 0eca03f

• Remove comment from before my enlightenment

  KaspariK

  committed
  d727236

  Copy full SHA for d727236

• Add remaining types, add tickets to valid TODOs, and delete irrelevant TODOs

  KaspariK

  committed
  a02aedc

  Copy full SHA for a02aedc

• Fix time_zone type. Remove unreachable code after typing trigger_downstreams

  KaspariK

  committed
  f26f855

  Copy full SHA for f26f855

• Maintain most of the parsing logic as is. Add a few more tests. Add TODO and ticket for regex issues

  KaspariK

  committed
  c3c8c77

  Copy full SHA for c3c8c77

• Add error handling in various to_json() funcs. Break JSON write out from pickle write so that we maintain writing pickles if JSON fails

  KaspariK

  committed
  7a0cb98

  Copy full SHA for 7a0cb98

• Update to_json exception logging. On DynamoDB write failure only pass the pickle back into the queue

  KaspariK

  committed
  e79ae25

  Copy full SHA for e79ae25

• Raise these so that we get the full picture on failure and can decide what to do in dynamodb_state_store

  KaspariK

  committed
  e8dcc8e

  Copy full SHA for e8dcc8e

• Add serialization error counter

  KaspariK

  committed
  b0186d1

  Copy full SHA for b0186d1