experiment: attach optional migration expression to actor (class) as actor [exp]?

[crusso]

Support the specification of an optional migration function on the definition of an actor or actor class.

The function must consume and produce records of stable types.

Roughly:

• Fields in the domain of the migration function override any stable variables of the same name in the regular stable signature.
  This determines the pre stable signature of the actor.
• The post stable signature is determined by the actors stable field declarations (as previously).
• Fields in the codomain/range of the migration function that also occur in the regular stable signature of the actor must be consumable (subtypes, ignoring top-level var/non-var distinctions) by the stable variables. Irrelevant, new fields produce errors.
• Fields neither in the domain nor codomain are treated as usual and transferred from the old actor, if present and a subtype, or initialized when absent.

Dynamically, on upgrade, the (implicit or explicit) post signature of the old actor must be compatible with the pre-signature of the new actor. This is either checked offline, statically in the classical compiler (and by Candid deserialization), and dynamically in the EOP compiler, using a pre signature type descriptor. All stable variables in the domain of the migration function must be non-null in the deserialized or transferred stable record. The migration function is applied to the subset of these fields to produce the codomain record. A new stable record is constructed using the values from the codomain and the values that were not overridden by the domain. Values in the domain are set to null (and will be initialized if present in the post signature).
(It would also be possible to transfer values from the domain, if not in the codomain, but required and compatible with the post-signature, but this design choice is not implemented)

A fresh install ignores the migration code; an upgrade applies it before entering the constructor.
All variables in the codomain must be present for the migration to succeed (since there is no way to provide an initial value).

You will need to remove the migration code to do a self-upgrade.

Follow on PRs implement support for EOP (#4829) and syntactic stable signatures (#4833) (both merged here but might simplify reviewing)

Extended stable signatures

A stable signature (stored in metadata) is now either a single interface of stable fields (as before) or (the new bit) a dual interface recording the pre and post upgrade interface when performing migration.

The pre and post fields are implicitly identical for a singleton interface and provide for backwards compatibility and ordinary upgrades (sans explicit migration).

Stability compatibility now checks the post-signature of the old actor is compatible with the pre-signature of the new actor.

<stab_sig> ::=
  <typ_dec>;* actor { <stab_field>;* }
  <typ_dec>;* actor ( { <stab_field>;* },  <stab_field>;* } )

Done this way, there should be no need to modify dfx, which defers to moc for the compatibility check anyway.

Example: adding record fields

To upgrade from:

persistent actor {
  type Card = {
    title : Text;
  };
  var map : [(Nat32, Card)] = [];
  var log : []
};

to incompatible stable type interface:

persistent actor {
  type Card = {
    title : Text;
    description : Text; // new field!
  };
  var map : [(Nat32, Card)] = [];
};

1. Define a migration module and function that transform the old stable variable, at its current type, into the new stable variable at its new type.

// CardMigration.mo
import Array "mo:base/Array";

module CardMigration {
  type OldCard = {
    title : Text;
  };

  type NewCard = {
    title : Text;
    description : Text;
  };

  // our migration function
  public func migrate( old : {
      var map : [(Nat32, OldCard)] // old type
    } } :
    {
      var map : [(Nat32, NewCard)] // new type
    } {
    { var map : [(Nat32, NewCard)] =
        Array.map<(Nat32, OldCard), (Nat32, NewCard)>(
          old.map,
          func(key, { title }) { (key, { title; description = "<empty>" }) }
  );

};

2. Specify the migration function as the migration expression of your actor declaration:

import CardMigration "CardMigration";

persistent actor
  [ CardMigration.migrate ] // Declare the migration function
  {
  type Card = {
    title : Text;
    description : Text;
  };

  var map : [(Nat32, Card)] = []; // Initialized by migration on upgrade
  var log : [Text] = []; // migrated implicitly as usual

};

After we have successfully upgraded to this new version, we can also upgrade once more to a version that drops the migration code.

persistent actor {
  type Card = {
    title : Text;
    description : Text;
  };

  var map : [(Nat32, Card)] = [];
  var log  : [Text] = []
};

TODOs

• Decide on final syntax, if keeping this one, perhaps tuck the migration expression into the obj_sort (to reduce diffs)
• Review and address remaining TODOS (some are actually done)
• Normalize and promote types of the migration function as appropriate.
• Check for field hash collisions
• Use better names in code (pre_ty, post_ty etc)
• Test actor classes
• Test more scenarios: variable retyping, variable removal, swap etc
• Devise some mechanism to detect repeated migration with the same migration code, or a coding pattern to prevent it?
  Some, but not all migrations might be reject as type-incompatible, but others may not be:
  * Maybe an implicit isMigrated field is enough, or a check that the post-signature of the new "not equals" the post-signature of the old canister?
• review/adjust error codes.
• fig grammar.sed to delete new production
• add some tests to test/cmp (tests: extended --stable-compatible checks #4850)
• avoid capture of shared pattern variables in migration function (or revise syntax)
• doc (Draft doc: migration expressions #4845)
• consider requiring migration exp to be static (thus pure)

[ggreif]

Did you consider persistent actor A (with migration = mf) { ... };?

[crusso]

Did you consider persistent actor A (with migration = mf) { ... };?

Not yet, but I also don't want A to be in scope in mf.

Maybe

persistent actor (with migration = mf) A { ... };

Or

(with migration = mf)
persistent actor  { ... };

But I expect parsing hell from the last one.

Are parentheticals only meant to attach to expressions?

[ggreif]

Are parentheticals only meant to attach to expressions?

Not specifically. Any AST node can be it.

[crusso]

Why a different error code from above? Cut'n'paste error?