Cleanup pointer serialization #16851

Job	Run time
build (8.10.7, ubuntu-latest)	28m 57s
cabal-format	5s
Check branch history	11s
build (9.2.8, ubuntu-latest)	16m 4s
fourmolu	44s
build (9.6.6, ubuntu-latest)	17m 25s
build (9.8.4, ubuntu-latest)	12m 20s
build (9.10.1, ubuntu-latest)	11m 10s
gen-hie	5m 11s
Check if formal-ledger-specifications SRP is valid	11s
${{ matrix.package }}-${{ matrix.ghc }}-${{ matrix.os }}	0s
Tests completed	1s
Send a slack notification on \#ledger-internal if the nightly build failed	0s
	1h 32m 19s

Provide feedback