*: add invocations to applicationlog

[ixje]

Problem

neo-project/neo#3386

Solution

Implement as extension. Moved the discussion from Dora's backend PR to here

To do

• copy arguments to avoid modifications
• limit the total number of argument stack items in a single transaction (for safety)
• make this a configurable feature
• include native contract calls

Do we want to limit the stack item depth (think MaxJSONDepth) or are we content with just limiting the total stack arguments?

[AnnaShaleva]

A good prototype, but I have several design questions that we should solve before review.

[AnnaShaleva]

@roman-khimov, I think we need some third opinion on these topics.

[roman-khimov]

How about System.Runtime.LoadScript calls, btw?

[AnnaShaleva]

How about System.Runtime.LoadScript calls

It leads to new execution context creation, thus it's a valid part of invocation tree. But is this information useful in practice? Dynamic invocations are identified by hash160 of the loaded script, as a result user can't get this script because he knows only its hash. But still we may include dynamic invocations into the resulting Invocations tree with some special field like isContractCall: false.

[ixje]

Picking this up again. I rebased the branch onto latest master and processed some of the feedback. In particular

• use stackitem.Serialize instead of deepcopy and re-use the results when storing the data
• make the behaviour configurable through a SaveInvocations config option

Note; It was unclear to me based on #3569 (comment) if I should have made it a tree or keep it flat. I kept it flat for now.

If the feature is enabled the applicationlog output looks as follows

"invocations": [
                    {
                        "contract_hash": "0xd2a4cff31913016155e38e474a2c06d08be276cf",
                        "method": "transfer",
                        "arguments": {
                            "type": "Array",
                            "value": [
                                {
                                    "type": "ByteString",
                                    "value": "krOcd6pg8ptXwXPO2Rfxf9Mhpus="
                                },
                                {
                                    "type": "ByteString",
                                    "value": "AZelPVEEY0csq+FRLl/HJ9cW+Qs="
                                },
                                {
                                    "type": "Integer",
                                    "value": "1000000000000"
                                },
                                {
                                    "type": "Any"
                                }
                            ]
                        },
                        "arguments_count": 4,
                        "is_valid": true
                    }
                ]

and in disabled state it returns

"invocations": []

I'm looking for feedback on the above before taking care of covering System.Runtime.LoadScript calls

[ixje]

@AnnaShaleva can this PR also get some review love please

[roman-khimov]

I don't see any limits for the overall size of saved data. While the feature is optional we still need to protect node from abuse.

[ixje]

ping

[ixje]

I feel there is a lot of resistance against this PR, but I can't really tell why. Other PRs I opened (later than this one, like #3674 and #3677) were swiftly reviewed (multiple times). This one however takes 2-3 weeks per update to get another response despite multiple pings.

Can any of you @AnnaShaleva @roman-khimov elaborate please? If I understand where this resistance is coming from I can perhaps do something about it.

[roman-khimov]

@ixje, zero resistance. Sorry, it's just that there are too many things to handle at once. @AnnaShaleva will get back to it soon.

[AnnaShaleva]

@ixje sorry for the delay. Actually, when we don't need #3569 (comment) and #3569 (comment) to be fixed, there are only a couple of non-critical issues left to be fixed, so the PR is almost ready and looks good.

[ixje]

@AnnaShaleva some RPC server tests and TestNEO_CommitteeEvents are failing due to changes requested by you in this comment. If I do write this 1 byte for the length then they pass (with some small updates where necessary).

I've spend some time trying to understand why TestNEO_CommitteeEvents is failing and for some reason the data fetched from the dao does contain the execution events, despite the body of this logic never being called (as per breakpoints not hitting in the debugger)

	if invocLen := len(aer.Invocations); invocLen > 0 {
		w.WriteVarUint(uint64(invocLen))
		for i := range aer.Invocations {
			aer.Invocations[i].EncodeBinaryWithContext(w, sc)
		}
	}

Putting a breakpoint in EncodeBinaryWithContext of ContractInvocation is also not hit. I'm not sure yet what/where I'm overlooking something. Any help/insight there is appreciated.

[codecov]

Codecov Report

Attention: Patch coverage is 62.39316% with 44 lines in your changes missing coverage. Please review.

Project coverage is 83.04%. Comparing base (e8b8c1a) to head (4b2ee9a).
Report is 221 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/core/state/contract_invocation.go	76.92%	10 Missing and 5 partials ⚠️
pkg/core/state/notification_event.go	20.00%	9 Missing and 3 partials ⚠️
pkg/core/interop/contract/call.go	9.09%	9 Missing and 1 partial ⚠️
pkg/core/blockchain.go	33.33%	3 Missing and 1 partial ⚠️
pkg/core/dao/dao.go	25.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3569      +/-   ##
==========================================
- Coverage   83.20%   83.04%   -0.17%     
==========================================
  Files         334      336       +2     
  Lines       46488    47005     +517     
==========================================
+ Hits        38681    39034     +353     
- Misses       6234     6373     +139     
- Partials     1573     1598      +25     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[AnnaShaleva]

Everything is almost done, so once finished, please, format commits according to https://github.com/nspcc-dev/.github/blob/master/git.md#commits and add a proper signoff to the commit messages.

[ixje]

Everything is almost done, so once finished, please, format commits according to https://github.com/nspcc-dev/.github/blob/master/git.md#commits and add a proper signoff to the commit messages.

I'll wait with the squashing and signoff until everything is really approved. I thought I was close a couple of times before and then the next review requests changes over parts that 'passed' the previous reviews.

[AnnaShaleva]

LGTM. Let's squash and reformat commits.

[AnnaShaleva]

Congratulations!

Let's wait for the testing jobs to finish before the merge.

[ixje]

🎉
Very happy to get this merged in time for the 0.108 release. Thanks for all the reviews, this was a lengthy one 💦