Implement support for externally developed plugins

[jpower432]

Continuing the conversation originating from oscal-compass/compliance-to-policy#23. The goal would be enable plugins to be developed externally without having to recompile C2P and reuse the plugins being developed in C2P Python.

Objectives

Define a plugin interface will allow externally developed plugins to be used with C2P core while allowing the reuse of existing plugins being developed in C2P Python.

Based on previous conversation with @yana1205, proposing the core plugin management logic be implemented in Go and the use of RPC based plugins.

Rationale

• External plugins allow more flexibility on what types of automation can be integrated and can be helpful when the automation or policy engine implementation is specialized (i.e. not general enough to be contributed back to C2P)
• Less to maintain in C2P and reduces the chance of bloat for every new implementation's dependencies
• Allows plugins to be reused in the different languages C2P supports. For example, the Python Auditree plugin could be reused.

Completion Criteria

I have added some high-level features I am proposing to support below. Some are brand new to C2P and some are based on the features of C2P Python for consistency.

Parity with C2P Python

• Add a plugin spec for interacting with policy engines (PVP) to include policy generate and results transformation Goal being that it is flexible to allow policy validation
• Add support for heterogeneous environments #11
• Support the plugin lifecycle for external plugins #9

New

• Add a plugin spec for interacting with remediation engines to provide remediation instructions/artifacts or implementation to remediate findings detected and stored in OSCAL Assessment Results during the scanning process. Proposing that the default behavior with no finding input be that all configured remediation data be generated or implemented.
• Add support for Assessment Plan as an input for RuleSet information (This may be partially implemented in oscal-sdk-go as a transformation from Component Definitions to Assessment Plans)

EDITED BELOW

Additional Notes:

• If go-plugin is being considered, a license exception is in place with CNCF. We may need to submit a license exception with our use case information.

[jpower432]

Adding a link to my fork where I am testing out some of these concepts - https://github.com/jpower432/compliance-to-policy-go/tree/feat/v2-module/v2

[jpower432]

@yana1205 @yuji-watanabe-jp If you are in support of this change, I am wondering how to get started on working on a contribution.

I realize that I am currently using a /v2 module in my fork. However, given that compliance-to-policy-go is still at v0, I believe incrementing to v2 would be unexpected since it would need corresponding v2.x.x tags in the repository and there are not v1.x.x tags.

I'm think there are a couple approaches we could take for this change:

• Modify the Existing Module: Make necessary changes to the current v0 module. If we choose this, I assume we should make a release first with the module as it is currently.
• Create a New Module: Develop a separate module within the repository if there is not enough overlap between the two modules (or they might be complementary).

Your input on the approach would be greatly appreciated.

[yana1205]

@jpower432 Thanks. The first one is reasonable because it avoids the overhead of managing multiple modules. v0 in Go is considered unstable, so users expect potential breaking changes when moving to v1. If the project needs some time before v2 is fully ready, I can add v1.0.0 tag to the current repository and it would be better to treat the v2 codes as v2.x version.
Once v2 codes is ready, we create v1 branch dedicated for v1 code, while "main" is dedicated for v2 codes by replacing all Go contents under the root directory with the contents under /v2. If any request for the v1 code base happens, we continue to update v1 branch with v1.x tag, similar to other projects (go-yaml, ginkgo).

[jpower432]

@yana1205 I appreciate your input and clarifications. I found the projects you linked helpful. Another project that manages this similarly to what you proposed is oras-go.

I agree the most straightforward solution while allowing for clear separation would be to tag the current repository with a v1.0.0 version. To ensure we are aligned on next steps, I wanted to restate the plan you proposed below with one small change:

1. Cut the first stable release of C2P Go and create a v1 branch
2. Submit a PR to update the README.md on the project status stating that v1 is in stable mode and can be found in the v1 branch and main is v2 in experimental mode. Similar to this approach here.
3. Bump the module version on main to v2
4. Once v2 is in a stable state, we can just remove the experimental information from the README.md

I am advocating to create the v1 branch now instead of when v2 is in a stable state so we do not have to manage pipelines and configuration files for both model versions on the same branch only to split them out later anyway. What do you think?

[yana1205]

@jpower432
Thanks for summarizing it! Yes, the plan sounds good and I totally agree with following the oras-go like documentation.