Parse (expr) step to extract JSON/YAML file contents

[jessesuen]

Proposed Feature

It has been requested to extract a portion of a file's contents after cloning, so that the file's contents can be used in a subsequent step. e.g. something like a parse-file step.

- uses: git-clone
  config:
    repoURL: ${{ vars.gitRepo }}
    checkout:
    - commit: ${{ commitFrom(vars.gitRepo, warehouse("base")).ID }}
      path: ./src
- uses: parse-file   # <<< NEW
  as: app_versions
  config:
    outputs:
    - path: config/app1/version.json
      fromExpression: "object.version"   # object would the file in object form (open to other names)
      name: app1
    - path: config/app2/version.yaml
      fromExpression: "object.version"   # object would the file in object form (open to other names)
      name: app2
- uses: yaml-update
  config:
    path: ./src/charts/my-chart/values.yaml
    updates:
    - key: app1.image.tag
      value: ${{ outputs.app_versions.app1 }}
    - key: app2.image.tag
      value: ${{ outputs.app_versions.app2 }}

Motivation

This would allow Kargo to use git repository monitoring feature in a more powerful way. e.g. A user could configure Kargo to watch a single file in for changes, similar to how we watch a OCI/Helm/Git repo for changes, but then act on specific changes to that file, possibly even in a different repo.

We already have a copy step which is close to this and consider this a variation of thecopy step.

Suggested Implementation

Implementation will probably be similar to our copy step, which deals with files on disk.

The big question would be how to extract information from a file. But we should be as good as:

• expr
• grep
• sed
• awk
• jq (not needed. use expr instead)
• yq (not needed. use expr instead)

EDIT:
Expr jq/yq should be our choice for structured data.

unstructured data may need grep/sed/awk-like syntax, but will be considered out of scope for this feature.

[krancour]

For the http step, we allowed expr-lang expressions to be used to extract stuff from the structured response body.

I'm not 100% against exec'ing out to jq/yq, but reusing the approach the http step used seems like a nice starting point because it introduces no new binaries to the image and has the added benefit of being consistent with something we've already done.

Unstructured data is a different story...

Is there a specific use case for unstructured data being used in a promotion process?

It might be worth tackling structured first and revisiting unstructured later.

[jessesuen]

Great suggestion. I forgot that expr already has a very powerful way to extract data, and we have precedent with http. I like the idea of using that over yq or jq syntax, so we can make a call to use expr as the method for extracting structured data.

I also agree we should try not to exec out to other binaries. Luckily, I think there would be a go library for all forms of parsing we would want to support.

Let me check with some stake holders how important dealing with unstructured data is.

[jessesuen]

Let me check with some stake holders how important dealing with unstructured data is.

I checked, and JSON/YAML seem to be the only need. So, I'm reducing the scope further to just support JSON/YAML files with expr extraction.