Add the ability to transform and split the results of the HTTP body

[jhorbulyk]

Special case of https://github.com/elasticio/platform/issues/77 This is a particularly common scenario for the REST and possibly SOAP API cases.

Consider the following flow case:
Simple Trigger -> REST API call that returns large array -> Splitter -> ...

Imagine that the REST API call returns an array with total size 1 MB with 9000 entries. Given that for each object produced by Splitter includes the entire result of the REST API call in every message emitted from the splitter, the above flow would produce 9 GB of data on execution easily exceeding the queue size limits. (This is further exasterbasted by elasticio/sailor-nodejs#80)

However, if these messages are 100% independent of each other, then there is no reason to attach each result to the larger REST body. It would be complex to change passthrough logic or message structure to handle the above case. In theory, the above could be tuned to emit things slower, but that also introduces problems. However, if we understand that the above is a common pairing, then it may make sense to add the following two inputs to the REST API component:

• Body JSONata transform expression (optional, default to idempotent $)
• Split result if it is an array (optional, default to false)

The component behavior would be changed to:
After the component has a JSON representation of the response body that they are going to emit, then:

1. Evaluate the provided JSON expression if it exists to the body of the HTTP response.
2. If the result is an array and splitting is true, emit a message per array entry. Otherwise emit the transformed result.

[zubairov]

@jhorbulyk good idea, I would, however, try to keep the 'configuration' options to the minimum. Do you think a single 'checkbox' for splitting (without additional JSONata field) would solve the immediate problem?

[jhorbulyk]

Do you think a single 'checkbox' for splitting (without additional JSONata field) would solve the immediate problem?

In this one instance yes. However, I disagree on the keep the configuration options to the minimum philosophy.

[stas-fomenko]

@zubairov @jhorbulyk so, how exactly solution of this issue should be implemented?

[zubairov]

Only additional checkbox for now, without the additional JSONata field.

[jhorbulyk]

Work (with transform case started) is in branch add-ability-to-transform-split-body

[stas-fomenko]

DoD

Component development prerequisites:

• Development team has all needed requirements
• Access to the needed service with all needed permissions is granted
• No open tasks that could block Component development process
• The E.io’s GitHub repository must be checked if there is another implementation of the component under development. If it exists, new repository should not be created
• Component architecture should be discussed and approved before start coding

Development:

• Code is written and formatted in accordance with the Styling guide
• Code (incl. configs, test samples, environment variables etc.) MUST NOT contain any sensitive data like passwords, tokens, API keys etc.
• Component logs does not contain any sensitive data (credentials, input/output requests, Environment variables)
• Code is covered with unit tests
• Component code is covered with integration tests (optional)
• Triggers/actions comply with OIH patterns
• Documentation (README.md file in the Github) is 100% ready
• CI is set up
• All development branches are reviewed

After QA:

• All development branches are merged into master branch
• Sprint review preparation is finished (demo flow is built)

QA:

• Test cases for the component are ready at the TestRail
• Functional testing is finished
• Regression testing is finished (optional)
• Component is deployed to the production stage (app.elastic.io) and/or other required stage
  • Component has a proper access level
  • Smoke test is finished
• Release notes are done
• Component spreadsheet is updated