Refinement of survey responses - setup (installation, integration)

[bertfrees]

We are currently in the process of analysing your responses to the Pipeline usage survey held earlier this year. Some of the feedback requires some more elaboration or discussion. Below are some responses in the topic "setup":

---

• Q: Would you encourage other organizations to consider using Pipeline in their production workflow?
  • One organisation indicated that special technical expertise is needed to set up Pipeline:

    That depends on the files they need to adapt and about their current workflow. Not every organization have a technical employee or volunteer that can set up Pipeline for production easily.

• Q: How easy has it been to integrate Pipeline into your production workflow?
  • One organisations mentioned client code that they had to write:

    For the Pipeline 2 we had to write clients for the API in both Python and Clojure

  • Another organisation mentioned scripting (with further specifying):

    • Production automation requires additional scripting.

• Q: Please describe any functionality you would like to see in Pipeline in future
  • One organisation reuested a RESTful interface to the web service:

    Having a RESTful interface to the API/web service would make integration with other systems easier.

• Q: What do you think are the 3 greatest weaknesses of Pipeline?

  Updatability

---

I would like to know:

• We would like more specific issues and suggestions for facilitating the setup.
• We would like suggestions for how we could make your effort of writing client code useful to others. Are you considering creating a library?
• Could you elaborate on the addition scripting that was needed? Are you referring to a missing feature in Pipeline?
• What makes our current web API not RESTful?
• Could you explain what exactly you mean by "updatability"? How would you ideally want to receive updates?

[josteinaj]

If the Web API is simple enough, and for simple use cases, maybe client libraries wouldn't be necessary? Some thoughts:

• JSON for requests and responses?
• For fast jobs, maybe a synchronous call could be useful? So you wouldn't need a while-loop checking the job status. However, I think most jobs are running for too long for this to be useful in our case
• Maybe query parameters would be easier to use instead of creating a job request document?
• job cleanup: I download the log, the reports and any output files exactly one time. After that, the job can be deleted to free up space. Can we avoid having to make a separate DELETE request?
  • maybe a query parameter for deleting the job could be part of the job endpoints? Something like ?delete=true
  • If I remember right, there's already some configuration for automatically deleting old jobs? Or is that only in the Web UI? Maybe a setting that deletes finished jobs that have not been queried for some configurable amount of time? 5 minutes would probably be a good default when using this

[bertfrees]

Thanks @josteinaj. Good suggestions.

• JSON for requests and responses

We've also thought of this. It would indeed simplify the usage if there was a choice between XML and JSON.

• Maybe query parameters would be easier to use instead of creating a job request document?

The idea is good, but I don't know if it's feasible, because the job request XML format is not totally trivial.

• For fast jobs, maybe a synchronous call could be useful?

Yes, good idea.

• Can we avoid having to make a separate DELETE request?

Good idea. But I don't think this needs to be settable on the level of individual jobs. I suggest that we try to accomplish this with a global setting. If you disable persistence (with the PIPELINE2_PERSISTENCE environment variable), jobs will not be persisted across different runs of the web service, however during the current run they will be kept in memory and temporary files until they are deleted.

We could add another setting for the time the jobs are kept in the volatile storage (currently forever).

If I remember right, there's already some configuration for automatically deleting old jobs?

Not sure if this is what you mean, but there is an undocumented org.daisy.pipeline.ws.cleanuponstartup property which cleans up idle or running jobs on starting the web service. But this doesn't delete finished jobs.

[egli]

We would like suggestions for how we could make your effort of writing client code useful to others. Are you considering creating a library?

I have a Python and a Clojure client library in production. These are quite rudimentary and just do the bare minimum get get the specific scripts working that we need. I do not plan to further improve the Python client library as I want to retire that code. In the case of the Clojure client libraries I have indeed thought about making it into a separate library. But I'm almost a bit embarrassed to do so as it is so rudimentary and doesn't even cover the whole API surface,

What makes our current web API not RESTful?

Did I say that? IMHO there is a bit of a mismatch in that a "classic" RESTful API deals with resources, their addition and removal. The pipeline is at its core a functional API: you invoke some function, give it some input and you get some output. In order to be RESTful, I guess, we introduce the concept of jobs, that you can add and delete. But fundamentally the end user doesn't care about jobs. They just want their conversion to happen.

So IMHO there is this conceptual mismatch. But do we need to do anything about it? I don't think so. RESTful is a useful concept and well established as an API.

[egli]

Maybe the idea of a synchronous API would be a step towards a more functional API? But then I'm not so sure. It can be asynchronous, just do not force me to create a jobs and dispose of it afterwards.

[bertfrees]

Did I say that?

No, it was Benetech who made this comment:

Having a RESTful interface to the API/web service would make integration with other systems easier.

A synchronous API would help, but comes with some other challenges:

• How do we indicate to the client that the submitted job is in the queue?
• The response must contain everything: the status, the messages, the results. How can this be done? Currently we return an XML document with links to the results.
• It does not make sense anymore to poll for new messages in a synchronous API (because there is no concept of jobs anymore). Also, the client would needs multi-threading in order to poll while it is waiting for a synchronous call.

I do like the idea of not having to delete jobs. But can that be RESTful?

[egli]

I did some research into how to do long running http calls. One good resource recommends the Asynchronous Request-Reply pattern. This still involves polling and uses 3 end points:

1. The API end point to send the request to
2. the status end point to poll the status of the request
3. and the resource URL fetch the final result

This doesn't really solve the problem of having to delete jobs as (from what I gather) you will still have to expire resources somehow.

The page on the Asynchronous Request-Reply pattern says that you should use this pattern if you cannot use more modern callback technologies such as WebSockets or webhooks.

All in all the Asynchronous Request-Reply pattern seems more like an incremental improvement. I am not sure if this outweighs all the changes involved in the docs, the client libs, etc

Might be better to leave the existing REST API and offer a additional WebSocket API instead

[josteinaj]

For deletion, you could maybe delete the previous job when a new job is posted (if queueing is disabled and you know there's only one job at a time).

You could also limit this to one job per client. If the client includes some ID that it uses when posting jobs, then the previous job can be deleted when a new job is posted by someone with the same ID.

Or maybe just a "max number of jobs to keep" and "max time to keep jobs after they're done" is good enough.

[bertfrees]

@egli Thanks for the research. That's interesting.

All in all the Asynchronous Request-Reply pattern seems more like an incremental improvement.

I think how the web API currently works is already very similar to what the pattern prescribes. The difference seems to lie mainly in the return codes and the use of the Operation-Location header. So I'm not sure whether the docs and client libs would need that much changes.

Might be better to leave the existing REST API and offer a additional WebSocket API instead.

I'm not sure whether a WebSocket API could solve the issue of having to delete jobs.

• A job may currently have multiple results. You may choose to get all the results, or the results of only one "port". You may even download individual files from a port. If we want to send the results through the websocket to the client automatically upon job completion (before deleting the job), the feature of downloading individual ports/files must be dropped.
• Is it even a good idea to include large files in a websocket message?

For deletion, you could maybe delete the previous job when a new job is posted (if queueing is disabled and you know there's only one job at a time).

Hm, I don't really like this idea. There are too many assumptions.

You could also limit this to one job per client. If the client includes some ID that it uses when posting jobs, then the previous job can be deleted when a new job is posted by someone with the same ID.

If the client authenticates with the server, such an ID is already included.

But still, I don't like that this would work only in the specific case of clients running one job at a time.

A setting for the time the jobs are kept in the (volatile) storage sounds simpler and better.

Ideally, I would like a solution where a job, once results are fetched (or possibly once the final job XML was returned), is automatically and immediately removed from the job list. The links to the result files (and perhaps the link to the the job XML as well) could possibly remain active for some time, but the job list would not include the job anymore. If a job completes but the results have not been fetched yet, the job should remain in the storage.

[egli]

I'm not sure whether a WebSocket API could solve the issue of having to delete jobs.

I thought if you have Websockets there is no longer a concept of jobs as everything "synchronous" in the sense that you start a pipeline script and all communication regarding that script will be on that Websocket until the script finishes.

• A job may currently have multiple results. You may choose to get all the results, or the results of only one "port". You may even download individual files from a port. If we want to send the results through the websocket to the client automatically upon job completion (before deleting the job), the feature of downloading individual ports/files must be dropped.

I guess you could still tell the server to only return parts of the result through the websocket.

• Is it even a good idea to include large files in a websocket message?

Good question

[bertfrees]

OK, so the server notifies the client when the job is completed, and sends it the result links. Then the client uses the links to download the results. When the websocket is closed, the job is deleted and the links don't work anymore. Yes, that should work.

I don't know if this is all gonna be worth it. But since I have already been working on some kind of websocket support last year (for messages and status updates only), perhaps it is not that much extra work. The reason I have not committed the work is because there are some issues, e.g. there is no websocket support in Restlet, so I'm running a separate websocket server (with Jetty) alongside the http server, and they run on different ports.

[josteinaj]

Are websockets generally well-supported in most programming languages? The last time I used websockets was for the webui.

[bertfrees]

I don't know. I started the websocket experiments for the new GUI (Electron app). It would be nice if it wouldn't have to poll for status updates and new messages.

We would have a websocket API in addition to the http API. It wouldn't replace it.