Refinement of survey responses - development (fixes, customizations, contributions)

[bertfrees]

We are currently in the process of analysing your responses to the Pipeline usage survey held earlier this year. Some of the feedback requires some more elaboration or discussion. Below are some responses in the topic "development":

---

Two organisations brought up the issue of complex source code, one specifically mentioned XProc:

• Q: What do you think are the 3 greatest weaknesses of Pipeline?

  • XProc implementations are hard to read, negating value of open source
  • Complex source code

One organisation says that it is hard to contribute changes, and to find development resources for Pipeline 2:

• Q: How happy do you think your organization is with Pipeline right now?

  • Limited resources in Daisy Consortium limits the development. A lot of planned development in Daisy Project we need and are waiting for. Difficult to find other sources for Pipeline development
  • In Pipeline 2, it is difficult to contribute changes without spending much time getting to know the project, and without breaking things that multiple scripts depend on.

One organisation mentioned the trouble they have building the aggregator project:

• Q: Have you used the Pipeline Source Code Repository? Any suggestions to make it more useful?

  I think the survey is talking about the pipeline aggregator GitHub repository, but I'm not sure and i'm not sure what to recommend to improved its use. all i can say is that I never managed to build the pipeline aggregator repository after multiple days of trying (both on Windows and Linux at the time), and former colleagues abandoned the idea to use or contribute to the pipeline 2 after trying to build the aggregator (with the makefile for linux, or trying to find the required maven profiles for windows) or avoided the aggregator build to only use the pipeline-assembly and the pipeline-modules repositories. Also, the repository main page (the readme) does not indicate itself how to built it : a direct link to the build page of the aggregator repository within a "Build the repository" of its readme could be useful.

---

I would like to know:

• We would like suggestions for the complex source code issue and the difficulty of finding developers. Should we reduce the required technical expertise? Reduce the amount of XProc and XSLT code? Extract code in standalone (Java) libraries? Further simplify the build system?
• How can we make it less easy to break things?
• Have things improved since you last tried to build the aggregator project?
• Do you have suggestions for improving the developer documentation (specifically about the aggregator project)?
• Have yave you tried to get in touch with Bert? Do you have suggestions for making it more clear how people can get in touch when they have issues with the build?

[egli]

We would like suggestions for the complex source code issue and the difficulty of finding developers. Should we reduce the required technical expertise?

Simpler is always better. But what exactly does that mean? Reduce the number of technologies?

Reduce the amount of XProc and XSLT code?

Personally I would have thought that XSLT makes a lot of sense in the domain we are in, converting to and fro mostly XML-based content.

XProc I find hard to get into as I could never find any decent material to learn it properly. It is also so niche that you don't even find SO answers.

Extract code in standalone (Java) libraries?

I am not so keen on that idea. I think some of the challenges in the pipeline can be expressed in XSLT very succinctly. In Java this would be much more verbose.

Further simplify the build system?

How? Certainly not by introducing yet another build tool.

What I'm dreaming of is that I could just use and deploy the standard pipeline. My modules I'd maintain and build separately. Then I could simply drop them in the standard deployed pipeline and everything would just work.

[bertfrees]

Simpler is always better. But what exactly does that mean? Reduce the number of technologies?

Yes, or it could be maximizing the amount of code in well-known technologies, to reduce the chance of developers coming into contact with the lesser-known technologies (read: XProc).

I think some of the challenges in the pipeline can be expressed in XSLT very succinctly. In Java this would be much more verbose.

With Java libraries we simply mean that the API would Java. XSLT can still be a part of the implementation.

How? Certainly not by introducing yet another build tool.

I'm not really thinking in terms of how many tools we use, but rather how easy and fast it is to compile the project and how tools can help with that. Of course the more tools you use the more things could also potentially go wrong, that needs to be part of the consideration. But the basic assumption is that tools can have a positive impact.

An example of an improvement to the build system we've explored is to make the incrementing of versions in poms less of a pain, and to do it as automatically as possible.

I'm also improving our module registry system and one of our Maven plugins with the goal to compute dependencies between non-Java components more automatically, again to reduce the amount of manual work needed by the developer.

This all adds a bit of complexity, but makes things more accessible for the occasional contributor.

What I'm dreaming of is that I could just use and deploy the standard pipeline. My modules I'd maintain and build separately.

You can. Pipeline allows that. (And I think you're almost there.)

I'm also looking for ways to further simplify the build of Pipeline itself, to remove as much obstacles as possible for potential contributors.