PDF/UA support, part 2

[hvbtup]

• Added some example PDF/UA reports.
• Add support for declaring newer PDF/A versions.
• Add support for declaring PDF/UA-2.
• Add support for declaring PDF 2.0.
• Use "auto" as default value for tagType property for some report element types to allow automatic context-depending setting as well as explicit setting.
• Support generating "Caption" tag for tables.
• Fix page-break handling of table rows and cells.
• Create corresponding PDF tags for some HTML elements.
• Add hyperlink support.
• Allow creation of a PDF that conforms to PDF/A and PDF/UA at the same time.
• Fixed specifying the document language based on report locale setting.
• Added THead, TBody, TFoot tags for tables.
• Fixed PDF syntax errors (contentByte.setColorStroke was called at wrong place in drawing sequence).
• Removed support for Acrobat Flash.
• Refactored attributes needed for split tracking into a dedicated class.
• Refactoring of some PDF-tag related constants.
• Added or improved some comments.
• Removed some warnings.

I originally developed this using my hobby account hvbargen over several weeks.

[hvbargen]

Reminder: Page numbering with the auto-text item causes and exception. I already added a commit and an example report with this commit,which I should add here when I'm at work again.

[speckyspooky]

Was the issue existing before or is it a reason of the changes?

[hvbargen]

The issue was existing before when you tried to generate PDF/UA.

[hvbtup]

I'll see what I can do this weekend.

[speckyspooky]

The most changes are added to get comments and some coding styles.
Also the question is added if it make sense to add PDF 2.0 support when we throw an exception if it is used through the report developer.

[speckyspooky]

The standard convention is to use name like "child" & "children" instead of "kid" and "kids".
Therefore change it to child & children.

[hvbtup]

The naming is used this way inside OpenPDF and the rare examples I could find in the net. Thus I used the same naming convention.

[speckyspooky]

But the BIRT developer used Schild & children, you are the first with Kids.
There is no Methode with "getKids" but you will finde "getChildren" and "getFirstChild()".

So it would be good to keep on child & children.

[merks]

Even for English it's poor English and I don't recall seeing such terminology used elsewhere.

[wimjongman]

Yes, lets use the children and child convention. I understand your idea to follow their pattern, but we should stick to our own.

[speckyspooky]

Better name for "kido" = "kid-object", normally we name it child e.r. "childPdfObj" or "childPdf"

[speckyspooky]

A description of the class is missing. You are the developer with the deepest knowledge and for other developers it would be helpfull to have a short description.

[speckyspooky]

A description of the class is missing. You are the developer with the deepest knowledge and for other developers it would be helpfull to have a short description.

[speckyspooky]

The core pain of the old code is that there is no documentation is given on class, methode and property level.
So it would be much better to avoid the "SuppressWarnings" and add regularly comments to all elements.

[speckyspooky]

Description is missing.

[speckyspooky]

Description is missing.

[speckyspooky]

Comment would be helpful which header details for which version woll be created.

[speckyspooky]

Add the class of the constant or remove it from the constants before.

[speckyspooky]

Add the class of the constant or remove it from the constants before.

[andreasrosdal]

Great work, I would like to encourage you to complete this and merge it. Well done.

[hvbargen]

Regarding adding a test with VeraPDF, I'll probably need help.

I added this to the pom.xml of org.eclipse.birt.report.engine.emitter.pdf.tests/pom.xml:

<dependencies>
	<dependency>
		<groupId>org.verapdf</groupId>
		<artifactId>validation-model</artifactId>
		<version>1.26.1</version>
	</dependency>
</dependencies>

But all the VeraPDF imports, e.g.

import org.verapdf.pdfa.PDFAParser;

do not work with an error message that the import cannot be resolved.
It seems that I am missing some additional step required to add a new Maven dependency.

[merks]

It needs to be in the target platform so added to the *.target and the dependency needs to be specified in the MAINFEST.MF. If it doesn’t work in the ide, it won’t work in the build. So I would need to add this to orbit and it would need ip review. I can look into that tomorrow. Hopefully it doesn’t drag in a whack of dependencies.

[merks]

There are a lot of dependencies

and they're not OSGi bundles. This looks like a significant workload for my free spare time...

[wimjongman]

Can't we take it as a fat jar? We only need it in a test bundle.

[merks]

The dependency fan-out is turning out not too bad. The most annoying questionable thing is this extremely old dependency:

        <dependency>
          <groupId>net.java.dev.stax-utils</groupId>
          <artifactId>stax-utils</artifactId>
          <version>20070216</version>
          <type>jar</type>
        </dependency>

But given we aren't shipping these dependencies in the product, that's of minimal concern.

I did have to make one package import optional:

According to the pom that is an optional dependency:

https://repo1.maven.org/maven2/org/verapdf/validation-model/1.26.5/validation-model-1.26.5.pom

I have no knowledge to know if this subset is fully functional for the intended purpose.

[hvbtup]

BTW there's a javax and a jakarta version of verapdf, see https://docs.verapdf.org/develop/

[merks]

I assume one would tend to prefer the jakarta version these days.

[merks]

BTW, I've hit a roadblock in that the source bundles produced retain the original jar signature and of course the jar has been modified to bundlize it such that the signature is invalid and the artifact will appear to be tampered which in fact is the case:

I'm not sure if the build machine, which signs these jars, will remove the original signature. I will need to investigate, but it's most inconvenient not to be able to test this locally. This also seems a lot of overhead for a test...

Note too that when I build a target platform where I let it pull in all recursive dependencies, it's an incredibly large set of 123 bundles:

So I'm not sure how fat the fat jar would be or how you create (and plan to maintain) the fat jar.

[hvbtup]

Right now, when I test manually, I use the command-line tool which is provided by the verapdf-greenfield-1.27.33-installer.zip.
But while this distribution is nice for manual tests once it is installed, it needs a manual installation step: The installer is a ZIP file, which needs to be extracted into a (temp) directory, there you call the actual installer program, which in turn extracts the final contents into a user-defined directory (a directory structure with 750 files).
Probably, once installed, I could call the Java API directly instead of using the shell and the CLI.
But extracting the ZIP and running the installer before being able to use it for tests seems not very elegant.

[merks]

This looks a bit like a fat jar with everything in it similar in size to the jar in the download you mentioned:

https://repo1.maven.org/maven2/org/verapdf/apps/pdfbox-apps/1.26.6/

It would be harder to get approval (and use) because sources are missing...

[wimjongman]

Can't we download and extract it on the fly in a @beforeAll context?

[merks]

@wimjongman

To which "it" do you refer?

This appears to contain an installer as a jar, but I don't know if that jar can be used directly to run the application.

https://software.verapdf.org/releases/verapdf-pdfbox-installer.zip

I'm also not sure about this one:

https://repo1.maven.org/maven2/org/verapdf/apps/pdfbox-apps/1.26.6/pdfbox-apps-1.26.6.jar

It appears to want to run the GUI:

Main-Class: org.verapdf.apps.PdfBoxGuiWrapper

Maybe if a different main class is specified on the command line it can do other things.

[wimjongman]

which "it"

I was referring to Hennings #2018 (comment) where he now downloads a zip and extracts it to run the test.

[wimjongman]

which "it"

I was referring to Hennings #2018 (comment) where he now downloads a zip and extracts it to run the test.

There seems to be a headless installer on this page https://docs.verapdf.org/install/. Look for section Automated Installation in the bottom.

[merks]

Yes that is the zip link I showed. But it’s an installer so one would need to run that first and after one could use the installation. Nicer would be if there exists a jar one can download and use, even if the jar was in a zip. Probably what’s needed is in the installer jar or the installer jar could be used that way.

[hvbtup]

I just saw the automated installation documentation Wim mentioned a few minutes ago.
It seems a bit cumbersome to use (I would prefer if I could just specify the target directory directly on the command line instead of creating an XML file for this), but usable.

So maybe it's possible to install VeraPDF on-the-fly before using it for tests.
But then we could still only use the command-line version in a subprocess, which means a bit of overhead per PDF file tested.

If accessing the Java API is not possible for various reasons, using the CLI is the second-best option IMO, despite the installer-, process- and Java initialization overhead.

@merks Ed, if we could solve the technical issues with using the installer, would we get an approval to use it?

Sigh... It seems rather complicated.
Should we move the "adding automated tests" topic out of this PR?

[hvbargen]

I'm going to test the following idea (in a stand-alone program) as a POC:

Initialization: Use the "automated installation" to install VeraPDF to a temp directory.

Figure out what's needed to call the API methods (by looking at the batch scripts to start the CLI.

Set up the class-path etc accordingly.

Write a Java method validatePDF which uses the API to validate a given PDF.
Call this method in a loop for several PDFs, to see if and what cleanup is needed inbetween.

Clean up: Remove the temp directory.

Once this works, rewrite the validatPDF method in a way which does not need the JARs at compile time.
For example, by using reflection or by using Rhino.

When this POC works, integrate this into the BIRT tests in such a ways that we can use the API directly from the tests, instead of creating and calling a child process for each PDF file.

[wimjongman]

Sounds great, Henning.

[hvbargen]

I created a POC in https://github.com/hvbargen/verapdf-install:

This uses the VeraPDFInstaller ZIP archive as a basis, runs the installer in automatic mode, then creates a new class loader which contains the extracted JAR, sets some system properties which VeraPDF uses (maybe this can be customized in others ways) and then uses Javascript (Mozilla Rhino) with this class loader to validate the PDF.

Well, that's how it should work.
But it doesn't, unfortunately.
Calling VeraPDF directly from Java worked fine (see https://github.com/hvbargen/verapdf-cal, yes, I know, there's a typo in the project name).
But calling it through Rhino, it complains that it needs javax, I don't know why. I give up for this weekend.

[speckyspooky]

If you have a "javax" error, my first idea would be that the module-handling of the JVM create some troubles.
But I don't know how you have tried to use Rhino - is it a kind of test report?

Because my thinking was if you use a test-report with Rhino the JVM would need according to your javax-error the global parameters to load directly your missing module(s) at starting moment of JVM.

But it is only an assumption because I don't know in which way you tried to use Rhino.

[hvbtup]

Ah, I see: I was going to write: You can take a look at the POC code at https://github.com/hvbargen/verapdf-install. It's really just a single class.
But I created this as a private repository. Will turn it into public tonight.