BFO PDF Library 2.27
We released version 2.27 of our PDF Library. If you're working with PDF/A this is a significant release - read on for details.
Many of the changes in this release come from work we've been doing with the Apache Tika Corpora, a collection of (currently) 32,580 PDF files submitted to public issue lists or bug trackers for various projects.
About a year ago we published an analysis of PDF/A conversion on about 15,000 files from another corpora, Govdocs1. If Govdocs1 contains typical PDF documents, the Apache Tika collection does not. It's comprised entirely of PDFs that were submitted as part of a bug report, so these files tend to be very poorly structured. Many were deliberately intended to stress-test PDF software.
Perhaps foolishly, our goal with this release was to convert as many of these as possible to PDF/A. We've largely achieved that, loading 98.8% of the files and successfully converting all but 4. Of the files we couldn't load, most are not acually PDF files, or are so truncated they have no catalog or pages.
We hope to do a proper write up of this later, but what matters for this article is that we had to do a lot of fixes to the API to handle the various types of corruption we encountered. Most of the changes in this release are fixes to ensure PDF/A conversion can better handle malformed files.
Some of the changes go further - partly to address bugs in our code, partly new functionality and partly bringing our PDF/A implementation up to date with the latest errata from the PDF/A Technical Working Group (of which BFO is a member). Here's a summary.
Our API wasn't previously identifying images that were damaged or truncated as being invalid for PDF/A. It isn't explicitly listed as a requirement for PDF/A, although it is implied, and a brief survey shows this test isn't consistently implemented across PDF/A verifiers. We've made the decision to disallow this in PDF/A, so we can repair it if encountered. The new ImageDamaged feature is part of all the PDF/A profiles, but this means that files previously certified as valid by our API may now appear invalid. To revert to the old behaviour, call
clearDenied()on your PDF/A target profile with this feature.
PDF/A-1 uses different language to PDF/A-2 and A-3 to describe how Dates are formatted in the PDF metadata, with PDF/A-1 disallowing dates with times if no time-zone is specified. In previous releases we didn't check for this. Like the change above, this may result in some files previously certified as valid now appearing as invalid. The feature for this change is XMPDateWithoutTimezone and the solution is the same as above.
- Some other very obscure issues (eg https://github.com/pdf-association/pdf-issues/issues/168) have been decided by the working group in a way that didn't match our validator. We've now updated our validation rules, although the impact of most of these should be negligable
Note that the first two items on this list remain open issues within the PDF/A working group - we're erring on the side of caution with these changes, and will update our validator to match when the issues are resolved.
We've also improved the functionality of the PDF/A conversion code.
- When converting to PDF/A, the API requires a list of fonts for potential substitution. We were scanning this list and searching for a single font that was the best match and had all the required glyphs - if no font matched, it would fail and the page would rasterized. This new release will instead search for list of fonts that can be used for substitution, meaning that the chances of success are much higher. The API of OutputProfiler.FontAction has changed slightly as a result - see the API docs for details on this, migration for any custom implementations is trivial.
- Profiling a PDF containing thousands of pages is a mullti-threaded operation, but while testing with some very large files (38000+ pages) containing text that shared a single font, we could see there was contention for some synchronized resources. We've rewritten these to use concurrent datastructures, and performance has jumped significantly. How much will vary depending on the number of threads, which defaults to the number of CPU cores available to Java. When testing with 10 threads we saw a 50% increase in speed.
- Many PDF processors incorrectly create the structure for PDF forms. We've adjusted the repair process in this release to try and handle some of the more exotic failures we encountered, and keep our results as close to Acrobat as possible. For a very small number of corrupt PDFs, the recovery process will change a result, which might mean the list of elements in the PDF Form is in a different order.
We've updated the
bfopdf-jj2000.jarincluded in the download, after we fixed an error in the decoder. It's quite obscure, but we recommend upgrading. If you choose not to upgrade to version 2.27 of our API, you can always update and rebuild just this Jar from the github source.
- If you're using the redactor, we recommend upgrading. We fixed an issue where the resulting PDF could sometimes be structured incorrectly, causing images to to missing or warnings in Acrobat.
- And finally, we dropped support for Java 6. The PDF Library now requires Java 7 or later to run.
The CHANGELOG has more details.
If you're working with PDF/A this release is a significant one. Conversion to PDF/A is more robust, with significantly less "Rebuild" processing - the step we run internally if our repair process has failed. Out PDF/A profiles are slightly stricter than in previous releases, ensuring that files we convert to PDF/A are unambiguously correct.
As always, the latest release is available at https://bfo.com/download.