LOCKSS Daemon 1.78.3

Features

• LOCKSS 1.78 to 2.0-beta1 migrator:

  • Now partially configures itself from 2.x.

  • Migrates metadata database, subscriptions, other configuration such as access lists, and user accounts.

  • During migration, all normal activities (polling, crawling, metadata extraction) continue for both migrated and unmigrated AUs.

  • Similarly, content access requests (to ServeContent or the proxy) are routed to 2.0 for migrated AUs, so they continue to be accessible during the migration process (experimental).

  • Display warnings on UI elements that should be used cautiously when migrating.

  • LCAP forwarding. The 1.x instance handles all LCAP communication with other nodes on behalf of both itself and the 2.x instance.

• LOCKSS 1.x and 2.x polling compatibility. In order to account for some small differences in the way 1.x and 2.x store URLs in their respective repositories, some incompatible changes were needed to the way LOCKSS 1.x generates and tallies votes. By default 1.x still operates compatibly with 1.77 and earlier releases, but cannot poll with 2.x. Setting org.lockss.poll.2.0Compatible=true causes 1.78 to poll compatibly with 2.x, and other 1.78's with the same setting.

• Optional configuration files may now be in XML format (.xml.opt).

• Allow AUs to be dropped into repository on the fly, for CLOCKSS triggered content and similar behind-the-scenes operations.

• Changed the default values of many configuration parameters to what is appropriate or likely for private LOCKSS networks, rather than the Global LOCKSS Network or CLOCKSS, which simplifies initial PLN setup. See also https://docs.lockss.org/en/latest/admin/starter.html.

• Added the plugin identifier to PluginReloaded alerts.

• Added a builtin plugin for preserving simple directory tree content on a Web server (DirTreePlugin).

• It is no longer necessary to define the standard titlesets (AllAus, ActiveAus, InactiveAus) in the props file. Disable with org.lockss.addStandardTitleSets=false.

Bugs

• The AU selection screen under AU Configuration could display erroneous disk choices.

• Disallowed or disabled servlets return a more appropriate status code (403 or 503).

• Files received as repairs in a poll may not have been findable (e.g. by ServeContent) if on a plugin's additional hosts.

• The RIS metadata extractor did not treat TY tag values case-independently.

• Added a CLOCKSS permission statement with open access qualification. Now accepting legacy Creative Commons 2.1, CC0, CERTIFICATION 1.0 and PDM 1.0 licenses.

Security

• Following best practices, we are removing unnecessary version number disclosures in HTTP responses and UI pages.

LOCKSS Daemon 1.77.6

Features

• The SOAP services can now be disabled by setting org.lockss.soap.enabled to false (for example in the Expert Config screen).

• Plugin registry servers that redirect from http: to https: are now supported. (https: loading of config props, as well as redirection from http:, has always been supported.)

Bug Fixes

• Fixed a bug preventing fetches from being retried after certain network or server errors causing crawls to fail. This was particularly affecting machines in the USDocs PLN.

LOCKSS Daemon 1.77.3

Features

• Self-generated SSL certificates for admin UI or ServeContent now have 2048 bit keys.

• Substantial reduction in memory requirements.

• The AUID is displayed in the AU detail page, and available in metadata index status tables.

• Minor migrator changes to improve reporting, increase compatibility with 2.0.

• All plugins and keystores used for unit tests are now generated by scripts.

Bug Fixes

• Plugin jar validation checked only jar members listed in the manifest. It may have been possible to exploit this to cause the system to load a class added to a plugin jar after it was signed.

• Fixed a unicode normalization vulnerability that might have allowed specially crafted bibliographic info in the title DB to cause the UI to misbehave.

• Fixed a bug preventing hash estimate padding from being fully configurable.

LOCKSS Daemon 1.76.5

Features

• This release includes a preview of the migration tool which will be used to transfer content from an existing LOCKSS 1.x system into a LOCKSS 2.x system. It is not quite complete (some aspects of the V1 configuration are not yet migrated, e.g. subscription info and user accounts), but we are interested in feedback and hope to surface any problems that we are not already aware of. You can find instructions on how to use the migration tool in the LOCKSS Community Wiki: https://github.com/lockss/community/wiki/Migration-Tool

• Triggers for plugin-specified actions during crawls now include redirection to URL pattern.

• Many non-standard DOI prefixes and decorations are now allowed in metadata extractors.

• Poll invitations that are refused because of poll version mismatch are identified as such in the poll status display.

• Added short_year() AU parameter functor.

• Changed default items/page in UI tables to 1,000.

• Allow plugins to declare minimum required daemon version for both 1.x and 2.x.

Bug fixes

• Handle split zips with less-standard extension numbering.

• Fixed excessive synchronization processing zip, etc. archives

• Fixed proxy to correctly forward POST request body.

• Fixed bug sending compressed servlet responses larger than 2GB.

• Prevent most cases of double form submission in UI.

• Ignore duplicate start URLs returned by crawl seed.

• Removed rate limit from plugin registry crawls.

• Updated dependent libraries.

LOCKSS Daemon 1.75.9

Security

• Removed Log4j 1.x from the release package.

  While the LOCKSS 1.x daemon itself does not use either Log4j 1.x or Log4j 2.x, some of the included third-party libraries do use Log4j 1.x. It is the case that there known security vulnerabilities against Log4j 1.x (CVE-2019-17571, CVE-2021-4104). It is impractical for us to audit those third-party libraries to determine whether they are affected by these vulnerabilities, so out of an abundance of caution, we have completely removed Log4j 1.x from the release package and replaced it with Log4j 2.17.1, the most recent version of Log4j 2.x. This version is not affected by Log4Shell (CVE-2021-44228) or the severe Log4j vulnerabilities discovered subsequently (CVE-2021-45046, CVE-2021-45105, CVE-2021-44832).

• Bring PostgreSQL JDBC driver up to date (CVE-2020-13692).

Bug Fixes

• Fixed a bug preventing AUs in title sets with certain Unicode characters in their name from being added.

• Switched from Commons Collections 3 to 4 and Commons Lang 2 to 3 (with exceptions required for plugin compatibility).

• Fixed some unclosed file errors.

LOCKSS Daemon 1.75.8

Features

• Added HTTP response and error categories; plugins can now map categories as well as individual responses/errors.

• Revamped HTTP error response handling. The default behavior for a few unusual responses has changed slightly.

• Added support for communicating with APIs via POST requests during crawls.

• Added support for deleting Cached URLs.

• Added PluginMissing alert (configured AU with no plugin) to distinguish from PluginNotLoaded (plugin load failure).

• Cause Json-Path to use Jackson instead of json-smart.

• Added Jonix library.

• Updated Jsoup (CVE-2021-37714), Commons CLI, Commons Compress, Commons IO, Commons Lang3, Json-Path, and Xerces (CVE-2012-0881) dependencies.

Bug Fixes

• Work around Java bug getting total/free space on filesystems larger than 8 exabytes.

• In unusual circumstances, a non-latest version of a plugin might have been loaded.

• Ensure UI response headers include correct charset in Content-Type.

• Ensure ServeContent doesn't return content encoded not in compliance with client's Accept-Encoding header.

• HTML and CSS parser now ignores data: URLs in CSS.

• SubTreeArticleIteratorBuilder now calls setFullTextFromRoles after setRoleFromOtherRoles.

LOCKSS Daemon 1.75.7

Features

• Java 1.8.0_291 (released on April 20, 2021) disabled the default encryption protocol used by LOCKSS for secure polling communication between boxes ("LCAP over SSL"). Only a few PLNs are using this -- the GLN and most PLNs are not affected. The 1.75.7 release changes the default protocol from the recently-disabled TLSv1 to TLSv1.2. PLNs that are using LCAP over SSL should install this version so that SSL communication continues to work. Alternatively, the default SSL protocol can be overridden in the PLN's props server, by setting org.lockss.scomm.sslProtocol=TLSv1.2. This should be done if 1.75.7 will not be installed promptly.

• Added the Jayway Json-Path library, so plugins can traverse and query JSON documents similarly to XPath for XML documents.

LOCKSS Daemon 1.75.5

Version 1.75 of the LOCKSS Daemon no longer supports Java 7 and requires Java 8.

Features

• Automatic processing of archive files now works for split Zip files.

• Transparently handle some malformed Content-Type headers such as ('image/jpeg', none).

• Link rewriter now handles protocol-relative links.

• Added X-Lockss-From-Auid response header to ServeContent to identify a file's AU.

• Added a OS Release Info daemon status table.

• TDB tools now accept directory paths as input files.

Bugs

• Link extractors now accept relative <base> tags.

• Do not force browser download of files that have an inline Content-Disposition in ServeContent.

• Ensure ServeContent uses legal filename in Content-Disposition.

• Do not let permission page fetch error prevent trying alternate permission pages.

• Guard against ArrayIndexOutOfBoundsException parsing large RDF on permission pages.

LOCKSS Daemon 1.74.10

Features

• Synchronizing subscriptions now requires confirmation, in order to reduce the likelihood of inadvertently overwriting manual changes to subscriptions.

• Added display of available AU count to subscription screens.

• Hashes started from the HashCUS UI are now visible to web services and vice-versa.

• JsoupHtmlLinkExtractor now processes <track> and <source> tags for media elements (<audio> and <video>).

• The crawler now retries SSLExceptions by default.

• Added hostIP conditional to the props.xml parser.

Bugs

• Malformed zip files caused metadata extraction of entire AU to abort.

• Charset guessing did not read up to the intended number of bytes.

• Deletions from TDB files did not always take effect.

• Fixed incorrect extensions to the future of some synchronized subscriptions.

• Substance checker improperly processed some files collected after a redirect.

• Some fetch exceptions weren't retried, depending on where in the transaction they occurred.

• Reduced overhead of parsing some large PDF files.

• Eliminated temporary file leak parsing large PDF files.

• Fixed some file descriptor leaks causing "too many open files" errors.

• The ContentService web service failed to serve content.

LOCKSS Daemon 1.74.7

Features

• The PDF filtering code has been hardened to withstand processing uncharacteristic PDF files with excessively large in-memory representations, without filling up the heap and without requiring changes to existing plugins.

Bugs

• The proxy failed to normalize URLs in requests that include an AUID.

• Cancelling hashes started from DebugPanel or HasherService frequently did not work, and sometimes crashed the daemon.

• Aborting crawls using crawlPriorityAuMap did not work.