FOIAXpress Release Notes 11.8.0

1 Release Overview

1.1 What’s New in v11.8.0?

Welcome to FOIAXpress v11.8.0, our summer release for 2024. This release was focused on addressing feedback we’re received from our customers, and making improvements to how you interact with our products every day.  

We’ve made some huge changes to EDR for this version that we’re excited to share, including detailed trackers for the sometimes-lengthy document upload and ingestion processes, added flexibility for data source management, and improved notifications and visibility when documents fail the ingestion process.  

We also streamlined the process for publishing to the Reading Room. Previously, you were limited to publishing one folder at a time, which added significant time and monotony to the publishing process. You can now bulk publish by selecting and adding multiple folders at once, saving time and getting documents into your Reading Room more quickly.

In Document Management, we’ve added customizations for fully withheld pages, and alert messages to help ensure pages marked as Withheld are properly redacted. Additional efforts were made to fix some potential security vulnerabilities, as well as issuing fixes for customer reported issues.

We want to provide you with an exceptional user experience, and we’re excited to provide real, meaningful improvements with FOIAXpress 11.8.0.

1.2 Important Notes for Deploying v11.8.0

IDs #81447, 77184

We have new requirements when upgrading to v11.8.0. Carefully review the updated Deployment Manual, including the Version History, for details on the new changes. One of these changes is a new pre-requisite: the IIS CORS Module must be installed on the application server. See the deployment manual for a link and additional details.

We also revised the FOIAXpress API inputs.json file. Previously, the file had a field for “Domain” which caused confusion. This now reads “DNS.” Also note that as of v11.7.0, the FOIAXpress API is required to be installed on the application server.

These updates are covered on detail in the Deployment Manual for v11.8.0.

Release Overview

1.3 Known API Issue

ID# 83694

There is a known issue with some GetRequestsCount/GetRequests API calls returning counts that were inconsistent with the in-application search results. After investigation, it was evident that the API was returning some duplicates within the results, causing incorrect counts. At the time of this release the root cause has not been identified, however we are working to understand and resolve this issue, and we look forward to including a solution in an upcoming hotfix.

2 Focus on Improving EDR

While EDR has provided a powerful platform for processing large volumes of documentation, the upload process was clunky, and left users in the dark about progress on uploads. File size limitations also hampered our users’ ability to effectively use the tool.  

With this release, we’ve made ease and usability of EDR file uploads a core focus, with a revamped process and interface that brings welcome flexibility and transparency to EDR file management. Read on to see all the new interfaces and improvements for EDR in 11.8.0.

2.1 New Data Source Management Options

When you’re creating data sources for EDR in 11.8.0, you’ll see a revamped process that allows for enhanced flexibility, visibility, and oversight. This starts at the updated Data Source screen (when you access Electronic Document Review from the Request menu):

You’ll have options to (A) Create Local Data Source (uploading local documents) or (B) Create Cloud Data Source, where you can create folders on the EDR file server to manage your EDR data. Creating a Cloud Data Source has the added benefit of being able to draw from existing EDR data already housed on your cloud server to build out the data set before adding it to your request.

2.2 Direct Uploads to the EDR File Server

ID #73431

Uploading and managing documents in the EDR File Server is significantly improved for v11.8.0. You can now create dedicated folders in the EDR File Server for each request, directly from the request, then populate these folders with documents to be added to EDR for processing this request.  

From the Electronic Document Review screen, there’s a new option to Create Cloud Data Source which allows you to manage folders and documents in the EDR file server. Click this option to open the EDR server browser, where you can click to create a (A) New Folder, name the folder and click (B) Create folder, then select your folder and click (C) Upload to open the new EDR document upload screen in a new tab:

From the EDR upload screen, you can drag and drop (or browse to select) files to add to this folder. Then, select these to add to EDR for this request. These cloud data sources provide the added benefit of being able to locate and reuse documents that already exists within the file server. From the Create Cloud Data Source screen, you can select (D) existing folders from the document tree, then click (E) Select to add these to EDR for this request:

2.3 Revamped EDR Document Upload Process

ID #70505, 73080

One area where EDR fell short was in clearly communicating the status of file uploads. Due to the larger size of some EDR file sets, it may take hours to fully ingest some EDR files, and this lack of clarity on status and progress lead to frustration for our users.

Our new EDR file upload center was created with some key improvements in mind. First, when you upload files to EDR, this interface opens in a new browser tab. This allows you to keep your application open in one tab, while you monitor upload progress in a separate, dedicated tab, as shown in the following example:

Here you’ll add a (A) name for your data source, then either (B) drag and drop the files into the workspace, or click (C) Add files to browse for the files you’d like to add. Once you’ve added all documents, you’ll click Start Upload to begin the upload process.

From this screen you can upload files into your data source and monitor the status of each (C) individual file, as well as the status of the (D) full data set, including number of files uploaded, size of the files uploaded, and total upload progress (expressed as a percentage).  

For detailed instructions on uploading files using the new upload process, see the EDR section of the User and Administrator Manuals.

2.4 File Limit Increased to 20 GB

ID #70507, 69852

Customers have noted limited ability to upload larger file sets, and uneven performance when they do upload these larger files. For this release, we’ve increased file size limit to 20 GB for each upload. Note that this is per upload, not per data set. You can upload multiple batches of documents, however for performance and processing time issues, we recommend a maximum of 20 GB for each upload.

In addition to increasing the file size, we’ve also stabilized upload performance. This enhanced stability ensures the application processes these large data sets without timeouts or other errors. Combined, these enhancements allow you to upload more data, more quickly, and more reliably than ever before.

Note: Processing times can be extremely long for larger file sets. You can safely navigate to the application while the processing occurs in the background, just leave the document upload tab open to monitor the upload.

2.5 New Ingestion Process

Regardless of how you create your EDR data set, after adding documents you’ll see these listed on the Data Source screen, with the name defined when you added the source.  

You can click to (A) view the list of files in the data set. From the list, you can (B) Remove files by clicking the X beside that file, or click Remove to remove all files from the set (the application will confirm before deleting the files). You can also add to the data source by clicking Add more files into Data Source which returns you to the upload screen.

When your Data Source is complete, you’ll configure your Clustering, Document Review, and Email Attachment settings just as you had before. Then you’ll click Add at the bottom of the screen to add the data source to EDR. However, we don’t leave you in the dark during the ingestion process anymore. See the Track EDR Document Ingestion section for details.

2.6 Track EDR Document Ingestion

ID #69851

After piecing together your Data Source, you’re ready for EDR to ingest that data. We’ve added a new tracking screen that shows exactly what’s happening in the background as your documents are moved into EDR.

This new tracker shows the (A) full EDR ingestion process, including what step is currently being executed, and a completion percentage for that step. There’s also a prominent (B) notice letting users know that they can navigate away from this screen without interrupting the ingestion process:

When the upload completes, you can click to view a (C) summary for the uploaded files. You can then click (D) Continue to Document Review to open the Data Source in EDR, or (E) Add More Data Sources to create additional data sources to add to EDR.

2.7 Failed Document Handling

ID #68772

When documents fail the EDR ingestion process, we wanted to highlight this in the UI. As part of the updated EDR ingestion tracker, we placed the Failed documents notification at the top of the documents list:

Click to expand the Failed list to see all failed documents, each of which you can (A) click to download. We also show the (B) Reason why the document failed:

Once you’re inside EDR, you’ll see another notice at the top of the EDR Home tab. Click Failed Documents to view the Failed Documents window, where you can view any failed documents, as well as Export All Items or Download Filenames.

2.8 EDR Document Page Count  

ID #69993

We’ve added a separate job for EDR page counts to each individual Data Source, allowing us to include these page counts for all EDR documents. The Pages column displays the page count for each EDR document:

2.9 PDF Image Document Handling

ID# 70379

We changed how the system handles PDF images and PDF documents containing only images. Previously these were treated as “blank” documents and excluded from processes like OCR and indexing. The system now checks these “blank” documents for images and then applies OCR and indexing as needed, ensuring these documents appear in the documents list.

3 Document Management

3.1 Alert Message for “Withheld” Flag

ID #73833  

We've added a layer of additional security to Review Flags to prevent unintended release of unredacted materials. When the "Withheld" flag is applied to pages that have not been redacted in full, the application now automatically prompts you to redact these pages in full. If prompted, we recommend applying the redactions to ensure the documents accurately reflect the "Withheld" flag. Click Yes to redact the pages in full:

3.2 Page Summary for Fully Withheld Pages by Request Type

ID #74163

When pages are fully withheld, we’ve provided a configurable page summary that is applied to these withheld pages. This functionality is expanded for this latest release, and you now have the option to create unique page summaries for each of your Request Types. While a default summary page can still be applied, you can now use the Request Type dropdown in the Page Summary for Fully Withheld Pages section (Administration > System Configuration > Document Management Configuration) to create custom summaries for each Request Type:

Document Management

The same customization options are available for each request type, including the Position, Format, Font, Header and Footer text, and the option to include a watermark. When creating a custom page summary, click Save on the configuration screen to retain the customizations for that request type.  

3.3 Watermark File Size Limit Increased

ID #68619

While users have had the option to upload custom stamps to use in Document Management, we've increased the file size limit for custom stamps to 2 MB. The ability to use larger images for custom stamps provides greater flexibility when reviewing and redacting documents in DM.

4 Application Enhancements

4.1 Total Days of Extension Insert Field for Correspondence

ID# 70635

We've added a new Total Days of Extension insert field for correspondence. The existing “Days of Extension” field only counted the most recent extension. This new field now captures the full extension length, inclusive of all extensions applied to a request:

4.2 Bulk Publish to the Reading Room

ID# 63441

Maintaining a robust Reading Room can help reduce the overall number of incoming requests. To help speed up building out your Reading Room, documents can now be added in bulk, allowing you to publish entire folders of documents, rather than one document at a time.  

Application Enhancements

At any point where the "Add Folder(s) to Reading Room" option is present (i.e. when closing a request, or when viewing documents in DM), all documents selected from the document tree are added to the selected Reading Room folder.  

To avoid duplication when publishing in bulk, we also alert if this document (or a document in the set) was previously published to the reading room. You’re also provided the opportunity to review already these previously published documents before adding any potential duplicates to the Reading Room.

4.3 SMS Options Hidden

ID #72488

We’ve hidden the SMS options for Multi-Factor Authentication (MFA). SMS-based MFA requires a third-party provider, which was not evident within the application’s configuration settings. To avoid confusion, these options no longer appear in the configuration UI unless an organization is currently configured to utilize SMS. Organizations currently using this feature will not be impacted by this change.

5 Improving the API

5.1 Request API Parameters

ID #77670

We’ve updated the Request API to include the following parameters:

• feeAmount: The fee amount associated with the request.

• requestPaymentConfirmationNumber: A unique confirmation number for payment.

5.2 GetRequestsCount

ID# 32960

The GetRequestsCount method can be used to get a count of all requests with specific “Where” clause fields, for example filtering for specific request type, action office, or request status.

5.3 API Request Error Messages

ID# 75676

We’ve revised the error messages received in response to failed API requests to provide more detail and context. This detail should assist users to identify and resolve API issues.  

6 Bug Fixes

We’ve corrected numerous bugs and issues in v11.8.0, as outlined in the following sections.

6.1 Request Processing

6.2 Document Management

6.3 Reports

6.4 Security