5.0 Release Notes

PopMedNet Distributed Query Tool

Release Notes

Based on Release 5.0

December 18, 2014

 

 

For a high level overview of the new features and enhancements in PopMedNet 5.0, please see New Features in 5.0

 


Releases

Date of Release1/5/20151/16/20152/23/20153/22/153/22/153/22/15
Networks UpdatedPCORnet DRNDemonstration SiteMDPHnetMini-SentinelNIHHDC
DataMart Client UpdateRequiredRequiredRequiredRequiredRequiredRequired

Introduction

PopMedNet 5.0 is a major release that focuses on speed and ease of use and introduces major new functionality.

A new desktop DataMart Client is required.Version 5.0 is not backwards compatible with requests from version 4.0, but can run side by side with version 4.0 of the DataMart Client with special considerations and instructions.

These release notes cover PopMedNet version 5.0. Enhancements, bug fixes, and other changes are listed below.

 

 


Enhancements

*Not all features are available on every network or to every user. Contact your PopMedNet Network Administrator for more information.

Infrastructure and Architecture Enhancements

Update
Description
5000+% speed increase in most scenariosAlmost the entire portal and its backing API have been majorly updated to vastly improve performance
Major security overhaulData storage for the security system has been simplified and the database "sharded" to result in 95% fewer records for security related operations in the database. This contributes to more than 60% of the system speed increase and significantly improves the scalability of the system.
Major document management overhaul

Previously documents were stored as binary large objects (BLOBs) within the database. Now, documents are stored as file streams within SQL Server 2008 R2 or later. This means that files are stored encrypted on the file system directly instead of in the database itself and are simply linked to the row in the database. As a result, files can now be streamed instead of read to their entirety and then sent across the wire. This results in several benefits:

  • Lower overhead with large documents
  • Better scalability over time.
  • Higher resiliency from upload/download and for disaster recovery as the master database is much smaller and thus less susceptible to errors
  • Improves the speed of the system from removing BLOB data that most databases do not respond well to. The speed improvements will be sustained regardless of how many documents are entered.
  • Prepares the system for document management within the Portal that would have previously required third party tools.
PopMedNet API

Previously, PopMedNet was an MVC.NET application that was completely self-contained and exposed certain Web Services as necessary for the DataMart Client. Now, PopMedNet is a fully multi-tier application with a full REST based API that supports both JSON, BSON, and XML request and response types. This API underpins all of the new PopMedNet functionality throughout the platform. This results in several benefits:

  • The API can be used by any software to communicate with the underlying system.
  • All pages in the Portal load asynchronously loading HTML while data is simultaneously downloading the background, resulting in a much faster response time for pages to load.
  • The DataMart Client can handle automatic downloading of components and faster updating of requests and responses.
Database schema upgrade

Almost every table in the database was optimized with better index, data types, and timestamps for optimistic concurrency. In addition, the Primary Keys on tables were upgraded to use Globally Unique Identifiers (GUIDs). This allows for records to be exchanged between multiple systems and maintain uniqueness constraints preventing conflicts. Furthermore, these GUIDs are sequential and thus can participate in clustered indexes, providing the best of both Integer based keys that are fast and typical GUIDs which provide uniqueness. Because the GUIDs were previously used as other fields in the same tables, the tables have shrunk in size by approximately 32 bits per record, a significant decrease across hundreds of thousands of records.

Tables names have also been updated to reflect the naming in the software. Foreign Keys and how they are linked were simplified and many Foreign keys were optimized to use composite keys where appropriate.

Web Portal Enhancements

UpdateDescription
Major user interface overhaul

Bootstrap has been implemented for an easier to use and more consistent user interface. The new user interface includes incremental lookup drop-down controls, combo boxes, pop-overs, and popup windows. Additionally, this prepares the system to support mobile browsers and customizing the user interface for each PopMedNet platform.

The system now provides greater feedback on errors immediately instead of only showing errors after submission or saving of the page. Data fields are also properly limited in size on the client side and data types have been strongly enforced with improved UI controls for data input.

Email notifications

All email notifications have been redesigned to include more information, more natural language, and bolded text for easier parsing of the email content. Subject lines now include the name of the notification to improve user's ability to manage the flow of incoming notifications.
Immediate email notificationsPreviously, email notification subscriptions set to send "Immediately" would send on a schedule of up to fifteen minutes. Now, email notifications are sent as soon as the triggering action is complete.
Viewing request details as a DataMart Administrator

Users may now view details of requests distributed to their DataMarts from the Portal, in addition to in their DataMart Client applications. They may view the request metadata and parameters, the status of their DataMart routing for the request, and any files or results uploaded in response to the request.

 

Table sorting and filtering enhancementsAll tables in the Portal feature new grid controls that allow any column to be filtered, sorted, and rearranged. Filtering and sorting preferences are saved across sessions.
Terms and Conditions agreementThe checkbox for agreeing to Terms and Conditions was removed so that users no longer have to check it each time they log into the Portal. Text was added to the login page to specify that users are agreeing to the Terms and Conditions of the network by clicking Login.
Request statuses rewritten

Request statuses were rewritten to more intuitively reflect the DataMart routing statuses within the request. Previously a request was only marked as "Complete" if all DataMarts in the request had uploaded results and would not be marked as "Complete" if one or more DataMarts rejected the request. Now, "Completed", "Rejected at DMC", and "Rejected After Upload" are all considered equal responses to the request.

In addition, corrections were made to appropriately adjust the number of responses received versus outstanding routings based on changes to the DataMart routings.

The new rules are as follow:

Routing statuses of DataMarts in the requestRequest status
All "Submitted" or "Held""Submitted"
All "Complete", "Rejected at DMC", or "Rejected After Upload""Complete"
All "Canceled""Canceled"
All "Resubmitted""Resubmitted"
All "Awaiting Request Approval""Awaiting Request Approval"
At least one DataMart routing "Awaiting Approval""Awaiting Response Approval"

At least one DataMart routing "Complete", "Rejected at DMC", or "Rejected After Upload"

No DataMarts "Awaiting Approval"

"x/x Responses Received"
"Additional Instructions" request metadata fieldA new "Additional Instructions" field has been added to the request metadata. The field is available to DataMart Administrators in the Request Details screen for a request and is included in the improved New Request Submitted email notification. The field has a 3000 character limit and respects line breaks.
"Description" request metadata character limitThe "Description" request metadata field has a new 400 character limit. This ensures that the request descriptions that appear in Network Activity Reports and Request Searches are limited for improved display. The larger "Additional Instructions" field was implemented in conjunction to ensure that requesters continue to have space to include all relevant information in the request metadata.
Code selector wizardThe Code Selector wizard for defining diagnosis and zip codes within request parameters was updated for improved responsiveness and a simpler code selection process.
New request dialogThe dialog for selecting a new request type was updated by removing the drop-down list and displaying request models and request types side by side for increased visibility of available request types.
Relocated DataMart Client installer downloadThe DataMart Client installer download is now located on the Resources page.
Canceling all DataMart routingsAll DataMart routings in a request may now be canceled. Previously, it was required that at least one routing remain in a request.
Rejected DataMart routings appear in Received Responses panel

Previously, if a DataMart routing for a request was rejected, the routing would remain in the Incomplete Responses panel. Now, rejections are considered an equally valid response as uploading results and appear in the Received Responses panel.

 

File upload controlsNew file upload controls were implemented to simply attaching files to a request. Previously, an attached file would remain in a "Pending" state until the user clicked "Upload". Now, attached files are automatically uploaded to the request.

Reason for account deactivation

When user accounts are deactivated, the active user is prompted to enter a reason for deactivation. This message is displayed on the deactivated account for users with permissions to manage the account.
User registration workflowThe user registration process has been changed such that the user registration screen is based on the same template at the user profile. A new user registration is considered an Inactive user profile and must be activated as part of the registration process. Upon activation, the user will receive an email that their account was approved.
Task list

The task list has been updated to include more functionality. Tasks may now be edited to include a priority, status, percent complete value, rich-text description, and start, due, estimated completion, and completion dates. The task list is relevant for the user registration workflow, but will be extended to other workflows in future development work.

Password strength indicatorNew password strength indicator when a user is registering for a new account. A password meets the criteria if it is at least 8 characters, contains at least 1 number, 1 lowercase letter, 1 uppercase letter, and 1 special character.

 

DataMart Client Enhancements

UpdateDescription
DataMart Client forward compatibilityThe DataMart Client has been enhanced to automatically download the packages for all requests that are needed to run it. If a request type is updated with more information or a different way of communicating with the data source behind it, these packages are automatically downloaded using the same protocol as the request itself and run within a highly secure sandbox environment. As a result, the DMC will be forward compatible with all new request types. This will allow for more frequent updates of PopMedNet functionality without increasing the burden on users to reinstall the DataMart Client. Updates to the DataMart Client will only be necessary when new functionality is added to the DataMart Client software itself.
New metadata fields viewable on Request Detail pagesAll request metadata entered by the requester is now available in the DataMart Client when viewing details of a request. Newly available fields include Purpose of Use, Level of PHI, Task Order, Activity, Activity Project, Requester Center, Workplan Type, and Additional Instructions.
Ability to manage multiple sets of network credentials in a single DataMart Client instancePreviously, if a single instance of the DataMart Client was configured to connect to a network with multiple sets of user credentials, only one set of credentials would be remembered when the DataMart Client was restarted. Now, all user credentials are saved.

 


Bug Fixes

UpdateDescription
Network messages for deactivated usersFixes a bug where network messages were sent to deactivated users. Now, only active users will receive network messages.
Floating tip textFixes a bug where tip text would persist, even after navigating away from the page.
Viewing files distributed with a requestFixes a bug where File Distribution and Modular Program requests would display "No files uploaded" in the "Uploaded Files" section of the Request Details page even if files had been successfully distributed with the request. Now, all files appear in the Request Criteria tab on the Request Details page.
Uploaded files in draft requestsFixes a bug where files previously uploaded to a saved draft request would not appear in the "Files previously uploaded" section of the request page, but would still be distributed to the DataMarts. Now, all files appear in the "Previously uploaded for this request" section of the request composer page.
Copying requests with attached filesFixes a bug with copying requests with attached files where the files would copy to the new request, but wouldn't be visible and would still be distributed to the DataMarts. Now, copied files appear in the "Previously uploaded for this request" section of the request composer page.

Viewing all DataMart responses

Fixes a bug where it would appear as if all DataMarts had been selected before viewing results, but only the first visible page of DataMarts would actually be selected. Now, all DataMarts appear in an unpaginated list and may be all selected at once.

Submitting a request with incompletely uploaded filesFixes a bug where requests could be submitted even if a file was still in the process of uploading to the request. Now, request will wait to submit until all attached files have finished uploading.
Resubmitting Metadata Search requestsFixes a bug where Metadata Search requests could not be resubmitted. Now, Metadata Search requests may be resubmitted and the system will automatically run the request. The page will not automatically redirect to the results, but the results may be viewed from the Request Details page.
DataMart Audit Report "Open Days" column

Fixes a bug with the calculation of the open days values for requests in the DataMart Audit Report where the values would always populate as days from submission to the date that the report was run, regardless of the status of the request. The new rules are as follows:

  • Canceled routings: 0 days
  • Incomplete routings ("Submitted", "Held"): current date - date submitted
  • Complete routings ("Completed", "Rejected Before Upload", "Rejected After Upload", "Awaiting Request Approval"): response date - date submitted

 


Features Removed

FeatureChange
Viewing access control inheritance

The ability to see when and where access controls were granted to a security group at a higher level in the access control hierarchy was removed due to a faulty implementation in the previous versions. Despite not being shown, inheritance is still respected. This feature will be re-implemented in future development work.

Event log reportThe Event Log Report was removed due to a faulty implementation in the previous versions. New reporting functionality is being researched and will be implemented in future development work.
Built-in security groupsBuilt-in security groups were removed from Organizations and Projects to reduce the number of security groups that appear when administering access controls.
Per-user permissionsPermissions may now only be applied to security groups and not to individual users.
Non-administrator visibility of personal security group membershipPreviously, users could view the security groups that they belonged to on their user profile. As a security enhancement, this feature was removed.
Default availability of Results Viewed and Results Reminder notification subscription optionsPreviously, every user had the default ability to subscribe to Results Viewed and Results Reminder notifications in relation to their own requests. Due to faulty implementation of these notification types, their default availability was removed.

 


Changes to Access Control Permissions

A number of changes were made to individual access control permissions. These changes are detailed below for reference for Network Administrators.

PermissionChange 
Site-wide permissionsPreviously only select permissions were available at the site-wide level. Now, almost all permissions may be applied at the site-wide level. 
LoginThis permission was removed. Any user with an active account has the right to login by default. To revoke access, a user's account should be deactivated. 
See DataMart Request QueuePreviously this permission only applied to visibility of requests in the DataMart Client application. Now it also controls whether users may view requests distributed to the DataMart on the Portal. Note that See DataMart Request Queue is not available at the site-wide or Project level. 
List RequestsThis permission was removed as its functionality duplicated the View Requests permission. 
Event: Project AssignmentThis permission was removed as it did not have any corresponding functionality. 
View Registry, Edit Registry, Manage Registry, Delete Registry

Previously these rights could only be applied at the site-wide level for every Registry/Research Data Set in the network. Now the rights are also available on each Registry/Research Data Set profile for a finer level of access control management for the entities.

 

 


Contact Us

PopMedNet Service Desk