5.0 Release Notes
PopMedNet Distributed Query Tool
Release Notes
Based on Release 5.0
December 18, 2014
For a high level overview of the new features and enhancements in PopMedNet 5.0, please see New Features in 5.0
Releases
Date of Release | 1/5/2015 | 1/16/2015 | 2/23/2015 | 3/22/15 | 3/22/15 | 3/22/15 |
---|---|---|---|---|---|---|
Networks Updated | PCORnet DRN | Demonstration Site | MDPHnet | Mini-Sentinel | NIH | HDC |
DataMart Client Update | Required | Required | Required | Required | Required | Required |
Introduction
PopMedNet 5.0 is a major release that focuses on speed and ease of use and introduces major new functionality.
A new desktop DataMart Client is required.Version 5.0 is not backwards compatible with requests from version 4.0, but can run side by side with version 4.0 of the DataMart Client with special considerations and instructions.
These release notes cover PopMedNet version 5.0. Enhancements, bug fixes, and other changes are listed below.
Enhancements
Infrastructure and Architecture Enhancements
Update | Description |
---|---|
5000+% speed increase in most scenarios | Almost the entire portal and its backing API have been majorly updated to vastly improve performance |
Major security overhaul | Data storage for the security system has been simplified and the database "sharded" to result in 95% fewer records for security related operations in the database. This contributes to more than 60% of the system speed increase and significantly improves the scalability of the system. |
Major document management overhaul | Previously documents were stored as binary large objects (BLOBs) within the database. Now, documents are stored as file streams within SQL Server 2008 R2 or later. This means that files are stored encrypted on the file system directly instead of in the database itself and are simply linked to the row in the database. As a result, files can now be streamed instead of read to their entirety and then sent across the wire. This results in several benefits:
|
PopMedNet API | Previously, PopMedNet was an MVC.NET application that was completely self-contained and exposed certain Web Services as necessary for the DataMart Client. Now, PopMedNet is a fully multi-tier application with a full REST based API that supports both JSON, BSON, and XML request and response types. This API underpins all of the new PopMedNet functionality throughout the platform. This results in several benefits:
|
Database schema upgrade | Almost every table in the database was optimized with better index, data types, and timestamps for optimistic concurrency. In addition, the Primary Keys on tables were upgraded to use Globally Unique Identifiers (GUIDs). This allows for records to be exchanged between multiple systems and maintain uniqueness constraints preventing conflicts. Furthermore, these GUIDs are sequential and thus can participate in clustered indexes, providing the best of both Integer based keys that are fast and typical GUIDs which provide uniqueness. Because the GUIDs were previously used as other fields in the same tables, the tables have shrunk in size by approximately 32 bits per record, a significant decrease across hundreds of thousands of records. Tables names have also been updated to reflect the naming in the software. Foreign Keys and how they are linked were simplified and many Foreign keys were optimized to use composite keys where appropriate. |
Web Portal Enhancements
Update | Description | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Major user interface overhaul | Bootstrap has been implemented for an easier to use and more consistent user interface. The new user interface includes incremental lookup drop-down controls, combo boxes, pop-overs, and popup windows. Additionally, this prepares the system to support mobile browsers and customizing the user interface for each PopMedNet platform. The system now provides greater feedback on errors immediately instead of only showing errors after submission or saving of the page. Data fields are also properly limited in size on the client side and data types have been strongly enforced with improved UI controls for data input. | ||||||||||||||||
Email notifications | All email notifications have been redesigned to include more information, more natural language, and bolded text for easier parsing of the email content. Subject lines now include the name of the notification to improve user's ability to manage the flow of incoming notifications. | ||||||||||||||||
Immediate email notifications | Previously, email notification subscriptions set to send "Immediately" would send on a schedule of up to fifteen minutes. Now, email notifications are sent as soon as the triggering action is complete. | ||||||||||||||||
Viewing request details as a DataMart Administrator | Users may now view details of requests distributed to their DataMarts from the Portal, in addition to in their DataMart Client applications. They may view the request metadata and parameters, the status of their DataMart routing for the request, and any files or results uploaded in response to the request.
| ||||||||||||||||
Table sorting and filtering enhancements | All tables in the Portal feature new grid controls that allow any column to be filtered, sorted, and rearranged. Filtering and sorting preferences are saved across sessions. | ||||||||||||||||
Terms and Conditions agreement | The checkbox for agreeing to Terms and Conditions was removed so that users no longer have to check it each time they log into the Portal. Text was added to the login page to specify that users are agreeing to the Terms and Conditions of the network by clicking Login. | ||||||||||||||||
Request statuses rewritten | Request statuses were rewritten to more intuitively reflect the DataMart routing statuses within the request. Previously a request was only marked as "Complete" if all DataMarts in the request had uploaded results and would not be marked as "Complete" if one or more DataMarts rejected the request. Now, "Completed", "Rejected at DMC", and "Rejected After Upload" are all considered equal responses to the request. In addition, corrections were made to appropriately adjust the number of responses received versus outstanding routings based on changes to the DataMart routings. The new rules are as follow:
| ||||||||||||||||
"Additional Instructions" request metadata field | A new "Additional Instructions" field has been added to the request metadata. The field is available to DataMart Administrators in the Request Details screen for a request and is included in the improved New Request Submitted email notification. The field has a 3000 character limit and respects line breaks. | ||||||||||||||||
"Description" request metadata character limit | The "Description" request metadata field has a new 400 character limit. This ensures that the request descriptions that appear in Network Activity Reports and Request Searches are limited for improved display. The larger "Additional Instructions" field was implemented in conjunction to ensure that requesters continue to have space to include all relevant information in the request metadata. | ||||||||||||||||
Code selector wizard | The Code Selector wizard for defining diagnosis and zip codes within request parameters was updated for improved responsiveness and a simpler code selection process. | ||||||||||||||||
New request dialog | The dialog for selecting a new request type was updated by removing the drop-down list and displaying request models and request types side by side for increased visibility of available request types. | ||||||||||||||||
Relocated DataMart Client installer download | The DataMart Client installer download is now located on the Resources page. | ||||||||||||||||
Canceling all DataMart routings | All DataMart routings in a request may now be canceled. Previously, it was required that at least one routing remain in a request. | ||||||||||||||||
Rejected DataMart routings appear in Received Responses panel | Previously, if a DataMart routing for a request was rejected, the routing would remain in the Incomplete Responses panel. Now, rejections are considered an equally valid response as uploading results and appear in the Received Responses panel.
| ||||||||||||||||
File upload controls | New file upload controls were implemented to simply attaching files to a request. Previously, an attached file would remain in a "Pending" state until the user clicked "Upload". Now, attached files are automatically uploaded to the request. | ||||||||||||||||
Reason for account deactivation | When user accounts are deactivated, the active user is prompted to enter a reason for deactivation. This message is displayed on the deactivated account for users with permissions to manage the account. | ||||||||||||||||
User registration workflow | The user registration process has been changed such that the user registration screen is based on the same template at the user profile. A new user registration is considered an Inactive user profile and must be activated as part of the registration process. Upon activation, the user will receive an email that their account was approved. | ||||||||||||||||
Task list | The task list has been updated to include more functionality. Tasks may now be edited to include a priority, status, percent complete value, rich-text description, and start, due, estimated completion, and completion dates. The task list is relevant for the user registration workflow, but will be extended to other workflows in future development work. | ||||||||||||||||
Password strength indicator | New password strength indicator when a user is registering for a new account. A password meets the criteria if it is at least 8 characters, contains at least 1 number, 1 lowercase letter, 1 uppercase letter, and 1 special character. |
DataMart Client Enhancements
Update | Description |
---|---|
DataMart Client forward compatibility | The DataMart Client has been enhanced to automatically download the packages for all requests that are needed to run it. If a request type is updated with more information or a different way of communicating with the data source behind it, these packages are automatically downloaded using the same protocol as the request itself and run within a highly secure sandbox environment. As a result, the DMC will be forward compatible with all new request types. This will allow for more frequent updates of PopMedNet functionality without increasing the burden on users to reinstall the DataMart Client. Updates to the DataMart Client will only be necessary when new functionality is added to the DataMart Client software itself. |
New metadata fields viewable on Request Detail pages | All request metadata entered by the requester is now available in the DataMart Client when viewing details of a request. Newly available fields include Purpose of Use, Level of PHI, Task Order, Activity, Activity Project, Requester Center, Workplan Type, and Additional Instructions. |
Ability to manage multiple sets of network credentials in a single DataMart Client instance | Previously, if a single instance of the DataMart Client was configured to connect to a network with multiple sets of user credentials, only one set of credentials would be remembered when the DataMart Client was restarted. Now, all user credentials are saved. |
Bug Fixes
Update | Description |
---|---|
Network messages for deactivated users | Fixes a bug where network messages were sent to deactivated users. Now, only active users will receive network messages. |
Floating tip text | Fixes a bug where tip text would persist, even after navigating away from the page. |
Viewing files distributed with a request | Fixes a bug where File Distribution and Modular Program requests would display "No files uploaded" in the "Uploaded Files" section of the Request Details page even if files had been successfully distributed with the request. Now, all files appear in the Request Criteria tab on the Request Details page. |
Uploaded files in draft requests | Fixes a bug where files previously uploaded to a saved draft request would not appear in the "Files previously uploaded" section of the request page, but would still be distributed to the DataMarts. Now, all files appear in the "Previously uploaded for this request" section of the request composer page. |
Copying requests with attached files | Fixes a bug with copying requests with attached files where the files would copy to the new request, but wouldn't be visible and would still be distributed to the DataMarts. Now, copied files appear in the "Previously uploaded for this request" section of the request composer page. |
Viewing all DataMart responses | Fixes a bug where it would appear as if all DataMarts had been selected before viewing results, but only the first visible page of DataMarts would actually be selected. Now, all DataMarts appear in an unpaginated list and may be all selected at once. |
Submitting a request with incompletely uploaded files | Fixes a bug where requests could be submitted even if a file was still in the process of uploading to the request. Now, request will wait to submit until all attached files have finished uploading. |
Resubmitting Metadata Search requests | Fixes a bug where Metadata Search requests could not be resubmitted. Now, Metadata Search requests may be resubmitted and the system will automatically run the request. The page will not automatically redirect to the results, but the results may be viewed from the Request Details page. |
DataMart Audit Report "Open Days" column | Fixes a bug with the calculation of the open days values for requests in the DataMart Audit Report where the values would always populate as days from submission to the date that the report was run, regardless of the status of the request. The new rules are as follows:
|
Features Removed
Feature | Change |
---|---|
Viewing access control inheritance | The ability to see when and where access controls were granted to a security group at a higher level in the access control hierarchy was removed due to a faulty implementation in the previous versions. Despite not being shown, inheritance is still respected. This feature will be re-implemented in future development work. |
Event log report | The Event Log Report was removed due to a faulty implementation in the previous versions. New reporting functionality is being researched and will be implemented in future development work. |
Built-in security groups | Built-in security groups were removed from Organizations and Projects to reduce the number of security groups that appear when administering access controls. |
Per-user permissions | Permissions may now only be applied to security groups and not to individual users. |
Non-administrator visibility of personal security group membership | Previously, users could view the security groups that they belonged to on their user profile. As a security enhancement, this feature was removed. |
Default availability of Results Viewed and Results Reminder notification subscription options | Previously, every user had the default ability to subscribe to Results Viewed and Results Reminder notifications in relation to their own requests. Due to faulty implementation of these notification types, their default availability was removed. |
Changes to Access Control Permissions
A number of changes were made to individual access control permissions. These changes are detailed below for reference for Network Administrators.
Permission | Change | |
---|---|---|
Site-wide permissions | Previously only select permissions were available at the site-wide level. Now, almost all permissions may be applied at the site-wide level. | |
Login | This permission was removed. Any user with an active account has the right to login by default. To revoke access, a user's account should be deactivated. | |
See DataMart Request Queue | Previously this permission only applied to visibility of requests in the DataMart Client application. Now it also controls whether users may view requests distributed to the DataMart on the Portal. Note that See DataMart Request Queue is not available at the site-wide or Project level. | |
List Requests | This permission was removed as its functionality duplicated the View Requests permission. | |
Event: Project Assignment | This permission was removed as it did not have any corresponding functionality. | |
View Registry, Edit Registry, Manage Registry, Delete Registry | Previously these rights could only be applied at the site-wide level for every Registry/Research Data Set in the network. Now the rights are also available on each Registry/Research Data Set profile for a finer level of access control management for the entities. |