The goal of this Framework is to encourage the use and development of high quality, safe, and effective digital health apps. The development of this first version reflects several key principles, including:
- Allowing only objective criteria to be used to ensure consistency of interpretation across different assessors
- Ensuring the use of only publicly available information (or information that should be made publicly available) to enable the framework to be used independently of app owners.
- Domains such as Technical Security that require app developer involvement are included as additional layers within the Framework.
- Considering differing levels and standards of review depending upon the functionality of an app, to ensure that the criteria used in the Framework are proportionate to the risk posed,
- Utilizing where relevant already existing and accepted standards, i.e. in Data Privacy or Usability and Accessibility, and aligning with emerging international standards and criteria
This Framework supports and augments the due diligence organizations may use and is not intended to replace internal due diligence of organizations wishing to integrate digital health apps into their systems of care. It should be viewed as a foundational layer to any such assessment and other layers of scrutiny should be applied dependent on the nature of the app and how it will be used.
The Assessment Framework score aims to deliver an evaluation where all Apps are treated equally and fairly. The primary mechanism is a ‘tariff’ based model, the aim of the scoring is ultimately to reward best practice and highlight poor practice and non compliance. The mechanisms used are designed to ensure that wherever possible the score reflects relative performance and properly differentiates between similar apps.
The line of compliance is set at 65%. Any score below 65% would indicate that an App has some issues that users should investigate further prior to using this App. Scores below 45% indicate that an App has considerable issues or challenges and in its current form is potentially unhelpful or unsafe.
What is included in the Framework?
The assessment begins with a series of questions to capture an app’s core purpose and functionality. These include the target audience, the type of data the app collects and the app’s primary functions and features. None of the questions are intended to have any scoring or risk implications and are purely used to capture important information about the core purpose and use of the app as well as allowing for assessors to gain an understanding of what ‘compliance’ criteria will be relevant to the app in question i.e. “does it collect data?”.
The Framework then comprises of assessment criteria across four categories:
| Data and Privacy | This includes adherence to federal laws designed to protect the rights and freedoms of individuals. It also takes a deeper dive into laws that are specific to individual products, such as the need for HIPAA compliance and COPPA. Find out more. |
| Clinical Assurance and Safety | This field looks to establish is the app safe and effective. This is by reference to the NICE Evidence Standards Framework, where requirements vary based on the functionality of the app. For patient safety, it also seeks to clarify if the app is a medical device as outlined by FDA Digital Health Requirements. Find out more. |
| Usability and Accessibility | Within this area, the Framework looks at design and development, checking for accessibility, usability and user support. It also looks for compliance with design standards. Find out more. |
| Technical Security and Stability | Criteria here looks to identify the technical fit of a product. For example, if an app is built in compliance with the secure coding practice guidelines of the OWASP (Open Web Application Security Project). Find out more. |
How is the Framework Scored?
The scoring follows closely the method used by ORCHA in many national and pan-national assessment programs, and has been subjected to extensive scrutiny over many thousands of assessments.
Technical Security is the only domain that does not contribute to the overall score, this is because it relies on non publicly available information. If an App is compliant with Technical Security they will receive a compliance badge that can be displayed alongside their overall score.
While a high scoring App is not guaranteed to be effective or safe or a poorly scoring App is not necessarily ineffective or unsafe, it does mean that the App has taken more or less care over the Apps compliance with these key Standards than other similar Apps. In the critical area of health and care, we believe that developers should take compliance with Standards extremely seriously. No matter how good the user experience of an App might be, if the App is not safe and robust or its treatment of often sensitive health data is not clear and correct, it should be treated with caution.
To view the full assessment framework documentation, please click here.
ACP’s collaboration on this project is an important step forward in identifying and creating digital health tools that are valuable and safe for our members and patients.
Ryan D. Mire, MD, FACP and ACP president