Model and data management 1, 2018

A Practical Guide to Managing Risk in Machine Learning Models

Global

Actors

Future of Privacy Forum, Immuta,

Tags

Risk management,

Resources


The Future of Privacy Forum is a US-based non-profit organisation dedicated to privacy leadership and scholarship.



This white paper provides a template for managing the practical risks of deploying an individual machine learning model. It is primarily aimed at lawyers, compliance personnel, data scientists and engineers. The paper highlights the importance of clearly documenting the initial objectives and underlying assumptions of any ML project.



Building on the US Federal Reserve’s SR 11-7 regulatory standard on model risk management, the authors of the white paper put forward a “three lines of defence” approach. This approach assigns specific roles and responsibilities to different actors involved in the creation, deployment, management and auditing of ML systems throughout the system’s entire lifecycle. The first line of defence is focused on the development and testing of models, the second line on model validation and legal and data review, and the third line on periodic auditing.



The framework also offers guidance on minimising the risks related to input data by documenting model requirements, assessing data quality, encapsulating the model, monitoring for data drift, and creating actionable alerts.



Finally, the framework offers recommendations on using outputs from the ML model as a way of assessing and understanding the workings of the model itself.



The paper describes effective risk management as an ongoing, never-ending task. The framework includes a comprehensive Model Management Checklist to help operationalise the 3 lines of defence outlined in the paper.

AI Governance
Database

This database is an information resource about global governance activities related to artificial intelligence. It is designed as a tool to help researchers, innovators, and policymakers better understand how AI governance is developing around the world.

We're collecting more examples as we find them. If you'd like to add yours to this map, please suggest an entry.

Suggest an entry
  • Filter by:
  • Current filters:
256 entries are listed below
  • Sort by: