Best Practices for Law Enforcement Data Collection and Transparency

Mar 30, 2023

(Illustration: Renzo Velez / POGO; Photos: Getty Images; Chad Davis (CC BY 2.0))

Arati Prabhakar
Director
Office of Science and Technology Policy
Executive Office of the President
Eisenhower Executive Office Building
1650 Pennsylvania Avenue
Washington, D.C. 20504

Subject: Best Practices for Law Enforcement Data Collection and Transparency

Dear Director Prabhakar:

The Project On Government Oversight (POGO) is pleased to submit this comment in response to the Office of Science and Technology Policy’s (OSTP) request for information regarding criminal justice statistics.

POGO is a nonpartisan independent watchdog that investigates and exposes waste, corruption, abuse of power, and when the government fails to serve the public or silences those who report wrongdoing. We champion reforms to achieve a more effective, ethical, and accountable federal government that safeguards constitutional principles.

This comment contains two sections. In the first, we offer general best practices for federal data programs, based on POGO’s longstanding work to support accurate and transparent government data. In recent years, the federal government has pursued an open data strategy, with the creation of Data.gov as a new portal to databases government-wide, and with agencies proactively posting databases both on Data.gov and their own websites. POGO supports such activities as a necessary and effective method of keeping the people informed, so they may better participate in our democracy. However, the recent steps forward on data transparency have not been evenly implemented across all agencies. Databases deemed challenging to disclose have often remained unaddressed. Data about the criminal legal system certainly falls into this category, as public disclosures remain seriously lacking.

In the second portion of this comment, we focus on one current data collection program that requires immediate reform: the Death in Custody Reporting Act (DCRA).¹ The problems with DCRA implementation, and their solutions, require attention and action because gathering accurate, comprehensive data on deaths in custody is an important step toward preventing similar deaths in the future. The lessons of DCRA also illustrate potential traps other data programs should avoid to ensure accurate, transparent data on law enforcement practices. This is especially true because DCRA is typical of data programs in which federal agencies must work with their counterparts in state, local, tribal, and territorial governments to ensure effective data collection.

Open Data Principles: Best Practices for Federal Data Programs

Access to timely, accurate, and complete data is a fundamental principle for an open and accountable government. Transparency and public disclosure of government data should always be the default position for government data, including law enforcement data, unless there are compelling reasons to limit access. Public disclosure of government data can increase accountability, oversight, trust, and informed decision-making, all of which are essential for ensuring effective and just law enforcement.

While there are concerns, such as privacy, that can require some limits on the public disclosure of government data, those concerns should inform how to disclose the data — the format and extent of the disclosure. They should not be used as an excuse to withhold information from the public.

POGO urges OSTP to maximize public disclosure of law enforcement data. There are several clear and important benefits to an open data approach.

Benefits of Open Data

The first and most direct benefit is increased trust. At a time when public trust in government, and in law enforcement specifically, is disappointingly low, it is important to take all available steps to restore that public trust. Transparency is key to improving public trust. When information is unavailable or intentionally withheld, many will assume the worst.

Greater accountability for law enforcement is another major benefit of data disclosure. Public access to law enforcement data can increase accountability by providing transparency into police actions including use of force incidents, complaints against officers, and investigations into misconduct. Public access to this data enables oversight by independent organizations, journalists, and members of the public. This can help ensure that law enforcement agencies are adhering to the law and upholding constitutional rights.

Public access to law enforcement data can also directly lead to better law enforcement outcomes. Making law enforcement data public means more people will review the data and help identify problem areas. Increased review means more chances to find things that might otherwise be missed or dismissed. Armed with information, community members can more effectively push for specific improvements in law enforcement performance, training and oversight.

Finally, disclosure of data can also lead to better data in the long run. When data is made public, more people review and use the data. They provide feedback on data problems and needed improvements. Data users with different perspectives, such as researchers, journalists, or community residents, can raise concerns or offer recommendations that would elude government officials. Transparency also encourages accountability: agencies are more motivated to fix and improve databases that are publicly available to avoid scrutiny and criticism.

Addressing Privacy Concerns

POGO acknowledges there can be privacy concerns, and some sensitive information related to investigations will need to be protected. But there are several data approaches used by other government agencies to anonymize or de-identify data so it can be made public while protecting individuals’ privacy. Simply withholding identifying information such as names, addresses, social security numbers, phone numbers, and other personal identifiers from a dataset can often allow the other data to be disclosed and used by the public.

Aggregation is another common method used to disclose data while ensuring that individuals cannot be identified. Data can be reported out for a whole city or county, or data for a particular facility can be reported out for a week or month to prevent individual details from being discovered. The agencies could establish a standard for the minimum number of data points needed in an aggregate collection to avoid easy identification of individuals. Collections below that threshold should be withheld and only disclosed when further aggregated with data from different locations, time periods, or other elements. However, it is essential that aggregation be used to the minimum extent that privacy laws require, and that it not be used to obscure merely unflattering information.

While no anonymization or de-identification method is foolproof, there are several widely practiced techniques that minimize the risk of identification. The DOJ should utilize these techniques to maximize disclosure of data. The databases should be monitored for potential problems and any identified issues or breaches should be addressed quickly.

Overall, making law enforcement data public can increase transparency, accountability, trust, and informed decision-making, all of which are essential for ensuring effective and just law enforcement.

Death in Custody Reporting

This section examines the problems with, and potential solutions to, the implementation of the Death in Custody Report Act (DCRA). DCRA requires states and territories that receive funding from the Byrne Justice Assistance Grant program to report data on deaths that occur in the custody of state, territorial, and local agencies. The law requires the Justice Department to study that data and issue a report to Congress on how the data can be used to reduce deaths in custody.

But as POGO and the Leadership Conference Education Fund recently detailed in our report, A Matter of Life and Death, the Justice Department has, for nearly a decade, failed to collect this statutorily mandated data.² In September 2022, the Government Accountability Office found that the department’s FY 2021 data was missing nearly 1,000 deaths that had been counted in other public sources.³ Over two thirds of the entries the department had were incomplete.

The ongoing failure of DCRA helps perpetuate the crisis of in-custody deaths: The program is intended to reduce deaths in custody by facilitating data-driven reforms. The Justice Department’s implementation missteps also raise questions about its ability to meaningfully engage with state, local, territorial, and tribal partners to collect robust data about the criminal legal system.

However, if properly implemented, the program could become a model for similar data efforts.

Challenges to Address

Most of our recommendations address four key areas where DCRA implementation can and should be improved: compliance, data collection methods, research scope, and transparency.

Compliance

DCRA cannot successfully serve its purpose if the agencies and jurisdictions that are required to report data fail to do so without consequence. The law gives the attorney general the authority and discretion to withhold up to 10% of a noncompliant state’s funding under the Byrne Justice Assistance Grant program.⁴ As mentioned above, state and local data collection under DCRA has yielded deeply flawed data. However, DOJ has yet to develop criteria for DCRA compliance, and it has indicated it is disinclined to use the statutory penalty.⁵

We recognize concerns that the compliance mechanism of a 10% grant reduction to a state’s grant allocation is a blunt tool, as that reduction may not affect all local agencies. However, as the penalty has never been imposed, we have been unable to gauge its effectiveness. While it is true that the state-level penalty may not give every local agency a financial incentive to comply with the reporting requirement, it may be a sufficient incentive for states to leverage their own power over local jurisdictions to ensure compliance. The only compliance tool guaranteed to fail is one that is never used.

Beyond enforcement actions, DOJ has not engaged sufficiently with states to ensure their own data collection programs are adequate. For instance, last fall the department announced it would require states to submit collection plans with their Byrne JAG applications.⁶ This practice, which was first proposed in 2016 but never implemented, would force states to clarify their plans and allow them to work with DOJ to improve their efforts, and should be standard practice for any program that relies on federal partnerships with other levels of government.⁷

Data Collection Methods

In addition to compliance, the methodological weaknesses in how data is currently collected would lead to such poor data quality as to jeopardize its usefulness. The significant flaws present in DOJ’s existing collection methods under DCRA must be fixed in their own right, but they also offer lessons for other data collection tools employed by the federal government to collect information for state and local agencies on a variety of criminal legal system matters.

Fundamentally, the current form used by the Bureau of Justice Assistance to collect state and local death data is not able to guarantee complete or accurate information.⁸ The form attempts to capture almost all non-demographic data in a single qualitative text field with the following instructions: “Please provide a brief description of the circumstances leading to the death (e.g., details surrounding an event that may have led to the death, the number and affiliation of any parties involved in the incident, the location and characteristics of an incident, other context related to the death, etc.).”

The open-ended approach runs the risk that not all relevant information will be captured and complicates subsequent data analysis by using text rather than standardized data fields.

Finally, the track record of earlier efforts to count arrest-related deaths suggests that relying solely on self-reported data from agencies will likely result in undercounts.⁹ The department recently revived plans to audit the data states submit.¹⁰ This sort of validation is essential to ensuring accurate data in DCRA context and other criminal legal system areas as well.

Research Scope

DCRA mandates that DOJ answer two broad research questions.¹¹ The first is to analyze how to use the data collected to reduce the number of deaths in custody. The second is to find relationships, if any, between deaths in custody and administrative policies. But there are no data collection plans that would produce the information necessary to deliver on this requirement, particularly relating to administrative policies. Most notably, answering these questions requires an analysis of agency and facility policies, such as those pertaining to use of force. These policies change over time and vary across agencies and departments. To obtain the data necessary to conduct an accurate and relevant analysis of the impact of policies on deaths, data collection on agency policies would have to be essentially ongoing. Based on the data currently being collected at this time, DOJ will be unable to fulfill the research requirement outlined in the statute.

Transparency

The DCRA statute does not explicitly require the public release of data beyond the report to Congress. However, the intent of the law is plainly to increase public understanding of deaths in custody. This congressional intent is reflected most recently in the Joint Explanatory Statement accompanying the FY 2023 Commerce, Justice, Science, and Related Agencies appropriations bill, which calls on the department to provide an explanation of how it plans to “improve the quality and transparency of future data” collected under DCRA.¹²

Troublingly, after initially promising to release DCRA data, the department has spent years walking back that position and is now actively fighting efforts to make the data public.¹³ The department claims that it cannot proactively release the data to any entity but Congress, and it is currently fighting efforts to obtain records through the Freedom of Information Act.¹⁴ These positions needlessly undermine the administration’s stated goal of increasing the transparency of key data about law enforcement and the corrections system.

A fundamental responsibility of government is to ensure the care and safety of individuals within its custody. Withholding crucial information from the public concerning deaths in custody not only undermines the spirit of DCRA, but also prevents the public from holding the government accountable and pursuing reforms. DCRA data, like other data about the functioning of the criminal legal system, addresses fundamental issues of life and liberty. It must be transparent in order to build trust and ensure the system functions in a just and equitable manner.

Recommendations¹⁵

Administration and Compliance

DOJ must ensure that reporting agencies have a clear understanding of their obligations and the consequences of noncompliance. The department should:

Refine and update coordination, definitions, and guidance for data collectors at the state and local levels. Providing consistent attention to updated, clear, standardized guidance and collaborative technical assistance will positively impact compliance, data collection, research, and transparency.
Begin annual compliance checks immediately, and make the findings public.
Use all available data, including from open sources, to assess compliance. Full compliance requires reporting complete and accurate information, which cannot be known without some sort of audit.
Issue clear standards for when and how the 10% penalty will be imposed. Impose the penalty on noncompliant states.

Data Collection Methods

The government cannot collect inferior data and expect quality research as a result. Collection forms must be designed to collect sufficient data to thoroughly study deaths in custody.

To the extent possible, questions should be specific, with checkbox or multiple choice answers that allow for clear data comparison.
Demographic variables for race, ethnicity, and age brackets should, at a minimum, meet the OMB Directive 15 reporting standards, which will allow for easy use and comparison across government data collections.¹⁶
All forms should include one question at the end of the survey with an open text box answer where additional details about the death in custody can be reported. However, the forms should not rely on text boxes as the primary or only means of eliciting key information.
The department should continue to enforce the recent requirement for states to submit data collection plans and review and revise those plans annually.

Research Scope

DCRA data, like many data collections, provides information about specific events. But to inform policy responses, it is also necessary to understand the policies that were in place at the time of the incidents. This information should be collected.

Develop a plan to gather data on relevant agency and facility policies.

Transparency

Without data transparency, policymakers and the public will not be able to use DCRA data to inform efforts to reduce deaths in custody. The Department of Justice should:

Release existing data at least at agency or facility level in machine-readable formats.
Ensure that public reports include highly salient data, including the number of killings by law enforcement officials.
Where applicable, clarify whether deaths occurred in agency-owned facilities or contract facilities.
Publish annual compliance plans for states that are available to the public.

We appreciate the administration’s attention to this urgent matter. If you have any questions, please don’t hesitate to contact Sean Moulton at [email protected] or David Janovsky at [email protected].