Give Now

We must close the loophole that allows law enforcement to buy our personal data without a warrant.

Championing Responsible National Security Policy

The History and Future of Mass Metadata Surveillance

Illustration by POGO.

This year, Congress has the chance to end the latest iteration of a long-running surveillance program that has violated Americans’ privacy and for a decade operated under secret and deeply problematic legal justifications. The Project On Government Oversight has called on Congress to end the “call detail records” program, which vacuums up a huge quantity of phone records from people the government does not suspect of wrongdoing. And when the law authorizing this program—Section 215 of the USA PATRIOT Act of 2001—expires in December, Congress should not only end the authority for the program, but enact additional reforms to protect Americans’ privacy rights from this type of improper surveillance.

If the history of these surveillance programs teaches us anything, it’s that the government can subvert seemingly clear legal standards and rules to accommodate its desire to move forward with surveillance activities.

Since the aftermath of 9/11, the executive branch has employed programs that collect domestic telephone metadata—that’s the data on who you call and receive calls from, when, and how long they last—on a massive scale, and skirted essential checks and balances. These programs, including the current call detail records program, have brought about unprecedented violations of Americans’ privacy, with little else to show for them.

If the history of these surveillance programs teaches us anything, it’s that the government can subvert seemingly clear legal standards and rules to move forward with surveillance activities. If Congress ends the call detail records program this year, it must make certain that it has really ended mass metadata surveillance, once and for all.

How Mass Collection of Telephone Metadata Operates

Originally, mass collection of domestic telephone metadata took the form of “bulk collection,” in which the government swept up the full call records of every customer of America’s biggest telephone companies, ostensibly so it could sift through the data and identify terrorism suspects based on call patterns. Bulk collection was banned in 2015 as part of the USA FREEDOM Act, which enacted a number of significant surveillance reforms, with this ban being the most notable.

But as a concession to gain intelligence community support, the sponsors of the law created the authority for the call detail records program as a new component of Section 215 of the USA PATRIOT Act of 2001. (In addition to the call detail records program, Section 215 authorizes the government to demand business records that are relevant to terrorism investigations.) In contrast to the bulk, nationwide orders the government had previously conducted under Section 215, the call detail records program was meant to preserve the technique of contact-chaining (checking if there are patterns between whom the contacts of a surveillance target calls and receives calls from) while reducing the scale of collection by only allowing the National Security Agency (NSA) to collect the phone records of a single target as well as phone records up to “two hops out” from the target.

This means with a call detail records order, the NSA acquires all metadata about the target’s incoming and outgoing calls (the “first hop”), and all phone records of the people who contacted the target (the “second hop”). Thus, under a single order, the NSA collects all the phone records of target and “first hop” individuals, and some of the phone records of “second hop” individuals.

Despite Major Reforms, Significant Action Is Needed

Even surveillance aimed at a single target under the call detail records program can quickly snowball, which means collection of phone metadata is highly damaging to individual privacy rights.

Newly released information confirms that the program has come at a huge cost to privacy. A recent annual transparency report from the Office of the Director of National Intelligence revealed that the NSA collected telephone metadata on 19,372,544 different phone numbers from late May 2018 until the end of the year. Although this is a substantial decrease from the old nationwide bulk collection program that swept up the full set of call records of over 100,000,000 individuals with a single order to AT&T or Verizon, it is nonetheless a massive invasion of privacy. (More on the old program and its history in a minute.)

The power of the program is clear when you consider that this collection of records from over 19 million phone numbers was based on a small number of targets. As the diagram below from the transparency report shows, collection rapidly expands to numerous targets.

There were 11 new targets in 2018, but collection that year may have carried over from orders for call detail records issued near the end of 2017, which encompassed 40 targets. This would make the average number of unique phone numbers collected per target at least 484,313. The frequency of robocalls likely inflates this number to some degree; as the transparency report notes, the over-19 million phone numbers include “numbers used by business entities for marketing purposes.” But even if we assumed that just 25 percent of these phone numbers were for individuals rather than automated calls, the average amount of phone numbers collected per target remains unacceptably large: If we considered the largest possible range of orders by additionally assuming that all 2017 orders carried over, and added that year’s 40 targets to 2018’s 11 targets, this would mean that each target designated under the call detail records program led to the collection of private call records for over 94,000 people.

Even without revealing the content of calls, the records of who you are calling and when can reveal the most intimate details about your life. Handing those data over to the government risks abuse, and gives the government power to stockpile information about our most intimate activities and interactions. And collecting that data from people who aren’t suspected of wrongdoing, as is the case with individuals in the first and second “hops” whose phone records are also collected, comes dangerously close to the type of general warrant the Founding Fathers believed must not be permitted in the United States.

In addition to this enormous cost to privacy, the call detail records program appears to have been a complete failure for intelligence collection. There is currently no public evidence the call detail records program has provided any unique value to national security. Technical problems with the program resulted in the NSA ending up with data it didn’t have the legal authority to possess, which in 2018 forced the NSA to purge all the call records it had collected since 2015. It reportedly later shut down the program entirely.

The impact of the call detail records program on Americans’ privacy is unacceptable. This program may have been a major step forward from nationwide bulk collection, but it is still mass collection of telephone metadata. It’s time for Congress to end the authority for the program.

As lawmakers consider ending the program and implementing further reforms, it’s important to look back on how this system began and evolved, and learn from our government’s past mistakes: secrecy, circumvention of checks and balances, and undue support for unscrupulous legal arguments and baseless national security claims.

Bulk Collection Begins Under the Terrorist Surveillance Program

Mass surveillance of telephone metadata began as a component of the George W. Bush Administration’s Terrorist Surveillance Program in the aftermath of the September 11 attacks. The Terrorist Surveillance Program (also called the “President’s Surveillance Program” and “Stellar Wind”) was based on a dubious interpretation of the executive power to wage the “war on terror.” Congress and the courts were entirely removed from the process of approving this system; nor were they notified when it went into effect.

This almost certainly violated constitutional requirements for separation of powers. The government generally treats Justice Robert H. Jackson’s concurrence in the watershed 1952 Youngstown Sheet & Tube Co. v. Sawyer decision as precedent, and abides by its argument that the president’s right to assert power is at its weakest when Congress has asserted its authority on an issue. Congress did just that in 1978 when it passed the Foreign Intelligence Surveillance Act (FISA), and many amendments and additions to it in the ensuing decades, including the USA PATRIOT Act of 2001, to set the parameters of national security surveillance.

However, the George W. Bush Administration put the Terrorist Surveillance program into action based on legal justifications written solely by then-Deputy Assistant Attorney General John Yoo during his tenure at the Justice Department’s Office of Legal Counsel (OLC). (Yoo is also notorious for writing the memoranda providing justification for torture during this period; the shoddiness of his legal analysis in both cases was attributed to the absence of the peer review typically required in the office.)

Disagreements Within the Executive Branch Over Bulk Collection

In 2004, a new head of OLC, Jack Goldsmith, found the justifications of aspects of the Terrorist Surveillance Program legally unsound, including for the part of the program that collected email metadata in bulk. This led to a conflict between the White House and the Department of Justice, culminating in the infamous “hospital visit.” On March 10, 2004, White House Chief of Staff Andrew Card and White House Counsel Alberto Gonzales went to the hospital where Attorney General John Ashcroft was recovering from surgery to ask him to reauthorize the email metadata collection program, but were confronted there by then-Acting Attorney General James Comey and Goldsmith. Ashcroft refused to sign the reauthorization, asserting that Comey had the relevant authority as acting attorney general. Comey shared Goldsmith’s view of the program and refused to reauthorize it.

Following this encounter, without Justice Department approval, Gonzales signed a new reauthorization absent DOJ approval. This was met with significant disapproval: Comey, then-FBI Director Robert Mueller, and a number of other high-ranking Justice Department and FBI personnel, including current FBI Director Christopher Wray, threatened to resign. In response, the White House merely shifted the legal justification for the program to a statutory authority.

Bulk Collection Shifts to PATRIOT Act Section 215

In 2006, a USA Today report revealed details of the telephone metadata bulk collection program, which had been running alongside the email metadata program, sparking increased public scrutiny. While the outlet reported that the government was conducting this surveillance without predicating it on suspicion of wrongdoing, the article did not include information on the scale of the program. In response, the Administration then argued, as it had with the email metadata collection program, that existing statutes permitted the telephone metadata program. 

Thus instead of ending bulk call detail record collection and asking Congress to authorize the surveillance systems it wanted—which would have complied with the constitutional separation of powers—the executive branch simply changed the legal justification for the program. Rather than claim unilateral executive authority, the White House said nationwide bulk collection of phone records was authorized by statute, specifically the provision of the PATRIOT Act known as “Section 215,” which broadly authorized the government to collect business records. This made the program subject to approval by the Foreign Intelligence Surveillance Court (commonly called the “FISA Court”) and more visible to Members of Congress, but the public was still unaware of its vast scale.

This legal justification was unsound. The executive branch’s rationale was that because Section 215 allowed the government to demand records that were “relevant” to terrorism investigations, it could demand that phone companies like AT&T and Verizon continuously supply all the phone records of all their customers in America because the network of connections was “relevant” to potentially assessing patterns. Courts, Congress, and independent experts would later blast this legal contention for rendering the concept of “relevance” virtually meaningless. However, the FISA Court accepted the government’s argument, effectively creating a secret law authorizing nationwide bulk collection under a provision of the PATRIOT Act intended to only permit targeted record requests.

The Fight to Ban Bulk Collection

The PATRIOT Act iteration of the telephone metadata bulk collection program was revealed to the public in June 2013 with the publication of the first story based on information provided by former NSA contractor Edward Snowden. The revelations that its scope was nationwide and that it was collecting the call records of every single American were met with outrage. Members of Congress, including the author of the PATRIOT Act, denounced the bulk collection program as beyond what Congress had authorized. For the first time since the September 11 attacks, more of the public believed that anti-terror policies had gone too far in diminishing civil liberties than believed such policies hadn’t gone far enough.

In the face of public scrutiny, the government’s paper-thin legal rationale for the program was shredded. The Privacy and Civil Liberties Oversight Board, an independent executive branch agency tasked with assessing the impact of counterterrorism activities on civil liberties, concluded in a scathing report that Section 215 of the PATRIOT Act did not in fact authorize the bulk collection program. A federal district court concluded that the program likely violated the Fourth Amendment. And the Second Circuit Court of Appeals ruled that the program was not permitted under Section 215.

The policy justification for the nationwide bulk collection program crumbled, too. The Obama Administration initially responded to the Snowden disclosures by claiming the programs had discovered or disrupted over 50 terrorist plots. But as Congress pressed the NSA to prove it, the number dropped from dozens to potentially four to just one, and that one case was revealed not to be a terrorist plot but a material support case involving just $8,000

The Privacy and Civil Liberties Oversight Board, as it announced in its report, also found that there was “no instance in which the program directly contributed to the discovery of a previously unknown terrorist plot or the disruption of a terrorist attack.” And the President’s Review Group on Surveillance, a specially created task force of experts including former high-ranking intelligence officials, concluded bulk collection “was not essential to preventing attacks and could readily have been obtained in a timely manner using conventional [i.e., targeted] section 215 orders.” Eventually, even the intelligence community acknowledged that ending the program would not harm national security.

Congress Passes the USA FREEDOM Act

Then, in 2015, as several provisions of the PATRIOT Act, including Section 215, were set to expire, Congress passed legislation ending bulk collection. The USA FREEDOM Act banned bulk collection by requiring that phone record requests be targeted by including a “specific selection term,” such as a name or phone number. The law also sought to improve the FISA Court by requiring disclosure of any decisions that created a “novel or significant” interpretation of law, and adding a role for an outside “special advocate” to defend privacy rights and bring broader perspective to important FISA Court deliberations.

In a compromise with the intelligence community, the bill created the authority for a new “call detail records” program to replace bulk collection—the “two hops” program on the books today. Lawmakers pushed back the expiration for Section 215 to December 2019. But it soon became apparent that the call detail records program, like its bulk collection predecessor, didn’t work.

What’s Next

Based on the call detail records program’s impact on the privacy of millions of Americans and its lack of security value, it’s clearly time for Congress to end the authority for it. Congress should also enact additional reforms to safeguard Americans’ privacy rights before considering extending the business records collection components of Section 215 other than the call detail records program.

First, Congress must fully ensure that secret law cannot be developed at the FISA Court and kept hidden from the public. The USA FREEDOM Act set an effective standard for requiring disclosure of FISA Court rulings that shape the law in a novel or significant way. But the law should also guarantee prompt public disclosure. The public needs to have reason to be confident that any major development or change to the law based on a court’s rulings behind closed doors is being revealed within a short, set timeframe.

Additionally, Congress needs to strengthen the “special advocate” role where outside lawyers are brought into important FISA Court proceedings as amici curiae—attorneys that act as “friends of the court” by providing legal insights without serving as an actual party in the case—to advance legal arguments to protect privacy rights. Unlike in other courts, there is otherwise nobody to argue against the government’s position, and often no notice of surveillance after the fact to those affected. The presence of a “special advocate” is a safeguard that helps prevent one-sided deliberations like those that enabled bulk collection for nearly a decade. So far, we’ve seen strong contributions from these amici to enhancing consideration of privacy and civil liberties during FISA Court deliberations. Legislative reforms would further strengthen and define their role. Such reforms should:

  • include a clear mandate for the special advocate to promote privacy and civil liberties;
  • include a measure to automatically trigger amici’s participation when the government is seeking surveillance orders that do not contain a particular target;
  • rescind FISA Court judges’ power to block amici from participating in designated situations;
  • grant amici access to the FISA Court’s full docket; and
  • grant amici the authority to appeal rulings to the FISA Court of Review.

Congress should also require that defendants receive notice when FISA surveillance is used in investigations of them. The Constitution requires that evidence that could aid a defendant—either as direct evidence or in challenging the legitimacy of an investigator’s conduct—be provided to them. Yet the government claims that it does not have a legal or constitutional obligation to disclose its evidence when it is derived from surveillance conducted pursuant to Section 215. This has stymied legitimate legal challenges to this type of surveillance. Any reauthorization of a narrowed Section 215 must include a clear requirement to protect defendants’ constitutional rights and facilitate their ability to challenge the legitimacy of surveillance programs used against them.

Finally, Congress should not reauthorize the remaining portions of Section 215 (separate from the call detail records program) or other expiring provisions of the PATRIOT Act unless lawmakers and other key oversight entities like the Privacy and Civil Liberties Oversight Board receive indisputable evidence that the call detail records program has not been secretly replaced by a similar mass collection system justified under another, as-yet-undisclosed legal authority.

However, establishing something akin to the call detail records program based on another statute would require a novel or significant interpretation of law by the FISA Court, and thus would trigger the public disclosure requirement. And given how clearly Congress spoke on this issue in passing the limits contained in the USA FREEDOM Act, any reasonable lawyer would conclude that, based on Youngstown, Congress has set clear bounds on the executive’s power to monitor telephone metadata, and as a result the executive’s power to recreate the Terrorist Surveillance Program has been cut off (former Attorney General Jeff Sessions conceded this point while answering a question from Senator Patrick Leahy (D-VT) during his confirmation hearing).

In light of all these factors, a shift of mass metadata surveillance to another hidden authority seems unlikely. But given the executive branch’s record of engaging in and concealing mass surveillance through strained interpretations of the law, it is essential for Congress to adopt a “trust, but verify” attitude before reauthorizing a narrowed Section 215 and permitting future surveillance pursuant to this law with its troublesome history.