A new report released by the New York City Comptroller reveals that the MTA has misrepresented subway performance and delay data for years and shifted the focus from the subways’ real operational problems. These practices endured despite repeated warnings from agency insiders that much of the delay data released to the public was all but meaningless.

The MTA responded that the report did not recognize the progress the agency has made since 2017, Crain's New York Businessreported. Byford assumed his post in January 2018.

"We appreciate the comptroller's focus on subway performance," an MTA statement said, "but this report is more history and politics than news, focusing on rejected practices of the past while glossing over recent reforms and NYC Transit's aggressive pursuit of additional transparency and accountability."

The investigation, based on internal documents primarily prepared by the MTA’s Performance Analysis Unit, as well as interviews with key MTA officials, makes clear that a history of distorted metrics has prevented transparency into what is going on underground in the subways. More specifically, the investigation found that the MTA failed to provide clear guidance on how to identify the causes of millions of delays, relied on distorted metrics to report system progress, and hid delay data where causes were unknown.

“While the new leadership deserves credit for trying to clean things up, the MTA still has a long way to go to ensure accuracy and reliability and to regain the public’s trust,” said Comptroller Scott M. Stringer. 

The Comptroller’s investigation found:

Incomplete and Unreliable Delay Data: From mid-2015 forward, numerous in-house MTA analyses concluded that the Authority’s internal systems could not accurately identify the causes of delays and, in particular, chronically misattributed delays to “Overcrowding.” A July 2015 internal MTA memo described the methodological breakdowns in stark terms: “No policy or guidance exists on how dispatchers should properly identify the cause of a particular delay or on how delays should be assigned to incidents… Dispatchers rely on train crews to report the cause of delays, and these explanations are suspect.”

Meanwhile, in January 2016, another memo noted that “much of the delay data is incomplete or unreliable, particularly the classification/categorization of delays and the assignment of delays to particular incidents.” Despite multiple warnings, MTA officials continued for years to publicly promote misleading information around delays, resulting in the agency being portrayed in a more positive light.

Meaningless and Disappearing “Wait Assessment” Results: Throughout 2016, MTA officials repeatedly asserted that subway service was improving based on apparent increases in “Wait Assessment” scores, a metric intended to measure the reliability and consistency of train service and touted by the MTA as its most important indicator of customer service quality.

However, internal analyses obtained by the Comptroller’s Office showed MTA analysts warning that alleged improvements in “Wait Assessment” were statistically insignificant and likely the result of “sample error.” After technological advancements in data collection finally made clear that “Wait Assessment” scores had actually gotten worse, not better, the MTA quietly stopped touting the metric without ever acknowledging that the prior reporting had been incorrect, according to the report.

Hidden “Unknown” Delay Causes: For nearly a decade, the MTA further distorted its disclosures by effectively hiding delays it had chalked up internally to “Unknown” causes. Instead of publicly listing these delays simply as “Unknown” in a separate category, MTA officials folded them into other publicly identified delay causes – burying the “Unknowns” without any explanation.

In this way, in Monthly Operations Reports provided to the public and the MTA Board from 2013 through mid-2018, the MTA hid 525,710 delays internally grouped under “Unknown.”

Flaws in New Metrics: Similar to the MTA’s misattribution of “Unknown” delays, the MTA’s current reporting of “Major Incidents” – defined as any event that delays 50 or more trains – conceals critical information. Specifically, “Major Incidents” exclude a large category of service disruptions tracked internally by the MTA – “Planned Work,” which includes track and signal maintenance that regularly bogs down whole subway lines. To date, the MTA has yet to acknowledge publicly that its “Major Incidents” metric excludes such work and, as such, fails to capture what is actually going on underground.

Under New York City Transit President Andy Byford, the MTA has recently acknowledged some of these issues, introduced reforms, and begun to reduce delays. Yet the report reveals systemic deficiencies, some of which remain embedded in the MTA’s performance reporting and continue to obscure the true causes of delays.

Comptroller Stringer is calling on the MTA to:

• Structure public reporting of delay data to maximize transparency, reliability and accountability.

• Publicly define all delay categories, specifically indicating what each one includes and, as necessary, omits.

• Ensure that all protocols relevant to performance reporting are formally codified in official policies and procedures, including establishing written definitions and instructions for all key terms, data categories, and work protocols.

• Adequately train all relevant personnel on protocols relevant to performance reporting.

• In the context of public reports of “Major Incidents,” provide the public with information about all categories of service disruptions tracked as incidents within its Subway Incident Reporting System (SIRS) that cause 50 or more delays, including specifically “Planned Work.”

• Transparently disclose in each Monthly Operations Report and on the MTA’s subway performance dashboard, the methodologies used to calculate performance metrics, including all exceptions and revisions to those methodologies, and including methodological weaknesses.

• Make available on the MTA’s website or through an Open Data portal each month all data in the SIRS database and any other databases relied on for public reporting.