Getty Images
Operations that run under extreme risk — such as nuclear power plants, aircraft carriers, and air traffic control centers — have long understood that safety depends not just on good technology but on how people exercise judgment under uncertainty. The scholarly field of high-reliability organizing (HRO) has identified a clear normative framework to distinguish organizations that manage catastrophic risk well from those that succumb to it: Reliable organizations protect front-line judgment, distribute decision authority to those closest to the problem, and treat emerging anomalies as signals rather than noise.
These principles are hard-won lessons from disasters. At Three Mile Island in 1979, for instance, operators of the U.S. nuclear power plant confronted more than 100 simultaneous alarms with no prioritization mechanism, producing cognitive paralysis rather than an informed response when the reactor began to melt down. Before the space shuttle Columbia’s deadly final flight in 2003, NASA engineers who suspected that a wing’s insulating foam had sustained damage during launch had escalation channels available but refrained from using them due to the organizational costs of challenging senior leaders. NASA’s accident investigation report said that “management techniques unknowingly imposed barriers that kept at bay both engineering concerns and dissenting views, and ultimately helped create ‘blind spots’ that prevented them from seeing the danger the foam strike posed.”
Today’s AI-enabled threat-detection systems — the adaptive machine-learning tools that surface fraudulent transactions, network intrusions, compliance violations, early signs of patient deterioration in nursing care, and the like — promise superhuman pattern recognition at scale. These threat-detection technologies, however, remain prone to bias, drift, and hallucination, and pose their own risks. A false positive can flood an organization with noise and lead to wasted resources. A missed threat can be catastrophic, as the cyber hacks a few years ago by Salt Typhoon against U.S. telecom networks and the massive 2024 data breach of Change Healthcare have shown.
Emerging AI governance frameworks, designed in part to protect organizations from AI technology falling short of its intended task, are paralleling underlying principles of the original HRO playbook (whether they realize it or not). Regulatory and academic debate is converging on governance practices that operationalize HRO principles for AI contexts, with edicts to, for example, keep a human in the loop where AI is implemented.
But my research on accountability systems suggests that these governance structures may harbor a hidden problem that both HRO theories and emerging AI governance frameworks do not adequately address. The paradox is that instead of protecting vigilance, expertise, and sensitivity to emerging failure, the processes being implemented may actually be subtly eroding the very behavioral foundations on which reliable threat detection depends.
Signals in Governance Policy Impact Employee Engagement
The accountability structures that underpin decision-making and HRO are not value-neutral conduits for oversight. In research with colleagues, I have shown that they instead send unspoken relational signals — about who is trusted, what kind of reasoning is valued, and when independent thinking is or is not welcome — in ways that can reshape how employees exercise judgment.
These are accountability signals: implicit messages that governance mechanisms transmit to employees. The danger is that these messages can encourage employees to step back from what their instinct tells them to do. Sometimes these signals produce mindless conformity, where employees default to procedurally defensible decisions rather than exercising independent judgment. At other times, they encourage reckless deviation, where employees stray from compliance procedures to protect their own interests.
The technologists, risk officers, and other leaders designing AI governance today all aim to follow the HRO-like playbook. Because AI technologies and governance are still maturing, there is not yet longitudinal evidence to document the effects of these design choices systematically. But latent behavioral mechanisms like relational signaling are well established in accountability scholarship, and such concepts suggest that as high-stakes organizations rush to implement AI technologies, the most common responsible AI governance principles may be setting companies up for less-secure environments.
How Accountability Signals Can Erode Reliability
Generally speaking, common best practices for AI governance are emerging across regulatory standards and academic discourse. In an ongoing collaboration with legal and policy scholars, we mapped 14 major governance frameworks — across financial services, health care, cybersecurity, and cross-sector AI governance — and found recurring features that center on preserving human oversight. Among the most prominent are designating a human in the loop (to preserve front-line oversight); implementing escalation and override mechanisms (to protect deference to expertise); ensuring documentation and auditability (to maintain sensitivity to operations); and scaling oversight proportionally to risk (to sustain preoccupation with failure where stakes are highest).
On their face, these emerging practices make sense. They are explicitly designed to correct for both “bad models” and “bad users” who may succumb to automation bias on the one hand and complacency on the other. Each practice is well intentioned.
But each one also embeds an implied accountability signal that can undermine the very reliability it was designed to protect. Here, I detail the signals inherent in four of the most recurring practices across AI governance frameworks.
1. Human-in-the-loop mandates send an implicit accountability signal that “your job is to review, not to decide.”
Responsible AI governance commonly assigns formal accountability to a designated individual. Front-line analysts are positioned as reviewers and approvers of AI-generated alerts, with responsibility codified in workflows and checklists. The intent is to preserve human oversight and prevent blame from being displaced onto the system.
In practice, this principle reshapes how analysts understand their role. They become auditors of machine output rather than originators of risk assessment. The distinction matters: Auditing i
