Why We Need Continuous Audit Testing in the Courts

Facebook Tweet LinkedIn

Here, I floated the idea of using continuous audit testing in hiring—an experimental mechanism where fake, government-inserted job applications could reveal employer bias and deter discrimination through behavioral uncertainty. It was a provocative proposal.

But now, I think I was thinking too small.

Because if we’re willing to use deception and experimentation to make sure someone gets a fair shot at a job, shouldn’t we be willing to do the same to make sure someone gets a fair trial?

Let’s talk about the legal system.

The Unsolvable Problem: Judicial Disparity

The idea that “justice is blind” has never fully aligned with reality. Study after study has shown that judicial decisions vary not only by facts, but by race, gender, class, and even the defendant’s physical appearance.

A famous study by Mustard (2001) found that African American men receive significantly longer sentences than White men for the same crimes, controlling for criminal history and other variables.
Another, by Rachlinski et al. (2009), found that even experienced judges exhibit implicit bias when making rulings, despite believing they are objective.
Add to that the variability across individual judges: one study found that sentencing decisions in asylum cases depend more on which judge you draw than the merits of your case (Ramji-Nogales et al., 2008).

And yet, very little has changed. Why?

Because judges are shielded by institutional inertia, vague legal standards, and enormous discretion. Even well-meaning ones are not immune to unconscious bias. Appeals are rare, complaints are rare, and systemic feedback is almost non-existent.

What if we changed that?

Judicial Testing: A Thought Experiment Becomes a Proposal

Imagine a world where every judge knows—at any given moment—that one of the defendants appearing before them might not be real.

Not real as in a hologram or an AI-generated transcript. Real as in a professionally trained actor sent by a regulatory body to play the role of a defendant. A test case. A control group. An ethical deception.

Just like in hiring audit studies, the key would be in the design:

The “test defendant” would be legally represented, charged with a real (minor) offense, and undergo the full judicial process.
Every variable—case facts, demeanor, legal argumentation—would be held constant except for one: race, gender, accent, religion, disability status, or other relevant identity markers.
The outcomes—bail decisions, plea deals offered, sentencing severity—would be statistically compared.

These synthetic trials wouldn’t need to happen constantly. Just randomly and regularly enough that every judge knows the possibility is real. And that bias is being measured, in the field, by data—not just complaints or headlines.

“Isn’t That… Entrapment?” – The Ethical Dilemma

This is the first and most legitimate objection.

Yes, the idea of inserting actors into the justice system sounds extreme. But is it more extreme than the injustice we tolerate?

Think about it:

Police departments already use undercover operations. The idea of deception is not foreign to law enforcement.
Mystery shoppers test banks for racial bias in mortgage lending—without telling the banker.
Food inspectors pretend to be customers. Airlines send fake passengers to evaluate service.

Why is justice the only field that assumes trust without verification?

Ethically, judicial audits would need to follow clear boundaries:

Only non-violent test cases.
No interference with real litigants’ court time.
Absolute transparency in aggregate results (no public judge-shaming).

Still, it would be disruptive—and that might be exactly what’s needed.

Cost vs. Cost of Injustice

Critics might also object that this is expensive. And it would be.

Recruiting actors. Running legal simulations. Paying for staff to manage audits. These aren’t cheap.

But compare that to the social cost of biased justice:

Thousands of people improperly sentenced.
Billions in lost economic productivity from incarceration.
Generational distrust in institutions among marginalized communities.
Millions spent litigating civil rights violations after the fact.

The economic cost of injustice far outweighs the operational cost of monitoring it.

As with any compliance mechanism, prevention is cheaper than punishment.

A Radical Idea, But Maybe Not So Radical

In medicine, we do randomized controlled trials to test for effectiveness. In finance, we run forensic audits. In education, we do blind grading.

In criminal justice, we still trust that a robe and gavel are enough.

It’s time to challenge that assumption.

Continuous judicial audit testing could provide the first real-time, field-based feedback mechanism to monitor and deter discrimination where it matters most.

It wouldn’t eliminate bias, but it would reduce its freedom to operate unchecked. And it would restore faith that, finally, someone is watching the watchers.

References

Mustard, D. B. (2001). Racial, ethnic, and gender disparities in sentencing: Evidence from the US federal courts. Journal of Law and Economics, 44(1), 285–314.
Rachlinski, J. J., Johnson, S. L., Wistrich, A. J., & Guthrie, C. (2009). Does unconscious racial bias affect trial judges? Notre Dame Law Review, 84(3), 1195–1246.
Ramji-Nogales, J., Schoenholtz, A. I., & Schrag, P. G. (2008). Refugee roulette: Disparities in asylum adjudication. Stanford Law Review, 60(2), 295–412.