rewrite this title The Black Box Problem: Why AI Needs Proof, Not Promises – Decrypt

When people think about artificial intelligence, they think about chatbots and large language models. Yet it’s easy to overlook that AI is becoming increasingly integrated with critical sectors in society. 

These systems don’t just recommend what to watch or buy anymore; they also diagnose illness, approve loans, detect fraud, and target threats.

As AI becomes more embedded into our everyday lives, we need to ensure it acts in our best interest. We need to make sure its outputs are provable.

Most AI systems operate in a black box, where we often have no way of knowing how they arrive at a decision or whether they’re acting as intended. 

It’s a lack of transparency that’s baked into how they work and makes it nearly impossible to audit or question AI decisions after the fact.

For certain applications, this is good enough. But in high-stakes sectors like healthcare, finance, and law enforcement, this opacity poses serious risks. 

AI models may unknowingly encode bias, manipulate outcomes, or behave in ways that conflict with legal or ethical norms. Without a verifiable trail, users are left guessing whether a decision was fair, valid, or even safe.

These concerns become existential when coupled with the fact that AI capabilities continue to grow exponentially. 

There is a broad consensus in the field that developing an Artificial Superintelligence (ASI) is inevitable.

Sooner or later, we will have an AI that surpasses human intelligence across all domains, from scientific reasoning to strategic planning, to creativity, and even emotional intelligence. 

Questioning rapid advances 

LLMs are already showing rapid gains in generalization and task autonomy. 

If a superintelligent system acts in ways humans can’t predict or understand, how do we ensure it aligns with our values? What happens if it interprets a command differently or pursues a goal with unintended consequences? What happens if it goes rogue?

Scenarios where such a thing could threaten humanity are apparent even to AI advocates. 

Geoffrey Hinton, a pioneer of deep learning, warns of AI systems capable of civilization-level cyberattacks or mass manipulation. Biosecurity experts fear AI-augmented labs could develop pathogens beyond human control. 

And Anduril founder Palmer Luckey has claimed that its Lattice AI system can jam, hack, or spoof military targets in seconds, making autonomous warfare an imminent reality.With so many possible scenarios, how will we ensure that an ASI doesn’t wipe us all out?

The imperative for transparent AI

The short answer to all of these questions is verifiability. 

Relying on promises from opaque models is no longer acceptable for their integration into critical infrastructure, much less at the scale of ASI. We need guarantees. We need proof.

There’s a growing consensus in policy and research communities that technical transparency measures are needed for AI. 

Regulatory discussions often mention audit trails for AI decisions. For example, the US NIST and EU AI Act have highlighted the importance of AI systems being “traceable” and “understandable.”

Luckily, AI research and development doesn’t happen in a vacuum. There have been important breakthroughs in other fields like advanced cryptography that can be applied to AI and make sure we keep today’s systems—and eventually an ASI system—in check and aligned with human interests.

The most relevant of these right now is zero-knowledge proofs. ZKPs offer a novel way to achieve traceability that is immediately applicable to AI systems.

In fact, ZKPs can embed this traceability into AI models from the ground up. More than just logging what an AI did, which could be tampered with, they can generate an immutable proof of what happened.Using zkML libraries, specifically, we can combine zero-knowledge proofs and machine learning that verify all the computations produced on these models.

In concrete terms, we can use zkML libraries to verify that an AI model was used correctly, that it ran the expected computations, and that its output followed specified logic—all without exposing internal model weights or sensitive data. 

The black box

This effectively takes AI out of a black box and lets us know exactly where it stands and how it got there. More importantly, it keeps humans in the loop.

AI development needs to be open, decentralized, and verifiable, and zkML needs to achieve this. 

This needs to happen today to maintain control over AI tomorrow. We need to make sure that human interests are protected from day one by being able to guarantee that AI is operating as we expect it to before it becomes autonomous.

ZkML isn’t just about stopping malicious ASI, however. 

In the short term, it’s about ensuring that we can trust AI with the automation of sensitive processes like loans, diagnoses, and policing because we have proof that it operates transparently and equitably. 

ZkML libraries can give us reasons to trust AI if they’re used at scale.

As helpful as having more powerful models may be, the next step in AI development is to guarantee that they’re learning and evolving correctly. 

The widespread use of effective and scalable zkML will soon be a crucial component in the AI race and the eventual creation of an ASI.

The path to Artificial Superintelligence cannot be paved with guesswork. As AI systems become more capable and integrated into critical domains, proving what they do—and how they do it—will be essential. 

Verifiability must move from a research concept to a design principle. With tools like zkML, we now have a viable path to embed transparency, security, and accountability into the foundations of AI. 

The question is no longer whether we can prove what AI does, but whether we choose to.

Edited by Sebastian Sinclair