Ask any CIO whether they want an AI system nobody can inspect, and the answer is no. Ask the same CIO to list the AI systems running in their organisation today, and the picture gets less comfortable. Somewhere in that list there's usually a tool that produces useful answers, that nobody in compliance can explain, and that would struggle to produce evidence if an inspector asked for it tomorrow.
That outcome is rarely the result of a bad decision. It's the result of a sequence of reasonable ones. Three procurement routes in particular produce black boxes reliably, and most organisations will recognise at least one of them in their own estate.
Nobody sets out to build a black box
A black box, in the sense that matters for governance, is a system where prompts go in and responses come out, but the people accountable for those responses cannot see what happened in between. They can't evaluate the answer, audit the decision, or change the system's behaviour without rebuilding it.
The reason this keeps happening is that the early signals are all positive. A prototype works, users like it, and the business need is met. The problem only becomes visible later, when someone from legal, compliance or information security asks a simple question: show me how this answer was produced. If the system can't answer that, it was a black box from the start, however well it performed.
Route one: commercial API adoption
The first route begins with a developer or a small team calling a commercial model API directly. It's the fastest way to prototype, which is exactly why it's so common. Within days, there's a working proof of concept, and within weeks it has quietly become something people depend on.
The trouble is that a direct model call is impossible to introspect. You don't know why the model gave one answer over another. You can't log its internal reasoning. And you can't audit it under the EU AI Act in any way that would satisfy a regulator asking for provenance. The prototype wasn't built to be governed, and retrofitting governance onto a direct API call usually means rebuilding the integration rather than adjusting it.
The warning sign is easy to spot: the system works, but the only record of what it did is the request and the response, if even that.
Route two: opaque SaaS AI
The second route doesn't involve building anything. You buy a product that has AI built into it, and the vendor treats the AI as a feature, one line on a feature list among many.
Here, the opacity is structural. You get no visibility into how the AI works, no control over which model runs underneath, and no audit trail beyond whatever the vendor's own dashboard chooses to show. That dashboard may report success rates and usage numbers, but it isn't evidence your team can inspect. If the vendor changes the model, you may not know. If a regulator asks how a specific answer was generated, the honest answer is that you'd have to ask the vendor, and hope.
This route is especially easy to miss because it often arrives through ordinary software procurement. Nobody flagged it as an AI purchase, so nobody applied AI governance to it. The first step in addressing it is simply knowing it's there, which means counting embedded and third-party AI features in your inventory, not just the systems you built on purpose.
Route three: RAG without governance
The third route is the most deceptive, because it's genuinely an improvement. Retrieval-augmented generation on a general-purpose model grounds answers in your own content, which is better than direct API calls with no grounding at all. Teams that choose it are usually doing the right thing in the right direction.
The catch is that it's still a black box if the surrounding machinery is opaque. Prompt handling, retrieval logic and response generation can all remain invisible to the compliance team. You may know that answers are grounded, but not which passages were retrieved for a given question, why those and not others, or which model version produced the final text. A reviewer who can't see the retrieval can't evaluate the answer, and "it's grounded in our content" is a claim rather than evidence.
This is the route where good intentions are most likely to be mistaken for good governance, because the system looks more trustworthy than it is.
Why all three pass the business test and fail the audit
What the three routes share is more important than what separates them. Each one satisfies the immediate business need. Each one produces answers people find useful. And each one fails the audit requirement, because the evidence of how an answer was produced either was never captured or isn't accessible to the people who need it.
That's why these problems tend to surface late. A demo succeeds, a pilot succeeds, a rollout succeeds, and then a governance review exposes that none of the success was inspectable. At that point, the options are limited: rebuild, migrate, or retire.
How to recognise which route you're on
You don't need an audit programme to find out. A few direct questions are usually enough.
- Can you reconstruct one specific interaction from request to response, including what was retrieved and which model answered?
- Can an accountable reviewer inspect the evidence behind an answer, not just a summary of it?
- Can you produce the record of an interaction on request without engineers piecing it together from several systems?
If the honest answer to any of these is no, or "I'd have to ask the vendor," you have a black box.
The most useful rule to adopt is a forward-looking one. Every new AI use case ships as a glass box, with traceability, observability and auditability built in from day one, or it doesn't ship at all.
Existing systems can be fixed over time through a simple sequence: inventory what you run, categorise it by risk, check what trace exists for each system, decide whether to rebuild, migrate or retire, and procure glass-box for everything new.
Where to go next
Our glass-box health check turns this into an assessment you can run against a system you already operate, with the evidence to ask for and the red flags to watch for on each property.
For the full argument, including the six architectural properties of a glass box and what EPSO's per-interaction record looks like, read Glass box vs black box: the enterprise AI transparency playbook.
If your concern is where data physically goes once a system runs, our page on hosting and sovereignty covers the infrastructure side, and Trusted Answers shows what source-cited, traceable answers look like in a public-facing deployment.
Where Dropsolid AI fits in this
Dropsolid is the Drupal Diamond Partner behind the sovereign AI use cases in production at EPSO and Pidpa, and has been building on Drupal since 2013. Dropsolid NV and Dropsolid AI NV are ISO 27001 certified, and we're headquartered in Ghent, Belgium. Both production deployments run as glass boxes, with an audit trace for every interaction.
If you want to check where your own estate stands, or scope a new use case that has to be inspectable from day one, talk to our team or book a demo and see the trace for yourself.