Most enterprise AI today runs as a black box. Prompts go in, responses come out, and the people accountable for those responses (legal, compliance, information security, the CIO) cannot inspect what happened in between. They can't evaluate the answer, audit the decision, or influence the model's behaviour without rebuilding it.
That isn't a tooling problem. It's a governance problem, and it starts with an assumption baked into almost every control framework in use today: that software behaves the same way twice.
Correct is a distribution, not a value
Legacy software is a system you can reason about. You provide an input, it converts it to an output, and when there's a bug you know, because for some input you got an unexpected result. Three properties make that governable: it's deterministic, it can be tested in isolation, and its results are reproducible.
AI-enabled applications break all three. Output for a given input can vary, because the system works with probabilistic reasoning, autonomous agents and external tools, context managed across many turns, and sometimes multiple models inside one workflow. "Correct" stops being a single value and becomes a distribution.
Your governance model was designed for the first kind of software. So the choice is blunt: you either update your governance model, or you accept that you don't have governance over AI.
Nobody sets out to build a black box
Almost no one decides to deploy something opaque. Black boxes emerge from procurement routes that seemed reasonable at the time, and most organisations will recognise at least one of the three below in their own estate.
- Commercial API adoption: A team starts calling commercial model APIs directly. It's fast to prototype, but it's impossible to introspect: you don't know why the model gave one answer over another, you can't log its internal reasoning, and you can't audit it under the EU AI Act.
- Opaque SaaS AI: You buy a product with AI built in, and the vendor treats the AI as a feature. There's no visibility into how it works, no control over which model runs, and no audit trail beyond the vendor's own dashboard.
- RAG without governance: Retrieval-augmented generation on a general-purpose model is better than direct API calls, because the answers are grounded. It's still a black box, though, because prompt handling, retrieval logic and response generation remain opaque to the compliance team.
All three routes satisfy the immediate business need and fail the audit requirement. That's why the problem tends to surface late, when someone asks to see the evidence and there isn't any.
Where the chain goes dark
The reason these systems resist governance becomes clearer if you follow a single request through them. Every AI request follows the same chain: a visitor or editor makes a request, the application sends input and context to the AI provider, the provider queries the language model, the model requests additional context from the vector database, and an answer comes back before the response is returned.
In a black box, most of that chain is invisible to the people accountable for the answer. In a glass box, every step is observable, every trace is auditable and every configuration is steerable. The difference isn't a report generated afterwards. Transparency has to be built into how the system runs, not produced as an afterthought.
What a glass box actually requires
A glass box is defined by six architectural properties. The first three decide what you can see, and the other three decide what you can do about it.
- Traceability of every interaction: Every input, retrieval, model call and output is logged in enough detail to reconstruct what happened: the prompt, the retrieved passages, the model version, the response and the citations.
- Observability of AI operations: Live signal on model usage, cost, latency and response quality, not weekly reports, so you see drift before it becomes an incident.
- Transparency of the decision process: The reasoning chain is exposed to a human reviewer. A reviewer who can't see the retrieval can't evaluate the answer.
- Feedback that corrects the source: User feedback and expert review are captured against each interaction, and answers that drifted are corrected at the source content as part of an improvement cycle.
- Auditability against a compliance framework: The audit trail can be filtered, exported and produced for an inspector on demand, as a standard capability rather than an ad-hoc exercise.
- Steerability without rebuild: You can influence behaviour through configuration, prompts, context templates and retrieval thresholds, without a code change and a redeploy.
The first three are about recording decisions rather than describing them. Anything not captured at the moment of the interaction can't be recovered afterwards, which is the same evidence test that runs through the EU AI Act's deployer obligations.
Most organisations are not starting from scratch
You already have AI in production, so the practical question is how to move from where you are to where you need to be without stopping work that's already running. The sequence is five steps.
- Inventory every AI-enabled system you run: Including prototypes, third-party features and embedded functionality. Under the EU AI Act, everything counts.
- Categorise them: which are high-risk, which touch personal data, and which produce output visible to citizens, customers or regulated audiences.
- Audit the trace for each system: what exists per interaction, where it's stored, and whether it can be exported. If the answer to any of these is unclear, you have a black box.
- Run a gap analysis: And decide for each incomplete system whether to rebuild it, migrate it to a glass-box platform, or retire it.
- Procure glass-box for anything new: The old systems you fix over time; the new ones you build right from day one.
If there's a rule worth adopting from all of this, it's the last one: every new AI use case ships as a glass box, or it doesn't ship at all.
What it looks like in production
EPSO, the European Personnel Selection Office, runs its candidate-facing AI as a glass box. For every interaction, the record includes the question as asked in the candidate's own language, the passages retrieved from EPSO's procedural documents, the model selected by the governed gateway, and the answer with clickable citations. Internal staff can inspect any interaction, filter by cycle and export subsets in the format the EU's information-governance framework expects.
That's the point of the whole argument: not a report about the system, but the system's own record.
Test a system you already run
If you want to know where your existing systems stand, the six properties above can be turned into an assessment you can run in an afternoon. Our glass-box health check does exactly that, and it's a good first step before the inventory and gap analysis described above.
Download the glass-box health check →
For the full argument, including the three pillars of transparency, observability and traceability and the EPSO trace in detail, read Glass box vs black box: the enterprise AI transparency playbook.
For how this plays out at the infrastructure level, our page on hosting and sovereignty covers where the data chain actually runs. And if you're responsible for answers that citizens or customers will see, Trusted Answers shows what source-cited, traceable responses look like in practice.
Where Dropsolid AI fits in this
Dropsolid is the Drupal Diamond Partner behind the sovereign AI use cases in production at EPSO and Pidpa, and has been building on Drupal since 2013. Dropsolid NV and Dropsolid AI NV are ISO 27001 certified, and we're headquartered in Ghent, Belgium. Both deployments run with all six glass-box properties in place, which is why they could satisfy institutional governance without bespoke work.
If you're mapping your current AI estate against the six properties, or scoping a new use case that has to ship as a glass box, talk to our team or book a demo to see the traceability and audit architecture directly rather than take our word for it.