EU AI Act Conformity Assessment
The EU AI Act (Regulation (EU) 2024/1689) entered into force on 1 August 2024, with obligations phasing in between February 2025 and August 2027. This assessment documents the ZOL Intelligent Search system's classification and compliance posture under the Act, citing specific articles rather than the Act in the abstract.
See EU AI Act (Regulation (EU) 2024/1689) for canonical text. See EU MDR (Regulation (EU) 2017/745) for canonical text. See HLEG Ethics Guidelines for Trustworthy AI 2019.
1. System Classification
1.1 Risk Category Determination
The EU AI Act classifies AI systems into four risk tiers. The classification depends on the system's intended purpose and deployment context.
| Risk Level | Article | Criteria | ZOL System? |
|---|---|---|---|
| Unacceptable | Art. 5 | Social scoring, real-time biometric identification, manipulation | No |
| High-risk | Art. 6 (read with Annex III) | Safety components; medical devices; critical infrastructure; education; employment; law enforcement | See analysis below |
| Limited risk | Art. 50 | AI systems interacting with natural persons; generating content | Yes |
| Minimal risk | (residual) | All other AI systems | Fallback if not limited risk |
1.2 High-Risk Analysis
The system operates in a healthcare setting, which raises the question of whether it qualifies as high-risk under Annex III, point 5(a): "AI systems intended to be used as safety components in the management and operation of [...] healthcare".
Classification rationale: NOT high-risk
| Factor | Analysis |
|---|---|
| Intended purpose | Information retrieval tool (zoektool), not clinical decision support |
| Output type | Navigational information (departments, doctors, contact details), not diagnoses or treatment recommendations |
| Decision impact | No legally binding or clinically significant decisions are made |
| MDR status | Does not meet Medical Device Regulation criteria (no clinical decision support functionality) |
| Safety architecture | Five-layer architecture explicitly prevents medical advice generation |
| User interaction | Voluntary website search, not integrated into clinical workflows |
The system is analogous to a hospital reception desk or information kiosk: it helps visitors find the right department or doctor, but does not participate in clinical decision-making. This places it outside Annex III scope.
This classification holds only while the system remains an information retrieval tool. If future development adds clinical decision support (e.g., symptom triage, treatment recommendations, or diagnostic assistance), the system would likely become high-risk and require full conformity assessment including CE marking.
1.3 Limited Risk Classification
As an AI system that interacts directly with natural persons (website visitors), the system falls under Article 50 (transparency obligations for certain AI systems). Additionally, as it generates text-based responses, Article 50(2) applies regarding AI-generated content disclosure.
2. Applicable Obligations
2.1 Transparency Requirements (Art. 50)
| Requirement | Implementation | Status |
|---|---|---|
| Inform users they interact with AI | System identifies as "ZOL Intelligente Zoekfunctie" (Intelligent Search Function). Responses include "Dit is geen medisch advies" disclaimer. A dedicated AIDisclaimer component displays an EU AI Act Art. 50 transparency notice in the chat interface. | Implemented |
| AI-generated content disclosure | Every response includes source citations linking to original hospital content. The distinction between human-authored source material and AI-generated synthesis is clear. | Implemented |
| Emotional manipulation prohibition | System provides factual, cited information only. No persuasive framing, urgency creation, or emotional language in responses. | Implemented |
2.2 Voluntary High-Risk Compliance
While not required for limited-risk systems, the ZOL system voluntarily implements several high-risk requirements as best practice for healthcare AI. Each row anchors to the specific AI Act article rather than the Act in the abstract:
| High-Risk Requirement | Article | Voluntary Implementation |
|---|---|---|
| Risk management system | Art. 9 | Five-layer safety architecture; DPIA conducted; risk register maintained |
| Data and data governance | Art. 10 | Only publicly available hospital data indexed; PII detection (PII Protection); per-class retention (Data Retention Policy) |
| Technical documentation | Art. 11 | Comprehensive Docusaurus site with architecture, safety, evaluation, and thesis documentation |
| Record-keeping (logging) | Art. 12 | Structured audit logs for all safety decisions, query processing, and PII detections; see Data Retention Policy |
| Transparency to deployers | Art. 13 | Source citations on every response; mandatory disclaimers; intent-classification visibility in admin UI |
| Human oversight | Art. 14 | Hospital helpdesk fallback (089 32 50 50); admin dashboard for monitoring; all blocked queries logged |
| Accuracy, robustness, cybersecurity | Art. 15 | Golden evaluation suite (run before every significant change); adversarial testing per Adversarial Hardening |
3. Technical Documentation (Art. 11 Equivalent)
Even though not required for limited-risk systems, the following documentation is maintained:
3.1 System Description
| Aspect | Detail |
|---|---|
| Purpose | Natural language search for hospital public information |
| Technology | Retrieval-Augmented Generation (RAG) with knowledge graph |
| Models | OpenAI GPT-4.1 / GPT-4.1-mini / GPT-4.1-nano (LLM tiering); OpenAI text-embedding-3-large for embeddings (1536 dim, hosted API per ADR-0048) |
| Training data | No fine-tuning. LLMs used via API. Knowledge base = ZOL public website content. |
| Languages | Dutch (primary), French, English, German, Turkish, Arabic, Polish, code-mixed |
3.2 Performance Metrics
These figures are the most recent measurement we publish; ongoing evaluation runs append to the Quality Evaluation page.
| Metric | Value | Measurement Method |
|---|---|---|
| Golden-eval pass rate | 100 % (most recent run, see Quality Evaluation) | Golden question evaluation with entity recall + safety checks |
| Safety refusal accuracy | 100 % on the safety + adversarial test cases | Zero false negatives on the dedicated safety + adversarial subset |
| Medical advice incidents | 0 | Continuous monitoring since deployment |
| Avg response time (text channel) | 8.3 s on the most recent pilot measurement | Pilot measurement (target: < 15 s) |
3.3 Risk Mitigations
See Safety Architecture for the complete five-layer defense-in-depth system and DPIA for the data protection risk assessment.
4. Human Oversight (Art. 14 Equivalent)
The system implements multiple levels of human oversight:
| Level | Mechanism |
|---|---|
| Real-time | Hospital helpdesk available via phone (089 32 50 50) as fallback for all queries |
| Operational | Admin dashboard shows query volumes, safety blocks, intent distribution, and error rates |
| Audit | Complete audit trail of all safety decisions, PII detections, and blocked queries |
| Override | Admin can disable any pipeline component (CRAG, safety layers, graph) via Settings API |
| Evaluation | 271-question golden evaluation suite (v3.3) run before every significant change |
5. Incident Response
5.1 Medical Advice Incident Protocol
In the event that the system provides content that could be interpreted as medical advice:
- Detection: Automated monitoring (regex + LLM judge) or user/staff report
- Immediate action: Disable the affected pipeline component via admin API
- Investigation: Root-cause analysis using audit logs and query replay
- Remediation: Fix applied, regression test added to golden evaluation suite
- Notification: Hospital DPO and communication team informed
- Documentation: Incident logged with timeline, cause, and corrective action
5.2 Data Breach Protocol
Follows the hospital's existing data breach notification procedures under GDPR Articles 33-34, with the addition of:
- Automated detection of unusual query patterns (rate limiting)
- Admin notification for PII detection spikes
- 72-hour notification window to supervisory authority if personal data is compromised
6. Compliance Timeline
| Date | EU AI Act Milestone | ZOL System Status |
|---|---|---|
| Feb 2025 | Prohibited practices take effect | Not applicable (no prohibited practices) |
| Aug 2025 | GPAI model obligations | Not applicable (using third-party models via API) |
| Aug 2026 | High-risk obligations take effect | System classified as limited-risk; obligations met voluntarily |
| Aug 2027 | Remaining provisions | Full compliance maintained |
7. Ongoing Compliance
| Activity | Frequency | Responsible |
|---|---|---|
| Risk classification review | Annually or on scope change | Development team + hospital DPO |
| Golden evaluation run | Before every production deployment | Development team |
| Safety incident review | Monthly | Hospital DPO + development team |
| Regulatory monitoring | Quarterly | Development team |
| Documentation update | On significant system changes | Development team |
8. Conclusion
The ZOL Intelligent Search system is classified as limited risk under the EU AI Act, subject to transparency obligations (Art. 50). The system voluntarily exceeds its regulatory requirements by implementing measures typically associated with high-risk systems, including comprehensive risk management, technical documentation, human oversight, and rigorous accuracy testing.
This proactive compliance posture:
- Demonstrates commitment to responsible AI in healthcare
- Provides a compliance buffer if the system's scope expands
- Builds trust with hospital partners and regulatory bodies
- Supports the thesis defense narrative of engineering excellence in a sensitive domain
Document version: 1.1 | Date: 2026-04-09 | Author: SOFT4U BV
References
- Regulation (EU) 2024/1689 — Artificial Intelligence Act, Articles 5, 6, 9, 10, 11, 12, 13, 14, 15, 50; Annex III.
- Regulation (EU) 2017/745 — Medical Device Regulation, Art. 2(1) (definition of medical device), Annex VIII Rule 11 (software classification).
- European Commission. (2021). Proposal for a Regulation laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). COM(2021) 206 final.
- High-Level Expert Group on AI. (2019). Ethics Guidelines for Trustworthy AI. European Commission.
- @owasp_llm_top10 — practitioner threat taxonomy used to structure the safety requirements that map to AI Act Art. 9 / 15.
- Safety Architecture overview, DPIA, Adversarial Hardening, PII Protection, Data Retention Policy — operational realisation of the obligations referenced above.