LatticeFlow’s Framework benchmarks for AI Act Compliance
The European Union has taken a significant lead in regulating artificial intelligence with the introduction of a risk-based framework that came into force in August. This framework is designed to ensure AI applications comply with specific legal obligations, with a focus on large language models (LLMs) that underpin many AI apps. LatticeFlow AI, a spinout from ETH Zurich, has developed a technical interpretation of the EU AI Act, offering an open-source LLM validation framework called Compl-AI. This initiative aims to map regulatory requirements to technical ones, providing a compliance evaluation for AI models.
LatticeFlow’s framework evaluates AI models across 27 benchmarks, such as toxic completions and prejudiced answers, to assess compliance with the EU AI Act. The evaluations reveal that while models generally perform well in avoiding harmful instructions, they struggle with fairness and recommendation consistency. Notably, smaller models often score poorly on technical robustness and safety, highlighting a need for balanced development that prioritizes compliance alongside capabilities.
The compliance leaderboard published by LatticeFlow ranks major AI models, including those from OpenAI and Meta, against the EU AI Act’s requirements. The results indicate that many models have been optimized for capabilities rather than compliance, with significant gaps in areas like cyberattack resilience and fairness. The framework is designed to evolve alongside the EU AI Act, encouraging contributions from the wider AI research community to refine benchmarks and expand its applicability.
LatticeFlow’s efforts underscore the ongoing challenge of aligning AI development with regulatory requirements. As compliance deadlines approach, AI developers will need to address areas of concern such as copyright and privacy. The framework offers a crucial step towards comprehensive compliance evaluation, inviting collaboration from researchers, developers, and regulators to ensure AI technologies are developed safely and responsibly.
