Testing RAG-Based AI Systems – Insights from a Client Project at Schaeffler Special Machinery
Retrieval-Augmented Generation (RAG) is a key building block of modern AI applications. By combining the power of large language models with company-specific knowledge, RAG enables highly customizable systems that deliver project-relevant answers and real business value.

But how do you reliably test such a system before it goes live?

Together with Schaeffler Special Machinery, we developed a tailored test strategy designed to do exactly that: systematically evaluate the quality, reliability, and safety of RAG-based systems—during the pilot phase, before production deployment.

In this talk, our colleagues from Schaeffler will first present their product, the technical specifics of a RAG system, and the value it creates for their customers. We will then dive into the testing challenges, the test concept we developed, and the frameworks we evaluated during the process.

A hands-on look at the challenges and considerations behind building and testing trustworthy AI systems—from real-world experience.