[Job - 28525] AI Quality Engineer Senior, QA

We are tech transformation specialists, uniting human expertise with AI to create scalable tech solutions.
With over 8,000 CI&Ters around the world, we’ve built partnerships with more than 1,000 clients during our 30 years of history. Artificial Intelligence is our reality.

Role Overview

The AI Agent Evaluation Engineer is responsible for ensuring the quality, accuracy, explainability, and reliability of AI agent systems across Proof-of-Concept, Pilot, and Production. The role focuses on establishing enterprise-grade evaluation frameworks for agentic AI, LLMs, and AI-driven workflows to ensure outputs are trustworthy, measurable, and continuously improving.

Key Responsibilities

• Design and implement evaluation frameworks for AI agents, LLMs, and RAG-based systems.

• Measure accuracy, relevance, consistency, hallucinations, and task success across AI outputs.

• Establish baseline and comparative evaluations across models, prompts, and agent strategies.

• Validate agent decision logic, reasoning paths, and tool usage for explainability and traceability.

• Support human-in-the-loop (HITL) evaluation for high-impact or high-risk use cases.

• Partner with engineering teams to improve prompts, retrieval strategies, and agent orchestration.

• Validate AI observability, monitoring, drift detection, and regression controls.

• Support vendor PoCs, pilots, and RFP evaluations with fact-based assessments.

Required Qualifications
• Experience evaluating Generative AI, LLMs, and agentic AI systems.

• Strong understanding of AI/ML evaluation metrics and error analysis.

• Hands-on experience with Python and AI evaluation workflows.

• Familiarity with RAG architectures, prompt evaluation, and agent orchestration.

• Experience with cloud AI platforms (Azure or GCP preferred).

Preferred Qualifications
• Experience in Education, Healthcare, or other regulated domains.

• Exposure to synthetic data generation and test scenario design.

• Familiarity with AI governance, risk, and compliance practices.

Success Measures

• Measurable improvement in AI accuracy, reliability, and trustworthiness.

• Clear visibility into why AI agents made specific decisions.

• Standardized evaluation frameworks adopted across AI initiatives.

• Increased leadership confidence in AI-driven outcomes.

Our benefits:

-Health and dental insurance
-Meal and food allowance
-Childcare assistance
-Extended paternity leave
-Partnership with gyms and health and wellness professionals via Wellhub (Gympass) TotalPass;
-Profit Sharing and Results Participation (PLR);
-Life insurance
-Continuous learning platform (CI&T University);
-Discount club
-Free online platform dedicated to physical, mental, and overall well-being
-Pregnancy and responsible parenting course
-Partnerships with online learning platforms
-Language learning platform
And many more!

More details about our benefits here: https://ciandt.com/br/pt-br/carreiras

At CI&T, inclusion starts at the first contact. If you are a person with a disability, it is important to present your assessment during the selection process. See which data needs to be included in the report by clicking here.This way, we can ensure the support and accommodations that you deserve. If you do not yet have the assessment, don't worry: we can support you in obtaining it.

We have a dedicated Health and Well-being team, inclusion specialists, and affinity groups who will be with you at every stage. Count on us to make this journey side by side.