Senior Product Manager, AI Evals

As part of our AI@LSEG Product Management team, we are seeking a Senior Product Manager with deep expertise in testing, automation, and evaluation systems to own our AI Evaluation Framework for the entire AI workflow from natural language requests to the execution to the response. This role will define how LSEG measures the accuracy, reliability and consistency of AI-powered interactions across our data, analytics and market solutions. You will establish the frameworks, processes and standards that evaluate LLM behaviour , MCP tool usage, and natural-language output quality across multiple AI stacks—ensuring our customers receive precise, trustworthy, and explainable results. Key Responsibilities & Accountabilities Framework Ownership Design and own the end-to-end evaluation framework for AI systems: natural-language prompting, LLM orchestration, MCP execution, and natural-language output quality. Codify a rubric-based scoring system covering task fulfilment, factual accuracy, reasoning quality, safety, coverage and instruction adherence. Develop the taxonomy for query types, complexity levels, and evaluation profiles across all LSEG MCPs and AI stacks. Test Design & Automation Partner with SMEs to create seed evaluation scenarios , and define the processes for generating structured prompt permutations using LLMs. Build the strategy for scalable automated evaluation pipelines , including LLM-based graders, confidence scoring, sampling strategies and regression detection. Define requirements for internal tooling: scenario catalogues, result dashboards, model-versus-model comparisons, performance tracking and release gates. Cross-Functional Leadership Collaborate with product teams across Data & Analytics, Markets, FTSE Russell and Risk Intelligence to integrate the evaluation framework into their AI initiatives. Work closely with Engineering, AI Governance, Model Risk, Compliance and GSSR to ensure alignment with responsible AI standards and regulatory expectations. Support product teams in interpreting evaluation results, identifying failure modes and improving model + MCP reliability. AI Strategy & Governance Contribute to group-wide AI governance by defining evaluation requirements that support responsible AI, explainability and bias mitigation. Provide insights on emerging testing and benchmarking methodologies across the AI ecosystem. Ensure evaluation frameworks adapt as LLMs, MCPs and customer workflows evolve. External & Internal Advocacy Serve as the internal subject matter expert for AI evaluation and testing. Present evaluation insights to senior stakeholders, ExCo , product leadership and external partners. Skills & Competencies Required Background in product management , QA/test automation , or ML evaluation frameworks with experience designing structured test systems at scale. Strong knowledge of LLMs , prompt engineering, and tool-augmented LLM workflows. Experience with test automation, scenario design, evaluation metrics and quality frameworks in complex systems. Familiarity with financial data, analytics workflows, or MCP-style interoperability . Ability to translate ambiguous real-world user tasks into formal evaluation structures and rubrics. Preferred Experience with LLM-based evaluators, rubric scoring systems, structured data validation, and regression frameworks. Familiarity with financial data pipelines, structured/unstructured data handling, analytics APIs, and Python/JS-based tooling ecosystems. Strong analytical skills: ability to identify quality gaps, failure patterns and reliability issues in AI systems. Comfortable partnering with senior internal stakeholders, engineering teams and policy groups. Measures of Success Quality & Accuracy Improvements in task fulfilment accuracy, reasoning clarity and factual correctness across MCP-powered workflows. Reduction in hallucinations, constraint violations, and output variability across permutations and AI stacks. Scalability & Adoption Uptake of the evaluation framework across LSEG product divisions. Level of automation achieved in test execution and evaluation pipelines. Stability of evaluation results and speed of identifying regressions. Organisational Impact Contribution to Responsible AI, model risk processes and governance frameworks. Improvements in transparency, auditability and explainability of AI behaviour . Ability to support product teams in shipping AI features with confidence. Career Stage: Manager London Stock Exchange Group (LSEG) Information: Join us and be part of a team that values innovation, quality, and continuous improvement. If you're ready to take your career to the next level and make a significant impact, we'd love to hear from you. LSEG is a leading global financial markets infrastructure and data provider. Our purpose is driving financial stability, empowering economies and enabling customers to create sustainable growth. Our purpose is the foundation on which our culture is built. Our values of Integrity, Partnership , Excellence and Change underpin our purpose and set the standard for everything we do, every day. They go to the heart of who we are and guide our decision making and everyday actions. Working with us means that you will be part of a dynamic organisation of 25,000 people across 65 countries. However, we will value your individuality and enable you to bring your true self to work so you can help enrich our diverse workforce. We are proud to be an equal opportunities employer. This means that we do not discriminate on the basis of anyone’s race, religion, colour, national origin, gender, sexual orientation, gender identity, gender expression, age, marital status, veteran status, pregnancy or disability, or any other basis protected under applicable law. Conforming with applicable law, we can reasonably accommodate applicants' and employees' religious practices and beliefs, as well as mental health or physical disability needs. You will be part of a collaborative and creative culture where we encourage new ideas. We are committed to sustainability across our global business and we are proud to partner with our customers to help them meet their sustainability objectives. Our charity, the LSEG Foundation provides charitable grants to community groups that help people access economic opportunities and build a secure future with financial independence. Colleagues can get involved through fundraising and volunteering. LSEG offers a range of tailored benefits and support, including healthcare, retirement planning, paid volunteering days and wellbeing initiatives. Please take a moment to read this privacy notice carefully, as it describes what personal information London Stock Exchange Group (LSEG) (we) may hold about you, what it’s used for, and how it’s obtained, your rights and how to contact us as a data subject . If you are submitting as a Recruitment Agency Partner, it is essential and your responsibility to ensure that candidates applying to LSEG are aware of this privacy notice.