LLM Ops & Evaluations

Why Sintra?

50,000+ small business owners use Sintra every day. Not because it's another SaaS tool — but because our AI helpers actually feel like teammates. They reply to customers, post on social media, analyze data, and much more. For someone running a business alone, this is the first time they've had help that actually helps.

We've built something people care about. Now we need to make it great.

Quick summary: We're looking for someone to own the quality of our AI employees - prompts, evaluations, datasets, and model selection. You'll build the systems that determine whether our AI helpers actually help. This is a foundational role: you'll design our evaluation framework, be our primary technical contact with LLM providers, and be responsible for improving LLM outputs across the app.

This role is for someone who's shipped LLM-powered products at scale, thinks in systems, and wants to define how AI quality works at a company scaling fast.

Who we're looking for

• 2+ years hands-on with LLMs in production.

• Has built evaluation systems, not just written prompts.

• Strong technical skill set - you'll automate evals and work closely with engineering.

• Systems thinker who can handle hundreds of use cases × user customization.

• Clear communicator - you'll be our point person with technical teams of leading AI labs.

What you'll do

• Own end-to-end quality of all AI outputs across the app.

• Design and build our evaluation framework - automated tests, human review loops, quality scoring.

• Create, version, and optimize prompts for every use case.

• Build and maintain test datasets that catch regressions before users do.

• Hire and lead a team of prompt engineers and eval specialists as we scale.

Salary range: €5,000-8,000/month gross, depending on experience.

Benefits

• Equity package: share in the company’s long-term success through an equity plan that rewards your impact and commitment.

• Flexible working model: enjoy a flexible approach to work that supports a healthy balance between your professional and personal life.

• Private health insurance: comprehensive private health coverage to support your well-being.

• Workout sessions with a personal trainer: access guided workout sessions with a personal trainer to help you stay active, healthy, and motivated.

• Daily office-provided lunch: fresh, complimentary lunch provided in the office every workday to keep you energized and focused.

By submitting your application you confirm that you have read and understood our Privacy Notice for Job Candidates: https://bit.ly/4b9hhlE