1033- AI Engineer (Evaluation Systems)

We Make Remote Work Remarkable • TopTalent from LatAm

Hello! We are GoFasti, a Talent-as-a-Service. GoFasti bridges the gap between world-class developers and designers from LatAm and first-class companies around the globe.

We need an English-fluent AI Engineer (Evaluation Systems), based in Latin America, available to work remotely.

We are looking for someone with exceptional communication and relationship-building skills, who embraces changes while maintaining strong attention to detail. An interested and proactive person, who's constantly learning and improving their skills.

Are you the one we are looking for?

Responsibilities:

• Design a structured, configurable evaluation engine

• Combine deterministic checks with LLM-as-judge verdicts

• Build calibration workflows using expert-labeled examples

• Measure precision and recall properly (not raw agreement)

• Handle delayed outcomes and low-confidence review flows

• Store structured verdicts that power dashboards and analytics

• Monitor drift and trigger recalibration when alignment drops

• Specialized judges for specific failure patterns

• Content-level evaluation (with careful PII handling)

• Error analysis pipelines that drive workflow redesign

• A/B testing for judge versions

• Eval-driven fine-tuning data curation

Requirements:

• 4+ years total backend / ML engineering experience

• 2+ years building production AI/LLM systems

• Experience with Python, Docker and PostgreSQL.

• Experience with AWS, OpenAI, Anthropic, and other LLM APIs

• Proven experience building LLM-based systems in production environments where output quality and reliability were critical.

• Experience developing evaluation, QA, or scoring pipelines to assess model performance.

• Strong understanding of precision/recall trade-offs and working with real-world data.

• Ability to design and implement systems that produce reliable, structured outputs from LLMs.

• Strong Python programming skills, including experience with asynchronous programming.

• Experience deploying and managing services on AWS.

It´s a Plus:

• Human-in-the-loop workflow design

• OpenTelemetry familiarity

Compensation:

• The Salary range offered for this position varies from (USD) $4,500 - $5,000 per month, depending on seniority and skillset.

• This position is for an independent contractor, through a payroll platform.

• The talent will work REMOTELY allocated at our client.

Here are the steps for this process:
Application review/approval > Screening interview with GoFasti's team > We build and send your profile to our client > Profile review/approval by client > Interview with the client > Client Assessment > Hiring and onboarding.

Once you apply for the job, our team will review your resume. If it meets the requirements, we will contact you and move forward in the process.

Note for Candidates Approached Directly:
If you were contacted directly by a member of our team and are interested in this opportunity, please do not apply through this link. Instead, reach out to the person who contacted you to coordinate a meeting.

Thank you!