Operations Manager- Data Annotation

About Us

We’re an AI-focused startup working with cutting-edge models and safety-critical use cases. A big part of our work relies on high-quality data annotation to train, evaluate, and monitor AI systems across complex domains (safety, fraud, compliance, content quality, etc.).

We’re looking for an Operations Lead to own our annotation operations end-to-end: managing projects, coordinating internal and external teams, and ensuring we deliver high-quality labeled data on time, at scale.

Role Overview

In this role, you will:

• Be the point person for all data annotation projects.

• Manage a team of annotators (external vendors).

• Design, refine, and enforce workflows, guidelines, and quality processes.

• Partner with product, research, and engineering to turn vague requirements into clear task specs and rubrics.

You’re the kind of person who loves structure, can keep many moving pieces aligned, and cares deeply about quality, throughput, and reliability.

What You’ll Do

Project & Workflow Management

• Own planning and execution for multiple annotation projects at once (scope, timelines, staffing, and priorities).

• Turn high-level requirements into clear task definitions, instructions, and edge-case guidance.

• Build and maintain project plans including milestones, SLAs, and communication cadences with stakeholders.

Team & Vendor Management

• Manage a team of annotators / reviewers (internal and/or external).

• Handle capacity planning, scheduling, and task assignment to hit deadlines.

• Coordinate with annotation vendors or agencies, ensuring they understand requirements and meet quality + throughput expectations.

• Provide feedback, coaching, and training to improve annotator performance.

Quality, Process & Tooling

• Define and iterate on rubrics, guidelines, and golden sets for consistent labeling.

• Design and manage QA workflows (spot checks, double label, adjudication, calibration sessions).

• Track and improve key metrics: accuracy, agreement, throughput, cost per label, and SLA adherence.

• Partner with the product/engineering team to improve annotation tooling, dashboards, and automation.

Stakeholder Communication

• Serve as primary contact for internal teams needing labeled data (research, product, T&S, etc.).

• Provide regular status updates: progress vs plan, blockers, quality metrics, and risks.

• Gather feedback on label quality, edge cases, and evolving requirements; turn those into updated guidelines and processes.

Continuous Improvement

• Identify and implement process improvements to increase speed, reduce errors, and lower costs.

• Run experiments to optimize task design, instructions, and QA strategies.

• Help codify best practices into playbooks and documentation as we scale.

What We’re Looking For

Experience

• 3–7+ years in operations, project management, or program management, ideally in:

• Data annotation / labeling

• Trust & Safety operations

• Customer support operations

• Or another high-volume, process-driven environment

• Experience managing small to mid-sized teams and/or external vendors.

• Prior work in AI / ML, data labeling, or content moderation is a strong plus.

Skills

• Strong project management skills: planning, prioritizing, and keeping multiple tracks on schedule.

• Excellent written communication; you can write clear guidelines and edge-case docs.

• Comfortable working with metrics and dashboards (e.g., spreadsheets, BI tools) to monitor performance.

• Detail-oriented and process-minded; you naturally look for ways to standardize and streamline.

• Familiarity with annotation tools (e.g., Labelbox, Scale, Doccano, custom tools) is a plus, but not required.

Mindset

• Ownership mentality: you feel responsible for outcomes, not just tasks.

• Calm under pressure; you can navigate ambiguity and shifting priorities.

• Collaborative; you work well with annotators, engineers, and leadership alike.

• Bias toward action: you don’t just spot problems—you propose and test fixes.

Nice-to-Haves

• Experience setting up calibration tests, golden sets, and inter-annotator agreement.

• Background in trust & safety, content policy, or compliance.

• Exposure to SQL or basic data analysis for monitoring volumes and quality trends.

• Experience in a startup or early-stage environment where processes are still being built.