Senior LLM Systems Engineer - OPS00041

We are a US-based outsource software development company that has been delivering exceptional software experience to our clients since 2011, helping technology companies to become industry leaders.

Over the past few years, we’ve been hiring specialists all over the world while our main development centers were in Ukraine. Now, we keep expanding and start growing our centers in different parts of the world. Dev.Pro is open to hire specialists from other countries as well as Ukrainians who live outside of Ukraine now. We stand with Ukraine and keep supporting our people by offering a friendly remote environment while adhering to the values of democracy, human rights, and state sovereignty.

As a company of professionals, Dev.Pro offers challenging and interesting projects with world-leading clients, a modern technology stack, and career opportunities for both technical and non-technical specialists.

Job Title: LLM Systems Engineer

We are a civic intelligence startup with a singular mission: to restore trust and accountability to the American political system. We are building the "credit score" for elected officials—a data-backed, transparent Trust Score derived from millions of fragmented government records.

We aggregate attendance, voting behavior, bill histories, and public statements, using AI to normalize this data and present it to voters with transparent sourcing (AP, Reuters, NPR). We are looking for a master-level engineer to help us scale the platform that will modernize how Americans evaluate their representatives.

We’re looking for an LLM Systems Engineer who can architect, implement, and optimize multi-node LangGraph pipelines — orchestrating LLMs, APIs, and guardrails into deterministic, testable, and production-ready systems.
The role focuses on graph-based orchestration, data integration, evaluation pipelines, and latency optimization, not model training.

Core Responsibilities1. LangGraph Architecture & Workflow Design

• Build multi-node LangGraph workflows with dynamic routing, shared state, and conditional edges.

• Define and manage the graph schema — nodes as active processors, edges as routing logic.

• Implement LLM-driven intent classification to route user input across nodes or mark it out-of-scope.

• Manage state synchronization so nodes only update relevant data (intent, metadata, context, etc.).

2. API & Data Integration

• Connect nodes to multiple external JSON-returning endpoints and normalize data structures.

• Design intermediate transformation layers to unify inconsistent API responses.

• Use mock endpoints and seamlessly transition them to live APIs without breaking architecture.

• Handle asynchronous and parallel API calls to minimize latency.

3. Guardrails & Security

• Implement prompt and query-level guardrails to block unsafe inputs and prevent data leakage.

• Design state-aware routing logic that enforces isolation and privacy constraints between nodes.

• Build internal validators for payloads, ensuring all node inputs/outputs conform to strict schemas.

4. Evaluation Framework & Ground-Truth Testing

• Develop synthetic JSON-based ground-truth datasets to test system output deterministically.

• Build automated evaluation scripts to calculate F1, precision, recall, and exact-match scores.

• Compare multiple LLMs (Claude, GPT, etc.) by node and metric to determine best performers.

• Automate regression testing for new model versions or prompt updates.

5. Performance & Latency Optimization

• Use async execution and parallel node evaluation to reduce latency.

• Stream partial responses to improve perceived performance.

• Profile system components to locate and fix slow or sequential bottlenecks.

• Implement caching or smart pre-fetching for frequently used data sources.

6. System Reliability & Observability

• Build logging, tracing, and metric dashboards for every node and edge in the graph.

• Define error-handling strategies for malformed API responses or timeouts.

• Maintain test coverage across orchestration logic, node isolation, and evaluation functions.

• Implement CI/CD hooks to automatically re-evaluate the system before deployment.

Required Technical Skills

• Python (advanced) — async I/O, FastAPI, type hinting, logging, pytest.

• LangGraph (expert-level orchestration) — state machines, conditional edges, node composition.

• LLM APIs (OpenAI, Anthropic, etc.) — structured prompting, JSON mode, token usage optimization.

• Data Engineering — JSON schema normalization, API integration, validation layers.

• Evaluation Systems — F1, precision/recall metrics, dataset design, automated scoring.

• Asynchronous & Parallel Processing — asyncio, concurrent futures, non-blocking execution.

• Security / Guardrails — prompt validation, regex filters, payload sanitization, sandboxing.

• DevOps / Tooling — Docker, CI/CD, logging, observability (Grafana, Prometheus, OpenTelemetry).

Nice to Have

• Experience with LangChain, LlamaIndex, or orchestration frameworks beyond LangGraph.

• Familiarity with vector databases (pgvector, Pinecone, Weaviate).

• Experience in deterministic or safety-critical LLM applications (finance, legal, etc.).

• Comfort with multi-model evaluation pipelines and A/B testing at node level.

What Success Looks Like

• A fully operational LangGraph-based orchestration handling multi-node, multi-model routing with measurable latency and correctness improvements.

• Deterministic test results with high F1 and zero hallucination or leakage.

• A clean, modular codebase that allows new nodes, APIs, and models to be added with minimal refactoring.

• Automated evaluation dashboards tracking node-level performance across models and data versions.