Principal Architect – Secure AI Agent Architecture & Engineering

About Coalfire

Coalfire is on a mission to make the world a safer place by solving our clients’ hardest cybersecurity challenges. We work at the cutting edge of technology to advise, assess, automate, and ultimately help companies navigate the ever-changing cybersecurity landscape. We are headquartered in Chicago, Illinois with offices across the U.S. and U.K., and we support clients around the world.

But that’s not who we are – that’s just what we do.

We are thought leaders, consultants, and cybersecurity experts, but above all else, we are a team of passionate problem-solvers who are hungry to learn, grow, and make a difference.

Position Summary

The Principal Architect is a senior strategic technical role responsible for designing, building, and maintaining scalable applications and automation workflows that will serve as the foundation for advanced AI security and compliance agents. The candidate should be able recognize which problems and tasks benefit from pattern-based decision-making and which are best solved with conventional programmatic logic.

This role ensures the seamless integration of front-end, back-end, and cloud-native services while enabling efficient and secure deployment pipelines. The developer will collaborate closely with AI engineering developers, client data and product teams to deliver robust, production-grade systems that can scale to enterprise-level demands.

What You'll Do:
• Design, develop, and maintain scalable full stack applications using Python to support security and compliance agent workflows.

• Define and implement frameworks and systems to measure confidence intervals and output reliability of non-deterministic systems ensuring statistically significant outcomes from agents.

• Build and optimize RESTful APIs to enable seamless data exchange across services and systems.

• Implement secure DevOps best practices including CI/CD pipelines, infrastructure-as-code, and automated deployment strategies.

• Develop automation frameworks and tools to reduce manual tasks, improve efficiency, and support AI-driven workflows.

• Collaborate with data engineering teams to integrate microservices with analytics platforms and big data pipelines.

• Ensure compliance with security standards, governance policies, applicable compliance standards based on client need, and cloud best practices.

• Monitor, troubleshoot, and optimize system performance, ensuring high availability and reliability of deployed applications.

• Provide technical mentorship and contribute to setting best practices for development, DevOps, and cloud engineering.

• Successful completion of required training is a core expectation of this role (e.g., Agentic AI course). Dedicated study hours will be allocated to support your preparation. In the event training is not passed/achieved, a structured improvement plan will be provided to guide you toward successful completion.

What You'll Bring:
• 10+ years in software development, with demonstrated expertise across front-end, back-end, and DevOps/cloud engineering.

• Proficiency in Python (TypeScript a strong plus)

• Proven experience in API development (RESTful/GraphQL)

• Hands-on experience with AWS services (EC2, Lambda, S3, RDS, EKS, etc.)

• Strong background in Kubernetes for container orchestration

• Expertise in DevOps practices, CI/CD pipeline design, and automation tools (e.g., Terraform, Ansible, Jenkins, GitHub Actions)

• Experience working with microservice architectures in production environments.

• AWS/GCP/Azure Solutions Architect, AWS/GCP/Azure DevOps Engineer, or equivalent cloud/devOps certifications.

• Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience). A degree is not required if you can demonstrate equivalent understanding of the subject matter.

• Strong problem-solving abilities with a focus on designing scalable and secure solutions.

• Excellent communication and collaboration skills, with the ability to work cross-functionally in a fast-paced environment.

• Highly adaptable and comfortable learning new AI/automation technologies.

• Strong organizational skills with the ability to manage multiple priorities under tight deadlines.

• Proactive, self-motivated, and able to take ownership of end-to-end development processes.

• Mentorship mindset, able to share expertise and uplift junior engineers.

• Legal eligibility to work in the U.S.; willingness to occasionally travel for team workshops or client engagements.

Preferred Qualifications:
• Advanced degree in Computer Science, Engineering, or a related discipline.

• Previous experience in security, compliance, or automation engineering roles.

• Hands-on experience with big data pipelines, analytics platforms, or data engineering practices.

• Familiarity with agentic AI/automation frameworks or GenAI application development.

• Additional certifications in Kubernetes (CKA/CKAD), security (CISSP, CISM), or DevOps tooling.

• Experience working within highly regulated industries (finance, healthcare, defense, etc.).

Why You’ll Want to Join Us

At Coalfire, you’ll find the support you need to thrive personally and professionally. In many cases, we provide a flexible work model that empowers you to choose when and where you’ll work most effectively – whether you’re at home or an office.

Regardless of location, you’ll experience a company that prioritizes connection and wellbeing and be part of a team where people care about each other and our communities. You’ll have opportunities to join employee resource groups, participate in in-person and virtual events, and more. And you’ll enjoy competitive perks and benefits to support you and your family, like paid parental leave, flexible time off, certification and training reimbursement, digital mental health and wellbeing support membership, and comprehensive insurance options.

At Coalfire, equal opportunity and pay equity is integral to the way we do business. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran. Coalfire is committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, its services, programs, and activities. To request reasonable accommodation to participate in the job application or interview process, contact our Human Resources team at [email protected].