Data Engineer

About Us

Abacus Insights is transforming how data works for health plans. Our mission is simple: make healthcare data usable, so the people responsible for care and cost decisions can act faster, with confidence.
We help health plans break down data silos to create a single, trusted data foundation. That foundation powers better decisions —so plans can improve outcomes, reduce waste, and deliver better experiences for members and providers alike.

Backed by $100M from top investors, we’re tackling big challenges in an industry that’s ready for change. Our platform enables GenAI use cases by delivering clean, connected, and reliable healthcare data that can support automation, prioritization, and decision workflows—and it’s why we are leading the way.

Our innovation begins with people. We are bold, curious, and collaborative—because the best ideas come from working together. Ready to make an impact? Join us and let's build the future together.

About the role

We are seeking a skilled Data Engineer to join our growing Tech Ops division. As the organization continues to expand, this role provides the opportunity to contribute meaningful technical impact across our data delivery ecosystem. In this position, you will collaborate with customers, data vendors, and internal engineering teams to design, implement, and maintain reliable data integration solutions within a modern, cloud‑based environment.

You will apply advanced knowledge of distributed computing, data architecture, and cloud‑native engineering to build scalable and resilient data ingestion and transformation pipelines. As a technical contributor, you will support the adoption of Abacus’s core data management platform and help ensure high‑quality, compliant data operations across the data lifecycle.

Your day‑to‑day

Design and develop data pipelines

• Build and maintain batch and real‑time data pipelines using PySpark, SparkSQL, Databricks Workflows, and distributed processing frameworks.

• Create ingestion frameworks integrating with Databricks, Snowflake, AWS services (S3, Lambda, SQS), and vendor data APIs, ensuring data quality, lineage, and schema evolution.

Support data architecture & modeling

• Develop data models, including star/snowflake schemas, and apply optimization techniques for analytical workloads in cloud data environments.

• Participate in design discussions for reliable and fault‑tolerant architectures across multi‑account AWS environments.

Translate business needs into engineering solutions

• Convert customer and internal business requirements into clear technical specifications, data flows, and engineering components.

• Contribute to reusable frameworks and standardized engineering patterns.

Implement security & compliance workflows

• Apply data security best practices including RBAC, encryption at rest/in transit, PHI handling, tokenization, auditing, and adherence to HIPAA and SOC 2 compliance requirements.

• Support automation of secure data operations and governance frameworks.

Advance engineering best practices

• Implement CI/CD workflows for data pipelines, including code versioning, automated testing, orchestration, observability, and logging standards.

• Conduct performance profiling and assist with compute optimization, cost efficiency, and tuning techniques for Databricks and Snowflake environments.

Documentation & Collaboration

• Produce clear technical documentation, runbooks, architecture diagrams, and operational standards.

• Assist senior engineers through collaborative code reviews, knowledge‑sharing sessions, and cross‑team engineering initiatives.

What you bring to the team

• Bachelor’s degree in Computer Science, Computer Engineering, or a closely related technical field.

• 3+ years of hands‑on experience as a Data Engineer working with distributed data processing systems in modern cloud environments.

• Working knowledge of U.S. healthcare data domains — including claims, eligibility, and provider datasets — and ability to apply this knowledge to ingestion and transformation workflows.

• Strong communication skills to articulate technical concepts across both technical and non‑technical stakeholders.

• Proficiency in Python, SQL, and PySpark, including distributed data transformations and performance‑optimized queries.

• Demonstrated experience building and operating production ETL/ELT pipelines using Databricks, Airflow, or similar orchestration frameworks.

• Familiarity with cloud‑native data platforms and services such as dbt, Kafka, Delta Lake, and event‑driven/streaming architectures.

• Experience with structured and semi‑structured data formats (Parquet, JSON, Avro, ORC), including schema evolution techniques.

• Working knowledge of AWS data ecosystem components—S3, SQS, Lambda, Glue, IAM—used to support scalable data engineering workloads.

• Experience with Terraform and modern CI/CD pipelines (e.g., GitLab) for automating deployment and versioning.

• Understanding of SQL optimization strategies including partitioning, pruning, Z‑Ordering, clustering, and caching for large analytical workloads.

• Experience using cloud data warehouse technologies such as Snowflake, BigQuery, or Redshift, including performance tuning and data modeling practices.

What we would like to see but not required:

• Experience in healthcare or payer data environments at scale.

What you’ll get in return

• Competitive Leave & Benefits

• Comprehensive health coverage

• Equity for every employee – share in our success

• Growth-focused environment – your development matters here

Working Arrangements

• Standard hours: 9 hours/day, 5 working days

• Location: Onsite

• Shift: 10 AM – 7 PM local time

Our Commitment as an Equal Opportunity Employer

As a mission-led technology company helping to drive better healthcare outcomes, Abacus Insights believes that the best innovation and value we can bring to our customers comes from diverse ideas, thoughts, experiences, and perspectives. Therefore, we dedicate resources to building diverse teams and providing equal employment opportunities to all applicants. Abacus prohibits discrimination and harassment regarding race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws.

At the heart of who we are is a commitment to continuously and intentionally building an inclusive culture—one that empowers every team member across the globe to do their best work and bring their authentic selves. We carry that same commitment into our hiring process, aiming to create an interview experience where you feel comfortable and confident showcasing your strengths. If there’s anything we can do to support that—big or small—please let us know.