Senior Infrastructure Engineer

About Toptal

Toptal is a global network of top talent in business, design, and technology that enables companies to scale their teams, on-demand. With $200+ million in annual revenue and team members based around the globe, Toptal is the world’s largest fully remote workforce.

We take the best elements of virtual teams and combine them with a support structure that encourages innovation, social interaction, and fun. We see no borders, move at a fast pace, and are never afraid to break the mold.

Job Summary:

We are looking for an experienced Engineer to build and scale services in a cloud environment within our Infrastructure team. Our Infrastructure Engineers work with a high-energy, fast-paced team responsible for supporting initiatives and operations across Toptal.

This is a remote position. We do not offer visa sponsorship or assistance. Resumes and communication must be submitted in English.

Responsibilities:

The following information is intended to describe the general nature and level of work being performed. It is not intended to be an exhaustive list of all duties, responsibilities, or required skills.

· Toptal services are deployed across hundreds of servers. You will be responsible for designing, deploying, and maintaining the core logging, database, networking, and monitoring infrastructure across hundreds of servers. This includes ensuring high availability and optimal performance of these critical systems.
· We are embracing DevOps practices, where the Infrastructure team develops systems, automation, tooling, and workflows, and consults/mentors Developer teams to enable them to own the whole lifecycle of the software they are making.
· Implement monitoring for automated system health checks, develop procedures, and maintain system troubleshooting and maintenance documentation.
· Collaborate regularly with Engineering teams to improve the company’s engineering tools, systems, procedures, and data security, not just administer clusters and cloud services.
· Join daily scrum standups. Expect pair programming, engaging in peer code reviews, and using collaboration tools like Slack and Zoom.
· Design, develop, document, analyze, create, test, or modify computer or cloud-based systems or programs.

In the first week, expect to:

· Join our team and get introduced to Toptal’s practices and processes.
· Learn about our team’s processes and get familiar with the code that maintains our infrastructure resources.

In the first month, expect to:

· Gain insight into our system topology and how the whole system is structured.
· Understand our monitoring systems, alerting systems, and security.
· Participate in team meetings and get familiar with the ongoing projects and initiatives.
· Talk and meet with people from the operations squad.

In the first three months, expect to:

· Start working on support tasks to familiarize yourself with the core tools, setup, and everyday challenges.
· Provide excellent customer service by understanding and addressing the team’s needs and expectations through effective communication and collaboration while learning about our infrastructure.
· Deliver internal infrastructure and services such as monitoring, logging, automation, and data services targeted at our internal users.

In the first six months, expect to:

· Support Infrastructure design, architecture, and implementation.
· Have opportunities to be involved in systems design, identify new technologies to support the business, and resolve infrastructure compatibility and performance problems as they arise.
· Participate in the on-call rotation schedule (during business and after hours) to support all infrastructure-related systems, while exercising discretion and independent judgment.
· Report any downtime or performance issues faced by the system, drill down to find out what caused it, and coordinate with other teams to resolve them.
· Handle incident resolution if a Developer is not needed.
· Participate in our disaster recovery, change control, and security standards initiatives.

In the first year, expect to:

· Communicate with key partners on project engagements.
· Partner closely with our Engineering teams to develop infrastructure automation and management solutions that focus on scalability, observability, automation, reliability, security, and quality in Google Cloud Platform.
· Plan and coordinate testing of changes, upgrades, patches, new releases, and new services.
· Participate in technology initiatives that enable Developers to deliver their services to our customers with minimal friction and high quality.

Qualifications and Job Requirements:

· 5+ years of experience in Linux debugging, networking, routing, IP addressing, Load Balancing, and VPNs. We don’t run our own AS and maintain thousands of BGP Routes but need to know how traceroute works or when to choose L4/L7 balancers.
· Previous experience managing infrastructure configuration and provisioning through code for large, distributed systems on public cloud platforms (preferably AWS and GCP, but if you have had experience with Azure, Oracle Cloud, DigitalOcean, Linode, and others, that’s fine too).
· Knowledge of Infrastructure-as-Code principles and hands-on experience with automation tools. We use Ansible and Terraform, but if you know Puppet or Chef very well, that’s good too. An understanding of version control is needed because our code is kept in Git.
· Experience running RDBMS, especially PostgreSQL experience, is an added advantage. You generally won’t need to manually optimize SQL queries, but it’s important to have a solid understanding when things break or aren’t performing as well as needed. Transferable knowledge from MySQL, SQLite, and others is preferred.
· Hands-on experience with system and application metric collection and alerting services like Graphite, Grafana, Prometheus, InfluxDB, Sensu, etc. A keen focus on what makes a system observable.
· Outstanding troubleshooting skills. Experience in resolving complex problems through various troubleshooting protocols and processes.
· A strong understanding of modern systems and service-oriented architecture.
· Proficiency in scripting languages such as Python, Bash, Ruby, etc.
· Experience with Docker, Docker Compose, and building optimized Dockerfiles is an advantage.
· Experience with Kubernetes environments: production operations, troubleshooting, debugging, cluster provisioning, and management is an advantage.
· Eagerness to help teammates, share knowledge with them, and learn from them.
· Outstanding written and verbal communication skills.
· Ability to work in a fast-paced, rapidly growing company and handle a wide variety of challenges, deadlines, and a diverse array of contacts.
· You must be a world-class individual contributor to thrive at Toptal. You will not be here just to tell other people what to do.