Docker in Production: Beyond docker-compose for Real Deployments

Docker has revolutionized application development, making it easier to package and deploy software. But while docker-compose is fantastic for local development, it falls short when you need to run your applications reliably in production.

Docker in Production: Beyond docker-compose for Real Deployments

At MisuJob, we leverage Docker extensively to build and deploy our AI-powered job matching platform, which processes 1M+ job listings and aggregates from multiple sources across Europe. We’ve learned firsthand that moving beyond docker-compose is crucial for achieving the scalability, reliability, and maintainability required for a production environment. This post outlines our journey and the tools we use to deploy Docker containers effectively.

The Limitations of docker-compose in Production

docker-compose shines during development. It allows you to define and manage multi-container applications with ease, simplifying the process of setting up and running dependent services like databases, message queues, and your application code itself. However, it has several significant limitations for production deployments:

• Single-Host Focus: docker-compose is designed primarily for single-host environments. It doesn’t inherently provide mechanisms for scaling applications across multiple servers or handling failures gracefully.
• Lack of Orchestration: It lacks the features needed for orchestrating complex deployments, such as automated rollouts, rollbacks, and health checks.
• Limited Monitoring & Logging: While you can configure logging drivers, docker-compose doesn’t offer robust monitoring or centralized logging capabilities out-of-the-box.
• Security Concerns: Managing secrets and sensitive data within docker-compose.yml files can introduce security vulnerabilities.
• Configuration Management: While environment variables help, managing complex configurations across multiple environments becomes cumbersome with just docker-compose.

Orchestration is Key: Kubernetes and Docker Swarm

To overcome these limitations, we adopted container orchestration. Orchestration tools automate the deployment, scaling, and management of containerized applications. Two popular options are Kubernetes and Docker Swarm. We evaluated both and ultimately chose Kubernetes for its rich feature set, strong community support, and extensive ecosystem.

• Kubernetes: Kubernetes is a powerful open-source container orchestration system for automating application deployment, scaling, and management. It provides features like:

  • Automated Deployments & Rollbacks: Deploy new versions of your application with zero downtime and automatically roll back to previous versions if issues arise.
  • Self-Healing: Automatically restarts failed containers and reschedules them on healthy nodes.
  • Scaling: Easily scale your application horizontally by increasing the number of replicas.
  • Service Discovery & Load Balancing: Provides built-in service discovery and load balancing for your applications.
  • Secret Management: Securely manage sensitive data like passwords and API keys.

• Docker Swarm: Docker Swarm is Docker’s native clustering solution. It’s simpler to set up and manage than Kubernetes, making it a good option for smaller deployments or teams with less experience. However, it lacks some of the advanced features and flexibility of Kubernetes.

While we use Kubernetes, the key principle is adopting some orchestration technology to move beyond the limitations of docker-compose.

Our Kubernetes Deployment Strategy

Our Kubernetes deployment strategy focuses on automation, reliability, and observability. Here’s a breakdown of the key components:

1. Infrastructure as Code (IaC): We use Terraform to provision and manage our Kubernetes cluster and underlying infrastructure on cloud providers like AWS and Azure. This ensures consistency and repeatability across environments.
2. Continuous Integration/Continuous Deployment (CI/CD): We use GitLab CI/CD to automate the build, test, and deployment process. When a code change is merged into the main branch, GitLab CI/CD automatically builds a Docker image, pushes it to our container registry (e.g., Docker Hub or AWS ECR), and deploys the new image to our Kubernetes cluster.
3. Declarative Configuration: We define our application deployments, services, and other Kubernetes resources using YAML files. These files are stored in Git and managed as code. This allows us to track changes, collaborate effectively, and easily roll back to previous configurations if needed.
4. Health Checks: We configure liveness and readiness probes for our containers to ensure that Kubernetes can automatically detect and restart unhealthy containers.
5. Monitoring & Logging: We use Prometheus and Grafana to monitor the performance of our applications and infrastructure. We also use Elasticsearch, Fluentd, and Kibana (EFK stack) for centralized logging.

Example Kubernetes Deployment YAML

Here’s a simplified example of a Kubernetes Deployment YAML file for one of our backend services:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
  labels:
    app: api-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
    spec:
      containers:
      - name: api-service
        image: misujob/api-service:latest
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        resources:
          requests:
            cpu: "200m"
            memory: "512Mi"
          limits:
            cpu: "500m"
            memory: "1Gi"

This YAML file defines a deployment with three replicas of the api-service container. It also configures liveness and readiness probes to ensure that the application is healthy and ready to serve traffic. Resource requests and limits are defined to ensure that the application has sufficient resources and doesn’t consume excessive resources.

Secrets Management with HashiCorp Vault

Storing secrets directly in Kubernetes manifests or environment variables is a security risk. We use HashiCorp Vault to securely manage secrets and dynamically inject them into our containers at runtime. Vault provides a centralized and auditable way to store, access, and distribute secrets.

Here’s an example of how we might use Vault to retrieve a database password:

vault read secret/data/production/api-service/database | jq -r '.data.data.password'

This command retrieves the password field from the secret/data/production/api-service/database path in Vault and prints it to standard output. We can then use this password to configure our application.

Observability: Monitoring, Logging, and Tracing

Observability is crucial for understanding the behavior of our applications in production and quickly identifying and resolving issues. We use a combination of tools to achieve comprehensive observability:

• Prometheus & Grafana: We use Prometheus to collect metrics from our applications and infrastructure, and Grafana to visualize those metrics. We monitor key performance indicators (KPIs) such as CPU usage, memory usage, request latency, and error rates.
• EFK Stack (Elasticsearch, Fluentd, Kibana): We use the EFK stack for centralized logging. Fluentd collects logs from our containers and forwards them to Elasticsearch, where they are indexed and stored. Kibana provides a web interface for querying and visualizing the logs.
• Jaeger/Zipkin: For distributed tracing, we are exploring Jaeger and Zipkin to trace requests as they propagate through our microservices architecture. This helps us identify performance bottlenecks and understand the dependencies between services.

Optimizing Docker Image Size

Smaller Docker images lead to faster deployments and reduced storage costs. We employ several techniques to minimize our image sizes:

• Multi-Stage Builds: We use multi-stage builds to separate the build environment from the runtime environment. This allows us to include build tools and dependencies in the build stage without including them in the final image.
• Base Image Selection: We choose minimal base images such as Alpine Linux or distroless images. These images contain only the essential components needed to run our applications.
• Removing Unnecessary Files: We carefully remove any unnecessary files from our images, such as documentation, examples, and development tools.

Performance Considerations and Optimizations

Deploying Docker containers in production requires careful attention to performance. Here are some of the optimizations we’ve implemented:

• Resource Limits & Requests: We configure resource limits and requests for our containers to ensure that they have sufficient resources and don’t consume excessive resources.
• Horizontal Pod Autoscaling (HPA): We use HPA to automatically scale the number of replicas based on CPU utilization or other metrics.
• Connection Pooling: We use connection pooling to reduce the overhead of establishing new connections to databases and other services.
• Caching: We implement caching at various levels (e.g., application-level caching, CDN caching) to reduce latency and improve performance.

Example: Optimizing Database Queries

Let’s look at an example of optimizing a database query. We identified a slow query in one of our services that was impacting performance. The original query looked something like this (PostgreSQL example):

SELECT *
FROM jobs
WHERE location LIKE '%Berlin%'
  AND salary > 50000
ORDER BY published_date DESC
LIMIT 100;

This query was slow because it was performing a full table scan on the jobs table due to the LIKE operator. To optimize the query, we added an index on the location column and used full-text search capabilities:

CREATE INDEX idx_jobs_location ON jobs USING gin (to_tsvector('english', location));

SELECT *
FROM jobs
WHERE to_tsvector('english', location) @@ to_tsquery('english', 'Berlin')
  AND salary > 50000
ORDER BY published_date DESC
LIMIT 100;

This optimization significantly improved the query performance, reducing the query time from several seconds to milliseconds.

Salary Benchmarking Across Europe

Understanding salary benchmarks is critical for both job seekers and employers. Here’s a simplified table showing estimated average annual salaries for Software Engineers (mid-level, 3-5 years experience) across different European countries:

These numbers are estimates and can vary depending on the specific role, company size, and location within each country. MisuJob helps you find more precise salary data tailored to your specific skills and experience.

Key Takeaways

• docker-compose is excellent for local development, but it’s not suitable for production deployments.
• Container orchestration tools like Kubernetes and Docker Swarm are essential for managing containerized applications at scale.
• Automate your deployment process with CI/CD pipelines and Infrastructure as Code.
• Implement comprehensive monitoring and logging to ensure the health and performance of your applications.
• Securely manage secrets with tools like HashiCorp Vault.
• Optimize Docker image sizes and application performance for faster deployments and reduced costs.

By following these best practices, you can leverage the power of Docker to build and deploy reliable, scalable, and maintainable applications in production. We’ve seen significant improvements in our deployment frequency, application uptime, and overall engineering efficiency since adopting these strategies at MisuJob.