Email Deliverability Engineering: SPF, DKIM, and DMARC Explained

Email deliverability: the bane of every engineer who’s ever had to send an email from their application. Getting your emails into the inbox, and not the spam folder, requires more than just pressing “send.” It demands a deep understanding of email authentication mechanisms.

Email Deliverability Engineering: SPF, DKIM, and DMARC Explained

At MisuJob, ensuring our users receive timely updates and job alerts is paramount. We aggregate from multiple sources and process 1M+ job listings, which translates to a significant volume of email communication. Poor email deliverability can directly impact user engagement and satisfaction. Therefore, we’ve invested heavily in understanding and implementing best practices for email authentication. In this post, we’ll share our experiences and knowledge about SPF, DKIM, and DMARC—essential tools for improving email deliverability.

The Problem: Spoofing and Phishing

Before diving into the solutions, let’s understand the problem. Email spoofing, where malicious actors forge the “From” address to impersonate legitimate senders, is a pervasive threat. Phishing attacks often rely on spoofing to trick recipients into divulging sensitive information.

Without proper authentication, email servers have no reliable way to verify the sender’s legitimacy, leading to increased spam filtering and missed opportunities for genuine communication. Imagine a crucial job alert from MisuJob ending up in a user’s spam folder – a missed career opportunity!

SPF: Sender Policy Framework

SPF (Sender Policy Framework) is an email authentication method that allows you to specify which mail servers are authorized to send emails on behalf of your domain. It works by publishing a TXT record in your domain’s DNS zone, listing the allowed IP addresses or hostnames.

How SPF Works:

When an email is sent, the receiving mail server checks the “Return-Path” address (also known as the “envelope sender”) to determine the domain responsible for the email.
The receiving server queries the DNS records of that domain for an SPF record.
If an SPF record exists, the receiving server evaluates whether the sending server’s IP address is authorized to send emails for that domain, based on the rules defined in the SPF record.
The receiving server assigns a “pass” or “fail” result to the SPF check, which influences its spam filtering decisions.

SPF Record Syntax:

An SPF record is a TXT record with the following syntax:

v=spf1 <mechanism1> <mechanism2> ... <modifier>

v=spf1: Specifies the SPF version (currently spf1).
<mechanism>: Defines authorization rules. Common mechanisms include:
- ip4: Matches specific IPv4 addresses or ranges.
- ip6: Matches specific IPv6 addresses or ranges.
- a: Matches the IP addresses of the domain’s A records.
- mx: Matches the IP addresses of the domain’s MX records.
- include: Includes another domain’s SPF record.
<modifier>: Modifies the behavior of the SPF record. The most common modifier is redirect, which redirects SPF evaluation to another domain.

Example SPF Record:

v=spf1 ip4:192.0.2.0/24 include:sendgrid.net -all

This record authorizes:

IP addresses in the range 192.0.2.0/24
IP addresses authorized by sendgrid.net’s SPF record
-all indicates that any other IP address is explicitly not authorized, and the email should be rejected (a “hard fail”).

SPF Considerations:

The 10 DNS Lookup Limit: SPF records are limited to 10 DNS lookups. Exceeding this limit can cause SPF checks to fail. This is particularly important when using multiple include mechanisms.
Multiple SPF Records: You should only have one SPF record per domain. Multiple SPF records can cause unpredictable behavior.
Testing Your SPF Record: Tools like dig or online SPF record checkers can help you verify the validity of your SPF record.

Example: Testing with dig

dig +short TXT misujob.com

This command will return the TXT records for misujob.com, including the SPF record (if present).

DKIM: DomainKeys Identified Mail

DKIM (DomainKeys Identified Mail) provides a way to digitally sign your emails, allowing receiving mail servers to verify that the email was indeed sent by you and hasn’t been tampered with during transit.

How DKIM Works:

The sending mail server generates a digital signature using a private key. This signature is added to the email header.
The public key corresponding to the private key is published in your domain’s DNS zone as a TXT record.
The receiving mail server retrieves the public key from the DNS record and uses it to verify the digital signature in the email header.
If the signature is valid, the receiving server can be confident that the email originated from the claimed sender and hasn’t been altered.

DKIM Record Syntax:

A DKIM record is a TXT record with the following syntax:

v=DKIM1; k=rsa; p=<public_key>; s=<selector>

v=DKIM1: Specifies the DKIM version (currently DKIM1).
k=rsa: Specifies the key type (usually RSA).
p=<public_key>: The public key used for signature verification.
s=<selector>: A selector that identifies the specific DKIM key pair used. This allows you to rotate keys without disrupting email delivery.

Example DKIM Record:

v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwJjP9h...; s=email

In this example:

email is the selector. This means the email sending server will use the private key linked to the selector to sign emails.
The long string after p= is the public key.

DKIM Considerations:

Key Length: Use a strong key length (e.g., 2048 bits) for security.
Key Rotation: Regularly rotate your DKIM keys to mitigate the risk of key compromise.
Selector Management: Choose meaningful selectors and keep track of which keys are associated with which selectors.

Example: Verifying DKIM Signature

While you can’t directly verify a DKIM signature using dig, you can examine the email headers to see if the DKIM signature is present and whether the receiving server has verified it. Look for headers like DKIM-Signature and Authentication-Results.

DMARC: Domain-based Message Authentication, Reporting & Conformance

DMARC (Domain-based Message Authentication, Reporting & Conformance) builds upon SPF and DKIM to provide a comprehensive email authentication framework. It allows domain owners to specify how receiving mail servers should handle emails that fail SPF and DKIM checks and provides reporting mechanisms to track email authentication results.

How DMARC Works:

The domain owner publishes a DMARC record in their DNS zone, specifying a policy for handling emails that fail SPF and/or DKIM checks.
When an email is received, the receiving mail server performs SPF and DKIM checks.
The receiving server evaluates the results of the SPF and DKIM checks against the DMARC policy.
Based on the policy, the receiving server may:
- Do nothing (monitor).
- Quarantine the email (e.g., move it to the spam folder).
- Reject the email outright.
The receiving server sends aggregate reports to the email address specified in the DMARC record, providing insights into email authentication results.

DMARC Record Syntax:

A DMARC record is a TXT record with the following syntax:

v=DMARC1; p=<policy>; rua=<email_address>; ruf=<email_address>; adkim=<alignment_mode>; aspf=<alignment_mode>; fo=<failure_options>

v=DMARC1: Specifies the DMARC version (currently DMARC1).
p=<policy>: Specifies the policy for handling emails that fail SPF and/or DKIM checks:
- none: Monitor only (no action).
- quarantine: Move to spam folder.
- reject: Reject the email.
rua=<email_address>: Email address to which aggregate reports should be sent.
ruf=<email_address>: Email address to which forensic reports (failure reports) should be sent.
adkim=<alignment_mode>: Specifies the DKIM alignment mode:
- r: Relaxed alignment (allows subdomain matching).
- s: Strict alignment (requires exact domain matching).
aspf=<alignment_mode>: Specifies the SPF alignment mode:
- r: Relaxed alignment (allows subdomain matching).
- s: Strict alignment (requires exact domain matching).
fo=<failure_options>: Specifies options for generating failure reports.

Example DMARC Record:

v=DMARC1; p=quarantine; rua=mailto:[email protected]; ruf=mailto:[email protected]; adkim=r; aspf=r; fo=1

This record specifies:

Emails that fail SPF and/or DKIM checks should be quarantined.
Aggregate reports should be sent to [email protected].
Forensic reports should be sent to [email protected].
Relaxed alignment is used for both DKIM and SPF.

DMARC Considerations:

Start with p=none: Begin by implementing DMARC in “monitor” mode (p=none) to gather data and identify any legitimate email sources that may not be properly authenticated.
Analyze Reports: Regularly analyze the DMARC reports to identify and address any email authentication issues.
Gradually Increase Policy: Gradually increase the policy from none to quarantine and eventually to reject as you gain confidence in your email authentication setup.
Monitor Impact: Closely monitor the impact of DMARC on your email deliverability and adjust your configuration as needed.

Example: Analyzing DMARC Aggregate Reports

DMARC aggregate reports are typically sent as XML files. Analyzing these reports can provide valuable insights into your email authentication performance. Here’s a simplified example of what a DMARC report might contain:

<record>
  <row>
    <source_ip>203.0.113.45</source_ip>
    <count>100</count>
    <policy_evaluated>
      <disposition>quarantine</disposition>
      <dkim>pass</dkim>
      <spf>fail</spf>
    </policy_evaluated>
  </row>
  <identifiers>
    <header_from>misujob.com</header_from>
  </identifiers>
</record>

This record indicates that 100 emails from IP address 203.0.113.45, claiming to be from misujob.com, passed DKIM but failed SPF. The DMARC policy was set to “quarantine,” so these emails were likely moved to the spam folder. This could indicate an issue with the SPF record or a legitimate sender not being properly authorized.

Practical Example: MisuJob’s Email Authentication Setup

At MisuJob, we use a combination of SPF, DKIM, and DMARC to protect our domain and ensure high email deliverability. Here’s a simplified overview of our setup:

SPF: We maintain an SPF record that includes all of our authorized sending servers, including our internal mail servers and third-party email service providers.
DKIM: We use DKIM to sign all of our outgoing emails, using a 2048-bit RSA key. We rotate our DKIM keys every six months.
DMARC: We started with a p=none policy and gradually increased it to p=quarantine after carefully analyzing our DMARC reports. We are now evaluating whether to move to p=reject.

We actively monitor our DMARC reports and use them to identify and address any email authentication issues. For example, we recently discovered that some of our marketing emails were failing SPF checks because they were being sent from a new IP address that wasn’t included in our SPF record. We quickly updated our SPF record to resolve the issue.

Impact on Deliverability and User Experience

Implementing SPF, DKIM, and DMARC has significantly improved our email deliverability rates. We’ve seen a noticeable decrease in the number of emails being marked as spam, which has resulted in better user engagement and satisfaction.

For example, before implementing DMARC, our email open rates for job alert emails in France were around 15%. After implementing DMARC with a p=quarantine policy, our open rates increased to 22%, a significant improvement.

Salary Implications for Email Engineers

Mastering email deliverability is a valuable skill, especially for engineers working on platforms that rely heavily on email communication. Here’s a table showing approximate salary ranges for email engineers in different European countries (in EUR per year):

Country	Junior (0-2 years)	Mid-Level (3-5 years)	Senior (5+ years)
Germany	50,000 - 65,000	70,000 - 90,000	95,000 - 120,000
UK	45,000 - 60,000	65,000 - 85,000	90,000 - 115,000
Netherlands	48,000 - 63,000	68,000 - 88,000	93,000 - 118,000
France	40,000 - 55,000	60,000 - 80,000	85,000 - 110,000
Spain	35,000 - 50,000	55,000 - 75,000	80,000 - 100,000
Nordic (Avg)	55,000 - 70,000	75,000 - 95,000	100,000 - 130,000

These figures are estimates and can vary depending on factors such as location within the country, company size, and specific skills.

Conclusion

Implementing SPF, DKIM, and DMARC is crucial for ensuring email deliverability and protecting your domain from spoofing and phishing attacks. While the initial setup may seem complex, the benefits of improved deliverability and enhanced security are well worth the effort. By understanding these authentication mechanisms and actively monitoring your email authentication results, you can significantly improve your email deliverability rates and ensure that your important messages reach their intended recipients.

Key Takeaways:

SPF, DKIM, and DMARC are essential for email authentication.
Start with p=none in DMARC and gradually increase the policy.
Monitor DMARC reports regularly to identify and address issues.
Email deliverability expertise is a valuable skill in the engineering job market.