iTranslated by AI
Technologies for Sending Emails
This is a memo of things I researched out of curiosity ✍
(Updated 2022/4/2) I have made additions and corrections based on various points and comments received on Twitter and Hatebu. In particular, I had completely incorrect explanations for several technologies, so this was extremely helpful... thank you very much 🙏
(Updated 2024/8/13) Earlier this year, a book titled 実務で使える メール技術の教科書 (Textbook of Email Technology for Practical Work) was published 🎉
I bought and read it, and it covers a wide range of topics related to email. While there are fewer descriptions regarding public clouds, it focuses on more fundamental content, such as how to build your own email server.
Why I researched it
When developing and operating web applications with email sending functionality, you often find yourself investigating causes when receiving inquiries like "Emails aren't reaching a specific address" or "I got an email from someone named MAILER-DAEMON."
In practice, we often use services like Amazon SES or SendGrid that handle email sending abstractly. However, specifically for troubleshooting, knowing the overview of the various underlying technologies makes it easier to grasp the situation and identify the cause.
Thankfully, email seems to be a mature technology, so there are many articles with old publication dates that are still helpful. However, as I researched to understand the big picture in the modern day, I felt it was wasteful to keep looking things up because there were various specifications that seemed to be defined ad-hoc for historical reasons. So, I decided to summarize them.
As the title suggests, this article is about sending emails, so I haven't touched on various technologies related to receiving emails or email content. Additionally, I am just a web application developer who uses email based on "vibes," so there is a possibility that it contains many errors.
Please keep that in mind 🦵
What I researched
To capture the big picture, I created a diagram to show where each of the technologies that frequently appear in research is located 🌏
Below, we will look at them in order from the email sender to the recipient.
MUA / MSA / MTA
To understand email sending roughly, it is good to be aware of these three main roles. While some articles may conflate MSA and MTA or have slight differences, understanding the general concept is sufficient.
- MUA (Message User Agent)
- Software used to send (or receive) emails.
- Originally, this referred to email clients running on PCs (such as Outlook) that perform communication via SMTP / IMAP.
- When using mechanisms like Amazon SES that allow email sending via API, the client program could also be considered an MUA in a broad sense.
- MSA (Message Submission Agent)
- Software that accepts email transmission requests from an MUA.
- It relays the accepted emails to an MTA.
- This corresponds to the source of the email; for example, Amazon SES configured on an AWS account or Postfix running on EC2 falls into this category.
- To prevent unauthorized users from sending emails, some kind of authentication mechanism is required for the MUA.
- Software that accepts email transmission requests from an MUA.
- MTA (Message Transfer Agent)
- Software that transfers emails to the destination server.
- It transfers emails sent from MSAs or other MTAs to the destination MTA.
- MSAs often also perform the role of an MTA.
- Basically, emails are passed through the route:
MUA -> MSA -> (possibly via other MTAs depending on the situation) -> Destination MTA.
- Basically, emails are passed through the route:
-
Email is a public and trust-based system, so one MTA (or MSA) can freely transfer emails to another MTA.
- If you know an email address, you can send an email to that person at any time.
- Because of this, a mechanism to verify whether the email sender is a legitimate server is necessary.
SMTP
The protocol used for sending emails is SMTP (Simple Mail Transfer Protocol). We won't go into detail here, but be careful not to confuse it with other protocols like IMAP (Internet Message Access Protocol) used for receiving emails.
In SMTP, commands are constructed using ASCII characters, similar to HTTP, and communication is performed via TCP. Therefore, if server access is possible, you can send emails using the telnet command or similar tools.
Submission Port
As mentioned in the previous article, the port numbers generally used when connecting to the destination server are 587 or 465, and these are called submission ports.
When people think of port numbers used for SMTP, 25 is probably the most famous, but this is used by MTAs to perform email transfer.
(Note that while there doesn't seem to be a common name like "submission port" for port 25, there were cases where it was referred to as the Standard Port).
The reason for this is that since SMTP originally did not require authentication and allowed anyone to send emails freely, spammers (Spam Mail) began to run rampant.
To address this, the ports used by MSAs and MTAs were separated. On the submission port, authentication is performed via SMTP AUTH etc. when using the MSA. Meanwhile, safe email transmission is achieved by verifying the reliability of the source MTA (checking if it is a spammer) that accesses port 25.
SMTPS
Nowadays, clients that send emails increasingly include web applications like Gmail and the AWS SDK. However, MUA originally referred to software like Outlook or Thunderbird that was installed on a client PC to communicate via SMTP.
For these, there is a standard called SMTPS (SMTP over SSL/TLS) that performs encrypted communication from the connection establishment stage using port 465.
Regarding this port, there is information stating that its use is not recommended due to historical reasons where it was temporarily deprecated. However, it is considered a valid port in RFC8314 (although it seems to overlap with other protocols).
Furthermore, it is still commonly used in practice. For example, BIGLOBE's email server connection settings required users to connect from MUA to MSA using port 465.
STARTTLS
In the previously mentioned SMTPS, the security of the communication path is high because encrypted communication is forced from the moment the port starts to be used. However, since SMTP was originally designed to send and receive in plaintext, there is a risk that an older destination MTA might not support encrypted communication, in which case the email might not be sent correctly.
To address this issue, instead of using the same port, SMTP can start communication in plaintext and switch to encrypted communication midway after confirming that both sides support TLS.
This can be implemented by issuing the STARTTLS command during SMTP communication, and it is used when using port 587 as a submission port or when using encrypted communication on port 25.
Specifications like STARTTLS, where the encryption method is changed during the connection process according to the connection target, are called Opportunistic Encryption, and this has practical advantages in SMTP where messages may pass through various MTAs during the transmission process.
In the context of SSL/TLS, this is sometimes explained with the terms Implicit TLS / Explicit TLS. SMTPS corresponds to Implicit TLS, and STARTTLS corresponds to Explicit TLS.
Additionally, regarding emails sent and received by Google, you can check what percentage of the total traffic is exchanged over encrypted paths.
As of 2022, it seems that approximately 85% of emails are sent over encrypted paths.
Quoted from https://transparencyreport.google.com/safer-email/overview
OP25B
Earlier, I explained that "email is a trust-based system." Initially, spammers bypassed MSAs that required authentication, set up their own rogue MTAs, and used bots to send spam or virus emails to port 25.
As a countermeasure, various ISPs (Internet Service Providers) took action by blocking outbound connections using port 25 from subscribed users. By limiting port 25 access to email servers operated by ISPs or email delivery providers, they block suspicious senders.
Reference: Sending emails using AWS
Taking Amazon SES as an example, you can choose to send emails via API or SMTP. In either case, a managed MSA (SES in the diagram) is used.
As a result, from the perspective of an MTA outside of AWS (Receivers in the diagram), the email source (SES in the diagram) is a sufficiently trusted entity. Furthermore, since the email sender (Sender in the diagram) is authenticated by the source, sending spam or virus emails would be traced back to the sender by the source.
Quoted from https://docs.aws.amazon.com/ja_jp/ses/latest/dg/send-email-concepts-process.html
Also, if you wonder if you can bypass these restrictions by running your own Postfix on EC2, that's not the case. Outbound communication on port 25 from AWS accounts is not allowed by default, and if you want to use it, you need to submit a request to AWS to remove the restriction.
SMTP Relay
Given the background explained above, using an MTA provided by a business specializing in email delivery, rather than operating one yourself, offers higher reliability and greater benefits from the recipient's perspective.
The aforementioned Amazon SES and SendGrid support SMTP relay, which allows you to transparently use their managed servers as the email source. This is intended for use cases where you delegate only the email sending portion from an MSA built independently within an internal network or similar environment. There are many providers specializing in such SMTP relay services.
SMTP relay between MTA servers is performed using port 25 and encrypted communication with STARTTLS.
You can check which SMTP servers an email passed through during delivery by obtaining the email headers and using the following site:
For example, in the display below, you can see that the email was sent through MTAs such as mail7.nyi.meetup.com → COL004-MC1F51.hotmail.com → COL004-OMC4S14.hotmail.com → mx.google.com.
Quoted from https://toolbox.googleapps.com/apps/messageheader/
DNS
I explained that verifying the validity of the email sender/relay source is important when sending emails between MTAs, and DNS (Domain Name System) plays a crucial role in this.
Email addresses are written in a format like hogehoge@example.com. The destination of the email is identified by resolving the IP address corresponding to the domain part after the @ symbol, example.com, via DNS.
To access DNS from the CLI, the dig command is used.
Below, we will look in detail at records specifically related to sending emails.
MX Record
In the process of transferring emails from an MSA to the destination MTA, the MX (Mail Exchange) record for the destination domain is referenced.
Note that the MX record of the sender domain is not required (it is irrelevant) for sending emails.
For example, investigating the MX record for the gmail.com domain returns the following, showing that communication with the destination MTA is possible by accessing gmail-smtp-in.l.google.com (though this is a guess since it cannot actually be tested due to OP25B...).
$ dig gmail.com MX
(omitted)
;; ANSWER SECTION:
gmail.com. 2159 IN MX 10 alt1.gmail-smtp-in.l.google.com.
gmail.com. 2159 IN MX 40 alt4.gmail-smtp-in.l.google.com.
gmail.com. 2159 IN MX 20 alt2.gmail-smtp-in.l.google.com.
gmail.com. 2159 IN MX 5 gmail-smtp-in.l.google.com.
gmail.com. 2159 IN MX 30 alt3.gmail-smtp-in.l.google.com.
(omitted)
If no MX record exists, it apparently attempts to connect to the address indicated by the A record or AAAA record as a fallback.
Therefore, during troubleshooting, the absence of an MX record does not immediately mean the domain cannot receive emails.
MTA-STS
As mentioned in the previous section, STARTTLS is vulnerable to operations where a malicious third party forces communication in plaintext, such as a downgrade attack, because the encryption level changes during the negotiation process with the connection target.
To address this situation, a new specification called RFC8461 MTA-STS (SMTP MTA Strict Transport Security) appeared in 2018.
MTA-STS allows the receiving MTA side to explicitly state policies such as forcing an encrypted connection for email senders; if the sender cannot support encryption, the email will not be sent.
For example, Gmail supports MTA-STS, and if you want to use it with your own domain, you can set it up following these steps:
The following article provides a thorough explanation of MTA-STS, including existing technologies:
SPF / DKIM / DMARC
Up to this point, I have explained the technologies used in the process where an email is transferred through MUA -> MSA -> MTA.
From here, I will explain the validity verification performed during the process where the destination MTA accepts the transmitted email.
Verification is performed by comparing information presented to the destination MTA during SMTP communication with several records registered in the DNS.
A brief summary is as follows:
- SPF
- Verification of the validity of the email sender's domain.
- DKIM
- Detection of tampering with email content.
- DMARC
- Deterrence of email sender domain spoofing.
- Definition of how to handle suspicious emails when they are detected.
Each of these settings is optional, and according to specifications, email transmission is possible without them. However, major email providers, including Gmail, have begun requiring these authentications to be passed when receiving emails from domains sending a certain volume of emails since February 2024. Therefore, it is practically essential to set them up, especially for business use.
The following document is helpful for understanding the requirements when implementing new email sending functionality in the future:
SPF
Sender domain validity verification confirms whether the original sender's IP address is associated with the email's sender domain.
For example, if an email with hoge@example.com as the From address is sent from the IP address xxx.xxx.xxx.xxx, it is sufficient if it can be determined that xxx.xxx.xxx.xxx is a valid sender from example.com.
This whole mechanism is called SPF (Sender Policy Framework), and it is implemented by registering a TXT record in a specific format in the DNS.
At this time, note that Envelope-From / Return-Path is used for checking the sender domain.
For instance, in Amazon SES, configuring it to allow amazonses.com as a sender completes the SPF setup. However, since SES's Return-Path defaults to something like xxxxx@ap-northeast-1.amazonses.com using amazonses.com, SPF configuration on the example.com domain side is not mandatory.
Therefore, if you have configured a custom domain for the Return-Path, you will need to specify the SPF record for that domain.
Here is a brief explanation of how to verify the settings.
SPF authentication status can be confirmed in the Authentication-Results header of the email.
The example.com domain shown as the sender xxxxx@example.com at this time is the domain used as the reference for the SPF record.
Authentication-Results: mx.google.com;
spf=pass (google.com: domain of xxxxx@example.com designates xxx.xxx.xxx.xxx as permitted sender) smtp.mailfrom=xxxxx@example.com;
If you have associated the example.com domain with SES as a valid sender, querying it with the dig command will return a value like the following.
Since only the person who owns the domain can register records in the DNS, this confirms that amazonses.com is recognized as a sender for example.com.
In addition, if a query for the A record of amazonses.com returns the aforementioned IP address xxx.xxx.xxx.xxx, it proves that the IP address xxx.xxx.xxx.xxx is appropriate as a sender for example.com.
$ dig example.com TXT
(omitted)
;; ANSWER SECTION:
example.com. 86400 IN TXT "v=spf1 include:amazonses.com ~all"
(omitted)
The following site can be used to check the operational status from outside the system.
Detailed specifications can be found here.
DKIM
While SPF verification allows checking the validity of the sender domain, this information refers to values set in the email header, assuming the email has not been tampered with by an MTA or similar during transit.
It is necessary to detect if the header From in the diagram below has been illegally rewritten.
Quoted from https://sendgrid.kke.co.jp/blog/?p=10121
This is achieved using a technology called DKIM (DomainKeys Identified Mail).
DKIM adds a digital signature to the email using public-key cryptography. By registering the public key in the DNS in the format selector._domainkey.example.com, it enables the destination MTA to detect tampering.
In Amazon SES, you can set up DKIM records using the following steps.
It is easy to use as it managedly handles the creation and management of digital signatures by simply operating from the console.
The flow for the destination MTA to verify the DKIM record is as follows:
Check the header information of an email received in Gmail or elsewhere, and refer to the part written as s=hogehoge; in the DKIM-Signature header.
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=hogehoge; d=example.com; (omitted)
When you query the TXT record using the dig command, the value returned as p=xxxxxx/yyyyyy/zzzzzz is used as the public key.
$ dig hogehoge._domainkey.example.com TXT
(omitted)
;; ANSWER SECTION:
hogehoge._domainkey.example.com. 7192 IN CNAME hogehoge.dkim.amazonses.com.
hogehoge.dkim.amazonses.com. 3570 IN TXT "p=xxxxxx/yyyyyy/zzzzzz"
(omitted)
The following site can be used to check the operational status from outside the system.
Detailed specifications can be found here.
DMARC
While SPF and DKIM verify the source domain and email content, they do not specify how to handle emails that fail these verifications, leaving it up to the recipient. To address this, a mechanism called DMARC (Domain-based Message Authentication, Reporting and Conformance) was devised for legitimate senders to express how unauthorized emails sent by malicious third parties should be handled.
The content can be verified by querying the DNS for a TXT record with _dmarc prepended to the source domain.
$ dig _dmarc.example.com TXT
(omitted)
;; ANSWER SECTION:
_dmarc.example.com. 86400 IN TXT "v=DMARC1; p=quarantine; rua=mailto:system@example.com; ruf=mailto:system@example.com; rf=afrf; pct=100"
(omitted)
Details of each value can be found below. In the case of the sample above, it means "for all emails that fail authentication, move them to the spam folder and report the authentication failure to system@example.com."
The following site can be used to check the operational status from outside the system.
The configuration method for Amazon SES is as follows:
BIMI
BIMI (Brand Indicators for Message Identification) is a standard that allows companies to display icons as the email sender in mailers, enabling users to verify if the email was sent from an official source.
When authentication is performed, an icon is displayed as the email sender, and in Gmail, a mark indicating that it has been authenticated is shown.
Quoted from https://workspaceupdates.googleblog.com/2023/05/expanding-gmail-security-BIMI.html
Note that enabling BIMI requires DMARC settings, and DMARC settings require either DKIM or SPF settings, so they must be addressed in order.
Verification of BIMI settings can be confirmed here:
The configuration method in SES is detailed below:
Reference: SPF / DKIM / DMARC configuration example when using Amazon SES + Terraform
The following article summarizes examples of setting this up with Terraform, so please feel free to check it out 🍢
Bounce / Complaint
Even if an email is sent in a state that clears various authentication issues, it may not be delivered due to circumstances on the recipient's side. The following classifications exist:
Especially in the case of hard bounces or complaints, the sender should no longer send emails to that destination, so it is necessary to suppress email transmission to the target email address using some method.
- Bounce
- Soft Bounce
- Occurs for temporary reasons, such as the destination mailbox being temporarily over capacity or the server being down.
- Hard Bounce
- Occurs for permanent reasons, such as the destination email address not existing or being rejected by the server.
- Soft Bounce
- Complaint
- Occurs when a user marks an email as spam after it has been received.
In addition to bounces caused by deliverability issues to email addresses as described above, bounces can also be caused by spam filters such as SpamAssassin.
List-Unsubscribe Header
If a user marks a received email as spam, the mail server operator (e.g., Google for Gmail) will begin discarding the mail. This negatively impacts the email reputation, so service operators want to avoid this as much as possible.
Furthermore, from the mail server operator's perspective, handling a large volume of emails destined for disposal is undesirable. Consequently, an email unsubscription function from the user to the service operator is recommended in RFC8058.
For implementation, setting the List-Unsubscribe and List-Unsubscribe-Post headers in the email header as shown below will display an unsubscribe button in supported email services.
List-Unsubscribe: <http://example.com/unsubscribe?user_key=xyz&mail_key=dm>
List-Unsubscribe-Post: List-Unsubscribe=One-Click
When a user clicks the button, a web request is sent to the service operator's server, which should then deactivate the email transmission settings.
While developing a dedicated endpoint is required, emails where the unsubscribe button is clicked are not classified as complaints, so this is expected to be effective in preventing reputation decline.
Image of an email with the List-Unsubscribe header
Image of an email with the List-Unsubscribe header
Envelope-From / Return-Path
When a bounce or complaint occurs due to the various reasons mentioned earlier, it is necessary to notify the sender of the email.
The email address used for this purpose is the Envelope-From, which indicates the email's sending server.
This is distinguished because the address from which the MUA requested the email transmission and the address used when it is delivered by the MSA can differ.
While the source of the transmission request is called Header-From, the source address at the time of delivery is called Envelope-From.
In Amazon SES, the specification is such that an address containing a unique ID generated for each email, like xxxxxx-yyyyyy-zzzzzz@ap-northeast-1.amazonses.com, is specified as the Envelope-From. It is automatically specified, and AWS interprets the received content.
Detailed specifications for the email format are as follows:
When a bounce occurs, the receiving MTA appends the address recorded in the Envelope-From to the email as the Return-path header.
By checking the Return-Path of a bounced email at the sender's end, you can confirm which address the bounce email passed through from the receiving MTA.
Through this sequence of operations, the same address as the Envelope-From will be set in the Return-Path, but it is worth noting whether the sending or receiving side writes each piece of information.
Reputation
How often bounces and complaints occur for a certain domain is managed by each email delivery provider through a mechanism similar to credit management, called reputation.
Email senders are required to keep these occurrence rates low.
You can check the reputation of the domain you are using on sites such as those listed in the following article.
In Amazon SES, you can receive notifications via SNS when bounces or complaints occur for an email.
Additionally, statistical information such as bounce and complaint rates is calculated for each domain and can be checked by the user.
If the bounce or complaint rate exceeds a certain value, restrictions on the use of SES may be imposed.
Suppression List
As a measure when a bounce email occurs, it is common to register the target address in a suppression list built on the email sender's side.
Suppression lists in Amazon SES include account-level and global lists; details are below.
MAILER-DAEMON / Mail Delivery Subsystem
In the process of bounce emails being returned to the Return-Path, emails may be sent to the source email address from senders such as MAILER-DAEMON or Mail Delivery Subsystem. These are intended to notify the email sender that a bounce has occurred, and by checking the content, you can confirm the detailed reason for the bounce.
The detailed reasons for a bounce can be difficult to understand because they are described in formats uniquely extended by each MTA in addition to SMTP status codes. The following site is very detailed and serves as a good reference.
Conclusion
I'm quite exhausted, but I'm glad I learned a lot 🍨
If there are any errors or other content that you think should be introduced, please let me know!
Discussion
俺もAmazonのパーソナル・ドキュメントサービスを利用したくてDebian LinuxへMSAをセットアップした。
まる2日も掛かったのはメールサーバーの仕組みが解ってなかったため。
わかっていれば1時間で済む話だった。
このサービスはMSAを使ってKindle Paperwhiteに紐付いたメールアドレスへhtmlファイルが添付した空のメールを飛ばせばKindle本として読めるというサービス。Documentを読むには超役立つサービスなのだ。
ただ、imgタグのsrcにはbase64エンコードした画像データを記述しないといけないのが困った所。
使ったMSAはExim4で、MTAにはGmailを使った。ハマりポイントはGmailとのSMTP AUTHの際にGmailがApplication-Specific Passwordを要求してきたこと(2022/03現在)。
Exim Internet Mailer
Googleは2段階認証を採用したユーザーにはこれを発行してくれるのだ。
Googleにログインする際につかうPasswordがSMTP AUTHにも使えるのだと思い込んでたら足をすくわれた。
おまけに、下記のWikiの通りにやっても上手く行かず絶望しかけた。
Exim - Debian Wiki
Exim4はPostfixとは異なりCyrus SASL(Simple Authentication and Security Layer)も同時インストールせずとも単独でGmailとのSMTP AUTHが成功した。
Googleは自社製品、たとえばGMailアプリ以外からのSMTP AUTHをする際にはApplication-Specific Passwordを要求するようだ。試しにmacOSに標準で入ってるメールアプリでGmailを利用しようとすると同じくApplication-Specific Passwordを与えないとメールが読めなかった。
Application-Specific PasswordはGitHubではAccess Tokenと呼ばれている。