iTranslated by AI
Building a mitmproxy-like tool in Rust: Lessons learned
Introduction
I released a library that allows you to build something like mitmproxy in Rust, so I'm writing about what I learned during the process. Here is the repository.
What is mitmproxy?
mitmproxy is an HTTP proxy server characterized by its ability to view the contents of HTTPS traffic.
HTTP Proxy
An HTTP proxy is a proxy that operates at the HTTP level. Therefore, the proxy itself must also understand HTTP.
HTTP Communication (Without Proxy)
printf "GET / HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n" | nc example.com 80
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 360610
Cache-Control: max-age=604800
Content-Type: text/html; charset=UTF-8
Date: Mon, 18 Nov 2024 06:44:56 GMT
Etag: "3147526947+gzip"
Expires: Mon, 25 Nov 2024 06:44:56 GMT
Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT
Server: ECAcc (sac/256D)
Vary: Accept-Encoding
X-Cache: HIT
Content-Length: 1256
Connection: close
... # Following Body (omitted)
HTTP Communication (Using Proxy)
printf "GET http://example.com/ HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n" | nc localhost 3003 # Running an HTTP proxy on http://localhost:3003
HTTP/1.1 200 OK
... # Omitted as it is the same
In this way, by sending GET http://example.com/ HTTP/1.1 instead of the usual GET / HTTP/1.1, you can tell the proxy where you want to access.
Since the proxy communicates with the destination and returns the result, you can view or edit the content on the proxy side.
HTTPS Communication (Without Proxy)
printf "GET / HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n" | ncat --ssl example.com 443
... # Omitted as it is the same
HTTPS Communication (Using Proxy)
Since this is special, let's try it with curl.
curl -v https://example.com -x http://localhost:3003
* Trying 127.0.0.1:3003...
* Connected to (nil) (127.0.0.1) port 3003 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to example.com:443
> CONNECT example.com:443 HTTP/1.1
> Host: example.com:443
> User-Agent: curl/7.81.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< Date: Mon, 18 Nov 2024 10:17:21 GMT
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=Los Angeles; O=InternetCorporationforAssignedNamesandNumbers; CN=www.example.org
* start date: Jan 30 00:00:00 2024 GMT
* expire date: Mar 1 23:59:59 2025 GMT
* subjectAltName: host "example.com" matched cert's "example.com"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x559036dbdeb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: example.com
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< age: 533083
< cache-control: max-age=604800
< content-type: text/html; charset=UTF-8
< date: Mon, 18 Nov 2024 10:17:23 GMT
< etag: "3147526947+gzip+ident"
< expires: Mon, 25 Nov 2024 10:17:23 GMT
< last-modified: Thu, 17 Oct 2019 07:18:26 GMT
< server: ECAcc (sac/251E)
< vary: Accept-Encoding
< x-cache: HIT
< content-length: 1256
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
... # Following Body omitted
* Connection #0 to host (nil) left intact
By first sending CONNECT example.com:443 HTTP/1.1 to the proxy, you request the proxy to communicate with example.com:443.
After that, tunneling occurs at the TCP level with example.com:443, from which normal HTTPS communication proceeds. (In this example, it communicates with example.com via HTTP/2).
Since the communication after tunneling is encrypted, it is impossible to view the content on the proxy side, and any edits will be detected.
How mitmproxy Works
As mentioned above, a normal proxy cannot see HTTPS traffic. However, mitmproxy can view all data sent by the client by having the client trust its own certificate as a root certificate beforehand.
The certificate for each destination domain is signed by mitmproxy on the fly.
From this point, the proxy can either act as a real proxy for example.com or simply create and return an arbitrary response.
HTTP Upgrade
HTTP/1.1 communication can be upgraded to another protocol in the middle. Proxies need to take this into account.
I suspect browsers can only upgrade to WebSocket (though I couldn't find a source for this).
Implementing in Rust
The assumption is that the proxy will primarily be used by a browser.
I designed it as a library, assuming that users will implement the proxy logic themselves.
Libraries Used and Impressions
hyper
An HTTP library. It's very good that it's low-level in a nice way. The interface is highly refined. Even if it feels a bit difficult to get into at first, it ends up making more sense in the long run.
Using hyper::body::Incoming makes it easy because it supports everything including:
- Body with length specified by Content-Length
- Body with Transfer-Encoding: chunked
- Server-Sent Events
- Whatever else might be there
It supports up to HTTP/2, but HTTP/3 is not yet supported.
rustls
A TLS library written in Rust. Since algorithms with security issues are not supported, there are some sites that can be accessed from a browser but not from rustls. It's fine for use on the server side.
native-tls
A wrapper that uses the TLS library available in the OS.
rcgen
A library for creating certificates. I use this to create certificates on the fly for the domains the client wants to communicate with.
Interface
After various trials, I decided to use hyper::service::HttpService to abstract the proxy.
As a side note, hyper's Service is not directly compatible with tower's Service: https://github.com/hyperium/hyper/blob/0bd4adfed2bf40dda6867260ded05542d5b921c5/docs/ROADMAP.md?plain=1#L359-L368.
However, hyper-util provides hyper_util::service::TowerToHyperService.
Users can implement the proxy logic by implementing HttpService, which receives a http::request::Request (usually the URI part only contains the path onwards, but here it includes everything like the scheme and host) and returns a http::response::Response.
let client = DefaultClient::new().unwrap();
let server = proxy
.bind(
("127.0.0.1", 3003),
service_fn(move |req| {
let client = client.clone();
async move {
let uri = req.uri().clone();
// You can modify request here
// or You can just return response anywhere
let (res, _upgrade) = client.send_request(req).await?;
println!("{} -> {}", uri, res.status());
// You can modify response here
Ok::<_, http_mitm_proxy::default_client::Error>(res)
}
}),
)
.await
.unwrap();
RFC 9110 contains provisions on how proxies should or should not behave, but these are intended to be left to the library users.
Considerations
Things I noticed when I actually set up the created tool as a Chrome proxy and tested it.
Using https://chromewebstore.google.com/detail/proxy-switchyomega/padekgcemlokbadohgkifijomclgjgif makes configuration convenient.
HTTP/2
Since the browser cannot determine whether to use HTTP/1 or HTTP/2 based on the URL alone, it uses a TLS feature called ALPN to decide whether to communicate with the server via HTTP/2.
Browsers never use HTTP/2 with http (without s).
Host Header Prohibition
For example, when a proxy receives an HTTP/1.1 request and relays it to the destination using HTTP/2, passing the HTTP/1.1 headers directly to HTTP/2 can lead to issues.
The Host header is mandatory in HTTP/1.1, but it is "Must Not" in HTTP/2, so it must be removed.
Actually, sending a Host header to https://www.google.com/ via HTTP/2 results in a rejection.
HTTP Version Matching
For example, https://echo.websocket.org/ upgrades to WebSocket when connected via HTTP/1.1, but returns a normal body when connected via HTTP/2 (since there is no WebSocket in HTTP/2, this is expected).
Therefore, if you want to relay as-is with mitmproxy, you need to relay using HTTP/1.1 when the client connects via HTTP/1.1, and relay using HTTP/2 when the client connects via HTTP/2.
TCP_NODELAY
There were some servers where communication would not work properly unless TCP_NODELAY was set on the TCP connection. I'm not entirely sure why.
Discussion