iTranslated by AI
Connecting to PostgreSQL with the workerd Socket API
It looks like the Socket API for workerd will be enabled starting from the next compatibility_date, so I checked the current status.
Previous article
Specification
First, let's check the interface.
Usage
The design involves reading from and writing to a Socket object created via connect() through reader/writer instances using streaming.
It's clear that it's a custom specification while keeping the draft Direct Sockets API in mind.
Are they considering standardization with Deno or WinterCG?
Since it's a Web API (specification for browsers), there are differences from Node.js's net.Socket.
const net = require('node:net');
const client = net.createConnection({ port: 8124 }, () => {
// 'connect' listener.
console.log('connected to server!');
client.write('world!\r\n');
});
client.on('data', (data) => {
console.log(data.toString());
client.end();
});
client.on('end', () => {
console.log('disconnected from server');
});
In other words, I think net.Socket needs to be provided even in Node.js compatibility for Node.js-compatible ORMs to function.
I thought the strategy was being discussed in discussions/198, but it seems they are rapidly implementing it internally with a sense of speed.
Environment Setup
Set up an environment where you can test the Socket API.
Currently, even if you enable the Socket API in Cloudflare Workers, you cannot deploy it to the production environment[1], but you can test it by running workerd locally.
There is a pre-compiled version of workerd available on npm, making it easy to try out via npx workerd.
However, for the latest version 1.20230419.0, you need to enable compatibility_flags.
npx wrangler generate cfw-tcp
cd cfw-tcp
compatibility_flags = ["tcp_sockets_support"]
npx wrangler dev --experimental-local
(Result of executing gopher.js)
Using the --experimental-local flag utilizes the Miniflare 3 series, which calls workerd internally, enabling tcp_sockets_support.
Tips: Compiling workerd yourself
If you want to try out features from versions newer than those published on npm, you will need to build it yourself using Bazel. While using Bazel might make it feel like a massive codebase, workerd is a relatively simple C++ codebase of around 100,000 lines with few dependencies. Generally, if you follow the README, it should be easier than you might expect.
Once built, the workerd command will be created at the following path, allowing you to start the Worker by passing a config as an argument.
./bazel-bin/src/workerd/server/workerd serve samples/hello-wasm/config.capnp
curl http://localhost:8080
HTTP Connection
Last time I used Gopher, so this time I'll try using HTTP. I was able to retrieve a response from neverssl.com:80 as shown below.
This was straightforward to implement.
import { connect } from 'cloudflare:sockets';
export default {
async fetch(req, env) {
const socket = connect({
hostname: 'neverssl.com',
port: 80
});
const writer = socket.writable.getWriter()
const encoder = new TextEncoder();
const encoded = encoder.encode("GET / HTTP/1.1\r\nHost: neverssl.com\r\n\r\n");
await writer.write(encoded);
const reader = socket.readable.getReader();
const decoder = new TextDecoder();
let response = "";
while (true) {
const res = await reader.read();
if (res.done) {
console.log("Stream done, socket connection has been closed.");
break;
}
response += decoder.decode(res.value);
}
return new Response(response);
}
}
❯ curl http://localhost:8080
HTTP/1.1 200 OK
Date: Tue, 09 May 2023 15:51:15 GMT
Server: Apache/2.4.56 ()
Upgrade: h2,h2c
Connection: Upgrade
Last-Modified: Wed, 29 Jun 2022 00:23:33 GMT
ETag: "f79-5e28b29d38e93"
Accept-Ranges: bytes
Content-Length: 3961
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
<html>
<head>
<title>NeverSSL - Connecting ... </title>
PostgreSQL Connection
Since HTTP connections can be replaced by fetch, I will try connecting to PostgreSQL as something that can only be done with a socket connection.
As shown in "How Vercel Postgres enables ORMs and connection pools on Edge Runtime", if TCP connection is possible, it is also possible to use Prisma with native drivers.
This time, I decided to use and modify a template called Template: worker-postgres which is available via wrangler generate.
npx wrangler generate postgres worker-postgres
This project has the following structure:
src/
├── deno
│ ├── LICENSE
│ ├── README.md
│ ├── buffer.d.ts
│ ├── buffer.js
│ ├── deferred.d.ts
│ ├── deferred.js
│ └── workers-override.ts
├── driver
│ └── postgres
│ ├── 62edfb469c0dbacd90273cf9a0d7a478.wasm
│ ├── LICENSE
│ ├── README.md
│ ├── edgeworkerizer.py
│ ├── index.d.ts
│ ├── index.js
│ └── postgres.js.deno
└── index.ts
A key feature is that it attempts to somehow convert and run deno-postgres.
The overall build flow is as follows:
- edgeworkerizer.py: Converts the Client class for Deno into JS for Workers
a. postgres.js.deno to index.js - edgeworkerizer.py: Embeds calls to 62edfb469c0dbacd90273cf9a0d7a478.wasm into index.js
- Overwrites the Deno code called within the Client class with the contents of workers-override.ts
- Outputs the above processes to dist/index.mjs using esbuild
cd src/driver/postgres/
python3 edgeworkerizer.py postgres.js.deno > index.js
cd -
esbuild --watch --bundle --sourcemap --outfile=dist/index.mjs --minify --format=esm ./src/index.js --external:*.wasm --external:cloudflare:* --inject:./src/deno/workers-override.ts
The entry point index.ts was written as follows:
import { Client } from './driver/postgres';
export default {
async fetch(request, env) {
try {
const client = new Client({
user: 'postgres',
database: 'postgres',
hostname: 'POSTGRES_HOST',
password: 'POSTGRES_PASSWORD',
port: '5432',
tls: {enabled: false}
});
await client.connect();
const result = await client.queryObject`select * from Event;`;
return new Response(JSON.stringify(result.rows));
} catch (err) {
return new Response((err as Error).message);
}
},
}
I initially tried localhost for POSTGRES_HOST, but since even establishing a connection didn't work, I used a Supabase remote host. This allowed the connection to be established (as described later).
The configuration is as follows. In addition to the esbuild output path, the path to the referenced WASM file also needs to be added.
using Workerd = import "/workerd/workerd.capnp";
const postgresExample :Workerd.Config = (
services = [
(name = "main", worker = .postgresWorker)
],
compatibilityDate = "2023-02-28",
);
const postgresWorker :Workerd.Worker = (
modules = [
(name = "worker", esModule = embed "dist/index.mjs"),
(name = "./62edfb469c0dbacd90273cf9a0d7a478.wasm", wasm = embed "dist/62edfb469c0dbacd90273cf9a0d7a478.wasm" )
]
);
In the existing generated code, there is logic to upgrade from HTTP to WebSocket to connect to a Cloudflare Tunnel host.
I predicted that I might be able to make it work by rewriting this part with the workerd Socket API.
First, since there's a part in workers-override.ts where Deno.connect() is implemented, I'll delete it entirely and replace it with the following.
import { connect as cfConnect } from 'cloudflare:sockets';
export function connect(options: ConnectOptions): Promise<any> {
return new Promise<Conn>((resolve, reject) => {
const socket = cfConnect(options);
resolve(socket);
});
}
Then, I'll rewrite the following part in workers-override.ts, which is called by Deno.connect.
async #createNonTlsConnection(options) {
this.#conn = await Deno.connect(options);
- this.#bufWriter = new BufWriter(this.#conn);
+ this.#bufWriter = new BufWriter(this.#conn.writable.getWriter());
- this.#bufReader = new BufReader(this.#conn);
+ this.#bufReader = new BufReader(this.#conn.readable.getReader());
}
Let's test it.
❯ curl http://localhost:8080 -v
* Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 13
< Content-Type: text/plain;charset=UTF-8
<
* Connection #0 to host localhost left intact
negative read
It failed, outputting only "negative read".
This was an assertion defined within the Client. Simply ignoring it didn't solve the problem, and it seemed like it was being triggered because the reading of the response from the server was invalid in the first place.
Further debugging would require monitoring the actual communication with the server, so I'll retreat for now.
-
If you deploy, you'll get an error saying 'The compatibility flag tcp_sockets_support is experimental and cannot yet be used in Workers deployed to Cloudflare. [code: 10021]' ↩︎
Discussion