Pitfalls of Sakura Cloud's Packet Filter

To everyone running servers on Sakura Cloud, have you ever fallen into an unexpected pitfall with network settings?

Today, I’d like to introduce a slightly unusual case I encountered with Sakura Cloud's packet filter settings.

In particular, you might face similar issues when migrating an old Linux environment to a containerized one.

What Happened?

I had built a virtual machine (VM) on Sakura Cloud and was running an old Linux environment containerized using incus (formerly LXD) on top of it.

I configured the Sakura Cloud packet filter and should have allowed all necessary traffic, but something felt off.

While I could connect to the host VM via SSH without any issues, trying to SSH directly into the container resulted in an unstable state where I could connect sometimes but not others.

Investigation: What tcpdump Told Me

To solve this mysterious phenomenon, I first investigated the communication status using the classic network debugging tool tcpdump. I ran tcpdump on both the host VM and inside the container to trace the packet flow during SSH connection attempts in detail.

This revealed a strange behavior. Connection requests from the SSH client were reaching the container, and the response from the container was also being sent.

However, it appeared that the response packets were somehow not leaving the host VM from the container. On the SSH client side, the connection would time out and drop.

Cause Identified: /proc/sys/net/ipv4/ip_local_port_range

To investigate why packets were not leaving the container, I dug deeper. While checking network-related configuration files and kernel parameters one by one, I finally arrived at the heart of the matter.

It was a kernel parameter within the container called /proc/sys/net/ipv4/ip_local_port_range. Upon checking this value, I found it was set to 1024 65000.

What is `/proc/sys/net/ipv4/ip_local_port_range`?

In TCP/IP communication, this is a kernel parameter that defines the range of "ephemeral ports" (temporary ports) used when a client connects to an external service. Normally, a port within this range is randomly selected and used as the source port.

As stated in the official documentation, Sakura Cloud's packet filter is basically stateless. This means it strictly checks the pair of source and destination ports; therefore, the source port of the return packet must be the same as the destination port of the initial connection or within a specific range.

The reason for the pitfall this time was that the range assumed by the Sakura Cloud packet filter as "ports for return packets" differed from the ip_local_port_range, causing return packets to be blocked by the packet filter.

Although I knew that Sakura Cloud's packet filter operates statelessly, since I was using the presets provided by Sakura Cloud, it didn't occur to me at first that the return packet port range might fall outside the range specified in the packet filter.

This served as a good lesson that in old Linux environments, the default ip_local_port_range may not match modern network environments or packet filter specifications.

Summary: Sakura Cloud's Packet Filter and Stateless Operation

What I learned from this incident is that Sakura Cloud's packet filter operates statelessly, as stated in the official documentation. When you encounter situations where connections are unstable like in this case, it might be a good idea to check the port range specifications for return packets as well.

Side Note

There might be performance issues and such, but it would be nice if the standard packet filter were provided with stateful operation so we wouldn't have to worry about things like this.