shapeio: Bandwidth Limiting for I/O in Go - fujiwara-ware advent calendar 2024 day 9

This article is the 9th day of the fujiwara-ware advent calendar 2024.

https://github.com/fujiwara/shapeio

What is shapeio

shapeio is a library for wrapping io.Reader/io.Writer in Go to apply bandwidth limiting.

Why I made it

stretcher, which I introduced yesterday, is a pull-type deployment tool. Since the deployment destination servers retrieve the deployment themselves, the I/O bandwidth of the source network storage server can become a bottleneck. By applying bandwidth limiting on the stretcher side that retrieves the files, you can adjust the load on the deployment source.

shapeio was created for this purpose.

Usage

Usage is simple. Wrap io.Reader/io.Writer and set the bandwidth limit with SetRateLimit.

import "github.com/fujiwara/shapeio"

func ExampleReader() {
	// example for downloading http body with rate limit.
	resp, _ := http.Get("http://example.com")
	defer resp.Body.Close()

	reader := shapeio.NewReader(resp.Body)
	reader.SetRateLimit(1024 * 10) // 10KB/sec
	io.Copy(ioutil.Discard, reader)
}

func ExampleWriter() {
	// example for writing file with rate limit.
	src := bytes.NewReader(bytes.Repeat([]byte{0}, 32*1024)) // 32KB
	f, _ := os.Create("/tmp/foo")
	writer := shapeio.NewWriter(f)
	writer.SetRateLimit(1024 * 10) // 10KB/sec
	io.Copy(writer, src)
	f.Close()
}

Implementation

shapeio's implementation uses the x/rate/limit package.

x/rate/limit is a package for rate limiting that implements the token bucket algorithm.

Please refer to the following material for how it is used.

https://speakerdeck.com/fujiwara3/rate

Summary

shapeio is a library that wraps io.Reader/io.Writer to apply bandwidth limits.

In Go, it is common to implement I/O processing for streaming, not limited to networking, using the interfaces of the io package. Since shapeio satisfies those interfaces, it can be used for bandwidth limiting in general I/O processing, not just for networking.

The first version of shapeio was created eight years ago, but because Go's backward compatibility has been maintained, it still works without any issues with the latest version of Go. It's great that the language itself has such high compatibility.

Stay tuned for tomorrow!