Proactively Use strings.Replacer

Introduction

To everyone who participated in the Go Advent Calendar 2024, thank you for your hard work. I hope you had a great Go life this year. I hope we have good "Go" connections next year as well.

Today, I'm going to talk about strings.Replacer.

Are you doing keyword replacement?

While browsing GitHub, I often come across code like this:

s = strings.ReplaceAll(s, "hoge", "moge")
s = strings.ReplaceAll(s, "foo", "bar")
s = strings.ReplaceAll(s, "fizz", "buzz")

This code replaces multiple specific keywords, but it is actually very inefficient. strings.ReplaceAll works by traversing the string in the first argument and replacing matches of the second argument with the third argument. In other words, these processes are executed repeatedly every time strings.ReplaceAll is called.

That's where strings.Replacer comes in

Have you ever used strings.Replacer? By using this, you can complete replacement processes that would otherwise be executed multiple times in a single pass.

rep := strings.NewReplacer(
    "hoge", "moge",
    "foo", "bar",
    "fizz", "buzz",
)
s = rep.Replace(s)

Comparison between strings.ReplaceAll and strings.Replacer

package replacer_test

import (
	"strings"
	"testing"
)

type Replacer []struct {
	from string
	to   string
}

func (r Replacer) Replace(s string) string {
	for _, ss := range r {
		s = strings.ReplaceAll(s, ss.from, ss.to)
	}
	return s
}

func BenchmarkReplacer1(b *testing.B) {
	rep := Replacer{
		{"foo", "bar"},
		{"hello", "world"},
		{"甲", "乙"},
	}
	s := strings.Repeat("このfooはhello-wideでfoo甲です。", 5000)

	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		_ = rep.Replace(s)
	}
}

func BenchmarkReplacer2(b *testing.B) {
	rep := strings.NewReplacer(
		"foo", "bar",
		"hello", "world",
		"甲", "乙",
	)
	s := strings.Repeat("このfooはhello-wideでfoo甲です。", 5000)

	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		_ = rep.Replace(s)
	}
}

Actually, at this level of benchmarking, there isn't much of a difference.

BenchmarkReplacer1-12               1173           1031753 ns/op          614401 B/op          3 allocs/op
BenchmarkReplacer2-12               1143            931853 ns/op          409624 B/op          3 allocs/op

Where the performance difference occurs depends on the number of replacements. For example, what happens if we increase the number of replacement candidates like this?

	rep := strings.NewReplacer(
		"foo", "bar",
		"hello", "world",
		"甲", "乙",
		"fizz", "buzz",
		"いちご", "strawberry",
		"りんご", "apple",
		"ばなな", "banana",
	)

cpu: AMD Ryzen 5 5600U with Radeon Graphics
BenchmarkReplacer1-12                702           1713491 ns/op          802819 B/op          4 allocs/op
BenchmarkReplacer2-12                757           1463078 ns/op          393244 B/op          3 allocs/op
PASS

A clear difference has started to emerge. If you are replacing 10 or 20 strings using something like a dictionary, it is better to use strings.Replacer. Of course, from the perspective of memory usage, it is more efficient to use it even when the number of replacements is small.

Is it okay not to use it if there are few replacement candidates?

strings.Replacer performs a single scan of the string. Therefore, the behavior differs in cases like the following:

• foo → bar
• bar → foo

In the case of the manual implementation mentioned earlier, the replacement target string "foo" would end up back as "foo" as a result. In many cases, this result is likely unintended. However, strings.Replacer keeps it as "bar".
If you want to avoid unintended behavior, it is better to use strings.Replacer. Also, in most cases, the replacement candidates are fixed. Once you create a Replacer with strings.NewReplacer, you can reuse it.

Replacer.WriteString is also convenient

Replacer also provides a method called WriteString. While the benchmark difference may not be significant if you are eventually storing the result in a string variable, a large difference occurs in cases where you output the result to a file or similar.

func BenchmarkWriter1(b *testing.B) {
	rep := Replacer{
		{"foo", "bar"},
		{"hello", "world"},
		{"甲", "乙"},
		{"fizz", "buzz"},
	}
	s := strings.Repeat("このfooはhello-wideでfoo甲です。", 5000)

	var buf bytes.Buffer
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		buf.WriteString(rep.Replace(s))
	}
}

func BenchmarkWriter2(b *testing.B) {
	rep := strings.NewReplacer(
		"foo", "bar",
		"hello", "world",
		"甲", "乙",
		"fizz", "buzz",
	)
	s := strings.Repeat("このfooはhello-wideでfoo甲です。", 5000)

	var buf bytes.Buffer
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		rep.WriteString(&buf, s)
	}
}

BenchmarkWriter1-12                  596           2150994 ns/op         1449942 B/op          4 allocs/op
BenchmarkWriter2-12                  674           1591118 ns/op          398274 B/op          0 allocs/op
PASS

This time, even with few replacement candidates, not only memory usage but also performance has changed. For example, in cases where you want to return the result of string replacement as an HTTP response, I think it's a good idea to actively use strings.Replacer.WriteString.

Summary

I compared strings.ReplaceAll and strings.Replacer.Replace. In cases with few replacement candidates, strings.Replacer was more advantageous in terms of memory usage, while the execution speed remained the same. Conversely, as the number of replacement candidates increases, strings.Replacer.Replace becomes more efficient in terms of both memory usage and execution speed.
In cases where multiple string replacements are performed, I think it is better to actively use strings.Replacer.Replace. Furthermore, in cases where you write the result of string replacements to a file or network communication, you can expect significant performance improvements by using strings.Replacer.WriteString.
Let's make active use of it!