From Bash to Go: A Journey in Self-Documenting Development Environment

＼スニダンを開発しているSODA inc.の Advent Calendar 2024 7日目の記事です!!!／

Ever stared at a years-old bash script and wondered what ancient developer secrets it holds? We've all been there. This is the story of how we transformed our development environment setup from a collection of mysterious bash incantations into a self-documenting, maintainable Go solution.

The Challenge: Taming the Development Environment

Our initial setup process was the stuff of developer nightmares: bash scripts that would occasionally fail silently, documentation scattered across various Notion pages, and a setup process that left failed states with all the clarity of a kernel panic message. New engineers joining the team had to piece together environment setup instructions like digital archaeologists.

Core Objectives

We set out to achieve several key improvements:

1. Streamline the environment setup process for new engineers
2. Standardize our scripting approach across the repository
3. Reduce external dependencies while maintaining production parity
4. Implement the principle of code-as-documentation

The Solution: Go All In with Go

The decision to migrate our setup scripts to Go might seem counterintuitive at first - why replace simple bash scripts with a compiled language? As it turns out, Go offered some unexpected advantages that made it the perfect fit for our use case.

From Bash Chaos to Go Control

Here's a prime example of how Go's standard library came to our rescue. Previously, checking database connectivity required external MySQL binaries. With Go, we got it down to this elegant solution:

func checkMySQLConnection(user, password, host, port, dbname string) (*sql.DB, error) {
    dsn := fmt.Sprintf("%s:%s@tcp(%s:%s)/%s", user, password, host, port, dbname)
    db, err := sql.Open("mysql", dsn)
    if err != nil {
        return nil, err
    }
    defer db.Close()

    for {
        err = db.Ping()
        if err == nil {
            fmt.Printf("Successfully connected to %s database.\n", dbname)
            return db, nil
        }
        fmt.Printf("Unable to connect to %s database. Retrying in 5 seconds...\n", dbname)
        time.Sleep(5 * time.Second)
    }
}

No more shell out to mysql -ping - just pure, dependency-free database connectivity checks.

Documentation that Lives and Breathes

Instead of maintaining separate documentation that would inevitably become outdated, we let the code speak for itself. Our Go scripts now serve as both implementation and documentation. Here's the run-down of the main function and what functions it calls. Error handling is omitted for brevity.

func main() {
    err := checkFiles() // check dependencies are present
    err = startDocker() // start our docker containers

    // wait until we can connect to the db and everything is ready to go
    _, err = checkMySQLConnection("root", "db", "127.0.0.1", "13306", "db")

    prompt := "Do you want to rebuild & seed the database with data?\n" +
        "Warning this will destroy current data. (y/n): "
    var input string
    fmt.Print(prompt)
    fmt.Scanln(&input)

    if input == "y" {
        err = remakeDatabase() // drop the whole database and remake it
        err = migrate()        // run migrations
        err = seedData()       // fixtures and seed data
        err = runBatch()       // run the initial batch tasks
    }
}

The function names and flow tell you exactly what's happening - no need to consult external docs or decipher cryptic bash variables.

Standardizing Script Development

With our setup script success, we expanded our Go-first approach to other repository scripts. Since our codebase is primarily Go, this ensures that engineers maintaining these scripts are already familiar with the language.

A Place for Everything: The scripts/ Directory

We established a clear organizational pattern where each script lives in its own directory under scripts/, following a consistent structure:

scripts/
├── country-ip-gen/
│   ├── Dockerfile
│   ├── Makefile
│   ├── main.go
│   └── README.md
├── email-blacklist/
│   ├── Dockerfile
│   ├── Makefile
│   ├── main.go
│   └── README.md
└── [other-script]/
    ├── Dockerfile
    ├── Makefile
    ├── main.go
    └── README.md

This standardized structure means engineers always know where to find critical information about any script in our repository.

Docker as the Great Equalizer

Each script directory contains its own Dockerfile, ensuring consistent execution environments. Even a barebones Dockerfile helps using it with a Makefile:

FROM golang:1.22
WORKDIR /app
COPY . /app
RUN go mod download
CMD ["go", "run", "main.go"]

Paired with a standardized Makefile:

build-run: build run

build:
    docker build . -t country-ip-gen

run:
    docker run -v $(PWD):/app --network network_default -it --rm country-ip-gen

Self-Documenting Scripts through README

Each script directory includes a README.md that follows a consistent pattern:

Country IP Blocks Generator

This script processes IPv4 address block files for various countries and generates 
a Go file containing a map of CIDR blocks to country codes for use in blocking 
or allowing traffic based on IP addresses.
It is used as part of the build process to keep an up-to-date record of countries 
and their IPs.

Files
* `/app/country-ip-blocks/ipv4`: Directory containing files with IPv4 address blocks. 
  Each file should be named after the country code it represents 
  (e.g., `us.txt` for the United States).
* `/app/country_ip_gen.go`: The generated Go file containing the map of CIDR 
  blocks to country codes.

Usage
* clone `https://github.com/herrbischoff/country-ip-blocks` in this directory.
* Run the script to generate the Go file
* The output is used in `go/pkg/countryip/`

This README structure provides:

1. A clear description of the script's purpose
2. Details about required files and their locations
3. Step-by-step usage instructions
4. Information about where the script's output is used

The Power of Standardization

This standardized approach delivers several benefits:

1. Predictable Structure: Engineers know exactly where to find documentation, code, and build instructions for any script.

2. Isolated Environments: Each script runs in its own Docker container, preventing dependency conflicts and ensuring consistent execution.

3. Self-Contained Units: All necessary files, including documentation and build instructions, live alongside the code they describe.

4. Build Process Integration: Scripts can be easily integrated into CI/CD pipelines since they all follow the same pattern.

The Results: A More Maintainable Future

Our migration to Go-based scripts has delivered several key benefits:

1. Self-Documenting Infrastructure: No more out-of-sync documentation. The code itself serves as living documentation that must be updated as requirements change.
2. Reliable Error Handling: Go's error handling ensures that when things go wrong (and they will), we fail gracefully with clear error messages.
3. Dependency Management: By leveraging Go's standard library and Docker, we've minimized external dependencies while maintaining a consistent environment.
4. Simplified Maintenance: With everything in one language and following similar patterns, maintaining and updating scripts has become significantly easier.

Lessons Learned

Documentation Belongs in Code: External documentation is a liability. When possible, make your code self-documenting and keep everything in the repository.
Embrace Standard Tools: Using the primary language of your codebase for scripts ensures maintainability and reduces cognitive overhead.
Containerize Everything: Docker provides consistency and eliminates "works on my machine" scenarios.

Looking Forward

This transformation has set a new standard for how we approach development tooling. By treating our scripts with the same care and attention as our production code, we've created a more robust and maintainable development environment.
The next time you're tempted to write a quick bash script, remember: your future self (and your teammates) will thank you for taking the time to do it right.