Zenn
👌

[connect][sqlc]gRPC API Development Tutorial

2024/07/12に公開
Go
gRPC
sqlc
Connect RPC
tech

gRPC API Development Tutorial (with Connect, gocraft/dbr, SQLC)

This tutorial explains the steps to develop a gRPC API in Go for a simple memo application. We will use technologies such as Connect, gocraft/dbr, and SQLC. We will clearly state the purpose and reason for each step and proceed according to best practices for each technology.

API Features

In this tutorial, we will create a gRPC API for a simple memo application with the following features:

  1. List Memos: Retrieve a list of memos for a specific user.
  2. Get Memo: Retrieve a specific memo corresponding to a user ID and memo ID.
  3. Create Memo: Create a new memo.

Technologies Used

  • Go: The programming language. It is simple, readable, and has strong concurrency features.
  • Protocol Buffers (protoc): The interface definition language for gRPC. Used for defining the API.
  • Connect: A library that supports both gRPC and HTTP APIs. Allows seamless switching between gRPC and HTTP APIs.
  • gocraft/dbr: A database access library. Allows writing SQL directly while maintaining type safety.
  • SQLC: A tool that generates Go code from SQL queries. Maintains consistency between SQL and Go.
  • PostgreSQL: A relational database. Open-source and widely used.
  • log/slog: A logging library. Allows structured logging.
  • direnv: An environment variable management tool. Allows switching environment variables per project.
  • dotenv: A library for loading environment variables from a file.
  • HTTP/2: A fast HTTP protocol. gRPC operates over HTTP/2.
  • h2c: A library for using HTTP/2 in cleartext.

Final Directory Structure

.
├── README.md                   # File for project description and usage instructions
├── api
│   └── proto
│       ├── memo.proto          # Protocol Buffers definition file
│       ├── memo.pb.go          # Automatically generated Go code
│       └── memo_grpc.pb.go     # Automatically generated gRPC server and client code
├── cmd
│   └── main.go                 # Main entry point. Starts the gRPC server.
├── go.mod                      # Go module configuration file
├── go.sum                      # File recording dependency versions
├── internal
│   ├── db
│   │   ├── db.go               # Database connection initialization
│   │   └── generated           # Code generated by SQLC
│   │       ├── db.go           # Database connection configuration
│   │       ├── models.go       # Database models
│   │       ├── querier.go      # SQL query execution functions
│   │       └── queries.sql.go  # SQL query definitions
│   └── server
│       └── memo_server.go      # gRPC server implementation
├── sqlc.yaml                   # SQLC configuration file
├── schema.sql                  # Database schema
├── queries.sql                 # SQL queries
├── .env                        # Environment variables file
├── .envrc                      # direnv configuration file
├── .gitignore                  # File specifying Git ignore patterns
├── Makefile                    # File defining build and test tasks
└── test_api.sh                 # API testing script

The roles of each directory and file are as follows:

  • README.md: File for project description and usage instructions.
  • api/proto: Contains Protocol Buffers definition files and generated Go code.
  • cmd: Contains main.go, the entry point of the application.
  • internal/db: Contains code related to database access.
  • internal/server: Contains the gRPC server implementation.
  • sqlc.yaml: SQLC configuration file.
  • schema.sql: File defining the database schema.
  • queries.sql: File defining SQL queries.
  • .env: File defining environment variables.
  • .envrc: direnv configuration file.
  • .gitignore: File specifying Git ignore patterns.
  • Makefile: File defining build and test tasks.
  • test_api.sh: Script for testing all API endpoints.

1. Project Setup

1.1 Create Project Directory

First, create and navigate to the project directory.

mkdir grpc-memo-api
cd grpc-memo-api

1.2 Initialize Go Module

Initialize a Go module. This makes dependency management easier.

go mod init grpc-memo-api

1.3 Create Necessary Directories

Create the project directory structure. This makes code organization easier.

mkdir -p api/proto
mkdir -p internal/server
mkdir -p internal/db/generated
mkdir -p cmd

1.4 Install Required Dependencies

Install the necessary Go packages.

go get github.com/bufbuild/connect-go
go get github.com/gocraft/dbr/v2
go get github.com/joho/godotenv
go get google.golang.org/protobuf
go get google.golang.org/grpc
go get golang.org/x/net/http2
go get golang.org/x/net/http2/h2c
go get golang.org/x/exp/slog

1.5 Create api/proto/memo.proto

Create the Protocol Buffers definition file. This defines the gRPC interface.

What is Protocol Buffers?

Protocol Buffers is a serialization format developed by Google. It is used to define the structure of data and efficiently serialize and deserialize that data. In gRPC, it is used to define the service interface.

Writing .proto Files

.proto files are used to define Protocol Buffers messages and services. They consist of the following elements:

  • syntax: Specifies the version of Protocol Buffers to use.
  • package: Specifies the package name.
  • option go_package: Specifies the package name when generating Go code.
  • message: Defines the structure of data.
  • service: Defines a gRPC service.
  • rpc: Defines a remote procedure call (RPC).

Write the following code in api/proto/memo.proto:

syntax = "proto3"; // Specify Protocol Buffers version

package memo; // Specify package name

option go_package = "grpc-memo-api/api/proto;proto"; // Specify package name for generated Go code

// MemoService service definition
service MemoService {
  rpc ListMemos (ListMemosRequest) returns (ListMemosResponse); // RPC to retrieve a list of memos
  rpc GetMemo (GetMemoRequest) returns (GetMemoResponse); // RPC to retrieve a specific memo
  rpc CreateMemo (CreateMemoRequest) returns (CreateMemoResponse); // RPC to create a new memo
}

// ListMemosRequest message definition
message ListMemosRequest {
  string user_id = 1; // User ID
}

// ListMemosResponse message definition
message ListMemosResponse {
  repeated Memo memos = 1; // List of memos
}

// GetMemoRequest message definition
message GetMemoRequest {
  string user_id = 1; // User ID
  string memo_id = 2; // Memo ID
}

// GetMemoResponse message definition
message GetMemoResponse {
  Memo memo = 1; // Memo
}

// CreateMemoRequest message definition
message CreateMemoRequest {
  string user_id = 1; // User ID
  string content = 2; // Memo content
}

// CreateMemoResponse message definition
message CreateMemoResponse {
  Memo memo = 1; // Created memo
}

// Memo message definition
message Memo {
  string id = 1; // Memo ID
  string user_id = 2; // User ID
  string content = 3; // Memo content
  string created_at = 4; // Creation timestamp
}

Tag Number Rules

Each field is assigned a tag number. Tag numbers start from 1 and are used as field identifiers. Be careful not to change tag numbers once assigned. The valid range for tag numbers is from 1 to 2^29-1.

1.6 Compile Protocol Buffers

Run the following command to generate Go code. This automatically generates the gRPC server and client code.

protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative api/proto/memo.proto

This command generates the following files:

api/proto/
├── memo.proto
├── memo.pb.go
└── memo_grpc.pb.go

2. Server Implementation

2.1 Create internal/server/memo_server.go

Implement the gRPC server. This defines the logic for handling requests from the client.

package server

import (
  "context"
  "grpc-memo-api/api/proto"
  "grpc-memo-api/internal/db/generated"
  "time"

  "github.com/bufbuild/connect-go"
  "github.com/google/uuid"
  "golang.org/x/exp/slog"
)

// MemoServer is the implementation of MemoService.
type MemoServer struct {
  DB *generated.Queries // Field for executing database queries
}

// ListMemos lists memos for the specified user.
func (s *MemoServer) ListMemos(ctx context.Context, req *connect.Request[proto.ListMemosRequest]) (*connect.Response[proto.ListMemosResponse], error) {
  // Retrieve memos from the database
  memos, err := s.DB.ListMemos(ctx, req.Msg.UserId)
  if (err != nil) {
    slog.Error("Failed to list memos", "error", err)
    return nil, err
  }

  // Convert to protocol buffer memo format
  var protoMemos []*proto.Memo
  for _, memo := range memos {
    protoMemos = append(protoMemos, &proto.Memo{
      Id:        memo.ID.String(),
      UserId:    memo.UserID,
      Content:   memo.Content,
      CreatedAt: memo.CreatedAt.Format(time.RFC3339),
    })
  }

  // Return the response
  return connect.NewResponse(&proto.ListMemosResponse{Memos: protoMemos}), nil
}

// GetMemo retrieves a memo corresponding to the specified user and memo ID.
func (s *MemoServer) GetMemo(ctx context.Context, req *connect.Request[proto.GetMemoRequest]) (*connect.Response[proto.GetMemoResponse], error) {
  // Retrieve the memo from the database
  memo, err := s.DB.GetMemo(ctx, generated.GetMemoParams{
    UserID: req.Msg.UserId,
    ID:     uuid.MustParse(req.Msg.MemoId),
  })
  if (err != nil) {
    slog.Error("Failed to get memo", "error", err)
    return nil, err
  }

  // Return the response
  return connect.NewResponse(&proto.GetMemoResponse{Memo: &proto.Memo{
    Id:        memo.ID.String(),
    UserId:    memo.UserID,
    Content:   memo.Content,
    CreatedAt: memo.CreatedAt.Format(time.RFC3339),
  }}), nil
}

// CreateMemo creates a new memo.
func (s *MemoServer) CreateMemo(ctx context.Context, req *connect.Request[proto.CreateMemoRequest]) (*connect.Response[proto.CreateMemoResponse], error) {
  // Generate a new memo ID and creation timestamp
  memoID := uuid.New()
  createdAt := time.Now()

  // Insert the memo into the database
  params := generated.CreateMemoParams{
    ID:        memoID,
    UserID:    req.Msg.UserId,
    Content:   req.Msg.Content,
    CreatedAt: createdAt,
  }

  err := s.DB.CreateMemo(ctx, params)
  if (err != nil) {
    slog.Error("Failed to create memo", "error", err)
    return nil, err
  }

  // Return the response
  return connect.NewResponse(&proto.CreateMemoResponse{Memo: &proto.Memo{
    Id:        memoID.String(),
    UserId:    req.Msg.UserId,
    Content:   req.Msg.Content,
    CreatedAt: createdAt.Format(time.RFC3339),
  }}), nil
}

3. Database Access Implementation

3.1 Create internal/db/db.go

Create a function to initialize the database connection. This establishes a connection to the database.

package db

import (
  "github.com/gocraft/dbr/v2"
  _ "github.com/lib/pq" // Import PostgreSQL driver
)

// NewDB initializes the database connection.
func NewDB(dataSourceName string) (*dbr.Connection, error) {
  // Open database connection
  conn, err := dbr.Open("postgres", dataSourceName, nil)
  if (err != nil) {
    return nil, err
  }
  return conn, nil
}

4. Main Function Implementation

4.1 Create cmd/main.go

Implement the main function. This starts the gRPC server and allows it to accept requests from clients.

What is a gRPC Server?

A gRPC server receives requests from clients and processes them based on the defined services. gRPC is an RPC (Remote Procedure Call) framework developed by Google that enables fast and efficient communication.

Write the following code in cmd/main.go:

package main

import (
	"net/http"
	"os"

	"grpc-memo-api/internal/db"
	"grpc-memo-api/internal/db/generated"
	"grpc-memo-api/internal/server"

	"github.com/bufbuild/connect-go"
	"github.com/joho/godotenv"
	"golang.org/x/exp/slog"
	"golang.org/x/net/http2"
	"golang.org/x/net/http2/h2c"
)

func main() {
	// Load .env file
	err := godotenv.Load()
	if (err != nil) {
		slog.Error("Error loading .env file", "error", err)
		return
	}

	// Get database URL from environment variables
	databaseURL := os.Getenv("DATABASE_URL")
	if (databaseURL == "") {
		slog.Error("DATABASE_URL is not set in the environment")
		return
	}

	// Initialize database connection
	conn, err := db.NewDB(databaseURL)
	if err != nil {
		slog.Error("Failed to connect to database", "error", err)
		return
	}
	queries := generated.New(conn)

	// Initialize gRPC server
	// MemoServer is a server struct that processes memo-related gRPC requests using the generated database queries.
	memoServer := &server.MemoServer{DB: queries}

	// Create a multiplexer to handle HTTP requests
	// http.NewServeMux() is used to route HTTP requests to the appropriate handler based on the path.
	// Since the gRPC server operates over the HTTP protocol, a multiplexer is needed to map requests to the appropriate gRPC methods.
	mux := http.NewServeMux()

	// Create a handler for the ListMemos endpoint
	// connect.NewUnaryHandler() creates an HTTP handler by specifying the path of the gRPC endpoint and the function to handle the request.
	// This exposes the gRPC method as an HTTP request.
	handler := connect.NewUnaryHandler(
		"/memo.MemoService/ListMemos", // Endpoint path
		memoServer.ListMemos,          // Function to handle the request
	)
	mux.Handle("/memo.MemoService/ListMemos", handler) // Register the handler with the multiplexer

	// Create a handler for the GetMemo endpoint
	handler = connect.NewUnaryHandler(
		"/memo.MemoService/GetMemo", // Endpoint path
		memoServer.GetMemo,          // Function to handle the request
	)
	mux.Handle("/memo.MemoService/GetMemo", handler) // Register the handler with the multiplexer

	// Create a handler for the CreateMemo endpoint
	handler = connect.NewUnaryHandler(
		"/memo.MemoService/CreateMemo", // Endpoint path
		memoServer.CreateMemo,          // Function to handle the request
	)
	mux.Handle("/memo.MemoService/CreateMemo", handler) // Register the handler with the multiplexer

	// Initialize HTTP/2 server
	// &http2.Server{} is a struct that represents the configuration of the HTTP/2 server.
	// This struct provides options to customize the behavior of HTTP/2.
	// Here, we use the default settings.
	h2s := &http2.Server{}

	// Create HTTP server
	// &http.Server{} is a struct that represents the configuration of the HTTP server.
	// The Addr field specifies the address and port on which the server will listen.
	// The Handler field specifies the handler to process requests.
	// h2c.NewHandler(mux, h2s) creates a handler to process HTTP/2 cleartext (unencrypted) requests.
	// This allows the gRPC server to use the HTTP/2 protocol.
	server := &http.Server{
		Addr:    ":50051",                // Server address and port
		Handler: h2c.NewHandler(mux, h2s), // Set HTTP/2 cleartext handler
	}

	// Log server startup message
	slog.Info("gRPC server is running on port 50051")

	// Start gRPC server
	// server.ListenAndServe() starts the HTTP server on the specified address and port.
	// This causes the gRPC server to start receiving requests.
	// If an error occurs, it is logged using slog.Error.
	if err := server.ListenAndServe(); err != nil {
		slog.Error("Failed to serve", "error", err)
	}
}

cmd/main.go Sequence Diagram

5. sqlc Configuration and Execution

5.1 Create sqlc.yaml

Create the sqlc configuration file. This generates Go code from SQL queries.

version: "2"
sql:
  - schema: "schema.sql" # Path to schema file
    queries: "queries.sql" # Path to query file
    engine: "postgresql" # Database engine to use
    gen:
      go:
        package: "generated" # Package name for generated Go code
        out: "internal/db/generated" # Output directory for generated Go code

5.2 Create schema.sql

Define the database schema. This creates the database tables.

CREATE TABLE memos (
  id UUID PRIMARY KEY, -- Memo ID
  user_id TEXT NOT NULL, -- User ID
  content TEXT NOT NULL, -- Memo content
  created_at TIMESTAMP NOT NULL -- Creation timestamp
);

5.3 Create queries.sql

Define SQL queries. This simplifies database operations.

-- name: ListMemos :many
SELECT id, user_id, content, created_at FROM memos WHERE user_id = $1;

-- name: GetMemo :one
SELECT id, user_id, content, created_at FROM memos WHERE user_id = $1 AND id = $2;

-- name: CreateMemo :exec
INSERT INTO memos (id, user_id, content, created_at) VALUES ($1, $2, $3, $4);

5.4 Run sqlc

Run the following command to generate Go code from SQL queries.

sqlc generate

This command generates the following files:

internal/db/generated/
├── db.go
├── models.go
├── querier.go
└── queries.sql.go

These files are used to simplify database access. Specifically, these functions are called in internal/server/memo_server.go to perform database operations.

6. Database Creation Steps

6.1 Install PostgreSQL

Install PostgreSQL and start the service.

brew install postgresql
brew services start postgresql@14

6.2 Create Database

Create a database.

createdb memo

6.3 Create Database User and Set Password

Enter the PostgreSQL prompt, create a user, and set a password.

psql -d memo

Once the PostgreSQL prompt appears, execute the following commands:

CREATE USER myuser WITH PASSWORD 'mypassword';
GRANT ALL PRIVILEGES ON DATABASE memo TO myuser;
ALTER DATABASE memo SET timezone TO 'UTC';

6.4 Execute schema.sql

Execute the schema file to create the tables.

psql -d memo -U myuser -f schema.sql

7. API Testing

7.1 Start Server

Start the gRPC server.

go run cmd/main.go

7.2 Testing with curl

Use the following curl commands to test the API.

CreateMemo

curl -v -X POST http://localhost:50051/memo.MemoService/CreateMemo -H "Content-Type: application/json" -d '{"user_id": "b7e72808-d71a-4502-a289-35cbafd1352c", "content": "test"}'

GetMemo

curl -v -X POST http://localhost:50051/memo.MemoService/GetMemo -H "Content-Type: application/json" -d '{"user_id": "b7e72808-d71a-4502-a289-35cbafd1352c", "memo_id": "fdf4fd15-3056-4853-a6ce-ef113db86f5c"}'

ListMemos

curl -v -X POST http://localhost:50051/memo.MemoService/ListMemos -H "Content-Type: application/json" -d '{"user_id": "b7e72808-d71a-4502-a289-35cbafd1352c"}'

7.3 Create Testing Script

Create a script to test all API endpoints.

Create test_api.sh 

#!/bin/bash

# CreateMemo
echo "Creating memo..."
curl -sS -X POST http://localhost:50051/memo.MemoService/CreateMemo -H "Content-Type: application/json" -d '{"user_id": "b7e72808-d71a-4502-a289-35cbafd1352c", "content": "test"}' | jq '.'

# GetMemo
echo "Getting memo..."
curl -sS -X POST http://localhost:50051/memo.MemoService/GetMemo -H "Content-Type: application/json" -d '{"user_id": "b7e72808-d71a-4502-a289-35cbafd1352c", "memo_id": "fdf4fd15-3056-4853-a6ce-ef113db86f5c"}' | jq '.'

# ListMemos
echo "Listing memos..."
curl -sS -X POST http://localhost:50051/memo.MemoService/ListMemos -H "Content-Type: application/json" -d '{"user_id": "b7e72808-d71a-4502-a289-35cbafd1352c"}' | jq '.'

This script allows you to test all API endpoints at once.

Execute the Script

Grant execute permission to the script and run it.

chmod +x test_api.sh
./test_api.sh

Summary

In this tutorial, we explained the steps to develop a gRPC API in Go for a simple memo application. We learned how to build a gRPC server and integrate it with a database through the following steps:

  1. Project setup
  2. gRPC server implementation
  3. Database access implementation
  4. Main function implementation
  5. sqlc configuration and execution
  6. Database creation
  7. API testing

By clearly stating the purpose and reason for each step and proceeding according to best practices, we can efficiently and effectively develop a gRPC API. This lays the foundation for building a scalable and maintainable application.

gRPC is an excellent choice for developing fast and efficient APIs. By using Protocol Buffers, we can define language-independent interfaces. Using Connect allows us to support both gRPC and HTTP APIs. Using SQLC allows us to maintain consistency between SQL queries and Go code.

By leveraging the technology stack introduced in this tutorial, you can develop more robust and extensible APIs. We encourage you to refer to this tutorial and introduce gRPC APIs into your own applications.

Discussion