Zenn
Open8

Complete Introduction to Protocol Buffers 3視聴メモ

ta.toshiota.toshio

Chapter 1 : Course Introduction

Protocol Buffersのデメリット紹介でシリアライズ（、圧縮されている？）からtext editorで確認することはできない、とのこと。

The need for Protocol Buffers

How are Protocol Buffers used?

ta.toshiota.toshio

Chapter 2 : Protocol Buffers Basics

First Message

Scalar Types

integerはint32を使いましょう
floating point numberはfloat（32bit）、より正確にするならdouble(64bit)

// The syntax for this file is proto3
syntax = "proto3";

/* Person is used to identify users
 * across our system */
message Person {
    // the age as of the person's creation
    int32 age = 1;
    // the first name as documented in the signup form
    string first_name = 2;
    string last_name = 3; // last name as documented in the signup form
    // small_picture represents a small .jpg file
    bytes small_picture = 4;
    bool is_profile_verified = 5;
    // height of the person in cms 
    float height = 6;

    // a list of phone numbers that is optional to provide at signup
    repeated string phone_numbers = 7;
}

Tags

19000 ~ 19999はgoogleが使っているから利用できない（or してはいけない）
2の29乗-1

Repeated Fields

list, arrayはrepeated fields

repeatedの反対はsinglar。書く必要はない

Default Values For Fields

https://protobuf.dev/programming-guides/proto3/#default

protocol buffers 3にはrequiredやoptionalのようなコンセプトはないとのこと

Enumerations (Enums)

https://protobuf.dev/programming-guides/proto3/#enum

Enum start by 0

message Person {
...
   repeated string phone_numbers = 7;

    // we currently consider only 3 eye colours
    enum EyeColour {
        UNKNOWN_EYE_COLOUR = 0;
        EYE_GREEN = 1;
        EYE_BROWN = 2;
        EYE_BLUE = 3;
    }

    // it's an enum as defined above
    EyeColour eye_colour = 8;
ta.toshiota.toshio

Chapter 3 : Protocol Buffers Basics II

Defining Multiple Messages in the Same File

// The syntax for this file is proto3
syntax = "proto3";

/* Person is used to identify users
 * across our system */
message Person {
  // the age as of the person's creation
  int32 age = 1;
  // the first name as documented in the signup form
  string first_name = 2;
  string last_name = 3; // last name as documented in the signup form
  // small_picture represents a small .jpg file
  bytes small_picture = 4;
  bool is_profile_verified = 5;
  // height of the person in cms
  float height = 6;

  // a list of phone numbers that is optional to provide at signup
  repeated string phone_numbers = 7;

  // we currently consider only 3 eye colours
  enum EyeColour {
    UNKNOWN_EYE_COLOUR = 0;
    EYE_GREEN = 1;
    EYE_BROWN = 2;
    EYE_BLUE = 3;
  }

  // it's an enum as defined above
  EyeColour eye_colour = 8;

  // Person's birthday
  Date birthday = 9;
}

message Date {
  // Year of date. Must be from 1 to 9999, or 0 if specifying a date without
  // a year.
  int32 year = 1;

  // Month of year. Must be from 1 to 12.
  int32 month = 2;

  // Day of month. Must be from 1 to 31 and valid for the year and month, or 0
  // if specifying a year/month where the day is not significant.
  int32 day = 3;
}

Nesting Messages

message Person {

  int32 age = 1;
...
  // we define the type Address within Person (full name is Person.Address)
  message Address {
    string address_line_1 = 1;
    string address_line_2 = 2;
    string zip_code = 3;
    string city = 4;
    string country = 5;
  }

  // multiple addresses
  repeated Address addresses = 10;
}

Imports


import "root/path/to/date.proto";

message Person {
...
  Date birthday = 9;
}

Packages

コンパイルしたとき、意図した場所に生成される。
名前のコンフリクトを防ぐのに役立つ

syntax = "proto3";

package my.date;

message Date {
...
}

// The syntax for this file is proto3
syntax = "proto3";

import "root/path/to/date.proto";

package person;

/* Person is used to identify users
 * across our system */
message Person {
...
  // Person's birthday
  my.date.Date birthday = 9;
...
}
ta.toshiota.toshio

Chapter 4 : Setting up Protoc Compiler

protoc generate code automatically based on protoc file

python

protoc -I proto --python_out=python_dir proto/*.proto

java

protoc -I proto --java_out=java_dir proto/*.proto

-Iオプションは.protoファイルでimportするファイルのPATHです。 --proto_pathの短縮形です。

-Iオプションを指定すると、
.protoファイルでのimportを使える
import pathを省略できる

https://christina04.hatenablog.com/entry/protoc-usage

ta.toshiota.toshio

Chapter 6 : Golang Programming with Protocol Buffers

go_package option

simple.proto

syntax = "proto3";

package example.simple;

option go_package = "simplepb"; // この記述で

message SimpleMessage {
  int32 id = 1;
  bool is_simple = 2;
  string name = 3;
  repeated int32 sample_list = 4;
}

simple.pb.go

// source: simple/simple.proto

package simplepb // こうなった

Reading and Writing to Disk

protoが生成した構造体の情報をファイルに書き出し、読み出すサンプル

import (
...
  "github.com/golang/protobuf/proto"
...
)

func main() {
	sm := doSimple()
	writeToFile("simple.bin", sm)
	sm2 := &simplepb.SimpleMessage{}
	readFromFile("simple.bin", sm2)
	fmt.Println("Read the content:", sm2)
}

func writeToFile(fname string, pb proto.Message) error {
	out, err := proto.Marshal(pb)
	if err != nil {
		log.Fatalln("Can't serialise to bytes", err)
		return err
	}

	if err := ioutil.WriteFile(fname, out, 0644); err != nil {
		log.Fatalln("Can't write to file", err)
		return err
	}

	fmt.Println("Data has been written!")
	return nil
}

func readFromFile(fname string, pb proto.Message) error {

	in, err := ioutil.ReadFile(fname)
	if err != nil {
		log.Fatalln("Something went wrong when reading the file", err)
		return err
	}

	err2 := proto.Unmarshal(in, pb)

	if err2 != nil {
		log.Fatalln("Couldn't put the bytes into the protocol buffers struct", err)
		return err2
	}

	return nil
}

Reading and Writing to JSON


func main() {
	sm := doSimple()

	smAsString := toJSON(sm)
	fmt.Println(smAsString)

	sm2 := &simplepb.SimpleMessage{}
	fromJSON(smAsString, sm2)
	fmt.Println("Successfully created proto struct:", sm2)
}

func toJSON(pb proto.Message) string {
	marshaler := jsonpb.Marshaler{}
	out, err := marshaler.MarshalToString(pb)
	if err != nil {
		log.Fatalln("Can't convert to JSON", err)
		return ""
	}
	return out
}

func fromJSON(in string, pb proto.Message) {
	err := jsonpb.UnmarshalString(in, pb)
	if err != nil {
		log.Fatalln("Couldn't unmarshal the JSON into the pb struct", err)
	}
}

Complex Proto Struct in Golang

struct example

syntax = "proto3";

package example.complex;

option go_package = "complexpb";

message ComplexMessage {
  DummyMessage one_dummy = 2;
  repeated DummyMessage multiple_dummy = 3;
}

message DummyMessage {
  int32 id = 1;
  string name = 2;
}

	cm := complexpb.ComplexMessage{
		OneDummy: &complexpb.DummyMessage{
			Id:   1,
			Name: "First message",
		},
		MultipleDummy: []*complexpb.DummyMessage{
			&complexpb.DummyMessage{
				Id:   2,
				Name: "Second message",
			},
			&complexpb.DummyMessage{
				Id:   3,
				Name: "Third message",
			},
		},
	}

	fmt.Println(cm)
ta.toshiota.toshio

Chapter 7 : Data Evolution with Protobuf

Rules for Data Evolution

  1. Don't change the numeric tags for any existing fields.
  2. You can add new fields, and old code will just ignore them.
  3. Likewise, if the old / new code reads unknown data, the default will take place.
  4. tag番号を使わなければ、フィールドを消しても良い。OBSOLETE_のようなprefixをつけて、フィールドをリネームしてもよい、ただしtag番号はそのままで。
  5. データ型の変更。でも変更しない運用のほうがいいよ、とのこと

Removing Fields

フィールドの削除したときは、タグ番号とフィールド名をreserveしなさい。

message MyMessage {
  int32 id = 1;
  string first_name = 2
}
↓
message MyMessage {
  reserved 2;
  reserved "first_name";
  int32 id = 1;
}

Reserved Keywords

message Foo {
  reserved 2, 15, 9 to 11;
   reserved "foo", "bar";
}

タグ番号とフィールド名は混ぜないように。
reservedタグ番号は絶対消さないように。

Evolving Enum Fields

enumは値がセットされなかったら最初の値がデフォルト値として扱われる。
おすすめとして、最初の値はUNKNOWN = 0として扱ったらどうか、とのこと。

ta.toshiota.toshio

Chapter 8 : Protocol Buffers Advanced

Integer Types Deep Dive

int32, int64 do ot encode negative values efficiently
sint32, sint64 encode negative values well (through the use of a technique called ZigZag)

Advanced Data Types (oneof, map, Timestamp and Duration)

oneof

多くのフィールドを持つメッセージで、同時に設定されるフィールドがせいぜい 1 つである場合、oneof 機能を使うことでこの動作を強制し、メモリを節約することができます。
oneofフィールドは通常のフィールドと同じですが、oneof内のすべてのフィールドがメモリを共有し、同時に設定できるフィールドは1つだけです。oneofのどのメンバを設定しても、他のすべてのメンバは自動的にクリアされます。oneofのどの値が設定されたかは、言語によって異なりますが、特殊な case() メソッドや WhichOneof() メソッドで確認できます。
複数の値が設定されている場合は、プロトの順序によって最後に設定された値が、それ以前の値をすべて上書きすることに注意してください。
oneof フィールドのフィールド番号は、メッセージ内で一意でなければなりません。

repeatedは使用できない

https://protobuf.dev/programming-guides/proto3/#oneof

maps

https://protobuf.dev/programming-guides/proto3/#maps

timestamp

A Timestamp represents a point in time independent of any time zone or calendar, represented as seconds and fractions of seconds at nanosecond resolution in UTC Epoch time

https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp

import "google/protobuf/timestamp.proto";

message MyMessage {
  google.protobuf.Timestamp timestamp = 1;
  string message = 2;
}

duration

it represents the tiem span between two timestamps

https://protobuf.dev/reference/protobuf/google.protobuf/#duration

import "google/protobuf/timestamp.proto";
import "google/protobuf/duration.proto";

message MyMessage {
  google.protobuf.Timestamp msg_date = 1;
  google.protobuf.Duration validaty = 2;
}

## porotocol buffers options

## Naming Convention From the doc

* Use CamelCase for message names
* Use underscore_spareted_names for fields
* Use CamelCase for Enums and CAPITAL_WITH_UNDERSCORE for values names

https://protobuf.dev/programming-guides/style/