Zenn
🐮

BFF + gRPC構成をMinikube with Istioで作ってみる

2021/09/13に公開
Rust
gRPC
minikube
Istio
BFF
tech

概要

週末とか時間ある時にこちょこちょ作ってた
ローカルの開発環境がほんのちょこっと動く雰囲気まできたので
記憶が風化しないうちに一旦メモに残しておくことにした🥸

作ったイメージ

BFFにはhttp、BFFからバックエンドサービスはgRPCというありがちな構成
ネットワークの制御はIstioで行いMinikube上に更新

言語は最近ちょこっと勉強したRustを使用
Webframeworkには最近出てたaxum
gRPCのClient, Serverにはtonicを使ってみた

開発環境のHot ReloadのためにSkaffoldを利用

ディレクトリ構成

最終的にはこんな感じ。

.
├── README.md
├── infra
│   └── kubernetes
│       ├── Makefile
│       ├── README.md
│       ├── capital-farm
│       │   ├── 00-namespace.yaml
│       │   ├── identity
│       │   │   ├── deployment.yaml
│       │   │   └── service.yaml
│       │   ├── istio
│       │   │   ├── destination-rule.yaml
│       │   │   └── virtual-service.yaml
│       │   └── web-bff
│       │       ├── deployment.yaml
│       │       └── service.yaml
│       ├── default
│       │   └── istio
│       │       └── gateway.yaml
│       ├── istio-system
│       │   └── istio-operator.yaml
│       ├── scripts
│       │   ├── browse.zsh
│       │   └── start-minikube.zsh
│       └── skaffold.yaml
└── src
    └── capital-farm
        ├── identity
        │   ├── Cargo.lock
        │   ├── Cargo.toml
        │   ├── Dockerfile
        │   ├── build.rs
        │   ├── proto
        │   │   └── helloworld.proto
        │   ├── src
        │   │   └── main.rs
        │   └── target
        └── web-bff
            ├── Cargo.lock
            ├── Cargo.toml
            ├── Dockerfile
            ├── build.rs
            ├── src
            │   └── main.rs
            └── target

環境構築Script

最終的にはこのスクリプトをぽこんと叩くと
Minikubeが立ち上がって、必要なリソース周りが諸々立ち上がって
webページにHello Tonic!が表示されるはず！

infra/kubernetes/scripts/start-minikube.zsh
#!/bin/zsh -eu

# start Minikube
minikube config set memory 16384
minikube config set cpus 4
minikube config set driver hyperkit
minikube start

# install Istio and deploy resources
istioctl operator init
kubectl apply -f istio-system -R

# TODO: wait until kubectl api-versions shows istio resources
sleep 5
kubectl apply -f default -f capital-farm -R

# update /etc/hosts
MINIKUBE_IP=$(minikube ip)
HOSTS_ENTRY="$MINIKUBE_IP capital-farm.ucwork.local"

if grep -Fq "capital-farm.ucwork.local" /etc/hosts > /dev/null
then
    sudo sed -i '.bk' "s/.*capital-farm.ucwork.local$/$HOSTS_ENTRY/" /etc/hosts
    echo "/etc/hosts is updated"
else
    echo "$HOSTS_ENTRY" | sudo tee -a /etc/hosts
    echo "/etc/hosts is added"
fi

スクリプトに書いてある通りですがこんな感じのことやってる

  1. Minikube起動
  2. IstioをMinikube上にInstall
  3. 自分のアプリ用リソースをデプロイ
  4. ローカルの/etc/hotsをMinikubeのipで書き換え

もう一声実行してる中身をメモしてく

Istio Install

Istioのインストールとかupgradeとか素敵にやってくれるIstio Operator

infra/kubernetes/istio-system/istio-operator.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istiocontrolplane
spec:
  profile: default
  components:
    pilot:
      k8s:
        resources:
          requests:
            memory: 3072Mi
    egressGateways:
      - name: istio-egressgateway
        enabled: true

capital-farm用のk8sリソース

自分の今作ってるアプリ（capital-farm）を動かすための
リソース一覧の中身をちょろっと書いてく

Istio Gateway

*.ucwork.localへのアクセスを全て受ける想定
サブドメイン切り替えていろんなサービス作っていきたいなという願望

infra/kubernetes/default/istio/gateway.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ucwork-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*.ucwork.local"

capital-farm namespace作成

namespace作成
istio-injection: enabledこれ指定忘れないこと大事！

infra/kubernetes/capital-farm/00-namespace.yaml
kind: Namespace
apiVersion: v1
metadata:
  name: capital-farm
  labels:
    name: capital-farm
    istio-injection: enabled

bff作成

deployment

infra/kubernetes/capital-farm/web-bff/deployment.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: capital-farm
  name: web-bff
  labels:
    account: web-bff
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: capital-farm
  name: web-bff-v1
  labels:
    app: web-bff
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-bff
      version: v1
  template:
    metadata:
      labels:
        app: web-bff
        version: v1
    spec:
      serviceAccountName: web-bff
      containers:
        - name: web-bff
          image: docker.io/ucwork/capital-farm-web-bff:0.0.1
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 9080
          volumeMounts:
            - name: tmp
              mountPath: /tmp
          securityContext:
            runAsUser: 1000
      volumes:
        - name: tmp
          emptyDir: {}

service

infra/kubernetes/capital-farm/web-bff/service.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: capital-farm
  name: web-bff
  labels:
    app: web-bff
    service: web-bff
spec:
  ports:
    - port: 9080
      name: http
  selector:
    app: web-bff

identity(backend service)作成

gRPCで疎通する対象のidentityサービス用のリソース

deployment

infra/kubernetes/capital-farm/identity/deployment.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: capital-farm
  name: identity
  labels:
    account: identity
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: capital-farm
  name: identity-v1
  labels:
    app: identity
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: identity
      version: v1
  template:
    metadata:
      labels:
        app: identity
        version: v1
    spec:
      serviceAccountName: identity
      containers:
        - name: identity
          image: docker.io/ucwork/capital-farm-identity:0.0.1
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 50051
          volumeMounts:
            - name: tmp
              mountPath: /tmp
          securityContext:
            runAsUser: 1000
      volumes:
        - name: tmp
          emptyDir: {}

service

infra/kubernetes/capital-farm/identity/service.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: capital-farm
  name: identity
  labels:
    app: identity
    service: identity
spec:
  ports:
    - port: 50051
      name: grpc
      appProtocol: grpc
      protocol: TCP
  selector:
    app: identity

Istioのリソース作成

virtual service

作成したIstio Gatewayを指定したVirtual Serviceの作成
capital-farm.ucwork.localにアクセスがきたらweb-bffに流してあげるように記載

infra/kubernetes/capital-farm/istio/virtual-service.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  namespace: capital-farm
  name: capital-farm
spec:
  hosts:
    - "capital-farm.ucwork.local"
  gateways:
    - default/ucwork-gateway
  http:
    - match:
        - uri:
            prefix: /web-bff
      route:
        - destination:
            host: web-bff
            port:
              number: 9080
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  namespace: capital-farm
  name: identity
spec:
  hosts:
    - identity
  http:
    - route:
        - destination:
            host: identity
            subset: v1
---

destination rule

ルーティングされたものにもう一声設定できるらしいdestination rule
今回大した設定してない

Istioのチュートリアルだとv1,v2とかいくつかバージョンの候補書いてあったりする

infra/kubernetes/capital-farm/istio/destination-rule.yaml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  namespace: capital-farm
  name: web-bff
spec:
  host: web-bff
  subsets:
  - name: v1
    labels:
      version: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  namespace: capital-farm
  name: identity
spec:
  host: identity
  subsets:
    - name: v1
      labels:
        version: v1
---

ソースコードの準備

backend(identity)とbffのソースを準備

identity

cd src/capital-farm/identity
cargo init

こんな感じでCargo.tomlを作成
そんで必要なパッケージ追加

src/capital-farm/identity/Cargo.toml
[package]
name = "identity"
version = "0.1.0"
edition = "2018"

[dependencies]
tonic = "0.5"
prost = "0.8"
tokio = { version = "1.0", features = ["macros", "rt-multi-thread"] }

[build-dependencies]
tonic-build = "0.5"

gRPC用のprotofile定義

src/capital-farm/identity/proto/helloworld.proto
syntax = "proto3";
package helloworld;

service Greeter {
    rpc SayHello (HelloRequest) returns (HelloReply);
}

message HelloRequest {
    string name = 1;
}

message HelloReply {
    string message = 1;
}

protofileをbuildしてくれるrustファイルを用意

src/capital-farm/identity/build.rs
fn main() -> Result<(), Box<dyn std::error::Error>> {
    tonic_build::compile_protos("proto/helloworld.proto")?;
    Ok(())
}

gRPCのserverソース作成

src/capital-farm/identity/src/main.rs
use tonic::{transport::Server, Request, Response, Status};

use hello_world::greeter_server::{Greeter, GreeterServer};
use hello_world::{HelloReply, HelloRequest};

pub mod hello_world {
    tonic::include_proto!("helloworld");
}

#[derive(Debug, Default)]
pub struct MyGreeter {}

#[tonic::async_trait]
impl Greeter for MyGreeter {
    async fn say_hello(
        &self,
        request: Request<HelloRequest>,
    ) -> Result<Response<HelloReply>, Status> {
        println!("Got a request: {:?}", request);

        let reply = hello_world::HelloReply {
            message: format!("Hello {}!", request.into_inner().name).into(),
        };

        Ok(Response::new(reply))
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let addr = "[::]:50051".parse()?;
    let greeter = MyGreeter::default();

    Server::builder()
        .add_service(GreeterServer::new(greeter))
        .serve(addr)
        .await?;

    Ok(())
}

Dockerfileも用意しておこう

src/capital-farm/identity/Dockerfile
FROM rust:1.54 as builder
WORKDIR /usr/src/myapp
COPY . .
RUN rustup component add rustfmt
RUN cargo install --path .

FROM debian:buster-slim
RUN apt-get update && rm -rf /var/lib/apt/lists/*
COPY --from=builder /usr/local/cargo/bin/identity /usr/local/bin/identity
CMD ["identity"]

自分のdocker hub環境に合わせてタグの名前は変えてね

docker build -t ucwork/capital-farm-web-bff:0.0.1 .
docker push ucwork/capital-farm-web-bff:0.0.1

bff

cd src/capital-farm/web-bff
cargo init

こんな感じでCargo.tomlを作成
必要なpackage追加

src/capital-farm/web-bff/Cargo.toml
[package]
name = "web-bff"
version = "0.1.0"
edition = "2018"

[dependencies]
axum = "0.1.3"
hyper = { version = "0.14", features = ["full"] }
tokio = { version = "1.0", features = ["full"] }
tonic = "0.5"
prost = "0.8"

[build-dependencies]
tonic-build = "0.5"

protofileをbuildしてくれるrustファイルを用意

src/capital-farm/web-bff/build.rs
fn main() -> Result<(), Box<dyn std::error::Error>> {
    tonic_build::compile_protos("../identity/proto/helloworld.proto")?;
    Ok(())
}

webサーバの受け口、gRPCのClientを作成

src/capital-farm/web-bff/src/main.rs
use axum::prelude::*;
use std::net::SocketAddr;
use hello_world::greeter_client::GreeterClient;
use hello_world::HelloRequest;

pub mod hello_world {
    tonic::include_proto!("helloworld");
}

#[tokio::main]
async fn main() {
    let app = route("/web-bff/test", get(root));
    let addr = SocketAddr::from(([0, 0, 0, 0], 9080));
    hyper::Server::bind(&addr)
        .serve(app.into_make_service())
        .await
        .unwrap();
}

async fn root() -> String {
    let mut client = GreeterClient::connect("http://identity:50051").await.unwrap();

    let request = tonic::Request::new(HelloRequest {
        name: "Tonic".into(),
    });

    let response = client.say_hello(request).await.unwrap();

    response.into_inner().message
}

Dockerfileも用意しておこう

src/capital-farm/web-bff/Dockerfile
FROM rust:1.54 as builder
WORKDIR /usr/src/myapp
COPY ./web-bff .
COPY ./identity/proto ../identity/proto

RUN rustup component add rustfmt
RUN cargo install --path .

FROM debian:buster-slim
RUN apt-get update && rm -rf /var/lib/apt/lists/*
COPY --from=builder /usr/local/cargo/bin/web-bff /usr/local/bin/web-bff
CMD ["web-bff"]

そしてbuild and pushじゃ

自分のdocker hub環境に合わせてタグの名前は変えてね

docker build -t ucwork/capital-farm-identity:0.0.1 .
docker push ucwork/capital-farm-identity:0.0.1

さぁ起動してみよう

infra/kubernetes/Makefile
start:
	zsh scripts/start-minikube.zsh
browse:
	zsh scripts/browse.zsh

こんな感じじゃ

make start
make browse

ちょっと時間かかるけどうまくいってたら
こんな感じでHello Tonic!が表示されるはず！

SkaffoldでHot Reload

skaffold向けのファイル準備

infra/kubernetes/skaffold.yaml
apiVersion: skaffold/v2beta21
kind: Config
build:
  artifacts:
    - image: docker.io/ucwork/capital-farm-identity
      context: ../../src/capital-farm/identity/
    - image: docker.io/ucwork/capital-farm-web-bff
      context: ../../src/capital-farm/
      docker:
        dockerfile: ./web-bff/Dockerfile
deploy:
  kubectl:
    manifests:
      - "./capital-farm/identity/deployment.yaml"
      - "./capital-farm/web-bff/deployment.yaml"

こんな感じでコマンド叩くと変更検知して画面の内容も変わってくれた！
別階層にあるprotofileを参照させちゃったので
contextを親階層にして、dockerfileを指定させてるのが癖ありげ

skaffold dev

まとめ

  1. cargo installじゃなくてcargo runにしたら流石にビルドもうちょい早くなるか
  2. KialiとかDashboard付けたい
  3. kustomizeとか入れてみたい

まだまだやりたいこと盛り沢山だけど、とりあえずキリがいいのでメモに残してみた

Istioでネットワークに関する関心事項が切り分けられるのは良さそう。
認証認可ももはやjwt絡めてIstioでやってみようと思う

Discussion