Closed2023/07/17にクローズ9

skaffold using helm

Helm

skaffold

ぺこ 2023/07/17

参考

ぺこ 2023/07/17に更新

Install

skaffold
- already installed by another step

% skaffold version
v2.6.1

kubectl
- already installed by Docker Desktop

% kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:20:07Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.4+k3s1", GitCommit:"8d0255af07e95b841952563253d27b0d10bd72f0", GitTreeState:"clean", BuildDate:"2023-04-20T00:33:18Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}

k3d
- already installed by another step

% k3d version
k3d version v5.5.1
k3s version v1.26.4-k3s1 (default)

helm

asdf plugin add helm
asdf install helm latest
helm version

Setup version

asdf current helm 3.12.2

ぺこ 2023/07/17

Generate application

参考にしたサイトと同じようにSpring Initializerで作成
- Project : Gradle - Groovy
- Language : Java
- Spring Boot : 3.1.1
- Project Metadata
  - Group : com.example
  - Artifact : demo
  - Name : demo
  - Description : Demo project for Spring Boot
  - Package name : com.example.demo
  - Packaging : jar
  - Java : 17
- Dependencies
  - Spring Web
Applicationを修正

src/main/java/com/example/demo/DemoApplication.java

package com.example.demo;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class DemoApplication {

	@Value("${skaffold.env}")
	private String env;

	@GetMapping("/envval")
	public String getEnv() {
		return env;
	}

	public static void main(String[] args) {
		SpringApplication.run(DemoApplication.class, args);
	}

}

application.propertiesを修正

src/main/resources/application.properties

skaffold.env=test

ぺこ 2023/07/17

Confirm application

ビルド

./gradlew clean build

起動

./gradlew bootRun

動作確認

curl localhost:8080/envval

output

test

ぺこ 2023/07/17に更新

Initialize skaffold

skaffold init -k helm

output

apiVersion: skaffold/v4beta6
kind: Config
metadata:
  name: helm
manifests:
  rawYaml:
    - helm

? Do you want to write this configuration to skaffold.yaml? Yes
Configuration skaffold.yaml was written
You can now run [skaffold build] to build the artifacts
or [skaffold run] to build and deploy
or [skaffold dev] to enter development mode, with auto-redeploy

ls -ln

output

total 64
-rw-r--r--@ 1 501  20   993  7 16 19:43 HELP.md
-rw-r--r--@ 1 501  20   448  7 16 19:43 build.gradle
drwxr-xr-x@ 3 501  20    96  7 16 19:43 gradle
-rwxr-xr-x@ 1 501  20  8496  7 16 19:43 gradlew
-rw-r--r--@ 1 501  20  2868  7 16 19:43 gradlew.bat
drwxr-xr-x  9 501  20   288  7 16 17:52 helm
-rw-r--r--@ 1 501  20    26  7 16 19:43 settings.gradle
-rw-r--r--  1 501  20   485  7 17 14:34 skaffold.yaml
drwxr-xr-x@ 4 501  20   128  7 16 19:43 src

helm create helm

output

Creating helm

tree helm

output

helm
├── Chart.yaml
├── charts
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── deployment.yaml
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── service.yaml
│   ├── serviceaccount.yaml
│   └── tests
│       └── test-connection.yaml
└── values.yaml

4 directories, 10 files

ぺこ 2023/07/17

Modify helm manifests

不要なファイルを削除

rm helm/templates/ingress.yaml
rm helm/templates/serviceaccount.yaml
rm helm/templates/tests/test-connection.yaml
rm helm/templates/_helpers.tpl
rm helm/templates/NOTES.txt

helm/templates/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-app
  labels:
    app: spring-app
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: spring-app
  template:
    metadata:
      labels:
        app: spring-app
    spec:
      containers:
        - name: spring-app
          image: {{ .Values.image }}
          ports:
            - name: http
              containerPort: {{ .Values.app.port }}
              protocol: TCP
          env:
            - name: SKAFFOLD_ENV
              value: {{ .Values.app.env }}
            - name: SERVER_PORT
              value: {{ .Values.app.port }}

helm/templates/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: spring-app
  labels:
    app: spring-app
spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.port }}
      targetPort: {{ .Values.app.port }}
      protocol: TCP
      name: http
  selector:
    app: spring-app

helm/values-dev.yaml

replicaCount: 2

image: spring-app

app:
  port: 8081
  env: dev

service:
  type: NodePort
  port: 8081

helm/values-prod.yaml

replicaCount: 4

image: spring-app

app:
  port: 8081
  env: prod

service:
  type: ClusterIP
  port: 8081

ぺこ 2023/07/17に更新

Modify skaffold settings

skaffold.yaml

apiVersion: skaffold/v4beta6
kind: Config
metadata:
  name: skaffold-deploy
build:
  artifacts:
    - image: spring-app
      buildpacks:
        builder: gcr.io/buildpacks/builder:v1
deploy:
  helm:
    releases:
      - name: spring-app
        namespace: default
        chartPath: helm
        valuesFiles:
          - "{{ .VALUES_FILE }}"
portForward:
  - resourceType: service
    resourceName: spring-app
    port: 8081

apiVersionは、現在(2023/7/17)の最新のものを設定
deploy > helm > releases > artifactOverridesを削除

ぺこ 2023/07/17に更新

Deploy application

VALUES_FILE=./helm/values-dev.yaml skaffold dev

output

Generating tags...
 - spring-app -> spring-app:latest
Some taggers failed. Rerun with -vdebug for errors.
Checking cache...
 - spring-app: Not found. Building
Starting build...
Found [k3d-my-cluster] context, using local docker daemon.
Building [spring-app]...
Target platforms: [linux/amd64]
Cleaning up...
Error: uninstall: Release not loaded: spring-app: release: not found
Cleaning up resources encountered an error, will continue to clean up other resources.
Build Failed. Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Check if docker is running.

エラーになった
以下のページを参考にした
- https://github.com/spring-projects/spring-boot/issues/32897
Docker Desktopの設定を変更
- Settings > Advanced
  - Allow the default Docker socket to be usedをオン
  - Apply & restart
Retry

output


~~~ 省略 ~~~

[builder] BUILD FAILED in 34s
[builder] --------------------------------------------------------------------------------
[builder] Sorry your project couldn't be built.
[builder] Our documentation explains ways to configure Buildpacks to better recognise your project:
[builder]  -> https://cloud.google.com/docs/buildpacks/overview
[builder] If you think you've found an issue, please report it:
[builder]  -> https://github.com/GoogleCloudPlatform/buildpacks/issues/new
[builder] --------------------------------------------------------------------------------
[builder] ERROR: failed to build: exit status 1
Cleaning up...
Error: uninstall: Release not loaded: spring-app: release: not found
Cleaning up resources encountered an error, will continue to clean up other resources.
build [spring-app] failed: executing lifecycle. This may be the result of using an untrusted builder: failed with status code: 51

Gradleは実行されていたが、ビルドエラーになった
ログを見ると以下のメッセージが出力されている

output

===> BUILDING
[builder] === Java - Runtime (google.java.runtime@0.9.1) ===
[builder] Using latest Java 11 runtime version. You can specify a different version with GOOGLE_RUNTIME_VERSION: https://github.com/GoogleCloudPlatform/buildpacks#configuration

アプリケーションはJava 17で作成したが、Java 11でビルドしようとしてエラーになった模様
https://github.com/GoogleCloudPlatform/buildpacks#configuration
ログ記載のURLからJavaのページを辿って
https://cloud.google.com/docs/buildpacks/java?hl=ja
環境変数GOOGLE_RUNTIME_VERSION=17 を設定すると良さそう
skaffold.yamlを以下のように修正

skaffold.yaml

apiVersion: skaffold/v4beta6
kind: Config
metadata:
  name: skaffold-deploy
build:
  artifacts:
    - image: spring-app
      buildpacks:
        builder: gcr.io/buildpacks/builder:v1
        env:
          - GOOGLE_RUNTIME_VERSION=17
deploy:
  helm:
    releases:
      - name: spring-app
        namespace: default
        chartPath: helm
        valuesFiles:
          - "{{ .VALUES_FILE }}"
portForward:
  - resourceType: service
    resourceName: spring-app
    port: 8081

Retry

output

~~~ 省略 ~~~

===> BUILDING
[builder] === Java - Runtime (google.java.runtime@0.9.1) ===
[builder] Using requested runtime feature version: 17

~~~ 省略 ~~~

[builder] Starting a Gradle Daemon (subsequent builds will be faster)

[builder] > Task :clean UP-TO-DATE
[builder] > Task :compileJava
[builder] > Task :processResources
[builder] > Task :classes
[builder] > Task :resolveMainClassName
[builder] > Task :bootJar
[builder] > Task :jar
[builder] > Task :assemble
[builder] 
[builder] BUILD SUCCESSFUL in 3m 35s

~~~ 省略 ~~~

Build [spring-app] succeeded
Tags used in deployment:
 - spring-app -> spring-app:633ce2fbf0aa454e4bb895df1d8dbe9b5b2520d2965707510a08271b559f9541
Starting deploy...
Loading images into k3d cluster nodes...
 - spring-app:633ce2fbf0aa454e4bb895df1d8dbe9b5b2520d2965707510a08271b559f9541 -> Loaded
Images loaded in 1 minute 13.382 seconds
Helm release spring-app not installed. Installing...
Error: INSTALLATION FAILED: 1 error occurred:
	* Deployment in version "v1" cannot be handled as a Deployment: json: cannot unmarshal number into Go struct field EnvVar.spec.template.spec.containers.env.value of type string


Cleaning up...
release "spring-app" uninstalled
deploying "spring-app": install: exit status 1

エラーになった
* Deployment in version "v1" cannot be handled as a Deployment: json: cannot unmarshal number into Go struct field EnvVar.spec.template.spec.containers.env.value of type stringのメッセージ
以下のようにhelm/templates/deployment.yamlを修正してみた

helm/templates/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-app
  labels:
    app: spring-app
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: spring-app
  template:
    metadata:
      labels:
        app: spring-app
    spec:
      containers:
        - name: spring-app
          image: {{ .Values.image }}
          ports:
            - name: http
              containerPort: {{ .Values.app.port }}
              protocol: TCP
          env:
            - name: SKAFFOLD_ENV
              value: "{{ .Values.app.env }}"
            - name: SERVER_PORT
              value: "{{ .Values.app.port }}"

Retry

output


~~~ 省略 ~~~

Helm release spring-app not installed. Installing...
NAME: spring-app
LAST DEPLOYED: Mon Jul 17 15:46:08 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
Waiting for deployments to stabilize...
 - deployment/spring-app is ready.
Deployments stabilized in 1.33 second
Port forwarding service/spring-app in namespace default, remote port 8081 -> http://127.0.0.1:8081
Listing files to watch...
 - spring-app
Press Ctrl+C to exit
Watching for changes...

~~~ 省略 ~~~

[spring-app] 2023-07-17T06:46:29.929Z  INFO 18 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8081 (http) with context path ''
[spring-app] 2023-07-17T06:46:30.024Z  INFO 18 --- [           main] com.example.demo.DemoApplication         : Started DemoApplication in 12.496 seconds (process running for 16.564)

起動できたっぽい
動作確認

% curl localhost:8081/envval       
dev

% kubectl get deployment/spring-app
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
spring-app   2/2     2            2           2m4s

helm/values-prod.yamlを指定して実行してみる
動作できた
動作確認

% curl localhost:8081/envval       
prod

% kubectl get deployment/spring-app
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
spring-app   4/4     4            4           2m20s

設定が反映されたことが確認できた

ヒロオカさんの記事では動いているのに... ?!

kubectlのバージョンが違うからでしょうか？

ぺこ 2023/07/17に更新

Buildpacks version

https://cloud.google.com/docs/buildpacks/builders?hl=ja
上記を見ると、現在(2023/7/17)の最新のタグはgoogle-22のよう
skaffold.yamlのbuild部分を以下のように修正

build part of skaffold.yaml

build:
  artifacts:
    - image: spring-app
      buildpacks:
        builder: gcr.io/buildpacks/builder:google-22
        env:
          - GOOGLE_RUNTIME_VERSION=17

正常に動作することを確認できた

このスクラップは2023/07/17にクローズされました