CANONICAL『Cloud Instance Initialisation with cloud-init』のメモ

参考：Cloud init | Ubuntu

Executive Summary

As a technology business decision maker you have a great deal of choice today when it comes to both open source operating systems and cloud execution environments (or just clouds) to run those operating systems on. We are using the term ‘cloud’ here in its widest possible sense: from private cloud (i.e. an automated data centre), through hybrid to public cloud. A common desire is to select the operating system which best supports your application stack and then be able to use that operating system as an abstraction layer between different clouds. However, in order to function together an operating system and its cloud must share some critical configuration data which tells the operating system how to ‘behave’ in the particular environment. By separating configuration data, one can avoid the tight coupling of operating system and cloud, or to use the more common industry term ‘locked-in’. Cloud-init provides a mechanism for separating out configuration data from both the operating system and the execution environment so that you maintain the ability to change either at any time. It serves as a useful abstraction which ensures that the investments you make in your application stack on a specific operating system can be leveraged across multiple clouds.

Introduction

Cloud-init is the industry standard method for cross-platform cloud instance initialisation. It is supported by all major public cloud providers, provisioning systems for private cloud infrastructure, and bare-metal installation. It enables the automated provisioning of an operating system image (which is simply the collection of packages) into a fully running state, complete with required access and applications. Typical task handled by cloud-init include networking and storage initialisation, optional package installation and configuration, user account creation and security key provisioning for remote access. cloud-init can these task within the first boot of an instance. This prevents end users from needing to spend time producing and maintaining images that are out-of-date, require building, and maintenance. Instead cloud-init does run in most operating systems each time a new instance is launched, independently from image source. This level of configuration is provided by cloud-init across all major Linux distributions and FreeBSD. With current support provided in the following.

• Ubuntu
• SLES/openSUSE
• RHEL/CentOS
• Fedora
• Gentoo Linux
• Debian
• ArchLinux
• FreeBSD

Cloud-init provides support across a wide ranging list of execution environments:

Public Cloud Infrastracture (IaaS)

• Amazon Web Services
• Microsoft Azure
• Google Cloud Platform
• Oracle Cloud Infrastructure
• Softlayer
• Rackspace Public Cloud
• IBM Cloud
• Digital Ocean
• Bigstep
• Hetzner
• Joyent
• CloudSigma
• Alibaba Cloud
• OVH

Private Cloud Infrastracture

• Bare metal install
• OpenStack
• LXD
• KVM
• Metal-as-a-Service (MAAS)

Every cloud is different and each platform provides information in a different format. Cloud-init takes the work out by standardising the platform interactions and abstracting the cloud’s implementation away from the user. Cloud-init has hooks to various init systems to perform a four-stage initialisation, described later. These stages are specific to an instance running in a ‘cloud-like’ environment and during this time, items such as storage, networking, DNS names, user accounts, access keys, and software packages can be setup. Cloud-init is written in Python and as mentioned is packaged for all major distributions of Linux. It is dual licenced under the GPLv3 and Apache 2.0 licenses. The following document will present end users with details of how a cloud instance is initialised from a disk image and the steps cloud-init takes on the image during boot.

序章

Cloud-initは、クロスプラットフォームのクラウドインスタンス初期化のための業界標準の方法です。これは、すべての主要なパブリッククラウドプロバイダー、プライベートクラウドインフラストラクチャのプロビジョニングシステム、ベアメタルインストールでサポートされています。これにより、オペレーティング・システム・イメージ（パッケージの集合体）を、必要なアクセスとアプリケーションを完全に実行した状態に自動プロビジョニングすることができます。cloud-init が処理する代表的なタスクには、ネットワークとストレージの初期化、オプションのパッケージのインストールと設定、ユーザーアカウントの作成、リモートアクセスのためのセキュリティキーのプロビジョニングなどがあります。これにより、エンドユーザーは、古いイメージを作成したり、構築やメンテナンスが必要なイメージを作成したり、メンテナンスに時間を費やす必要がなくなります。その代わり、ほとんどのオペレーティング・システムでは、イメージ・ソースとは関係なく、新しいインスタンスが起動するたびに cloud-init が実行されます。このレベルの設定は、すべての主要な Linux ディストリビューションと FreeBSD で cloud-init によって提供されています。現在提供されているサポートでは、以下の通りです。

• Ubuntu
• SLES/openSUSE
• RHEL/CentOS
• Fedora
• Gentoo Linux
• Debian
• ArchLinux
• FreeBSD

Cloud-initは、幅広い実行環境に対応しています。

パブリッククラウドインフラストラクチャー(IaaS)

• Amazon Web Services
• Microsoft Azure
• Google Cloud Platform
• Oracle Cloud Infrastructure
• Softlayer
• Rackspace Public Cloud
• IBM Cloud
• Digital Ocean
• Bigstep
• Hetzner
• Joyent
• CloudSigma
• Alibaba Cloud
• OVH

プライベートクラウドのインフラストラクチャー

• Bare metal install
• OpenStack
• LXD
• KVM
• Metal-as-a-Service (MAAS)

クラウドごとに異なり、プラットフォームごとに異なる情報を提供する フォーマットを使用しています。Cloud-init は、プラットフォームのインタラクションを標準化することで作業を軽減します。また、クラウドの実装をユーザーから離れて抽象化しています。Cloud-init には 様々な init システムへのフックを使用して 4 段階の初期化を行います。の後になります。これらのステージは、「クラウドのような」環境で実行されているインスタンスに特有のものです。この間、ストレージ、ネットワーク、DNSなどのアイテムが 名前、ユーザーアカウント、アクセスキー、ソフトウェアパッケージを設定することができます。Cloud-initはPythonで書かれており、前述したようにすべての主要な Linux のディストリビューションで使用されています。これは GPLv3 と Apache 2.0 の二重ライセンスです。以下のドキュメントでは、エンドユーザにクラウドがどのようにして インスタンスはディスクイメージから初期化され、cloud-init のステップは イメージを起動します。

Instance Initialisation

When a user launches a new instance on a cloud, how does the instance get provisioned and configured to enable remote access by the user? Cloud-init is the solution to allow the user and cloud to initialise the instance.

An individual running system (or instance) on a cloud is typically composed of the following parts:

• Disk Image
• Metadata
• User data (optional)
• Vendor data (optional)

Disk Image

A disk image contains the operating system for the instance. This allows the user to quickly and easily pick a necessary image and start deploying, rather than needing to create and upload an image before use. Many operating systems offer images on cloud platforms with cloud-init pre-installed.

Below is an example of an Ubuntu Server 18.04 LTS disk image found on Amazon Web Services.

Metadata

Each cloud provider produces metadata, which is information to aid in setting up the instance. Depending on what data the cloud exposes the output can contain information about the disk image, storage, networking, users, and other information.

The source and format of the metadata varies from cloud to cloud. Cloud-init refers to the cloud’s metadata source as the datasource. Some clouds utilise a drive attached to the instance (e.g. Microsoft Azure and LXD) and other times it is an API endpoint (e.g. Amazon Web Services, Google Compute Engine, OpenStack) reached over the network.

A user can view the exposed metadata from the datasource after deployment using cloud-init’s query subcommand.

Amazon Web Services example on next page...

User Data

Data provided by end users to initialise the instance is called user data. This is the mechanism in which users interact with cloud-init. This data can range from simple shell scripts to more structured “cloud config” that utilise cloud-init’s built-in config modules. This type of data is completely optional and not required for an instance initialisation.

As an example, user data can be as simple as a basic shell script.

#!/bin/sh
sudo apt-get update
sudo apt-get install --yes pastebinit
cloud-init query userdata | pastebinit

For more advanced configuration, cloud-init also has built in config modules that enable setting up specific parts of system. These modules are used via a cloudconfig based user data and includes special configuration options to configure the following and more.

Customisation of these modules is available via “cloud config”. Cloud config is a YAML syntax file prefaced with ‘#cloud-config’ at the top of the file. The same shell script above written like cloud-config is shown below which demonstrates how instead of needing to provide numerous apt or dnf commands the user can provide a list of packages to install and cloud-init will install those packages.

#cloud-config
hostname: my_cloud_instance
packages:
    pastebinit
runcmd:
    cloud-init query userdata | pastebinit

Cloud-init’s cloud-config also provides the ability to utilise Jinja templating. The user can make use of the previously mentioned metadata in their own user data to make decisions on how to setup an instance. This allows for custom user data settings depending on the cloud, region, or even storage and networking configurations.

## template: jinja
#cloud-config
{% set NAME=’system-’ ~ v1.platform ~ ‘-’ ~ v1.region ~ ‘-’ ~ range(9) | random
%}
hostname: {{ NAME }}
{% if v1.cloud_name == ‘aws’ and v1.region == ‘us-east-2’ -%}
echo Installing custom packages
...
{%- endif %}

The mechanism to provide user data varies by cloud. However, most provide a method via the UI as well as via some API or CLI mechanism as well.

Below is an example found on Microsoft’s Azure Cloud Computing Platform launch screen.

Cloud provided command-line tools to launch instances have user data flags to pass a user data file during instances creation.

aws ec2 run-instances --image-id ami-abc1234 --count 1 \
    --instance-type m4.large --key-name keypair \
    --subnet-id subnet-abcd1234 --security-group-ids sg-abcd1234 \
    --user-data file://user_data.yaml

Finally, a user can view the passed in user data with cloud-init’s query subcommand.

$ cloud-init query userdata
#cloud-config
users:
- name: powersj
    ssh-import-id: lp:powersj
    shell: /bin/bash
    sudo: [‘ALL=(ALL) NOPASSWD:ALL’]

Vendor Data

Vendor data is the same as user data, however, it comes from the cloud platform vendor itself. User data directives always takes precedence over vendor data. The value of vendor data is it allows the cloud vendor to specify potentially platform or region specific values for instances. For example, setting up package mirrors, NTP servers, etc. Also note that vendor data is completely optional and not required for configuring an instance.

$ cloud-init query vendordata
#cloud-config
{}

The four stages of cloud-init

During the boot process cloud-init executes four stages. Each of these stages are setup to allow cloud-init to configure the instance from very early on to allow for network and storage configurations to occur before the rest of the system is configured.

The following will describe each stage and what configuration occurs during:

1. Init Local

The first stage runs as early as possible in order to allow cloud-init to generate network configuration and activate networking. As such, this is run as soon as the root filesystem is mounted read-write. Cloud-init will go and search for any local data source objects that may allow it to obtain metadata to configure the system. If any information is received cloud-init will proceed to setup networking devices, including bridges, bonds, vlans, etc.

2. Init

The second stage runs after networking is up and searches for any network data sources to find metadata for the system. This allows for custom storage configurations and expanding disk volumes to occur very early on. At this stage cloud-init will also make any changes it is requested to make to the block devices as well as setup file systems. Finally, the user-data for the system is obtained for use during the next stage.

3. Modules Config

The third stage follows immediately after the previous one to guaranteed networking and begins running the cloud-init config modules. Here SSH keys are imported, apt or yum is configured, and time related services are configured.

4. Modules Final

The final stage will then run at the very end of the boot process. This stage involves package install and configuration via user-scripts. Puppet and chef tools are setup and configured, and finally the final message module is run.

Summary

Cloud-init is an essential tool for cloud instance initialisation. It provides the ability to setup and configure instances across clouds and distributions. Instances can be configured with special storage and networking and with customised software configurations.