iTranslated by AI
Architecture of an OSS-based Home Cloud
Introduction
When it comes to the cloud, the three major providers—AWS, Azure, and GCP—are mainstream. However, since various feature-specific cloud-native OSS projects are currently being actively developed, combining them makes it possible to build a cloud environment at home with functions equivalent to the services provided by each vendor.
Until now, I have written articles about several components (mainly CNCF products), but it can be difficult to see how each component functions within the whole system based on individual articles alone.
Therefore, in this article, I will classify the components I use at home and those I have written about previously by function and provide an overview of their respective roles.
Classification Map by Function
Here is a diagram classifying the projects I use at home or have written about by function (including some not yet covered in articles).
While the functions (such as Provisioning and Monitoring) are set based on my own judgment, some parts refer to the classifications in the CNCF Landscape.
Conceptual diagram of components by function.
Lining up the logos of projects I've covered so far gives it that "professional" look.
Infrastructure
First of all, to do anything, you need an environment equipped with servers (physical PCs, virtual machines, etc.), networks, and storage to run components and perform work.
In my home environment, I configure virtual machines on OpenStack using physical PCs, build a Kubernetes cluster on top of them, and use a setup to test OSS for K8s.
OpenStack
OpenStack is a virtualization platform for building private cloud environments. By using OpenStack, you can build multiple hypervisor-based virtual machines on a physical PC as far as the specs allow.
In my home environment, I have built an All-in-One OpenStack on a Minisforum mini PC, where I spin up and tear down virtual machines for verification.
There are several ways to build OpenStack, but I think using kolla ansible, which installs docker-containerized OpenStack components via Ansible, is the easiest way.
Kubernetes
The well-known standard for container orchestration. Since it is famous, I will omit the explanation.
Provisioning
Provisioning here refers to the task of creating VMs and their associated resources on the aforementioned OpenStack. When performing various verifications and operations, it is desirable to isolate individual environments to minimize impact (as they can be quickly deleted and recreated when no longer needed, and recovery is tedious if the existing environment is affected). However, manually creating/deleting VMs and installing dependency tools every time is cumbersome, so the process needs to be automated.
Ansible and Terraform, which are widely used in the fields of Provisioning and IaC, mainly have the following characteristics:
- Resources and processes can be described declaratively.
- The same result is obtained no matter how many times the process is performed (idempotency).
- A wide range of modules is available, reducing the effort to create processes from scratch. Most tasks can be handled by standard processes or modules and plugins managed by the community.
While I could write scripts to perform the necessary processes myself, I use these tools in my home environment because the above properties make adding and managing processes easy.
Terraform
Terraform is a well-known product for achieving IaC. By using the Terraform Openstack provider, you can create resources such as virtual machines on OpenStack. This allows you to create virtual machines with pre-determined specs, flavors, floating IP addresses, and other properties, and delete them all at once when they are no longer needed.
Ansible
Ansible is effective for installing verification tools and configuring settings to improve debugging efficiency (such as shell completion and host name resolution) on virtual machines created for verification.
Since there are some inconveniences (though not impossible) in performing software or middleware provisioning with Terraform, I use them for different purposes: infrastructure construction with Terraform, and software/middleware setup and other general-purpose automation tasks with Ansible.
Repository
Repositories are used to store code, documents, and container images.
GitLab
A Git repository for storing source code and other files. Since it is famous, I will omit the explanation.
Gitea
A Go-based Git repository for storing source code and other files. Compared to GitLab, it is lightweight and has fewer internal components, making it easier to manage. If you don't need the extensive features of GitLab, using Gitea is also recommended.
Distribution
A private registry for storing container images. It was originally named "Docker Registry," but the name seems to have changed to "Distribution" when the governing body moved to the CNCF.
Since it can be built with Docker, I use this when I want to quickly set up and use a private registry.
Harbor
Harbor is also a private registry for storing container images. While its core image storage functionality is the same as Distribution, it comes with various features suitable for production environments, such as a web UI, access control, and integration with image vulnerability scanning tools. It can be used with Docker or Kubernetes.
CI/CD
Here, tools and platforms that can comprehensively perform integration, such as building and testing, and deployment to target environments are classified under CI/CD.
Concourse
Concourse is a cloud-native CI/CD platform that allows you to achieve CI/CD by describing processes to be executed on a command basis, similar to GitHub Actions or CircleCI.
Refer to the following articles for more about Concourse.
Tekton
Tekton is also a cloud-native CI/CD platform that allows you to achieve CI/CD by describing processes to be executed on a command basis.
Refer to the following article for more about Tekton.
Argo CD
Argo CD is also a familiar tool for achieving deployment based on GitOps principles in K8s.
Flagger
Regarding deployment to K8s, at least in a home environment, manual deployment of manifests or deployment using Helm, Argo CD, etc., is sufficient. However, if you want to try more advanced deployment methods such as Canary or Blue/Green deployment, Flagger can be used.
Refer to the following article for more about Flagger.
StackStorm
StackStorm is an event-driven workflow engine that performs various processes and integrations. Rather than being exclusively for CI/CD, it can be used for more general-purpose process automation. However, since image building, testing, and deployment are possible by building them in, it is classified under CI/CD here.
Refer to the following article for more about StackStorm.
GitLab CI/CD
GitLab also comes with CI/CD by default, which can be executed by placing a YAML for CI/CD in the repository. Since it can be used without additional setup if you prepare a runner, it is an effective method to use GitLab CI/CD when advanced CI/CD features are not required. Like GitHub, it has the advantage of integrating secrets, variable storage, and variables referenced in pipelines with the repository. On the other hand, the CD part requires some creativity.
Gitea Actions
Gitea is equipped with Gitea Act Runner, similar to GitHub Runner, which can execute CI/CD jobs described in YAML format. It also allows downloading and using public GitHub Actions. Since YAML can be written in the same format as GitHub Workflows, it is a strong option if you are already familiar with workflows or if you primarily use Gitea.
Observability
In cloud environments, since numerous workloads such as VMs and containers operate, monitoring is essential to ensure they work correctly.
Logging
Needless to say, log collection is important for various aspects such as root cause analysis during errors, troubleshooting, and event monitoring. Basically, the configuration involves installing a logging agent on VMs or K8s and managing them centrally on a separate log monitoring platform.
Logging agents also have many choices, but a previous article introduced a method using Filebeat, which is part of the Elastic stack.
Metrics
Metrics are also important for monitoring workload usage and state changes, such as CPU, memory, storage utilization, and network In/Out.
Metrics collection agents also have many choices, but currently, the method of collecting metrics exposed by various exporters using Prometheus seems to be widely used.
A previous article described a method for collecting K8s cluster metrics using Prometheus and Thanos.
Tracing
For Tracing, the configuration using OpenTelemetry for tracing data collection and Jaeger for visualization seems to be widely used. The article is below.
Profile
Profile in the field of Observability is also called Continuous Profiling and is utilized for continuously monitoring and analyzing application memory, CPU usage, and other information. While resource usage such as memory and CPU for the entire application can be collected with metrics, profiles allow for the collection of resources used down to the function or package level within the application. This is particularly useful for identifying bottleneck processes and addressing events where resource consumption worsens over time, especially when continuously improving an application.
There are various options for continuous profiling, and the following is a summary of an article about using Pyroscope, an OSS project managed by Grafana.
Other
Conventionally, the general configuration was to combine specialized software for each basic element of Observability, such as Logging and Metrics. However, in recent years, solutions that can comprehensively collect and monitor these elements have been increasing. For example, as mentioned earlier, development is underway for OpenTelemetry to allow for the collection and management of elements like Logging, Metrics, Tracing, and Profiling within OpenTelemetry alone.
Using a centralized platform instead of different software for each element offers the following expected benefits:
- Reduction in learning, implementation, and management costs for each software
- Integrated operation of data collection, processing, management, and analysis on a single platform
- Information search and troubleshooting may become easier as there is no need to cross different software
Of course, traditional software specialized in each field, such as Filebeat for Logging, has advantages such as a wealth of features, stability from long-term use, and accumulated expertise. Therefore, a centralized management solution is not always the best option in all cases, but it is still worth considering based on factors such as the number of users, the size of the community, and whether it is being actively developed.
Vector
The basics of Observability involve a flow of collection, processing (such as structuring, tagging, and noise removal), and output to a visualization platform for monitoring and analysis. Vector is an application that can centrally manage this series of processes as an Observability data pipeline.
While the intermediate data processing and transformation part is Vector's main area of responsibility, it supports input from 43 types of Sources like Prometheus and OpenTelemetry, and output to 59 types like Datadog and AWS CloudWatch. Thus, it has excellent integration with other systems and allows for building an end-to-end Observability foundation.
As of now, it supports Log and Metric, with Tracing not yet supported.
I wrote an article verifying a configuration where logs collected by Filebeat are processed by Vector and output to Elasticsearch and OpenObserve.
Data Storage
Since many elements within Observability do not store data for long periods by default, data storage is necessary to store collected data for long periods or for centralized management and reference.
That being said, for example, in the case of Filebeat, it's common to use Elasticsearch for data storage, and the standard data storage destinations for each component are usually well-established. Therefore, unless there is a specific reason, it is best to use those (as information searching is easier).
Elasticsearch
Since Filebeat for Logging and Jaeger for Tracing support sending to Elasticsearch by default, Elasticsearch is recommended as a data storage destination.
Elasticsearch itself has powerful search, analysis, and aggregation functions, and can be used for reference and visualization in cooperation with Kibana, mentioned later (though it's so feature-rich that it can be a challenge to master).
MinIO
Prometheus itself supports writing to various Remote storage, which includes Elasticsearch.
On the other hand, since Thanos stores data long-term in object storage, MinIO, an OSS object storage, can be used.
Also, since MinIO itself is an S3-compatible object storage, it is not specialized only for metrics and can be used as a general data storage destination.
Rook
Ceph is an OSS distributed storage system for building and managing large-scale storage clusters of thousands of units that operate in production environments.
Rook is a system that can be called the cloud-native version of this, allowing you to build a distributed storage system on K8s with Ceph as the storage provider.
Ceph Storage Provider
Rook orchestrates the Ceph storage solution, with a specialized Kubernetes Operator to automate management. Rook ensures that Ceph will run well on Kubernetes and simplify the deployment and management experience.
Using Rook makes the following major storage types available:
- Block storage
- Object storage
- Shared file system
Rook's status as a CNCF Graduated project indicates its high level of maturity.
Visualization
An important point when considering Observability is being able to refer to accumulated data in an appropriate format when needed.
Since analysis and visualization tools commonly used with specific types of data storage are mostly established, it is best to use those unless there is a specific reason otherwise.
Kibana
Kibana is often used in combination with Elasticsearch. Since Kibana itself is part of the Elastic Stack products, it has high affinity with Elasticsearch, allowing you to reference and analyze stored data under various conditions (though this one is also a challenge to master due to its abundance of features).
Grafana
Grafana is often used in combination with Prometheus (Thanos). Using PromQL to query Prometheus metrics, you can filter the necessary metrics and create your own dashboards to flexibly visualize only the information you need. In addition to Prometheus, it supports a wide range of data sources such as Relational DBs and Time Series DBs.
OpenObserve
OpenObserve is positioned as a lightweight version of Elasticsearch + Kibana, serving as a platform for storing and visualizing logs, metrics, and trace data. By narrowing its focus to these purposes, it can be used with lower learning and operational costs compared to the original Elasticsearch.
Network
Traefik
Traefik Proxy is a modern and feature-rich OSS proxy server. Like Nginx, it can perform proxying to other servers, load balancing, and user authentication, and it can also be used as an Ingress Controller when used with K8s. My personal impression is that it feels like the web server functions were removed from Nginx and replaced with YAML-based configuration to make it easier to manage. Additionally, it features TCP, UDP, and gRPC proxying as well as Load Balancing capabilities.
Refer to the following article for more about Traefik.
CoreDNS
CoreDNS provides DNS and Service Discovery functions.
When you create a K8s cluster with tools like kubeadm, it is used by default for DNS within the cluster, but it can also be built separately and used as a DNS server.
Linkerd
Linkerd is a proxy for achieving a Service Mesh within a K8s cluster.
While Istio is famous in the Service Mesh field, Linkerd is also a CNCF Graduated project, indicating the project's high maturity and large user base.
Security
Security is obviously important, as terms like DevSecOps suggest, but to be honest, if you don't expose your home environment to the network, you might not need to worry about it that much.
However, its importance remains unchanged, so if you are aiming for production-like operations, you need to pay attention to the security aspect.
Trivy
Trivy is an OSS tool for detecting vulnerabilities in container images and scanning for flaws in IaC and manifest files.
Refer to the following article for more about Trivy.
Tetragon
Tetragon is an OSS project for enhancing the security and observability of target environments using eBPF. By utilizing eBPF, it can perform container runtime security protection and apply unique policies, allowing you to enhance security in areas that the three major components might not fully cover. The article is below.
Notary Project
As container workloads have become mainstream, image-related security has increased in importance, such as whether a container image was created by a trusted issuer and whether the image has been tampered with by a third party. The Notary Project is a project for managing specifications and tools for artifact signing to ensure the security of OCI artifacts, including container images.
Ratify
Ratify is a tool for verifying signatures of OCI artifacts. It is mainly used on K8s clusters in conjunction with OPA Gatekeeper to verify OCI artifacts such as image signatures, vulnerability scan results, and SBOMs when a Pod is created. This allows for enhanced security by ensuring only trusted images are deployed to the K8s cluster.
Refer to the following article for verification of Ratify.
Development
Backstage
Backstage is an OSS framework for building developer portals to centrally manage software dependencies, code, infrastructure, and more.
As the number of tools to manage increases, understanding their dependencies becomes difficult, requiring developers to traverse numerous UIs and documents. Backstage consolidates such information into a web UI called a portal, providing an experience that allows development work to proceed more efficiently.
This field is classified as Platform Engineering. It offers benefits such as clarifying dependencies between various software, surrounding tool information, and management teams when multiple teams within an organization are developing multiple pieces of software. Additionally, since there are not many OSS options for building developer portals, its strengths include being vendor-independent and free to use, and allowing functional expansion by developing your own plugins.
Refer to the following article for more about Backstage.
Other
Items that do not fit into the above categories are classified as "Other."
Litmus
Litmus is a chaos engineering framework for intentionally injecting failures into K8s clusters to evaluate the availability and resilience of the current configuration.
While chaos engineering is rarely necessary in a home environment, it's quite interesting to try out operations like applying load to target Pods or bringing down nodes.
Also, since some companies actually incorporate chaos engineering in large-scale environments, it's worth trying to get a feel for that.
Refer to the following article for more about Litmus.
k8sGPT
k8sGPT is a service that uses public services or local LLMs to assist in troubleshooting and enhancing security within K8s clusters.
Although it is still a CNCF Sandbox project, services that use AI to manage K8s more efficiently are novel, and future developments are expected.
Identity Management
As the number of services increases, managing users, groups, and credentials becomes more challenging. For example, in the overview diagram, GitLab, Harbor, StackStorm, Elasticsearch, MinIO, and Grafana are services that manage user information internally. Tasks such as adding individual users to each application, setting roles and permissions according to user roles, reflecting new passwords when updating them, and deleting no longer needed users from each service's registration information become quite tedious processes.
Therefore, as the scale of services or organizations grows, the importance of applications for efficiently managing identities—such as users, their associated groups, and access restrictions—increases.
LDAP
LDAP is a protocol for efficiently querying users managed within a server. Applications or software with the function of centrally managing users and organizations using this are called LDAP servers. Like Active Directory, it is useful for achieving efficient operations by centrally managing users and integrating with related services based on user-specific information (identity).
There are several types of LDAP servers, each with different functions and features.
- OpenLDAP
- FreeIPA
- 389 Directory Server
- LLDAP
Refer to the following article for more about LLDAP.
Authelia
Authelia is an application with features such as an authentication/authorization server, access control, and single sign-on via OpenID Connect. By combining it with LDAP to manage users efficiently and integrating with each application via OpenID Connect, management overhead can be reduced.
Refer to the following article for more about Authelia.
HashiCorp Vault
HashiCorp Vault is a Secret Manager for storing various sensitive information, including user passwords. As services increase, strictly handled information such as administrator passwords, information used for inter-service authentication, certificates, and tokens also increases, but these can be centrally managed and protected.
Refer to the following article for more about Vault.
Multi-cluster Management
In cases of simple configurations or operations, a single cluster is often sufficient. However, when operating larger workloads, there is an increasing demand for handling multiple clusters—such as multi-cluster setups that run multiple K8s clusters together, multi-cloud setups that utilize clusters on various clouds, or hybrid clouds. While using multiple clusters offers many benefits such as scheduling optimization, disaster recovery, and improved availability, it also increases management complexity as the number of management targets grows. Therefore, tools for efficiently managing multiple clusters become important.
Major cloud providers have few features for managing multiple clusters since they assume the use of their own managed K8s services. In contrast, many OSS products support clusters across various environments—likely to avoid vendor lock-in—though EKS, GKE, AKS, and on-premises (bare-metal) clusters are the most common.
Rancher
Rancher is an integrated platform for managing multiple clusters, creating clusters, and monitoring resources.
It supports managed cluster services from major public clouds as well as bare-metal clusters, allowing for centralized tracking and management of cluster-wide resource usage and workload/network resource lists in a single platform. It is also integrated with other tools like kube-metrics-stack and Istio, making it easy to combine essential elements for multi-cluster operations, such as metrics monitoring and service mesh construction.
Refer to the following article for more about Rancher.
Karmada
Karmada is a K8s multi-cloud and multi-cluster management system. It can manage multiple K8s clusters independently of cloud providers and enables application deployment across clusters as well as inter-cluster failover.
Refer to the following article for more about Karmada.
Bonus
Correspondence with AWS Services
The following table shows the correspondence between the functions of the introduced OSS and AWS services.
Items that are functionally a bit ambiguous are marked with a ?.
| OSS | AWS Service | Function |
|---|---|---|
| Tekton Concourse |
CodeBuild | CI |
| ArgoCD Flagger |
CodeDeploy | CD |
| GitLab Gitea |
CodeCommit, etc | Code Repository |
| Harbor Distribution |
ECR | Container Registry |
| StackStorm | EventBridge, Step Functions, etc | Event-driven workflow |
| Kubernetes | EKS | Container orchestration |
| Rook | EBS S3 EFS |
Storage |
| MinIO | S3 | Object Storage |
| Elasticsearch, Kibana OpenObserve |
OpenSearch | Distributed search and analytics engine |
| OpenTelemetry | AWS Distro for OpenTelemetry | Trace |
| Tetragon | ? | eBPF |
| Pyroscope | CodeGuru Profiler? | Profiler |
| Ansible | Systems Manager? | Provisioning |
| Terraform | CloudFormation, etc | IaC |
| Trivy | ECR Image Scan, etc | Vulnerability Scanner |
| Litmus | Fault Injection Service | Chaos Engineering |
| LDAP | ? | Authentication and Authorization |
| Authelia | Cognito?, IAM? | Authentication and Authorization |
| HashiCorp Vault | Secrets Manager | Secrets Management |
| CoreDNS | Route 53 | DNS |
| Linkerd | App Mesh? | Service Mesh |
| Traefik | ELB? | Proxy, LoadBalancer |
| Vector | CloudWatch | Logging, Metrics |
Conclusion
I have categorized the components of my home environment and the projects I have written about by function, and organized their respective roles and positions. By looking at them from an overview, you can visually recognize which areas each component covers and which functions are missing.
While services from cloud vendors are basically pay-as-you-go, a home cloud is free no matter how much you use it (though there are electricity costs for the PC). If you want to try out CI/CD or Observability but are concerned about the cost, or if you want to understand how services work by getting your hands dirty, it might be worth considering building a home cloud environment.
Discussion