Zenn
Open16

#BuildingEventDrivenMicroservices

読書メモ
読書会
技術書を英語で読む会
gkzgkz

Building Event-Driven Microservices
https://learning.oreilly.com/library/view/building-event-driven-microservices/9781492057888/

Preface

  1. Why Event-Driven Microservices
  2. Event-Driven Microservice Fundamentals
  3. Communication and Data Contracts
  4. Integrating Event-Driven Architectures with Existing Systems
  5. Event-Driven Processing Basics
  6. Deterministic Stream Processing
  7. Stateful Streaming
  8. Building Workflows with Microservices
  9. Microservices Using Function-as-a-Service
  10. Basic Producer and Consumer Microservices
  11. Heavyweight Framework Microservices
  12. Lightweight Framework Microservices
  13. Integrating Event-Driven and Request-Response Microservices
  14. Supportive Tooling
  15. Testing Event-Driven Microservices
  16. Deploying Event-Driven Microservices
  17. Conclusion
gkzgkz

Preface

mediumの変遷が人対人の関係性、社会構造に変化をもたらしたように、computer system architecturesもまた人対人との関係性に変化をもたらした

each new medium has fundamentally changed people’s relationship with computing
The medium of the asynchronously produced and consumed event has been fundamentally shifted by modern technology

Chap. 1 Why Event-Driven Microservices

  • Event-based communication itself is not new
  • The following perspectives are new and noteworthy points of view
- handling big data sets, at scale and in real time
- can be stateful or stateless, complex or simple
- might be implemented as long-running, standalone applications or executed as a function using Functions-as-a-Service
  • The following is the benefits of using event-driven microservices
- granularity/scalability/flexibility on tech and biz requirements/loosely coupling/CD support/high testability
gkzgkz

Chap. 2 Event-Driven Microservice Fundamentals

  • producer microservices v.s. consumer microservices
producer consumer
produce events to event streams consume from input event streams

Communication between event-driven microservices is completely asynchronous.

  • event brokers v.s. message brokers
event brokers message brokers
retains them for as long as the organization needs deletes events after acknowledgment
  • The microservice tax

includes the cost of managing, deploying, and operating the event broker, CMS, deployment pipelines, monitoring solutions, and logging services.

paid by the organization or by each microservices

gkzgkz

Chap.3 Communication and Data Contracts

  • convey

    • verb
    • meaning: transport or carry to a place.

  • misinterpret

    • veb
    • meaning: misunderstand, misconceive, misconstrue, misapprehend, mistake, misread, put a wrong

  • event is as follows

    • is a statement of fact

    • when combined with all the other events in a system, provides a complete history of what has happened.

  • adhere

    • verb
    • meaning: stick fast to (a surface or substance)

  • explict schemas v.s. implicit schemas as data contracts

explict schemas implicit schemas
be capable of enforcing data contracts
event format is prospective consumers data contracts are susceptible
give security and stability to both consumers and producers
gkzgkz

Chap.4 Integrating Event-Driven Architectures with Existing Systems

  • data liberation

    • Data liberation is the practice of allowing users to view and export the data that you have about them
      Source: What is Data Liberation? - Simplicable

    • Data liberation enforces two primary features of event-driven architecture: the single source of truth and the elimination of direct coupling between systems.

    • This data needs to be liberated from these legacy systems to enable other areas of the organization to compose new, decoupled products and services.

    • Three patterns

      • Query-based, Log-based, Table-based

  • double duty

    • noun
    • meaning: a twofold function or role.

  • spectrum

    • noun
    • meaning: used to classify something in terms of its position on a scale between two extreme points.

  • emit

    • verb
    • meaning: produce and discharge (something, especially gas or radiation).
      • Syn: discharge, release

  • disrupt

    • verb
    • meaning: interrupt (an event, activity, or process) by causing a disturbance or problem.

  • interrupt

    • verb
    • meaning: stop the continuous progress of (an activity or process)
gkzgkz

Chap. 5 Event-Driven Processing Basics

Basic idea

Event-driven microservices' process:

  • create a producer client and a consumer client

  • register itself with any necessary consumer groups

  • starts a loop to poll the consumer client for new events, processing them as they come in and emitting any required output events.

  • traverse

    • verb
    • meaning: move back and forth or sideways.

Common transformations:

  • Filter

    • emits zero/one

  • Map

    • Changes the key and/or value of the event, emitting exactly one event.

    • if you change the key, you may need to repartition to ensure data locality.

  • MapValue

    • Change only the value of the event, not the key.

    • Repartitioning will not be required.

  • Custom transforms

    • Apply custom logic, look up state, and even communicate with other systems synchronously.

Necessities to branch event streams or to merge streams

  • to branch event streams

    • to output it to a new stream based on the result
    • streamが増える

  • to merge streams

    • to output to a single output stream
    • from multiple input streams are consumed
    • streamが減る、以前あったものを結合する

See the details in Chap. 6

how to handle consuming and processing events from multiple input streams in a consistent and reproducible order.

Repartitioning

  • the act of producing a new event stream with one or more of the following properties:

    • Different partition count
    • Different event key
    • Different event partitioner

  • delegate

    • verb
    • meaning: entrust (a task or responsibility) to another person, typically one who is less senior than oneself.
      • Syn: assign, entrust, give, pass on

  • colocate

    • verb
    • meaning: place side by side or in a particular relation.

  • data localityについて深堀した

  • jsonやNoSQLはlocalityの考えが使われている

    • JSONの表現であれば、すべての関連情報は1カ所に集まっているので、一度のクエリだけで十分

    • リレーショナルの例でプロフィールをフェッチしようとすれば、複数のクエリを実行する

Source: 同上

gkzgkz

Chap. 6 Deterministic Stream Processing

  • 本でベストエフォートって書かれているの、初めてみたかもしれない。そんなことない？

    • The overarching goal of deterministic processing is that a microservice should produce the same output whether it is processing in real time or catching up to the present time.

    • 同じにすることは難しいと思っていたのでベストエフォートというスタンスそのものは同意。

  • 本書で出てくるウォーターマークとは

    • 著作権保護の文脈で使われるものとはまた違うものとは思っている。

    • 本書ではwatermarkについてこのように。

      • Watermarking is used to track the progress of event time through a processing topology and to declare that all data of a given event time (or earlier) has been processed. This is a common technique used by many of the leading stream-processing frameworks, such as Apache Spark, Apache Flink, Apache Samza, and Apache Beam.

    • ↓にちかいニュアンスかなと。

      Apache Beam が追跡するウォーターマークは、特定のウィンドウのすべてのデータがパイプラインに届くときを予測するためのシステムの概念です。

      Source: Apache Beamのプログラミング モデル

  • 本書で挙げられていた参考資料：A whitepaper from Google

  • fundamental issues in stream processingという観点での落とし所

    • you can never be sure that you have received all of the events.

    • how much state to store

  • fault

    • noun
    • meaning: .responsibility for an accident or misfortune.

  • watermark

    • noun
    • meaning: A watermark is an identifying image or pattern in paper that appears as various shades of lightness/darkness when viewed by transmitted light (or when viewed by reflected light, atop a dark background), caused by thickness or density variations in the paper.

  • concise

    • adj
    • meaning: giving a lot of information clearly and in a few words; brief but comprehensive.
    • Syn: succinct, short, brief

Two processing states

typical of long-running microservices common for underscaled and new services
state at near–real time from the past in an effort to catch up to the present time

best-effort determinism

  • consistent timestamps

  • well-selected event keys

  • partition assignment

  • event scheduling

  • strategies to handle late-arriving events.

  • reconcile

    • verb
    • meaning: restore friendly relations between.

  • Event time

    • assigned to the even by the producer at the time the event occurred.

  • Broker ingestion time

    • assigned to the even by the event broker.

  • Consumer ingestion time

    • ingested by the consumer

  • Processing time

    • processed by the consumer

分散システムでは「どちらのイベントが先に発生したか」を断定するのが不可能なことがある

Source: 『Time, Clocks, and the Ordering of Events in a Distributed System』の要約

stream times, Kafka Streams

Out-of-Order and Late-Arriving Events

  • Microserviceごとに「遅れているとみなす基準が違う」
    -> One microservice may consider any out-of-order events as late, whereas another may be fairly tolerant and require many hours of wall-clock or event time to pass before considering an event to be late

  • 原因

    • out-of-order data

      • when data is consumed from a stream that is already out of order or when events are being sourced from an external system with existing out-of-order timestamps.
        　 - > Multiple producers writing to multiple output partitions

      • e.g. Repartitioning

Windowing

  • grouping events together by time.
  • particularly useful for events with the same key, where you want to see what happened with events of that key in that period of time.
Tumbling Sliding Session
a fixed size a fixed window size and incremental step not-fized
gkzgkz

Chap. 7 Stateful Streaming

how to build, manage, and use state for event-driven microservices.

Materialized state/State store

  • Materialized state

    • immutable

    • enable you to use common business entities in your microservice applications

  • State store

    • mmutable

    • enable you to store business state and intermediate computations.

  • materialize

    • verb
    • meaning: become actual fact; happen.
      • e.g.: "the forecast rate of increase did not materialize"
      • Syn: happen, occur, come about take place

where the service will store its data

Internally, such that the data is stored in the same container as the processor, allocated in memory or on disk
Externally, such that the data is stored outside of the processor’s container, in some form of external storage service. This is often done across a network.

  • allowcation

    • noun
    • meaning: the action or process of allocating or sharing out something.
      • e.g.: "more efficient allocation of resources"
      • Syn: allotment, assignment, issuing, issuance, awarding, grant

  • duality

  • noun

  • meaning: : the quality or state of having two different or opposite parts or elements

RockDB について深堀

  • また、LevelDBのシングルスレッドの圧縮プロセスが特定のサーバワークロードに対して適切に機能しないのに対し、RocksDBはそうしたI/OバウンドワークロードでLevelDBを上回る性能を発揮できるという。

  • 用途としては、閲覧履歴やユーザーの状態を保存するWebサイトのアプリケーション、スパム検出アプリケーション、グラフ検索クエリ、Hadoopへのリアルタイムクエリが必要なアプリなど、低遅延のデータベースアクセスが必要なアプリケーションを挙げている。Facebookでは現在、各種アプリケーションのペタバイトに近いデータをRocksDBで管理しているという。

Source: Facebook、Key-Valueストア「RocksDB」をオープンソース化：C++ライブラリとして構築 - ＠IT

  • deterministic
    • adj
    • meaning: relating to the philosophical doctrine that all events, including human action, are ultimately determined by causes regarded as external to the will.

Pros/Cons of Internal State

  • Pros:

    • Scalability requirements are offloaded from the developer

    • Physically attached local disk can be quite performant for the majority of modern microservice use cases.

    • Flexibility to use network-attached disk

  • Cons:

    • Limited to using runtime-defined disk

    • Wasted disk space

    • The new or recovered instance needs to materialize any state defined by its topology before it can begin processing new events. The quickest way to do so is to reload the changelog topic for each stateful store materialized in the application.

Pros/Cons of External State

  • Pros:

    • Full data locality

  • Cons:

    • Performance loss due to network latency

    • Financial cost of external state store services

    • Full data locality

Disadvantages of Using Internal State | Limited to using runtime-defined disk

In addition, many compute resource management solutions allow only for volume size to be increased, as decreasing a volume’s size means that data would need to be deleted.

cf. EC2にアタッチしているEBSボリュームのサイズは縮小できますか？ – 株式会社サーバーワークス サポートページ

Apache Kafka has this functionality built into its Streams framework via a simple configuration setting. This setting provides highly available state stores and enables the microservice to tolerate instance failures with zero downtime.

  • num.standby.replicas 深堀

The number of standby replicas. Standby replicas are shadow copies of local state stores. Kafka Streams attempts to create the specified number of replicas per store and keep them up to date as long as there are enough instances running. Standby replicas are used to minimize the latency of task failover. A task that was previously running on a failed instance is preferred to restart on an instance that has standby replicas so that the local state store restoration process from its changelog can be minimized. Details about how Kafka Streams makes use of the standby replicas to minimize the cost of resuming tasks on failover can be found in the State section.

Apache Kafka

Q. （ Each stateful partitionが）Materialized、実体化されるとは、「何がどうなる」か？
↓の2つのmaterializedは別のニュアンス？後者のとは、 Materialized state になることか？
冒頭

Materialized state
A projection of events from the source event stream (immutable)

Using hot replicas

Figure 7-5 shows a three-instance deployment with an internal state store replication factor of 2. Each stateful partition is materialized twice, once as the leader and once as a replica.

gkzgkz

Chap. 8 Building Workflows with Microservices

各サービスに何をすべきかを告げるようなオーケストレータは必要ありません。対照的に，すべてのコンポーネントがイベントを発行し，他のコンポーネントはそれに対応することができます。
https://www.infoq.com/jp/articles/monitor-workflow-collaborating-microservices/

  • exacerbate

    • verb
    • meaning: make (a problem, bad situation, or negative feeling) worse.
      • e.g.: "the exorbitant cost of land in urban areas only exacerbated the problem"
      • Syn: aggravate, make worse, worsen, inflame, compound, intensify

  • brittle

    • adj
    • meaning: 1. delicate and easily broken: 2. unkind and unpleasant: 3. delicate and easily broken

  • mitigate

    • adj
    • meaning: make (something bad) less severe, serious, or painful.
      • Syn: alleviate, reduce, diminish, lessen, weaken, lighten, attenuate

  • discern

    • verb
    • meaning: recognize or find out.
    • Syn: perceive, make out, pick out, detect, recognize, notice

  • resilent

    • adj
    • meaning: (of a person or animal) able to withstand or recover quickly from difficult conditions.
      • e.g.: "babies are generally far more resilient than new parents realize"
      • Syn: strong, tough, hardy

  • meanwhile

    • adv
    • meaning: in the intervening period of time.
      • e.g.: "meanwhile, I will give you a prescription for some pills"
      • Syn: for now, for the moment, for the present, for the time being

  • meaning: on the other hand.

    • e.g.: "he has said little, meanwhile, about how he plans to live his life"

choreography v.s. orchestration

In choreography, each service calls other services as needed, without a central mediator.

Figure 17-7. Using choreography in microservices to manage coordination

In Figure 17-8, the developers create a service whose sole responsibility is coordinating the call to get all information for a particular customer. T

Figure 17-8. Using orchestration in microservices

Source: 17. Microservices Architecture - Fundamentals of Software Architecture

どのサービスを呼び出すかを、アプリケーション側は知らなくてよいので、同じように新規会員登録のイベントを作成しパブリッシュするだけです。

マイクロサービスアーキテクチャにおけるオーケストレーションとコレオグラフィ - Qiita

  • subordinate
    • adj
    • maeniang: placed in or occupying a lower class, rank, or position
      • e..g: inferior a subordinate officer.
  • encapsulation
    • noun
    • meaning: the action of enclosing something in or as if in a capsule.

重要なのは、FaaSは自然とコレオグラフィを指向(略)マネージドサービス間を非同期イベントで連携して接続するアーキテクチャなので、これ以外に作りようがない

伊藤直也氏が語る、サーバーレスアーキテクチャの性質を解剖する（中編）。QCon Tokyo 2016 － Publickey

エラー特定のためにログは中央に集約。

AWS Lambda を利⽤してマイクロサービスを実装している場合は、AWS Lambda がサポートする
各開発⾔語で標準出⼒に対してログを送信すると Amazon CloudWatch Logs にログ情報が送られ
る

https://pages.awscloud.com/rs/112-TZM-766/images/wp_mad_201906.pdf

gkzgkz

Chap. 9 Microservices Using Function-as-a-Service

  • 要確認

    • Committing offsets only after processing has completed for a given event or batch of events is a FaaS best practice. It is important for you to understand how offsets for your specific function-based solutions are handled, so let’s take a look at the implications of each approach.

    • オフセットとは、Record番号のことであり、ConsumerがRecordをどこまで読みだしたのかを示す

    • Consumerはアプリケーションの再起動時に、自身のOffsetから再開できるようOffsetを永続化する必要があります。そのため、Consumerは自身のOffsetをKafka上のOffset用Topicや、任意のデータストアに保存します。

再利用のしすぎは要注意

An often-cited feature of FaaS frameworks is that they make it easy to write a single function and reuse it in multiple services.

ひとつのFaaSサービス（lambdaひとつ）にひとつの機能を

FaaS solutions may incorporate multiple functions to solve the business requirements of the bounded context, and while this is not an uncommon or bad practice, a good rule of thumb for FaaS solutions is that fewer functions are better than many granular functions. Testing, debugging, and managing just one function is much easier than doing the same for multiple functions.

Cold Start and Warm Starts

currently do not support triggering directly from open source brokers.ということは？？？

  • FaaS solutions from Google, Microsoft, and Amazon provide this trigger for usage with their proprietary event brokers, but currently do not support triggering directly from open source brokers.

  • Figure 9-1. Integrated event-stream listener with FaaS framework

  • Figure 9-2. External event-stream listener application provided by Kafka Connect

Triggering Based on Consumer Group Lag

Triggering

  • Based on Consumer Group Lag
  • on a Schedule
  • Using Webhooks
  • on Resource Events

Maintaining Stateここから読むのが辛くなってきた。。

  • albeit

    • conjunction
    • meaning: though.
      • e.g.: "he was making progress, albeit rather slowly"

  • comprise

    • verb
    • meaning: consist of; be made up of.
      • Syn: contian, take in

  • granularity

    • noun
    • meaning: the scale or level of detail in a set of data.

  • blur

    • verb
    • meaning: make or become unclear or less distinct.
      • e.g.: "tears blurred her vision"
      • Syn: unfocus, soften, obscure, dim

  • a rule of thub

    • meaning: A rule of thumb is a guideline, idea, or principle that helps you make decisions.
      • e.g.: "Arrive early" is a good rule of thumb for most appointments.
gkzgkz

Chap. 10 Basic Producer and Consumer Microservices

Basic producer and consumer (BPC) microservices

  • usecases are simple patterns such as stateless transformations

  • External state stores are more commonly used

  • sidecar pattern

  • The BPC is also a suitable approach when the underlying data layer performs most of the business logic, such as a geospatial data store; free-text search; and machine learning, AI, and neural network applications

gkzgkz

Chap. 11 Heavyweight Framework Microservices

Google Dataflowすごそう。。

  • This chapter covers
    • aspects of Apache Spark, Apache Flink, Apache Storm, Apache Heron, and the Apache Beam model

Some defining characteristics of a heavyweight streaming framework

  • 1st
    • requires an independent cluster of processing resources to perform its operations.
  • 2nd
    • uses its own internal mechanisms for handling failures, recovery, resource allocation, task distribution, data storage, communication, and coordination between processing instances and tasks.

A Brief History of Heavyweight Frameworks

  • MapReduce -> Storm/Heron or Spark/Flink
  • MapReduce
    • was powerful, it executes slowly in comparison to many of today’s options.
    • As these data sets have grown so has the demand for faster processing, more powerful options, simpler execution options, and solutions that can provide near-real-time stream-processing capabilities.

Figure 11-1. A generic view of a heavyweight stream-processing framework

  • Zookeeper provides highly reliable distributed coordination and is used to determine which master node is in charge (as it is not uncommon for nodes to fail, be they workers, masters, or Zookeeper nodes). Upon failure of a master node, Zookeeper helps decide which of the remaining masters is the new leader to ensure continuity of operations.

The heavyweight frameworks' usecases

  • Extract data, transform it, and load it into a new data store (ETL)
  • Perform session- and window-based analysis
  • Detect abnormal patterns of behaviorAggregate streams and maintain state
  • Perform any sort of stateless streaming operations

The heavyweight frameworks' shortcomings

  • not originally designed with microservice-style deployment in mind
  • most of these frameworks are JVM (Java Virtual Machine)-based, which limits the implementation languages you can use to create singular microservice applications.
  • materializing an entity stream into an indefinitely retained table is not supported out of the box by all frameworks.

Figure 11-3. Single job deployed on and managed by Kubernetes cluster

How to overcome the potential risks of internal staets

  • the potential risks of internal staets
    • carry some risks, such as state loss due to disk failure, node failures, and temporary state outages due to aggressive scaling by the CMS.
  • How to overcome them
    • use of checkpoints, which are snapshots of the application’s current internal state, and are used to rebuild state after scaling or node failures.

Google Dataflow

  • Google Dataflow, which executes applications written with the Beam API, provides built-in scaling of both resources and worker instances. Heron provides a (currently experimental) Health Manager that can make a topology dynamic and self-regulating. This feature is still under development but is meant to enable real-time, stateful scaling of topologies.

Autscaling Applications

  • Some frameworks may have autoscaling options built in, such as Google’s Dataflow engine, Heron’s Health Manager, and Spark Streaming’s dynamic allocation functionality.

  • 14章に書いてあり

  • Others may require you to collect your own performance and resource utilization metrics and wire them up to the scaling mechanism of your framework, such as the lag monitor tooling discussed in “Consumer Offset Lag Monitoring”.

  • aforementioned

    • adj
    • meaning: mentioned earlier:

  • shortcoming

    • adj
    • meaning: a fault or failure to meet a certain standard, typically in a person's character, a plan, or a system.
      • Syn: defect, fault, flaw, imperfection, deficiency, limitation, blemish

  • preclude

    • verb
    • meaning: prevent from happening; make impossible.
      • Syn: prevent

  • dedicate v.s. devote

gkzgkz

Chap. 12 Lightweight Framework Microservices

Benefits

  • applications can be dynamically scaled as they are under execution.
  • There is no need to restart an application just to change parallelism, though there may be a delay in processing due to consumer group rebalancing and rematerialization of state from the changelog.

Considerations of scaling a lightweight application

  • Event Shuffling
  • State Assignment
  • State Replication and Hot Replicas

Frameworks

  • Apache Kafka Streams
  • Apache Samza: Embedded Mode
gkzgkz

Chapter 13. Integrating Event-Driven and Request-Response Microservices

13-8, 13-9の2つの結合のカタチがあることがなるほどというかんじ。

  • Figure 13-8. An all-in-one microservice serving from external state store; note that either instance could serve the request

    • doesn’t require much in the way of deployment coordination. This is a single all-in-one microservice that can continue to serve state from the external state store regardless of the current instance count.

  • Figure 13-9. A microservice composed of separate executables—one for serving requests, the other for processing events

    • the request-response API is fully independent of the event processor, you can choose the implementation languages and scaling needs independently.

    • isolates any failures in the event processing logic from the request-response handling application.

    • 注意点

      • While this pattern has two microservices operating on a single data store, there’s still just a single bounded context. These two microservices are treated as a single composite service. They reside within the same code repository and are tested, built, and deployed together.

  • デメリット

    • Coordinating changes between two otherwise independent applications is risky, as altering the data structures, topologies, and request patterns can necessitate dependent changes in both services.

  • Request-response endpoints provide the means to serve important data in real time, particularly when you are:

Two main types of externally generated events to consider.

Serving Real-Time Requests with External State Stores

Micro-Frontends in Request-Response Applications

  • cease

    • verb
    • meaning: come or bring to an end.
      • Syn: come to an end

  • await

  • precedence

    • noun
    • meaning: the condition of being considered more important than someone or something else; priority in importance, order, or rank.

  • predominantly

    • adv
      • meaning: mainly; for the most part.

  • autonomous

    • noun
    • meaning: a person or entity that is self-controlling and not governed by outside forces.

  • adhere

    • verb
    • meaning: stick fast to (a surface or substance).
    • Syn: stick, stick fast, cling, hold fast, cohere, bond, attach
gkzgkz

Chapter 14. Supportive Tooling

  • By following the single writer principle (see “Microservice Single Writer Principle”), you can attribute event stream ownership back to the microservice that owns the write permissions.

    • Chapter 2. has Microservice Single Writer Principle
    • Chapter 9.

      • Triggering Based on Consumer Group Lag

One useful technique for assigning ownership

  • to tag streams with metadata

  • Examples:

    • Stream owner (service), Personally identifiable information (PII), Financial information, Namespace, Deprecation, Custom tags
    • Usecase of deprecation

      • The new events can be put into the new stream, while the old stream is maintained until dependent microservices can be migrated over. Finally, the deprecated event stream owner can be notified when there are no more registered consumers of the deprecated stream, at which point it may be safely deleted.

  • Figure 14-1. Schema registry workflow for producing and consuming an event

Permissions and Access Control Lists for Event Streams

  • WARNING

    • you enable and enforce identification for your event broker and services as soon as possible,

    • Adding identification after the fact is extremely painful, as it requires updating and reviewing every single service that connects to the event broker.

  • Table 14-1. Typical event stream permissions for a given microservice

Cross-Cluster Event Data Replication

  • When selecting a replication tool implementation, consider the following:

    • Does it replicate newly added event streams automatically?

    • How does it handle the replication of deleted or modified event streams?

    • Is the data replicated exactly, with the same offsets, partitions, and timestamps, or approximately?

    • What is the latency in replication? Is it acceptable to the business needs?

    • What are the performance characteristics? Can it scale according to business needs?

  • dearth

    • noun
    • meaning: lack, scarcity, scarceness, shortage, shortfall, want, deficiency, insufficiency, inadequac
      y

  • deprecation

    • noun
    • meaning: In several fields, deprecation is the discouragement of use of some terminology, feature, design, or practice, typically because it has been superseded or is no longer considered efficient or safe, without completely removing it or prohibiting its use.

  • quota

    • noun
    • meaning: a fixed share of something that a person or group is entitled to receive or is bound to contribute.
      • Syn: allocation

  • hysteresis

    • noun

    • Hysteresis is the dependence of the state of a system on its history. For example, a magnet may have more than one possible magnetic moment in a given magnetic field, depending on how the field changed in the past.

    • Source: Hysteresis - Wikipedia
gkzgkz

Chapter 15. Testing Event-Driven Microservices

基本的なテストの設計戦略は各サービスは疎結合とし、サービスが外部APIを使う場合はmockを使うことであるとすれば、Stateful Functionsのようなものを使ってテストするときは？テストケースに漏れがないようにどうするか？という気持ちで読み進めていた。

Populating with events from production

  • DevOps 技術: テストデータの管理  |  Google Cloud

    • テストデータの管理は困難な場合があります。テストデータの管理に関する一般的な問題には、次のようなものがあります。

      • テストでのデータへの過度の依存。特に、単体テストではテスト外部のデータや状態に依存しないようにしてください。

      • テストに関連する部分や重要な部分を特定するのではなく、本番環境データベース全体のコピーを使用する。

      • 機密データのマスキングやハッシュ化を行わない。

      • 古いデータや関連性がなくなったデータに依存する。

    • テストデータ管理の改善方法

  • A great thing of testing event-driven microservices is they are very modular.

General Testing Principles

  • Event-driven microservices share the testing best practices that are common to all applications.

  • Functional testing, such as unit, integration, system, and regression testing,

Stateful Functions

  • Stateful unit testing also requires that persistent state, whether in the form of a mocked external data store or a temporary internal data store, is available for the duration of the test.

  • Mocking the endpoint is one way of providing a reliable implementation of the data store for the duration of the test. Another option is creating a locally available version of the data store, though this is more akin to integration testing, which will be covered in more detail shortly.

The topology testing frameworks

  • time-based aggregations, event scheduling, and stateful functions perform as expected.

  • What data do you need to determine success or failure? Is manually crafted event data sufficient? Programmatically created? Does it need to be real production data? If so, how much?

How to prepare remote testing environment

  • By populating with events from production
  • By populating with events from a curated testing source
  • By creating mock events using schemas
  • By testing using a shared environment
  • Testing using the production environment

Summay

  • The tradeoff is the increased difficulty in managing the event data, ensuring validity, and clarifying ownership.