iTranslated by AI
Testing Cross-Region Failover with AWS FIS Network Connectivity Disruption Action
Looking back at the 2022 updates, I tried out the network connectivity disruption action in FIS.
What is AWS Fault Injection Simulator (FIS)?
A service that allows you to trigger simulated attacks and disruptions in an AWS environment.
It enables you to perform what is known as chaos engineering on AWS.
Network Connectivity Disruption Action
In October 2022, an update to FIS added the network connectivity disruption action.
This made it possible to simulate network failures.
This time, I used the above to test cross-region failover.
Architecture Diagram
Identical configurations are prepared in the Tokyo region and the Osaka region as a hot standby.
Using Route 53 failover routing, the primary points to the Tokyo region and the secondary points to the Osaka region.
The FIS network connectivity disruption action is executed against the subnet where the EC2 instances in the Tokyo region are located. The verification is successful if the DNS destination switches to the secondary Osaka region via Route 53 failover routing.
For clarity, I have configured them to display "ap-northeast-1" for the Tokyo side and "ap-northeast-3" for the Osaka side.
Steps to Execute FIS
① Creating an Experiment Template
Set up the actions and targets you want to trigger.
② Starting the Experiment
An experiment is started based on the created experiment template.
① Creating an Experiment Template
Target Settings
Set where the target will be.
| Resource Type | Resource ID |
|---|---|
| Which resource to target | Specify the target resource by ID |
Action Settings
Select the failure or anomaly you want to trigger. In this case, select "aws:network:disrupt-connectivity" to execute the network connectivity disruption action.
In addition, actions for API errors, EC2, and RDS are also supported.
Scope Settings
Select between which traffic endpoints the anomaly should occur.
Since we want to trigger Route 53 failover routing, select "ALL".
| ALL | Availability-zone | S3,DynamoDB |
|---|---|---|
| All traffic | Traffic between AZs | Traffic between services |
IAM Role Settings
You need to create an IAM role for FIS execution.
The required permissions vary depending on the action being performed. Required permissions are listed in the documentation.
Stop Condition
Experiments can be automatically stopped using pre-configured Amazon CloudWatch alarms, for example, if the service enters an abnormal state.
This allows for minimizing the impact on users if unexpected issues occur during the experiment.
This completes the experiment template.
② Starting the Experiment
Now that the experiment template is ready, let's start the experiment right away. Selecting the experiment template and clicking [Start experiment] begins the process.
After waiting a few minutes from the start...
First, I confirmed that the target site was no longer accessible.
Next, I confirmed that the Route 53 health check was failing on the Tokyo region side.
After waiting a little longer for DNS propagation, I accessed the same URL and confirmed that the screen display had switched, meaning the DNS destination had successfully changed to the Osaka region.
(Aside) EC2 Status Check Did Not Fail
Loss of network connectivity is included as a cause of EC2 system status check failure.
・The following are examples of problems that can cause system status check failures:
・Loss of network connectivity
・Loss of system power
・Software issues on the physical host
・Hardware issues on the physical host that affect network reachability
So I was watching to see if the EC2 status check on the Tokyo region side would fail, but there was no change even after starting the experiment.
I suspect that FIS network connectivity disruption actions do not affect communications on the AWS system side. Therefore, it seems that a complete AZ failure cannot be reproduced with FIS...
Impressions
This was my first time using FIS, and since I had an image that generating such simulated failures would require complex settings or agents, I was impressed that I could get it running without complex configurations.
Also, while this verification was a simple DR configuration, I felt that DR testing could be conducted in a production environment in a similar way.
Presentation Material
Discussion