iTranslated by AI
How to Preview and Apply Security Rules for Suspicious Access Logs in Google Cloud Armor
Cloud Armor is Google Cloud's managed WAF.
In this article, I will mainly explain the flow of blocking suspicious access. I will not cover Adaptive Protection, a highlight feature that recently reached GA, or Managed Protection, which is available for an additional fee.
Cloud Armor Architecture
The configuration is quite similar to AWS WAF, featuring:
- Blocking or allowing requests to the load balancer (Cloud Load Balancing)
- Creating rules to block or allow requests
- Additional capabilities available for an extra fee
These are the main features. Here, I will introduce the basic flow using the Standard tier pricing model.
WAF Challenges: Not knowing the impact until a rule is applied
Suppose you want to block a certain suspicious request. One challenge I have felt with Web Application Firewalls is that "you don't know how many requests will be blocked when you apply this block rule." While some unintentional blocking is inevitable to protect yourself from attacks, it directly harms the user experience (UX). It makes sense to change the conditions if you find unintentional blocks after applying, but this forces inconvenience on some users.
Preview Mode for Rules
One of the features of Cloud Armor is Preview Mode. This solves the aforementioned challenge; although the block rule is applied, it does not actually return a 403, but instead returns a 200 as normal. Instead, it logs to Cloud Logging, saying, "This is currently in preview mode, but it would be blocked under the current rule." By checking these logs, we can verify that the rules are applied within the intended scope before actually applying them. This is very helpful. Let's try it out.
Prerequisites: Backend Service is Created
Cloud Armor is associated with the backend service of an HTTP Load Balancer. First, build the application under the load balancer.
External HTTP(S) Load Balancing Architecture for Cloud Run Applications
Although not strictly required, it is recommended to enable log output for the backend service. You can determine whether a request was blocked by a Cloud Armor security policy from these backend service logs.
Because Google Cloud Armor logs are part of the Cloud Load Balancing logs, the log sampling rate configured for the load balancer is applied to the generation of Google Cloud Armor logs. If you decrease the sampling rate for HTTP(S) Load Balancing, TCP Proxy Load Balancing, or SSL Proxy Load Balancing, Google Cloud Armor request logs will be sampled at that rate.
https://cloud.google.com/armor/docs/request-logging?hl=en
Refer here for information on how to enable logging.
Below is an example of applying this via Terraform with a sampling rate of 0.1.
# Load balancer backend: Cloud Run api
resource "google_compute_backend_service" "cloudrun_api_backend" {
load_balancing_scheme = "EXTERNAL"
name = "${var.project_id}-cloudrun-api"
description = "Cloud Run User API"
port_name = "http"
protocol = "HTTP2"
session_affinity = "NONE"
timeout_sec = 30
connection_draining_timeout_sec = 0
security_policy = google_compute_security_policy.user_app_armor_policy.self_link
# Configure to view Cloud Armor detection logs
+ log_config {
+ enable = true
+ sample_rate = 0.1
+ }
backend {
group = google_compute_region_network_endpoint_group.cloudrun_api_neg.self_link
}
}
Once the backend service and logging are prepared, create the Cloud Armor security policy.
Creating Cloud Armor
In the Management Console, go to Cloud Armor > Create security policy.
- Under Apply policy to target, select the backend service you want to apply the policy to.
- Although we will not use Adaptive Protection in this example, it is a good idea to enable it as it can provide alerts for potential attacks even in the Standard tier.
Once the settings are finalized, create the policy.
The Blocking Flow
Now that we are prepared, let's follow the flow until the blocking is active.
Finding Suspicious Request Logs
While you might be tempted to ask, "Is this a manual process?", think of it as "The system automatically gathers various materials for judgment, but a human makes the final decision on whether to block the request." In most cases, you would likely block based on a combination of IP address and User-Agent. For example, suppose you found the following suspicious request in Cloud Logging:
{
"insertId": "62d0cf9200019f2b843c1093",
"httpRequest": {
"requestMethod": "GET",
"requestUrl": "https://example.com/articles/super-power?wesdf12fsef",
"requestSize": "406",
"status": 200,
"responseSize": "16321",
"userAgent": "curl/7.79.1",
"remoteIp": "3.5.140.1",
"serverIp": "142.250.207.14",
"latency": "0.067959112s",
"protocol": "HTTP/1.1"
},
"timestamp": "2022-07-15T02:23:14.106283Z",
"severity": "INFO",
"receiveTimestamp": "2022-07-15T02:23:14.155967694Z",
"spanId": "474634380232661585"
}
- IP Address:
"remoteIp": "3.5.140.1"[1] - User-Agent:
userAgent: "curl/7.79.1"
So, we will use this information to block the request. By the way, "serverIp": "142.250.207.14"[2] should be the load balancer's IP address.
Creating a Block Rule
Since we have identified the IP address and User-Agent, let's add a rule to the security policy.
For the "Condition", refer to the syntax and enter the following:
inIpRange(origin.ip, '3.5.140.1/32') && has(request.headers['user-agent']) && request.headers['user-agent'].contains('curl/7.79.1')
Furthermore, check the box to enable preview only.
By checking preview, the request will not actually be blocked, but instead, the system will log that it matches the criteria.
Checking Preview Logs
We have created a rule that matches our criteria. Now, let's try accessing it. It will not be blocked, but the following log will be output as a load balancer log (resource.type="http_load_balancer").
{
"insertId": "1s3h2rcg12f1i1q",
"jsonPayload": {
"enforcedSecurityPolicy": {
"name": "test-policy",
"outcome": "ACCEPT",
"priority": 2147483647,
"configuredAction": "ALLOW"
},
"statusDetails": "client_disconnected_after_partial_response",
+ "previewSecurityPolicy": {
+ "priority": 0,
+ "name": "test-policy",
+ "configuredAction": "DENY",
+ "outcome": "DENY"
+ },
"@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
"cacheId": "NRT-cf0517a3",
"remoteIp": "3.5.140.1"
},
"httpRequest": {
"requestMethod": "GET",
"requestSize": "61",
"status": 200,
"responseSize": "63560",
"userAgent": "curl/7.79.1",
"remoteIp": "3.5.140.1",
"cacheLookup": true,
"latency": "0.388584s"
},
"resource": {
"type": "http_load_balancer",
},
"timestamp": "2022-07-15T05:50:35.809583Z",
"severity": "INFO"
}
previewSecurityPolicy is the record I mentioned at the beginning, confirming that "this request is currently in preview mode but would be blocked under the active rules."
Checking for Unintended Blocks
Review the preview logs to ensure that no unintended, legitimate requests are blocked. You can list them using the following query in Cloud Logging:
resource.type="http_load_balancer"
jsonPayload.previewSecurityPolicy.configuredAction="DENY"
I especially recommend checking rules based solely on IP addresses or common browser User-Agents, as they might involve other users. If you can narrow it down further by referrers or request destinations, you can restrict the block scope more precisely.
Disabling Preview and Applying the Rule
Once you have verified the block target in preview, edit the rule to disable preview mode. Now, requests matching the conditions will be blocked (returning 403).
<!doctype html><meta charset="utf-8"><title>403</title>403 Forbidden
You will receive a response like this, indicating that it is being blocked.
Checking Block Logs
Run the following query in Cloud Logging to list logs where the rule was enforced or allowed.
resource.type="http_load_balancer"
jsonPayload.enforcedSecurityPolicy.configuredAction="DENY"
You will find request logs like the following, confirming that the block is working as intended.
{
"insertId": "1rtzymifc41of2",
"jsonPayload": {
"cacheId": "NRT-33e460bb",
"@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
+ "enforcedSecurityPolicy": {
+ "outcome": "DENY",
+ "name": "test-policy",
+ "priority": 0,
+ "configuredAction": "DENY"
+ },
"statusDetails": "denied_by_security_policy",
"remoteIp": "3.5.140.1"
},
"httpRequest": {
"requestMethod": "GET",
"requestSize": "61",
"status": 403,
"responseSize": "353",
"userAgent": "curl/7.79.1",
"remoteIp": "3.5.140.1",
"cacheLookup": true,
"latency": "0.248457s"
},
"resource": {
"type": "http_load_balancer"
},
"timestamp": "2022-07-15T06:46:54.337457Z",
"severity": "WARNING",
"receiveTimestamp": "2022-07-15T06:46:55.367075272Z",
"spanId": "64f80aac2648396f"
}
This completes the process of using Cloud Armor to block requests.
Conclusion
We tested the flow of creating, previewing, and applying Cloud Armor rules. It is helpful that the pricing is based on pay-as-you-go, which is kind to the wallet.
- WAF requests: $0.75 per 1 million requests
- WAF security policies: $5 per month per policy
- WAF rules: $1 per month per rule
In this example, with one security policy and one rule (I am unsure if the basic rule is included, sorry), the estimate is around 1,000 yen. I hope this helps someone.
References
https://qiita.com/pict3/items/7b21f034006f6ee18b77", "completed": true}
Discussion