I can't find my Cloud Armour logs despite enabling them 15 mins ago. Is that kind of delay normal?

Question

I'm trying to debug a waf rule. I've set it to preview in Cloud Armour and checked the policy is applied to all 3 of my services. I then went into Load Balancer and clicked edit (there's only one load balancer) then chose "Backend configuration" and clicked the pencil icon next to the service that runs my API. In the sidebar I checked "Enable logging" and set the rate to 1 (capture all events) and clicked Update to save my changes, then update in the page above for good measure. I then went back in to verify the changes had taken and they had.

Now it's been 15 minutes and there's no sign of any logs despite me doing several actions that hit my api in that project.

I've tried looking in the Cloud Console with the filter:

resource.type:(http_load_balancer) AND jsonPayload.enforcedSecurityPolicy.name:(owasp-policy)

I've tried looking for DENY outcomes in Cloud Shell:

gcloud logging read 'resource.type="http_load_balancer" AND jsonPayload.previewSecurityPolicy.outcome="DENY"' --format=json --limit=1

I've tried looking for ACCEPT outcomes too:

gcloud logging read 'resource.type="http_load_balancer" AND jsonPayload.previewSecurityPolicy.outcome="ACCEPT"' --format=json --limit=1

And I get nothing back. No results in the Console, and just an empty array [] in the Shell.

Is it normal for it to take so long for logs to start appearing or is something wrong? If so I would appreciate any pointers towards steps I might have missed, or troubleshooting techniques I could use to figure out why I'm getting no results.

update: It's been over half an hour now and this less specific query also returns nothing, which has to be wrong if load balancer logging is actually happening:

gcloud logging read 'resource.type="http_load_balancer"' --format=json --limit=1

:/

Answer 1

So we got logging enabled by adding the following to our load_balancer/main.tf Terraform file, under resource...

log_config {
    enable = true
}

Note we only did this for one of our dev envirionments, as all that logging might start to add up in prod.

:)