Incident Detection and Response in Google Cloud Platform (GCP)

Author Eric Evans

Photo by Nacho Rochon on Unsplash

Introduction

Detecting events and having the ability to react to these events in a rapid fashion is extremely important to bolstering security posture in the cloud. One of the benefits of having infrastructure in the cloud is having the ability to quickly develop and iterate on solutions, usually driven by event-driven architecture via the use of serverless functions that key off events of interest. These events of interest typically reside within the logs that are generated by your cloud service provider (CSP), ingested by a system that can detect patterns within these logs to look for indicators of compromise (IoC), and then finally having infrastructure in place that can react to the events for remediating any findings. In this article, we will explore the options that are natively available in Google Cloud Platform (GCP). Please note that there are third-party solutions to event detection and response in GCP, each with their own pros and cons.

Google Cloud Platform

In GCP terms, there is an entire offering called Operations (formerly known as Stackdriver) that is used to monitor your cloud environment by enhancing observability in the form of logs, metrics, alerts, and so on. This service is where a good detection strategy foundation starts with GCP. Once the data that is coming into the platform is optimized in the right areas, filters can be used to export detections wherever they may need to go (known as destination sinks). Cloud-native detection systems such as Security Health Analytics (SHA) and Event Threat Detection (ETD) can be used to bring protentional IoCs to light using these logs as a source of truth, and finally the use of Pub/Sub and Cloud Functions can be used to respond to findings generated by these tools.

Operations

As noted above, Operations is where a robust security foundation starts in Google Cloud. The main component of Operations that is useful for security practitioners in GCP is Cloud Logging. Through Cloud Logging, you can have a variety of sources to give information about events that happen within your cloud environment. These include:

• Data Access Logs – contain information about API calls that happen within Google Cloud. This includes any events that read, create, or modify data within the platform.

• Admin Activity Logs – contain information about API calls that performs administrative actions on resources within Google Cloud. This includes events that create or modify infrastructure.

• System Event Logs – contain information about administrative actions that modify your environment that happens via Google’s systems (not API calls).

• Agent Logs – contain logs that are forwarded to Cloud Logging via an installed agent. These are useful for custom or hybrid logging solutions that may not fit into the other categories

• Access Transparency Logs – contain information about actions that Google performs for support and/or troubleshooting purposes.

• VPC Flow Logs – contains information about network traffic within your Virtual Private Clouds (VPCs). Very useful for network monitoring, forensics, security analysis, and cost management.

• Firewall Logs – contains information about actions that happen because of a firewall within GCP. Useful for debugging firewall configurations and checking on allowed/denied traffic based on Firewall rules.

Security Health Analytics

Security Health Analytics (SHA) is a service that monitors your GCP environment for particular events of interest that may be of security concern. SHA uses Cloud Audit logs such as Admin Activity and Data Access logs to generate findings that are shown in Security Command Center (SCC). Some findings that SHA can find are:

• Users that don’t use two-step verification (2SV)

• Unrestricted GCP API keys

• API keys that are not rotated for more than 90 days

• Public Google Compute Engine (GCE) images

• Project-wide SSH keys that are in use

• Instances with public Internet Protocol (IP) addresses

• Weak SSL policies

• Container vulnerability findings

• DNS vulnerability findings

• Firewall vulnerability findings

• Identity and Access Management (IAM) and Key Management Service (KMS) findings

Event Threat Detection

Similar to SHA, Event Threat Detection (ETD) keys off of Cloud Logging to find events of interest. Instead of focusing on misconfigurations and vulnerabilities, ETD instead looks for threats that are currently happening in your GCP environment and can perform actions on them in near-real time. Some examples of what ETD can detect are:

• VMs connecting to a known bad domain for malware

• VMs connection to a known bad IP address for malware

• Cryptomining that is happening on resources in your environment

• Brute force SSH attacks on hosts in your environment

• Currently happening denial of service (DoS) attacks

• IAM anomalous grants, such as granting permissions to accounts that aren’t members of the organization

Security Command Center

Both SHA and ETD have integrations with SCC, which makes it a great place to centralize all detected findings. Findings from these services can be sent to a Pub/Sub topic to allow for real-time streaming events to be exported to your security operations center (SoC), or even better, have the events sent to a Cloud Function where automated remediations can occur.

Cloud Functions & Security Response Automation

Google Cloud Functions are very helpful when it comes to automating response to findings in GCP. These serverless functions contain code that can perform actions on your cloud environment in response to Pub/Sub notifications that can come from sources such as SCC (which has findings coming from ETD & SHA). Security Response Automation (SRA) is a set of open-source tools that Google provides to get you started on using Cloud Functions for automatic remediations.

Packet Mirroring

All of the options discussed so far does not perform deep packet (content) inspection on network traffic that occurs with GCP. As a best practice, it is recommended not to perform this type of inspection within cloud environments and instead rely on net-flow logging (such as VPC Flow Logs) to detect IoCs in the cloud. However, if for any reason (e.g. compliance or security) packet content inspection is necessary, GCP offers Packet Mirroring which clones network packets from Virtual Machines (VMs) in you VPC network and forwards it to a destination for inspection.

About the author: Eric Evans is a Senior Cloud Security Consultant at ScaleSec, where he enjoys innovating and making the cloud a safer place. Originally a software developer his passion is now focused on DevOps & CyberSecurity.