Motivation:
I was looking into ways where I can have a centralized auditing/accounting pipeline related to various GCP Cloud services .
This article provides one such pipeline that can be used to have a centralized mechanism for auditing of cloud logs using Opensource Tools.
Tools Used:
1) Cloud Router and Pub/Sub topic with Subscription to route various GCP services Cloud Logs.
2) Grafana Agent: Agent that is configured to have Pull Based Subscription with the Pub/Sub topic and push to Long term Logging tool.
3) Loki: A Long term Logging Tool where logs can be pushed and analyzed for Auditing purposes with the help of Grafana.
Data Flow:
