Forwarding logs from GCP storage bucket

The Google Cloud Platform (GCP) Cloud Storage acts as a scalable container that stores a large volume of data. Site24x7 uses the Google Cloud Function to gather new logs added in the Cloud Storage bucket and sends them to Site24x7 for indexing.

Define the Log Type

A Log type defines the specific format in which an application generates its logs. Since applications like IIS, Cassandra, Apache, or MySQL produce logs in varying formats, creating distinct log types helps organize them, making log access easier and searches more efficient. To create a log type, follow the below steps:

1. Log into Site24x7.
2. Navigate to Admin > AppLogs > Log Types > Add Log Type. You can also define a custom log type to group logs from various applications for easier access and more efficient searching.
3. If you choose Custom Log Type, enter a display name.
4. Provide the logs stored in your Cloud Storage bucket to identify the log pattern.
5. Click Save.
   
   

Once you define a Log Type for your logs stored in your GCP, list it under a Log Profile and start managing your logs by performing search queries.

Create a Log Profile

To collect logs from GCP Cloud Storage bucket into Site24x7, begin by setting up a Log Profile and configuring a Google Cloud function to route logs from your Cloud Storage bucket to Site24x7's log receiver. Follow the below steps to create a log profile:

1. Navigate to Admin > AppLogs > Log Profile > Add Log Profile.
2. Enter the following details:
   
   1. Profile Name: Enter a unique name to identify your Log Profile.
   2. Log Type: Choose the log type, the one that was created using the above steps.
   3. Log Source: Choose GCP Functions (used here as a placeholder to set up a generic log source).
   4. Timezone: Select the appropriate timezone for your logs.
3. Click Save to create the log profile.
   
   

Configure Google Cloud functions

1. Sign in to the Google Cloud Console.
2. Go to the hamburger icon on the top-left corner.
3. Click on Cloud Run > Services > Write a function.
   
   
4. On the Create service page that opens, set up the Cloud function:
   
   1. Choose Use an Inline editor to create a function.
   2. Provide your service name and select your Region.
   3. Choose Python 3.11 (the latest available) as the Runtime.
   4. Under Trigger, select Cloud Storage trigger that opens the Eventarc Trigger window.
      
      
   5. Choose the your object name created in the bucket in the Event type.
   6. Choose application/json in the Event data content type.
   7. Browse your bucket name and click Select to add it under the Bucket field to make the Google Cloud Storage bucket to subscribe to your newly created bucket.
   8. Choose the Compute Engine default service account as the Service account.
5. Click Save trigger.
   
   
   The trigger details are added successfully.
6. Under the Authentication section:
   1. Choose Required authentication and enable the Identify and Access Management (IAM) checkbox.
7. Click Create.
   
   

This setup will create a Eventarc trigger function that listens to incoming log messages from Cloud Storage trigger.

Deploy the function code

1. Once the Eventarc trigger is created, edit the cloud function (function code) in the main.py file.
   
   
2. Replace the main.py file with the code from the below link:
   https://github.com/site24x7/applogs-gcp-functions/blob/main/cloudbucket/main.py
   After replacing the code, change the Function entry point value to "process_gcs_event".
   
   
3. Replace requirements.txt file contents with the below content:
   https://github.com/site24x7/applogs-gcp-functions/blob/main/cloudbucket/requirements.txt
   
   
4. After entering the code, go to the Site24x7 web client. 
   
   1. Navigate to Admin > Applogs > Log Profile
   2. Select the created Log Profile, and copy the displayed code for the logTypeConfig.
      
      
5. Back in the Google Console:
   
   1. Navigate to the Configuration section of the created Cloud Function.
   2. Under Environment variables, add a new key-value pair:
      
      • Key: logTypeConfig
      • Value: Paste the logTypeConfig copied from the Site24x7 Log Profile page.  
6. Click Deploy.
   
   

Related article

• Log forwarding from Google Cloud Platform