Help Docs

Google Cloud Dataproc monitoring integration

Monitor your Google Cloud Dataproc resources to track job submissions and failures, monitor YARN resource allocation, and keep an eye on HDFS storage health. Google Cloud Dataproc monitor lets you maintain reliable big data processing clusters.

Benefits

Monitoring Dataproc ensures that your service remains reliable, secure, and high-performing. Key benefits include:

  • Identify job failures or resource exhaustion before they impact batch processing SLAs.
  • Track YARN and HDFS resource utilization to fine-tune cluster sizing.
  • Receive automated alerts and use IT automation to maintain SLA compliance and reduce downtime.
  • Consolidate Dataproc metrics with other Google Cloud Platform services for a single pane of glass view.
  • Rest assured that bulk configuration and agent upgrades ensure smooth monitoring across multiple clusters.

Use cases

Here are some use cases where Google Cloud Dataproc monitoring will be helpful:

  • Monitor running and failed job counts to detect processing issues in real time.
  • Capture HDFS unhealthy block trends to quickly identify storage layer problems.
  • Visualize YARN memory and container allocation to plan capacity and optimize cluster scaling.
  • Set thresholds for job completion time and node failures to ensure contractual obligations are met.
  • Track operation failure counts to strengthen cluster management reliability.
  • Integrate Dataproc monitoring with other Google Cloud Platform services (e.g., BigQuery, Cloud Storage, Vertex AI) for holistic visibility.

Setup and configuration

  • Adding Google Cloud Dataproc while configuring a new Google Cloud monitor

    If you have not configured a Google Cloud monitor yet, add one by following the steps below:

    1. Go to Cloud > GCP > Add GCP Monitor or Admin > Cloud Monitoring > Google Cloud Platform (GCP).
    2. Provide a unique display name for identification purposes.
    3. Upload the JSON file that contains the private key of the service account to authenticate and perform resource discovery.
    4. Select Dataproc from the Select the Resources for Monitoring list.
    5. Select existing Notification Profiles, User Alerts Groups, Tags, and IT Automation Templates or add new ones. You can also integrate alarms with your preferred third-party service.
    6. Click Start GCP Monitoring.
  • Adding Google Cloud Dataproc to an existing Google Cloud monitor

    If you already have a Google Cloud monitor configured for the service account, you can add Google Cloud Dataproc by following the steps below:

    1. Go to Cloud > GCP and select your Google Cloud Platform monitor.
    2. Click the hamburger Hamburger icon icon next to Service View and select Edit, which brings you to the Edit GCP Monitor page.
    3. On the Edit GCP Monitor page, select Dataproc from the Select the Resources for Monitoring list and click Save.
    4. After successful configuration, go to Cloud > GCP > Dataproc. Now you can view the discovered Google Cloud Dataproc resources.
Note

It will take approximately five minutes to discover new Google Cloud Platform resources.

Licensing

Each Google Cloud Dataproc monitor consumes one basic monitor license.

Polling frequency

Google Cloud Dataproc monitor collects minute-wise metric data, and the statuses of your Google Dataproc resources are reported every five minutes.

Supported metrics

Metric nameDescriptionStatisticUnit
Running Jobs The number of jobs currently running on the Dataproc cluster. Average Count
Failed Jobs The number of jobs that failed on the Dataproc cluster. Total Count
YARN Allocated Memory Percentage The percentage of total YARN memory currently allocated on the cluster. Average Percentage (%)
HDFS Storage Utilization The percentage of total HDFS storage capacity currently in use. Average Percentage (%)
Submitted Jobs The number of jobs submitted to the Dataproc cluster. Total Count
Job Completion Time The time taken for jobs to complete on the Dataproc cluster. Average Seconds
Job Duration The duration of jobs run on the Dataproc cluster. Average Seconds
Job State The current state of a Dataproc job. Average Count
YARN Applications The number of YARN applications running on the cluster, grouped by state. Average Count
YARN Containers The number of YARN containers currently allocated on the cluster. Average Count
YARN Memory Size The total amount of YARN memory available on the cluster. Average GiB
YARN Pending Memory Size The amount of YARN memory requested but not yet allocated. Average GiB
YARN Virtual Cores The number of YARN virtual CPU cores available on the cluster. Average Count
YARN NodeManagers The number of active YARN NodeManagers in the cluster. Average Count
HDFS Storage Capacity The total HDFS storage capacity available on the cluster. Average GiB
HDFS DataNodes The number of active HDFS DataNodes in the cluster. Average Count
HDFS Unhealthy Blocks The number of HDFS blocks reported as unhealthy, such as corrupt or missing blocks. Average Count
Running Nodes The number of nodes currently running in the Dataproc cluster. Average Count
Expected Nodes The number of nodes expected to be running based on the cluster's configuration. Average Count
Failed Nodes The number of nodes that failed in the Dataproc cluster. Total Count
Capacity Deviation The deviation between the cluster's expected and actual node capacity. Average Count
Batch Spark Executors The number of Spark executors running for a Dataproc serverless batch workload. Average Count
Operation Completion Time The time taken for cluster operations to complete. Average Seconds
Operation Duration The duration of operations performed on the Dataproc cluster. Average Seconds
Failed Operations The number of cluster operations that failed. Total Count
Running Operations The number of cluster operations currently running. Average Count
Submitted Operations The number of operations submitted for the Dataproc cluster. Total Count
Session Spark Executors The number of Spark executors running for a Dataproc serverless interactive session. Average Count

Threshold configuration

  • Global configuration
    1. In the web client, go to the Admin section on the left navigation pane.
    2. Select Configuration Profiles from the left pane, then select Threshold and Availability from the drop-down menu.
    3. Click Add Threshold Profile in the top-right corner.
    4. For Monitor Type, select Dataproc.
    5. Specify the threshold values for the required metrics, then click Save.
  • Monitor-level configuration
    1. In the web client, go to Cloud > GCP > Dataproc.
    2. Select a resource you would like to set a threshold for, then click the hamburger Hamburger icon icon.
    3. Select Edit, which directs you to the Edit Dataproc Monitor page.
    4. Specify the threshold values for the required metrics, then click Save.

IT Automation

IT Automation tools automatically resolve performance degradation issues. These tools react to events proactively rather than waiting for manual intervention. The IT Automation tools automate repetitive tasks and automatically remediate threshold breaches. The alarm engine continually evaluates system events for which thresholds are set and executes the mapped automation when there is a breach.

How to configure IT Automation for a monitor

Configuration Rules

Editing multiple monitors to associate different monitor groups or add a different tag can be a tedious process. With Configuration Rules, you can automate the configuration settings of your monitoring resources. Create custom rules to track configuration changes continuously and achieve the ideal configuration settings.

How to add Configuration Rules

Summary

The Summary tab will give you the performance data organized by time for the metrics listed above. To view the summary:

  1. Go to Cloud > GCP > Dataproc.
  2. Select a resource.
  3. Click the Summary tab.

Configuration Details

The Configuration Details tab provides details on the configurations of application instances. To get the configuration details:

  1. Go to Cloud > GCP > Dataproc.
  2. Select a resource.
  3. Click the Configuration Details tab.

Reports

Gain in-depth data about the various parameters of your monitored resources and accentuate your service performance using our insightful reports.

To view reports for a Google Dataproc resource:

  1. Go to the Reports section on the left navigation pane.
  2. Select Dataproc from the menu on the left.
  3. You can find the Availability Summary Report, Performance Report, and Inventory Report for one selected monitor. Or you can get the Summary Report, Availability Summary Report, Health Trend Report, and Performance Report for all the Google Dataproc monitors.

You can also get reports from the Summary tab of the Google Dataproc monitor:

  1. Click the Summary tab.
  2. Get the Availability Summary Report of the monitor by clicking Availability.
  3. You can also find the Performance Report of the monitor by clicking any chart title.

Related content

Was this document helpful?

Would you like to help us improve our documents? Tell us what you think we could do better.


We're sorry to hear that you're not satisfied with the document. We'd love to learn what we could do to improve the experience.


Thanks for taking the time to share your feedback. We'll use your feedback to improve our online help resources.

Shortlink has been copied!