Real-Time Cost Anomaly Detection Now Available for Google BigQuery On-Demand & Reservations

We’re excited to roll out a major upgrade to how DoiT helps you stay in control of your Google Cloud spend: real-time anomaly detection for Google BigQuery, including both on-demand and reservation workloads. Once enabled, you’ll receive alerts about unexpected BigQuery cost spikes in under an hour.

What’s new

Until now, anomaly insights for BigQuery relied on next-day billing-file ingestion. That meant if a bad query ran at 2 PM today, you wouldn’t know until tomorrow.

With our new real-time detection pipeline for BigQuery on-demand, we continuously ingest and analyze live BigQuery usage metadata, flag unusual usage patterns, and send you Slack or email alerts in less than an hour-not the next day.

Why it matters

Real-time detection for BigQuery means that you can:

• Catch runaway queries in minutes: Prevent accidental or inefficient queries from racking up costs before anyone notices.
• Protect against operational mistakes: Get alerted when abnormal query activity starts impacting spend.
• Strengthen your security posture: Real-time cost changes can signal unauthorized data access or compromised systems.

As DoiT’s BigQuery expert Sayle Matthews can attest, the risk of unchecked queries racking up significant costs in a short period of time is very high:

“One of the largest issues is seeing how much their bill is at any given moment and being able to alert them when a ‘runaway query’ hits. We have seen some examples where customers have single queries that cost $2,000 USD and run in less than a minute, and of course, these were run multiple times in quick succession. These mistakes lead to massive bills that aren't caught for days or weeks later.”

Watch this video for a more detailed breakdown of one such example of a runaway query that could have cost a customer thousands in unnecessary charges.

What you need to do

This feature is available for all DoiT customers with a paid Enhanced, Premium, or Enterprise subscription and a connected Google Cloud account. 

You can enable real-time anomaly detection for BigQuery when connecting a new project or editing an existing one. For a new project, take the following steps:

1. Sign in to the DoiT Cloud Intelligence console, select Integrate from the top navigation bar, and then select Google Cloud.
2. From the Connect drop-down, select Project.
3. Select the Real-time Anomalies – BigQuery checkbox. You can expand the feature to view its required permissions
4. Select Generate gcloud commands
5. Follow the instructions displayed in the side panel to configure your service account. See Connect a Google Cloud project for more information.
6. If the connection has been set up successfully, the status of the Real-time Anomalies – BigQuery feature will show Healthy.

To enable the feature on multiple projects, you'll need to repeat the process for each one. To add real-time anomalies to a connected project:

1. From the Integrate tab, select Google Cloud and find the project you want to enable it on.
2. Select the kebab menu (⋮) next to the project connection, and then select Edit.
3. Select the Real-time Anomalies – BigQuery checkbox to add the feature.
4. Select Generate gcloud commands.
5. Follow the instructions displayed in the side panel to update your custom role.
6. Select Done to enable the feature

Next steps

For more information about real-time anomaly detection, consult our Help documentation or raise a support ticket.