Introducing Real-Time Cost Anomaly Detection for Google BigQuery On-Demand

We’re excited to roll out a major upgrade to how DoiT helps you stay in control of your Google Cloud spend: real-time anomaly detection for Google BigQuery on-demand. For the first time, you’ll receive alerts about unexpected BigQuery cost spikes in under an hour.

What’s new

Until now, anomaly insights for BigQuery relied on next-day billing-file ingestion. That meant if a bad query ran at 2 PM today, you wouldn’t know until tomorrow.

With our new real-time detection pipeline for BigQuery on-demand, we continuously ingest and analyze live BigQuery usage metadata, flag unusual usage patterns, and send you Slack or email alerts in less than an hour – not the next day.

Why it matters

As DoiT’s BigQuery expert Sayle Matthews will tell you, the risk of unchecked queries racking up significant costs in a short period of time is very high:

“One of the largest issues is seeing how much their bill is at any given moment and being able to alert them when a ‘runaway query’ hits. We have seen some examples where customers have single queries that cost $2,000 USD and run in less than a minute, and of course, these were run multiple times in quick succession. These mistakes lead to massive bills that aren't caught for days or weeks later.”

Real-time detection for BigQuery on-demand means that you can:

• Catch runaway queries in minutes: Prevent accidental or inefficient queries from racking up costs before anyone notices.
• Protect against operational mistakes: Get alerted when abnormal query activity starts impacting spend.
• Strengthen your security posture: Real-time cost changes can signal unauthorized data access or compromised systems.

What you need to do

This feature is available for all DoiT customers with a paid Enhanced, Premium, or Enterprise subscription and a connected Google Cloud account. To enable it, take the following steps:

1. Locate the service account of interest on the Google Cloud access & features page.
2. Select the kebab menu (⋮) next to the project connection, and then select Edit.
3. Select the Real-time Anomalies – BigQuery checkbox to add the feature.
4. Select Generate gcloud commands.
5. Follow the instructions displayed in the side panel to update your custom role.
6. Select Done to enable the feature

Enable BQ real-time anomaly detection

Next steps

Up next, we’ll be releasing the same support for BigQuery reservations to bring the same real-time intelligence across your full BigQuery footprint.

For more information about enabling real-time anomaly detection, consult our Help documentation or raise a support ticket.