Idle SQL Warehouse
Compare query timestamps, Warehouse active intervals, auto-stop, and starts. A long gap between the last query and stop suggests idle tail. Confirm that frequent polling is not keeping it alive before recommending a shorter auto-stop.Repeated SQL or cache misses
Group by SQL hash, principal-safe project tag, and minute. Look for stable repeated statements, multiple browser tabs, overlapping Agents, or retries. Distinguish:- ThinkPlatform response-cache hits, which avoid Databricks execution;
- Databricks result-cache hits, which still appear in Query History and can wake compute;
- explicit fresh/manual requests, which intentionally bypass the response cache.
Expensive scans
Rank query hashes by DBU proxy, duration, and bytes read. Check partition pruning, bounded dates, selected columns, reused Gold tables, and repeated equivalent aggregates. Recommend materialization only when reuse outweighs its refresh/storage cost.Jobs, failures, and retries
Calculate attempts per successful run and cost per successful run. Check overlapping schedules,max_concurrent_runs, timeout, retry count, skew, dependency failures, and repeated full recomputation. Do not treat a necessary retry policy as waste without failure evidence.
Compute selection
Flag production Jobs on All-Purpose/existing clusters. For compatible workloads compare five representative runs on Serverless and a policy-managed Job Cluster using output equality, success rate, duration, and billed cost. Keep the cheaper valid mode; do not optimize from DBU rate alone.Lakehouse AI
Verify representative sample size, AI expression count, materialization, error rate, and DBU per successful row. Re-evaluating an AI expression in multiple actions can multiply work; materialize it once before validation and downstream reads.Dashboard refresh
Group Query History by SQL hash and minute while the page is idle.refreshSeconds=0 should produce only the initial/manual request; allowed automatic refresh must be at least 900 seconds. Consolidate widgets that share filters or source SQL into one dataset.
Missing tags or billing coverage
List untagged resources and unallocated cost separately. Required tags aretp_org_id, tp_project_id, tp_system, and workload_type; SQL also needs request-source Query Tags. Fix observability before asserting project chargeback.
Recommendation order
- Stop accidental idle/repeated work.
- Bound previews, scans, samples, retries, and concurrency.
- Reuse response cache and materialized results.
- Right-size Warehouse and Job compute after representative comparison.
- Improve tags and billing coverage.