SUMMARY:
See how XTIVIA’s Insight360 assessment uncovered critical gaps in a utility data lakehouse — reshaping five Gold data products before they ever reached production.
Table of contents
Introduction
An integrated U.S. energy company serving more than 3 million customers across Arkansas, Louisiana, Mississippi, and Texas had built a Databricks lakehouse ingesting SCADA telemetry, ETRM trade data, maintenance records, environmental feeds, and wholesale billing records. The Bronze and Silver layers were working. But when leadership asked whether the Gold data products were trustworthy enough to retire legacy reports, the platform team could not answer with confidence.
XTIVIA was engaged to run an Insight360 assessment—529 automated checks across governance, security, jobs, cost, MLOps —and a 32-check Industry Utilities Pack, purpose-built for energy platforms. What followed did not just validate the platform — it fundamentally shaped the Gold data products the client deployed into production.
The Gold Products at Stake
The platform was built to serve five enterprise data products: Generation Dispatch Performance feeding MISO day-ahead bidding decisions, Energy Trading P&L with PPA valuations and FERC-reportable settlement reconciliation, Wholesale Billing with demand determinants and rate schedules, Regulatory Compliance covering NERC CIP posture and EPA emissions reporting, and Asset Health models driving capital planning and outage scheduling. Each depended on the integrity of every layer beneath it.
The Utility Checks That Changed Everything
The Industry Utilities Pack delivered the most consequential findings. Three checks directly reshaped the Gold product design before launch.
Generation Telemetry Freshness (UTL.002) — The SCADA-to-Bronze pipeline had not refreshed within the 24-hour operational threshold. The Generation Dispatch Gold product was computing capacity factor and availability from stale data. In MISO day-ahead markets, a dispatch bid based on stale availability can trigger reliability penalties or leave revenue on the table. The finding led to a freshness SLA enforced by automated alerting tied directly to this check.
Billing Determinant Completeness (UTL.007) — The billing tables were missing MWh quantity fields and commercial rate determinants. The ingestion pipeline extracted position and pricing data, but not the settlement-specific fields needed to compute wholesale invoices. The Wholesale Billing Gold product would have launched structurally incomplete, with invoices lacking demand-charge components and rate-schedule references. The finding triggered a pipeline redesign before promotion.
PPA Pricing Data Quality (UTL.018) — PPA pricing records lacked contract ID linkage, pricing period boundaries, escalation rates, and index references. The ETRM source had this data, but the extraction pulled only current mark-to-market values — not the contractual terms needed to audit a valuation back to its basis. For a utility managing 15-20 year PPAs with complex escalation formulas, this is both a financial risk and a FERC compliance gap. The finding drove a full redesign of the PPA data model in Silver.
The pack also confirmed strengths: EPA Environmental compliance scored 100% across all CEMS, emissions, and monitoring checks — giving leadership confidence to retire the legacy compliance extract. The maintenance and reliability data linkage passed, confirming that the foundation of the Asset Health Gold product was sound.
The Platform Foundation
Beyond utility-specific checks, three platform patterns threatened every Gold product simultaneously. Across 17 production jobs, zero had failure notifications, 14 lacked task timeouts, and 15 ran under personal user identities. A single expired token would have silently halted all five Gold product pipelines. The fix was one standardized job template — resolving 58 findings with a single operational change.
The assessment found 465 broad table grants to generic principals, meaning every user could access trading positions, billing data, and employee records. Group-based RBAC through Unity Catalog replaced blanket access with function-scoped permissions. Table documentation sat at 16% comment coverage against a 70% threshold—meaning consumers of a column like gen_output_mw had no way to know if it was gross or net output.
AI Analysis: What Scoring Missed
Insight360’s AI layer — powered by Databricks Foundation Models — found that 14 of 17 jobs “passing” the service principal check actually had the value “Not specified.” The true adoption rate was 6%, not the 82% the score reflected. The AI also identified a compound risk: no IP access lists + 465 broad grants + no monitoring = an undetectable unauthorized access path to all 306 managed tables. Each finding alone was High severity; together, a Critical exposure chain.
The Outcome
The assessment was completed in under two weeks. Remediation took 90 days across three phases: identity and access hardening (0-30 days), operational resilience (30-90 days), and governance maturity (90-180 days). The five Gold products launched with validated data quality profiles, enforced freshness SLAs, and governed access controls. The legacy reports were retired. Insight360 now runs monthly, catching regressions before they reach the Gold layer.
The transformation was not in the pipeline — those were always well built. It was in the assurance that surrounded them. Insight360 moved the client from “we think it’s ready” to “we’ve verified it against 529 checks, including 32 utility-specific quality standards, and closed every gap.”
XTIVIA is a Databricks partner specializing in data engineering, platform governance, and AI/ML for energy, healthcare, financial services, and retail.
Schedule an Insight360 assessment for your utility data lakehouse.