← Professional Data Engineer: pipelines and data decisions
07 / 8 · 40 MIN

Analysis, sharing, and ML evaluation

Deliver useful data while preserving access scope and evaluation validity.

Concept and mechanism

Data consumption needs a contract as clear as ingestion. An authorized view can expose an appropriate projection without direct base-table reads, but does not automatically bypass column controls. Test with the actual consumer identity and include fields allowed by requirements. A linked dataset in BigQuery sharing is a read-only reference rather than an independent recovery copy. For dashboards, materialized views can reduce repeated work but require analysis of supported SQL, freshness, and refresh, query, and storage costs. BI Engine uses memory managed separately from slot reservations; do not confuse the units when planning capacity.

Guided application

In a fictional case, a model routes APS tickets. If variants of the same incident appear in training and testing, the metric can reflect familiarity rather than generalization. If it uses resolution notes available only after routing, it receives information absent in production. Define the splitting unit and each feature’s temporal availability while retaining representative evaluation. In vector search, an index can reduce latency using approximate nearest neighbor but retrieve fewer neighbors than brute force. Measure recall with relevant examples and decide the cost and performance trade-off. Search speed does not remove document authorization requirements. Result quality should be evaluated across the complete path the user actually uses.

IN PRACTICE

Excellent testing with cross-set duplicates does not demonstrate good routing of new incidents.

Common pitfalls

View as bypass; linked dataset as backup; index as perfect recall; future information as available feature.

Related topics: Design, governance, and identity · Quality, migration, and cutover · Streaming, time, and effects

Take this idea with you

Validate access, freshness, and representativeness in actual consumption.

Create account

Reference: BigQuery authorized views · Current linked standard guide (document title v4.2); edition date unconfirmed (2026-09-30 inspection)