You Say ‘What’, I Hear ‘Where’ and ‘Why’ — (Mis-)Interpreting SQL to Derive Fine-Grained Provenance

Tobias MüllerBenjamin DietrichTorsten Grust

Proceedings of the 44th Int’l Conference on Very Large Databases. PVLDB 11(11), pages 1536–1549. Rio de Janeiro, Brazil, August 2018.

SQL declaratively specifies what the desired output of a query is. This work shows that a non-standard interpretation of the SQL semantics can, instead, disclose where a piece of the output originated in the input and why that piece found its way into the result. We derive such data provenance for very rich SQL dialects — including recursion, windowed aggregates, and user-defined functions — at the fine-grained level of individual table cells. The approach is non-invasive and implemented as a compositional source-level SQL rewrite: an input SQL query is transformed into its own interpreter that wields data dependencies instead of regular values. We deliberately design this transformation to preserve the shape of both data and query, which allows provenance derivation to scale to complex queries without overwhelming the underlying database system.

An extended version of the paper (PDF) is available on arXiv.