Updating Dependent Data Models #52

xerial · 2024-08-29T18:59:49Z

Now that wvlet has added save to, append to, delete operators in #229, it's ready to start building materialization plans for data models.

Key questions to consider:

subscribe my_model as  my_model_stream

subscribe as my_stream = 
   from ... 
end

Related work:

paper list https://github.com/xerial/streamdb-readings?tab=readme-ov-file#incremental-processing-with-materialized-views
dbt has different types of materialization https://docs.getdbt.com/docs/build/materializations
- table (full-refresh the entire model) -> full-refresh
- view (using DBMS native view) -> unnecessary for wvlet as data model works as a view
- incremental (use conditional switch of SQL statements) -> Need to specify watermark columns (e.g., time)
- ephemeral (expand dependent models as subqueries) -> This is the default behavior when running a query using models
- materialized view (uses DBMS's native materialized view feature) -> We will not work as covering DBMS differences will be complicated
Snowflake introduced CHANGE stream query to incrementally materialize queries https://docs.snowflake.com/en/user-guide/streams-intro

The text was updated successfully, but these errors were encountered:

xerial added this to wvlet: Roadmap 2024-2025 Aug 29, 2024

xerial converted this from a draft issue Aug 29, 2024

xerial moved this from Ready to In Progress in wvlet: Roadmap 2024-2025 Aug 29, 2024

xerial modified the milestones: M2: Model Management and Materialization, model: Model Management and Materialization, Release 2024.09 Aug 29, 2024

xerial changed the title ~~LogicalPlan -> Materialization Plan with CTAS/INSERT INTO~~ Updating Dependent Data Models Nov 6, 2024

xerial modified the milestones: Release 2024.09, Release 2024.12 Nov 14, 2024

xerial closed this as completed in b563254 Dec 19, 2024

github-project-automation bot moved this from In Progress to Done in wvlet: Roadmap 2024-2025 Dec 19, 2024

xerial reopened this Dec 19, 2024

xerial moved this from Done to In Progress in wvlet: Roadmap 2024-2025 Dec 19, 2024

xerial mentioned this issue Jun 8, 2025

Implementation Plan for Issue #52: Updating Dependent Data Models #986

Open

33 tasks

Provide feedback