Skip to content

[PLAT-440] roll up routing in metrics view#9180

Open
pjain1 wants to merge 8 commits intomainfrom
rollup_mv
Open

[PLAT-440] roll up routing in metrics view#9180
pjain1 wants to merge 8 commits intomainfrom
rollup_mv

Conversation

@pjain1
Copy link
Copy Markdown
Member

@pjain1 pjain1 commented Apr 3, 2026

  • Add rollup table config to metrics view proto and YAML parser
  • Implement query routing: eligible rollups are selected based on grain derivability, dimension/measure coverage, timezone match, time range alignment, and time coverage
  • Prefer coarsest grain among eligible rollups; break ties by smallest data range
  • For no-time-range queries ("all data"), verify the rollup covers the base table's full range rather than skipping coverage checks

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

@pjain1 pjain1 closed this Apr 3, 2026
@pjain1 pjain1 reopened this Apr 3, 2026
@pjain1 pjain1 changed the title roll up routing in metrics view [PLAT-440] roll up routing in metrics view Apr 5, 2026
@pjain1 pjain1 requested a review from begelundmuller April 6, 2026 05:32
return wm.min, wm.max, true
}

mn, mx, err := e.fetchTimestamps(ctx, rollup.Database, rollup.DatabaseSchema, rollup.Table)
Copy link
Copy Markdown
Member Author

@pjain1 pjain1 Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of directly querying for watermarks here, I explored using metrics_time_range resolver approach so that we can rely on user defined cache_key_ttl and cache_key_sql to not fetch watermark until needed. It also helps with automatically invalidating cache if rollups are Rill managed as the resolver cache key will rely on metrics view status updated on.

However, the only issue I see is for external olap (majority of cases), mv cache is disabled by default so resolver ends up querying watermarks for all eligible rollups for every single query (contrast with current behaviour where time range is only queried once for time picker). In this case I was thinking of an L1 cache having simple time based ttl of lets say 1 or 5 minutes in this file and only if that expires then call metrics_time_range resolver. I already have the changes locally if needed. Thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant