8000 test: add coverage check for slt files by BugenZhao · Pull Request #21703 · risingwavelabs/risingwave · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

test: add coverage check for slt files #21703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
May 6, 2025
Merged

test: add coverage check for slt files #21703

merged 12 commits into from
May 6, 2025

Conversation

BugenZhao
Copy link
Member
@BugenZhao BugenZhao commented May 5, 2025 •
edited
Loading

Signed-off-by: Bugen Zhao i@bugenzhao.comI hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Coded with 95% vibe.

Add a script to check whether all .slt and .slt.part are covered in CI.

This is only a rough estimation, because..

  • we only perform static analysis for ci/scripts/*.sh, while some runtime conditionals cannot be evaluated
  • not all scripts located in ci/scripts/ are guaranteed to be covered in Buildkite workflows
  • runtime conditionals in .slt are not evaluated either, skipif .. include cannot be handled accurately

However, the results still seem quite useful after I introduce a .coverageignore file to fix the corner cases.

List of uncovered files goes below. Opened an issue (#21704) to track them.

"e2e_test/backfill/backfill_order_control_recovery.slt",
"e2e_test/background_ddl/common/create_bg_mv.slt",
"e2e_test/background_ddl/common/create_fg_mv.slt",
"e2e_test/background_ddl/common/create_index.slt",
"e2e_test/background_ddl/common/create_sink.slt",
"e2e_test/background_ddl/common/create_table.slt",
"e2e_test/background_ddl/common/drop_index.slt",
"e2e_test/background_ddl/common/drop_mv.slt",
"e2e_test/background_ddl/common/drop_sink.slt",
"e2e_test/background_ddl/common/drop_table.slt",
"e2e_test/background_ddl/common/validate_backfilled_mv.slt",
"e2e_test/background_ddl/common/validate_no_jobs.slt",
"e2e_test/background_ddl/common/validate_one_job.slt",
"e2e_test/background_ddl/sim.slt",
"e2e_test/background_ddl/sim/basic.slt",
"e2e_test/compaction/ingest_rows.slt",
"e2e_test/database/timezone.slt",
"e2e_test/nexmark/create_source_bug_fix.slt.part",
"e2e_test/over_window/generated/batch/ignore_nulls/mod.slt.part",
"e2e_test/over_window/generated/batch/odd_frames/mod.slt.part",
"e2e_test/over_window/generated/streaming/ignore_nulls/mod.slt.part",
"e2e_test/over_window/generated/streaming/odd_frames/mod.slt.part",
"e2e_test/sink/kafka/avro-decimal.slt",
"e2e_test/sink/license.slt",
"e2e_test/sink/pg_native_vs_jdbc.slt",
"e2e_test/sink/rate_limit.slt",
"e2e_test/streaming/except.slt.part",
"e2e_test/streaming/intersect.slt.part",
"e2e_test/streaming/nexmark/sinks/q19.slt.part",
"e2e_test/streaming/nexmark/sinks/q6_group_top1.slt.part",
"e2e_test/user_doc/user_doc.slt"

Also add this check into CI 10000 . Currently not marked as required.

Preview of a failed run:

https://github.com/risingwavelabs/risingwave/actions/runs/14830705413/job/41631146788?pr=21703

image

A successful run:

https://github.com/risingwavelabs/risingwave/actions/runs/14830737859/job/41631236003?pr=21703

image

Checklist

  • I have written necessary rustdoc comments.
  • I have added necessary unit tests and integration tests.
  • I have added test labels as necessary.
  • I have added fuzzing tests or opened an issue to track them.
  • My PR contains breaking changes.
  • My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
  • I have checked the Release Timeline and Currently Supported Versions to determine which release branches I need to cherry-pick this PR into.

Documentation

  • My PR needs documentation updates.
Release note
Signed-off-by: Bugen Zhao

BugenZhao added 6 commits May 5, 2025 14:31
@BugenZhao
initial commit
14da8f9 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao
correctly handle include_ignored
5a1ede0 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao
move to e2e_test dir
25cfe21 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao
add workflow
1d45a9a 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao
use latest version always
fac07f2 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao
remove output json
c52dcdd 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@github-actions github-actions bot added the A-test Area: Testing framework & misc tests not belonging to any specific component. label May 5, 2025
BugenZhao added 4 commits May 5, 2025 14:38
@BugenZhao
fix grep
0a931aa 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao
add a bad slt
1cf6ca6 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao
add a bad part
767f59e 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao
refine error reporting
328d83e 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao
remove bad files
7cec4fa 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao BugenZhao marked this pull request as ready for review May 5, 2025 07:50
Copilot
Copilot AI reviewed May 5, 2025
View reviewed changes
Copy link
Contributor
@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces a script to perform SLT file coverage analysis and integrates the check into CI to prevent new uncovered SLT files from being introduced.

  • Added documentation in README on using the SLT coverage check script
  • Introduced a GitHub Actions workflow to run the coverage script and identify regressions in SLT coverage

Reviewed Changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.

File Description
e2e_test/README.md Added documentation explaining the SLT coverage check usage
.github/workflows/slt-coverage-check.yml Added a new CI workflow to run the SLT coverage check script
Files not reviewed (1)
  • e2e_test/.coverageignore: Language not supported

@BugenZhao BugenZhao mentioned this pull request May 5, 2025
Open
@BugenZhao
ignore subscription
dd648a7 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao BugenZhao mentioned this pull request May 5, 2025
Merged
8 tasks
@BugenZhao BugenZhao requested a review from stdrc May 6, 2025 08:55
@BugenZhao BugenZhao added this pull request to the merge queue May 6, 2025
Merged via the queue into main with commit 3b7e146 May 6, 2025
34 checks passed
@BugenZhao BugenZhao deleted the bz/slt-coverage branch May 6, 2025 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

Copilot code review Copilot Copilot left review comments

@kwannoel kwannoel kwannoel approved these changes

@xiangjinwu xiangjinwu Awaiting requested review from xiangjinwu

@xxchan xxchan A 57F0 waiting requested review from xxchan

@stdrc stdrc Awaiting requested review from stdrc

Assignees
No one assigned
Labels
A-test Area: Testing framework & misc tests not belonging to any specific component.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@BugenZhao @kwannoel
0