8000 test: better e2e test framework · Issue #20525 · risingwavelabs/risingwave · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

test: better e2e test framework #20525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
stdrc opened this issue Feb 18, 2025 · 8 comments
Open

test: better e2e test framework #20525

stdrc opened this issue Feb 18, 2025 · 8 comments
Assignees
@stdrc @BugenZhao @xxchan
Labels
A-test Area: Testing framework & misc tests not belonging to any specific component. S-need-design Status: A detailed design is needed before coding. Typically used for feat/refactor issues. type/refactor Type: Refactoring.

Comments

@stdrc
Copy link
Member
stdrc commented Feb 18, 2025 
# file structure
e2e_test
    |- streaming
        |- main.py # discover all tests in `tests`
        |- risedev.yml
        |- tests
            |- a.slt
            |- b.slt
    |- batch
        |- ...

risedev e2e streaming
@github-actions github-actions bot added this to the release-2.3 milestone Feb 18, 2025
@xxchan
Copy link
Member
xxchan commented Feb 18, 2025

What problem does it solve?

@stdrc stdrc removed this from the release-2.3 milestone Feb 18, 2025
@xxchan
Copy link
Member
xxchan commented Apr 17, 2025 •
edited
Loading

Some discussion: SLT itself isn't a very good DSL.

@xxchan xxchan mentioned this issue Apr 17, 2025
Merged
8 tasks
@xxchan
Copy link
Member
xxchan commented Apr 17, 2025 •
edited
Loading

batch e2e tests is especially bad. It has multiple layers of control:

e2e_test/visibility_mode/
            |- barrier.slt (include)
            |      |- batch/local_mode.slt (include)
            |                    |- **/*.slt.part
            |      |- batch/distribution_mode.slt
            |- checkpoint.slt

If we switch to main.py, we don't need to include any more.

@stdrc
Copy link
Member Author
stdrc commented Apr 17, 2025
  • At the end of day, why do we need a DSL? We can still do data-driven tests as a library, using rust or python.

Agree. DSL, NO; Lib, YES!

@BugenZhao
Copy link
Member

Looking forward to further design. From my own perspective, however, I still believe the flexibility is always a trade-off: with limited expressiveness, we still gain some benefits with sqllogictest-rs.

@xxchan
Copy link
Member
xxchan commented Apr 25, 2025

I agree for (maybe most) simple cases, DSL may be convenient. SLT + system commands are working fine. Maybe I just w 8000 ant a less verbose DSL, e.g., using markers like > to run SQL. 🤣 Besides, it can support some built-in commands, instead of full-fledged bash, as you dislike/

@xxchan
Copy link
Member
xxchan commented Apr 25, 2025

Maybe we can experimentally support an alternative syntax in sqllogictest-rs...

@xxchan xxchan mentioned this issue May 22, 2025
Open
8 tasks
@xxchan
Copy link
Member
xxchan commented May 22, 2025

#21082 is kind of this

@stdrc stdrc added type/refactor Type: Refactoring. A-test Area: Testing framework & misc tests not belonging to any specific component. S-need-design Status: A detailed design is needed before coding. Typically used for feat/refactor issues. labels May 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stdrc stdrc

@BugenZhao BugenZhao

@xxchan xxchan

Labels
A-test Area: Testing framework & misc tests not belonging to any specific component. S-need-design Status: A detailed design is needed before coding. Typically used for feat/refactor issues. type/refactor Type: Refactoring.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@stdrc @BugenZhao @xxchan
0