8000 feat(source): Support AWS privatelink for CREATE TABLE command by StrikeW · Pull Request #9728 · risingwavelabs/risingwave · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat(source): Support AWS privatelink for CREATE TABLE command #9728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
May 12, 2023
Merged

feat(source): Support AWS privatelink for CREATE TABLE command #9728

merged 9 commits into from
May 12, 2023

Conversation

StrikeW
Copy link
Contributor
@StrikeW StrikeW commented May 10, 2023 •
edited
Loading

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Support AWS PrivateLink for the CREATE TABLE command that uses a cloud-hosted Kafka source (MSK).

CREATE TABLE mytable with (
  connector = 'kafka',
  topic = 'kafka_1_partition_topic',
  properties.bootstrap.server = '127.0.0.1:29092',
  connection.name = 'pl1',
  privatelink.targets = '[{"port": 8001}, {"port": 8002}]'
) ROW FORMAT JSON;

🤖 Generated by Copilot at 7088ca4

This pull request refactors and simplifies the code for handling Kafka connector connections in various handlers and catalog modules. It introduces a common utility function resolve_connection_in_with_option to get and validate connection ids from with options. It also improves the reference counting logic of connections in the CatalogManager by using two helper functions from utils.rs.

related: #8771
close #9793

Checklist For Contributors

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • I have demonstrated that backward compatibility is not broken by breaking changes and created issues to track deprecated features to be removed in the future. (Please refer to the issue)
  • All checks passed in ./risedev check (or alias, ./risedev c)

Checklist For Reviewers

  • I have requested macro/micro-benchmarks as this PR can affect performance substantially, and the results are shown.

Documentation

Click here for Documentation

Types of user-facing changes

Please keep the types that apply to your changes, and remove the others.

  • Connector (sources & sinks)
  • SQL commands, functions, and operators

Release note

Support AWS PrivateLink for the CREATE TABLE command that uses a cloud-hosted Kafka source

@github-actions github-actions bot added the type/feature Type: New feature. label May 10, 2023
@StrikeW StrikeW changed the title feat(source): Add private link for CREATE TABLE command feat(source): Support AWS privatelink for CREATE TABLE command May 10, 2023
@StrikeW StrikeW marked this pull request as ready for review May 10, 2023 08:32
@StrikeW StrikeW added the user-facing-changes Contains changes that are visible to users label May 10, 2023
@github-actions github-actions bot removed the user-facing-changes Contains changes that are visible to users label May 10, 2023
@neverchanje
Copy link
Contributor
neverchanje commented May 11, 2023 •
edited
Loading

Is it a user-facing change? Besides, what's its difference from our former CREATE CONNECTION command?

@github-actions github-actions bot added the user-facing-changes Contains changes that are visible to users label May 11, 2023
@StrikeW
Copy link
Contributor Author
StrikeW commented May 11, 2023

Is it a user-facing change? Besides, what's its difference from our former CREATE CONNECTION command?

CREATE CONNECTION is only for creating a privatelink connection, then the connection can be used by source, sink, table that needs to access a MSK service in another vpc.

@StrikeW StrikeW requested review from WillyKidd and yezizp2012 May 11, 2023 09:08
@codecov
Copy link
codecov bot commented May 11, 2023 •
edited
Loading

Codecov Report

Merging #9728 (6774689) into main (f583d6d) will increase coverage by 0.00%.
The diff coverage is 48.57%.

@@           Coverage Diff           @@
##             main    #9728   +/-   ##
=======================================
  Coverage   70.98%   70.98%           
=======================================
  Files        1246     1246           
  Lines      208380   208393   +13     
=======================================
+ Hits       147921   147936   +15     
+ Misses      60459    60457    -2
Flag Coverage Δ
rust 70.98% <48.57%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/meta/src/manager/catalog/mod.rs 25.16% <0.00%> (+0.15%) ⬆️
src/meta/src/manager/catalog/utils.rs 51.98% <0.00%> (-2.77%) ⬇️
src/frontend/src/utils/with_options.rs 74.74% <59.25%> (-9.98%) ⬇️
src/frontend/src/handler/create_sink.rs 97.35% <100.00%> (+2.44%) ⬆️
src/frontend/src/handler/create_source.rs 44.94% <100.00%> (+0.30%) ⬆️
src/frontend/src/handler/create_table.rs 89.83% <100.00%> (+0.06%) ⬆️
src/frontend/src/handler/util.rs 88.54% <100.00%> (+0.22%) ⬆️

... and 3 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

WillyKidd
WillyKidd approved these changes May 11, 2023
View reviewed changes
Copy link
Contributor
@WillyKidd WillyKidd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM :)

@StrikeW StrikeW added this pull request to the merge queue May 12, 2023
Merged via the queue into main with commit f7b7b9f May 12, 2023
@StrikeW StrikeW deleted the siyuan/private-link-table branch May 12, 2023 14:19
@CharlieSYH CharlieSYH added the 📖✓ Covered or will be covered in the user docs. label May 29, 2023
@CharlieSYH
Copy link
Contributor

Documented at https://www.risingwave.dev/docs/upcoming/create-source-kafka/#create-source-with-aws-privatelink-connection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WillyKidd WillyKidd WillyKidd approved these changes

@yezizp2012 yezizp2012 yezizp2012 approved these changes

Assignees
No one assigned
Labels
type/feature Type: New feature. user-facing-changes Contains changes that are visible to users 📖✓ Covered or will be covered in the user docs.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

feat: Support AWS privatelink for CREATE TABLE command
5 participants
@StrikeW @neverchanje @CharlieSYH @yezizp2012 @WillyKidd
0