8000 Generalize HTTP interface and use the new HTTP interface in `httpfs` by Mytherin · Pull Request #17464 · duckdb/duckdb · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Generalize HTTP interface and use the new HTTP interface in httpfs #17464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 20 commits into from
May 13, 2025
Merged

Generalize HTTP interface and use the new HTTP interface in httpfs #17464

merged 20 commits into from
May 13, 2025

Conversation

Mytherin
Copy link
Collaborator
@Mytherin Mytherin commented May 13, 2025 •
edited
Loading

Follow-up from #17420

Interface

This PR generalizes the HTTP interface into a well-defined interface that lives in the DBConfig in the HTTPUtil class.

This interface is as follows:

class HTTPUtil {
public:
	virtual unique_ptr<HTTPClient> InitializeClient(HTTPParams &http_params, const string &proto_host_port);
};

class HTTPClient {
public:
	virtual unique_ptr<HTTPResponse> Get(GetRequestInfo &info) = 0;
	virtual unique_ptr<HTTPResponse> Put(PutRequestInfo &info) = 0;
	virtual unique_ptr<HTTPResponse> Head(HeadRequestInfo &info) = 0;
	virtual unique_ptr<HTTPResponse> Delete(DeleteRequestInfo &info) = 0;
	virtual unique_ptr<HTTPResponse> Post(PostRequestInfo &info) = 0;
};

The RequestInfo structs hold information pertaining to the request, for example:

struct BaseRequest {
    // RequestType, (GET)
	RequestType type;
	// full URL of request (www.website.com/file.csv)
	const string &url;
	// relative path (/file.csv)
	string path;
	// host + port (www.website.com)
	string proto_host_port;
	// set of headers for the request
	const HTTPHeaders &headers;
	// additional parameters (e.g. retry count, etc)
	HTTPParams &params;
};

struct GetRequestInfo : public BaseRequest {
	std::function<bool(const_data_ptr_t data, idx_t data_length)> content_handler;
	std::function<bool(const HTTPResponse &response)> response_handler;
};

Usage

The HTTPUtil class lives in the DBConfig, and a default implementation is provided using httplib in the main library. This is necessary to be able to install extensions without relying on httpfs.

The httpfs extension overrides this implementation with a new implementation based on httplib + openssl, allowing SSL requests to be made. From a user perspective, HTTP requests can now be made by getting the HTTPUtil from the database through the following method:

class HTTPUtil {
public:
	static HTTPUtil &Get(DatabaseInstance &db);
};

And then constructing a Request object and using that, e.g.:

auto &http_util = HTTPUtil::Get(db);
GetRequestInfo get_request(url, ...);
auto response = http_util.Request(get_request);
// do something with the response

CC @carlopi

@Mytherin Mytherin merged commit e36f028 into duckdb:main May 13, 2025
58 of 69 checks passed
This was referenced May 13, 2025
Merged
Merged
Mytherin added a commit that referenced this pull request May 14, 2025
@Mytherin
HTTPFS Parameters fix (#17486)
9b4d61c 
Follow-up fix from #17464

* Provide an additional overload to initialize parameters in `HTTPUtil`
- `HTTPUtil::InitializeParameters`
* Add HTTPUtil::GetName method
* Minor clean-up - move `HTTPStatusCode` to its own files

CC @carlopi
krlmlr added a commit to duckdb/duckdb-r that referenced this pull request May 18, 2025
@krlmlr
vendor: Update vendored sources to duckdb/duckdb@e36f028 8000
b201981 
Generalize HTTP interface and use the new HTTP interface in `httpfs` (duckdb/duckdb#17464)
krlmlr added a commit to duckdb/duckdb-r that referenced this pull request May 19, 2025
@krlmlr
vendor: Update vendored sources to duckdb/duckdb@e36f028
908119c 
Generalize HTTP interface and use the new HTTP interface in `httpfs` (duckdb/duckdb#17464)
krlmlr added a commit to duckdb/duckdb-r that referenced this pull request May 19, 2025
@krlmlr
vendor: Update vendored sources to duckdb/duckdb@e36f028
31e6cd1 
Generalize HTTP interface and use the new HTTP interface in `httpfs` (duckdb/duckdb#17464)
@dentiny dentiny mentioned this pull request May 21, 2025
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@Mytherin
0