8000 DOS vulnerability in /tx_search RPC endpoint · Issue #1387 · cometbft/cometbft · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

DOS vulnerability in /tx_search RPC endpoint #1387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mrlp4 opened this issue Sep 22, 2023 · 5 comments
Closed

DOS vulnerability in /tx_search RPC endpoint #1387

mrlp4 opened this issue Sep 22, 2023 · 5 comments
Labels
bug Something isn't working rpc

Comments

@mrlp4
Copy link
mrlp4 commented Sep 22, 2023 •
edited
Loading

Summary

RPC query /tx_search?query=tx.height>1&prove=true triggers enormous RAM utilization in cometbft \ tendermint node, which triggers oom killer and node restart.
Tested and confirmed with various cometbft versions (up to v0.37.2) and tendermint (at least v0.34.21).

Details & PoC

query to RPC http://127.0.0.1:26657/tx_search?query=tx.height>1&prove=true triggers complete RAM utilization and oom-kill for node process by system.

Impact

Most of the RPC endpoints, including many listed in chain-registry are vulnerable.
tbn: only heavy-state nodes are affected, pruned one's react with short RAM spikes only.

Similar behavior but for another endpoint as been described in issue 517
@mrlp4 mrlp4 added bug Something isn't working needs-triage This issue/PR has not yet been triaged by the team. labels Sep 22, 2023
@alijnmerchant21
Copy link
Contributor

Hi @mrlp4 thanks for reporting the issue. We understand that this issue can impact the stability and performance of CometBFT nodes, and we are actively working to address it.

Some quick suggestions:

  • Try not to expose RPC endpoints like - tx_search or block_search directly to the internet.

  • Maybe implement robust request validation and rate-limiting mechanisms to protect RPC endpoints.

  • Implement pagination for RPC endpoints that return large datasets. By limiting the number of items per page and returning data in smaller, manageable chunks, you can reduce RAM and CPU usage during queries.

For a more long-term solution, we are working on - ADR 102

The ADR focuses on - Resource Offloading, Scalability, and Minimal Effort Integration

@alijnmerchant21 alijnmerchant21 removed the needs-triage This issue/PR has not yet been triaged by the team. label Sep 25, 2023
@jmalicevic
Copy link
Contributor

To follow up on Ali's comment, we have merged an implementation of ADR-101 #82. This is an API which allows users to extract data from Comet via gRPC , build their custom indexing and querying scheme and instruct Comet to prune this data from the full nodes. ADR-102 is an example companion that should use this API. But users are not constrained to using this particular application.

@andynog
Copy link
Contributor
andynog commented Sep 25, 2023 •
edited
Loading

Thank you for bringing this to our attention, @mrlp4. As we have previously mentioned, this is a known issue that has been reported multiple times over the years. While we understand the concerns that this may cause for public nodes, we are confident that the secure operator's default setup ensures that this issue poses no threat to the chain's consensus or liveness.

While we understand there's a concern with bringing down publicly available nodes, most of them are nodes set up to provide data to users on a best-effort basis, like the ones in Cosmos Directory. From a consensus standpoint, validators don't expose their validating nodes to the public internet (mainly the RPC endpoint), and indexing services (e.g., block explorers) also maintain their nodes and keep them private.

As @jmalicevic mentioned, our goal is to make available the features that will allow the creation of Data Companions (ADR-101), which can offload indexing to external nodes, and these will be able to better index and serve the data.

There are other things that they can do to mitigate some of the risk. For example, these public nodes can use a reverse proxy where they don't expose some particular endpoints related to the indexing, such as /tx_search and /block_search. They can also run the servers with indexing disabled to prevent out-of-memory issues.

We understand that this is not the final solution. The Data Companion route will be the best solution in the future for publicly available nodes. In the meantime, until the Data Companion is not fully functional, we will investigate this further to see if there's a way to mitigate this issue without causing any breaking changes or that would require a lot of effort.

@baabeetaa
Copy link

reverse proxy (nginx) could block uri-over-http (/tx_search)

but could not block jsonrpc-over-http:
{ "jsonrpc": "2.0", "id": "0", "method": "tx_search", "params": {"query": "tx.height > 1", "prove": false, "page": "1", "per_page": "1", "order_by": "asc"}}

I have to write a custom proxy/gateway.

@adizere
Copy link
Member
adizere commented Jan 31, 2024

Closing as hopefully the issue was resolved with the suggestions above. In case it still poses as a vulnerability, please reopen (or submit a hacker1 report).

@adizere adizere closed this as not planned Won't fix, can't repro, duplicate, stale Jan 31, 2024
@faddat faddat mentioned this issue Nov 3, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working rpc
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@adizere @jmalicevic @baabeetaa @mrlp4 @alijnmerchant21 @andynog
0