HADOOP-18528. Disable readAhead/prefetching by default #5134
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
where do you test it?
what do you all think? this is us disabling the readahead so we can get on with the real fix without being in a rush and really trying to get the tests together. that should include taking the AbfsInputStream test, have it create real CSV rows (use my mkcsv code as a foundation) and then have use the mapreduce TextInputFormat code to read it back across multiple threads, just have hive and spark do |
4 tasks
|
🎊 +1 overall
This message was automatically generated. |
|
Okay, ran the test suite on trunk as well. There are no additional errors/failures after the changes, it's showing some errors on the base version. Not sure why this is a failure and not just skipping the test in case the config isn't set. Seeing it fail as a stand-alone test as well. Intermittent failure, saw this in trunk once. Also had a timeout error in ITestAbfsTerasort. |
|
the lease one plays up on me, as does ITestSmallWriteOptimization; especially on parallel runs. i'm not worried there |
steveloughran
approved these changes
Nov 15, 2022
asfgit
pushed a commit
that referenced
this pull request
Nov 15, 2022
Disables block prefetching on ABFS InputStreams, by setting fs.azure.enable.readahead to false in core-default.xml and the matching java constant. This prevents HADOOP-18521. ABFS ReadBufferManager buffer sharing across concurrent HTTP requests. Once a fix for that is committed, this change can be reverted. Contributed by Mehakmeet Singh.
asfgit
pushed a commit
that referenced
this pull request
Nov 15, 2022
Disables block prefetching on ABFS InputStreams, by setting fs.azure.enable.readahead to false in core-default.xml and the matching java constant. This prevents HADOOP-18521. ABFS ReadBufferManager buffer sharing across concurrent HTTP requests. Once a fix for that is committed, this change can be reverted. Contributed by Mehakmeet Singh.
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
Disables block prefetching on ABFS InputStreams, by setting fs.azure.enable.readahead to false in core-default.xml and the matching java constant. This prevents HADOOP-18521. ABFS ReadBufferManager buffer sharing across concurrent HTTP requests. Once a fix for that is committed, this change can be reverted. Contributed by Mehakmeet Singh.
saxenapranav
added a commit
to saxenapranav/hadoop
that referenced
this pull request
Dec 1, 2022
This reverts commit 69e50c7.
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Setting readAhead config to false by default.
How was this patch tested?
Region:
us-west-2mvn -Dparallel-tests=abfs -DtestsThreadCount=8 -Dscale clean verifySeeing some timeout errors, going to test the latest ABFS trunk code to see the differences tomorrow.
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?