8000 indexes: Stop using node internal types by ryanofsky · Pull Request #25494 · bitcoin/bitcoin · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

indexes: Stop using node internal types #25494

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jul 19, 2022
Merged

Conversation

ryanofsky
Copy link
Contributor
@ryanofsky ryanofsky commented Jun 28, 2022

Start transitioning index code away from using internal node types like CBlockIndex and CChain so index code is less coupled to node code and index code will later be able to stop locking cs_main and sync without having to deal with validationinterface race conditions, and so new indexes are easier to write and can run as plugins or separate processes.

This PR contains the first 7 commits from #24230 (comment) which have been split off for easier review. Previous review comments can be found in #24230

@DrahtBot
Copy link
Contributor
DrahtBot commented Jun 29, 2022

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #25623 ([kernel 3e/n] Decouple CDBWrapper and CBlockTreeDB from ArgsManager by dongcarl)
  • #25297 (WIP, wallet: speedup transactions sync, rescan and load not flushing to db constantly by furszy)
  • #25222 (refactor: Pass reference to LookUpStats by MarcoFalke)
  • #24150 (refactor: move index class members from protected to private by jonatack)
  • #19888 (rpc, test: Improve getblockstats for unspendables by fjahr)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

info.prev_hash = index->pprev ? index->pprev->phashBlock : nullptr;
info.height = index->nHeight;
LOCK(::cs_main);
info.file_number = index->nFile;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that this will only copy/alias the internal memory to a different type. I wonder if this is actually useful, because it disables lock annotations for the memory where they may actually still be needed? Also, I wonder how useful an interface is that returns a data structure that points into memory that is owned by a different process (I am assuming each blockindex runs in a different multiprocess)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #25494 (comment)

In commit "interfaces, refactor: Add more block information to block connected notifications" (0a93a16)

It seems that this will only copy/alias the internal memory to a different type. I wonder if this is actually useful, because it disables lock annotations for the memory where they may actually still be needed?

I don't think there's actually a concern here, and this is what current code already does. cs_main is locked while reading nFile and nDataPos values, but not when ReadBlockFromDisk is called. Values are copied first and then used.

Also, I wonder how useful an interface is that returns a data structure that points into memory that is owned by a different process (I am assuming each blockindex runs in a different multiprocess)

The change in this commit is to replace individual block & height parameters with a BlockInfo struct parameter that can hold more information about the block:

-        virtual void blockConnected(const CBlock& block, int height) {}
-        virtual void blockDisconnected(const CBlock& block, int height) {}
+        virtual void blockConnected(const BlockInfo& block) {}
+        virtual void blockDisconnected(const BlockInfo& block) {}

The point of using the struct is to make code more readable and avoid the need to update blockConnected/blockDisconnected declarations when more block information is needed.

Semantically there's no difference between passing a const uint256& hash parameter vs adding a const uint256& hash; struct member, or passing an int file_number parameter vs. adding an int file_number; struct member. All the copy and reference and lifetime concerns are the same.

It is true that because referencing memory in other processes is not possible, multiprocess code will create local copies of values when references are passed, and the local copies may have shorter lifetimes than original references. But this is true wherever references and pointers are used in src/interface/ classes, so not a new thing.

@fjahr
Copy link
Contributor
fjahr commented Jul 2, 2022

Linter warns of a circular dependency

A new circular dependency in the form of "index/base -> node/context -> net_processing -> index/blockfilterindex -> index/base" appears to have been introduced.

Copy link
Contributor Author
@ryanofsky ryanofsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #25494 (comment)

Linter warns of a circular dependency

Thanks! Should be fixed


Updated 3c82fa5 -> ddbbdcf (pr/ind.1 -> pr/ind.2, compare) to fix lint-circular-dependencies.py error

info.prev_hash = index->pprev ? index->pprev->phashBlock : nullptr;
info.height = index->nHeight;
LOCK(::cs_main);
info.file_number = index->nFile;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #25494 (comment)

In commit "interfaces, refactor: Add more block information to block connected notifications" (0a93a16)

It seems that this will only copy/alias the internal memory to a different type. I wonder if this is actually useful, because it disables lock annotations for the memory where they may actually still be needed?

I don't think there's actually a concern here, and this is what current code already does. cs_main is locked while reading nFile and nDataPos values, but not when ReadBlockFromDisk is called. Values are copied first and then used.

Also, I wonder how useful an interface is that returns a data structure that points into memory that is owned by a different process (I am assuming each blockindex runs in a different multiprocess)

The change in this commit is to replace individual block & height parameters with a BlockInfo struct parameter that can hold more information about the block:

-        virtual void blockConnected(const CBlock& block, int height) {}
-        virtual void blockDisconnected(const CBlock& block, int height) {}
+        virtual void blockConnected(const BlockInfo& block) {}
+        virtual void blockDisconnected(const BlockInfo& block) {}

The point of using the struct is to make code more readable and avoid the need to update blockConnected/blockDisconnected declarations when more block information is needed.

Semantically there's no difference between passing a const uint256& hash parameter vs adding a const uint256& hash; struct member, or passing an int file_number parameter vs. adding an int file_number; struct member. All the copy and reference and lifetime concerns are the same.

It is true that because referencing memory in other processes is not possible, multiprocess code will create local copies of values when references are passed, and the local copies may have shorter lifetimes than original references. But this is true wherever references and pointers are used in src/interface/ classes, so not a new thing.

Copy link
Contributor
@mzumsande mzumsande left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK ddbbdcf

I reviewed these commits multiple times as part of #24230, and think they are good.
I also did some manual testing for this PR on regtest and signet and didn't see any differences in behavior.

While it's a bit unfortunate that this introduces a circular dependency, this is transitory and will get removed with the last commits of #24230 after all internal uses of CChainState are removed.

nit: clang-format suggests a few minor changes (spacing) over these commits

Copy link
Contributor
@fjahr fjahr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tACK ddbbdcf

Reviewed code and ran it, syncing txindex from intermittently synced state to fully synced and fully synced coinstatsindex from scratch.

a.wallet->blockDisconnected(block, chain.size() - 1);
b.wallet->blockDisconnected(block, chain.size() - 1);
const uint256& hash = block.GetHash();
const uint256& prev_hash = chain.size() >= 2 ? std::get<1>(chain[chain.size() - 2]).GetHash() : uint256();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think this line could be made a little easier to parse but since it's in fuzzing it's not so important

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #25494 (comment)

nit: I think this line could be made a little easier to parse but since it's in fuzzing it's not so important

Simplified

int file_number = -1;
unsigned data_pos = 0;
const CBlock* data = nullptr;
const CBlockUndo* undo_data = nullptr;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note for other reviewers: in this PR this is unused but will be used in follow-ups, see #24230

Copy link
Member
@maflcko maflcko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to merge as-is. Let me know what you think.

review ACK ddbbdcf 🐯

Show signature

Signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

review ACK ddbbdcf4112c3a84e3287e6bf701e0eb592b24c 🐯
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
pUjssQwAplJ1ZbuRcMBriV0HxqVNf02T4stpYurghGghrlmhm8pidPOD4r+gcr+g
ECrmBrsbS9Ka/ycxwoM0nel0hEMsXLZWy6ueY0ufksVY8TFgIdVSQJtSLZ9MNuUE
7n9/Qz+z1tl2isShx5nbEQ+UcRiCfXVryfV0OuGO4RLPSJV9fqYaPBKh85af6hxt
P+q4qD4Ql6kWZLPyrdZ+M3JO3KIgRDdGLjXdvFQvNDVDos68HT+UTRvRod2B3kkM
5I84K4L6SfO/WW5sGXM/G8t7qyC4Sdv906qBP9zGkxFV/D5jj3n+cpgb8vog1RGr
6riVrC9pPnCC8kSopDgHdHIEiNdnqxeQJ2eiFutmI2B2Gz6ex7YjvBGyeJeRSka5
sNtrD+9EKtUE5Ta/Iby8mpl1+lSpzC9vFGJGuhTQg1kNzOV80MSVw480wDZcUrTM
vtZoxXpOH3/CnL+8Y+MvxanIahrvS/UApp60iZ848v8pEXtI4cGzUsysRIekUkeD
fmaWQU9d
=52YB
-----END PGP SIGNATURE-----

return false;
}

uint256 expected_block_hash = pindex->pprev->GetBlockHash();
uint256 expected_block_hash = *block.prev_hash; 6D40
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

88cad3c: I wonder where UB should be avoided with asserts and where not

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #25494 (comment)

88cad3c: I wonder where UB should be avoided with asserts and where not

I think it's reasonable to add Asserts all these places or add a const uint256& PrevHash() helper method which asserts, if the Assert is too verbose. Just to be clear, there shouldn't be any UB presently here due to the block.height > 0 check above. The Assert is just an extra check against possible future UB that could occur if a bug is introduced somewhere else in the code.


bool BaseIndex::CommitInternal(CDBBatch& batch)
{
LOCK(cs_main);
// Don't commit anything if we haven't indexed any block yet
// (this could happen if init is interrupted).
if (m_best_block_index == nullptr) {
return false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ddbbdcf: This changes behavior: Previously there was a call to error(), now there is none?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #25494 (comment)

ddbbdcf: This changes behavior: Previously there was a call to error(), now there is none?

Good catch. The error print was lost because of a bad rebase. The m_best_block_index == nullptr check was added in #24117 after this was written. Fixed now

@ryanofsky
Copy link
Contributor Author

Happy to merge as-is. Let me know what you think.

Thanks for review Marco. I definitely want to make some improvements based on your comments, so better not to merge for now

@@ -0,0 +1,23 @@
// Copyright (c) 2022 The Bitcoin Core developers
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add this new file to iwyu?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #25494 (comment)

Maybe add this new file to iwyu?

Thanks, added

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like an iwyu bug, so maybe undo for now?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #25494 (comment)

https://cirrus-ci.com/task/4504627910541312?logs=ci#L6998

Looks like an iwyu bug, so maybe undo for now?

I'm not actually sure what the bug is but I added some missing includes and undid change that adds the file to IWYU for now. The larger PR #24230 this was part of adds more to this file so there is a chance to enable IWYU again later

Copy link
Contributor Author
@ryanofsky ryanofsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated ddbbdcf -> fb245ec (pr/ind.2 -> pr/ind.3, compare) with suggested changes like moving the new file from node/ to kernel/
Updated fb245ec -> f678151 (pr/ind.3 -> pr/ind.4, compare) to fix lint error https://cirrus-ci.com/task/6068993915092992

@@ -0,0 +1,23 @@
// Copyright (c) 2022 The Bitcoin Core developers
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #25494 (comment)

Maybe add this new file to iwyu?

Thanks, added

return false;
}
}

for (const auto& filter_type : g_enabled_filter_types) {
InitBlockFilterIndex(filter_type, cache_sizes.filter_index, false, fReindex);
if (!GetBlockFilterIndex(filter_type)->Start(chainman.ActiveChainstate())) {
InitBlockFilterIndex([&]{ return interfaces::MakeChain(node); }, filter_type, cache_sizes.filter_index, false, fReindex);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #25494 (comment)

2f8a530: Wrapping this into a function to avoid moving a pointer seems overkill?

Reason this is using a std::function is I was looking for a simple way to allow the Init*Index functions to stay where they are in src/index/ without referencing the NodeContext struct. Maybe this isn't a critical goal though, or there is a better way of achieving it. Since I want to extend #10102 to run indexes separately I will have to revisit this code anyway later, so happy to go with whatever would be suggested now if this is not good.

a.wallet->blockDisconnected(block, chain.size() - 1);
b.wallet->blockDisconnected(block, chain.size() - 1);
const uint256& hash = block.GetHash();
const uint256& prev_hash = chain.size() >= 2 ? std::get<1>(chain[chain.size() - 2]).GetHash() : uint256();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #25494 (comment)

nit: I think this line could be made a little easier to parse but since it's in fuzzing it's not so important

Simplified

Copy link
Member
@maflcko maflcko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

f678151 📭

Show signature

Signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

f678151ff9a0a5934965b1c85ec97a1dfe45a67e 📭
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
pUhL5AwAtnEh0cX8QFjSw7c20rH3WCFif2w8QDJpwnRDHHmc8qSzVamRRmmjGaZ2
uT2IvZCVzzLqvcJKWJR6Woan9pS5GJtF/pqBxv1olJd8CBNO1VrL3o5NnBzKIDAN
6PWq4Tl50TrIwnR5Wnd8pEYfbK77M4ghdSZ5TYtzL+gHTL8wQcYGmJCw7pDAgeAp
iMMe82rhI/sYeU9nqo3l5MF2IgJkzEuQ/wpxD4rmkDyf+9yLNtquHCBzR5BSPhDl
5LoR7v9LR5WqHZzOWyrrY/FhUmnbYDpmub8TAmYGj6cy+in37jLheDY4EDU2mzJG
CMtE/6yRGY8dPFohG//yUquEfHG7lVacW+9tMXrRZ0+zcmlBFcobOtwNRJ4Oc4qZ
IcmULASpFcVpFbLrOTqS5n/zrIvicQSxH4nXGXNnB7gvZFT4lmpnOin6txgN/Sic
sf20wlYEbQpJBWgBtRjSny8SuY5BHJeAypVzIbw1Sc78GpZs2m3bmjHZHuwud97p
FKbfDUd0
=d7Fu
-----END PGP SIGNATURE-----

if (m_best_block_index == nullptr) {
return false;
bool ok = m_best_block_index != nullptr;
if (ok) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still wrong?

Previously the call order was CoinStatsIndex::CustomCommit , then BaseIndex::CommitInternal (which may be skipped).

Now, everything is skipped?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #25494 F438 (comment)

In commit "indexes, refactor: Remove CChainState use in index CommitInternal method" (f678151)

This is still wrong?

Previously the call order was CoinStatsIndex::CustomCommit , then BaseIndex::CommitInternal (which may be skipped).

Now, everything is skipped?

You are referring to the m_best_block_index == nullptr case? If so then yes the whole commit is skipped now when previously only half of the commit was skipped (custom part was not skipped, base part was skipped).

I didn't notice yesterday that only the base half of the commit was skipped, but it's definitely better to skip the whole commit. This m_block_index == nullptr case was only added very recently in 0243907 as part of an attempt to avoid index corruption if state was flushed during startup that would happen because GetLocator can return null on startup. It was an imperfect attempt and according to comments in #24117 (comment), corruption continued to happen after that PR. It's possible this commit might inadvertently fix this by skipping the whole commit instead of half of it, but it's not clear. In any case, it should be better behavior and it makes more sense to skip the whole commit.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that skipping the error message in the m_block_index == nullptr case would have actually been ok: Right now, everytime one starts to sync a new index with an existing chain, this would appear once in the error log, even though it is expected and not actually an error. But possibly this is fixed anyway by a later commit from #24230 rewriting the init logic anyway into not attempting to call commit before actually doing some indexing?

I agree that skipping the entire thing custom+base commit makes more sense, but I can't see how this could possibly avoid additional cases of corruption or be the cause of #24117 (comment), since the custom code in both coinstatsindex and blockfilterindex doesn't make use of m_block_index and just prepares custom data in the batch without committing.

Copy link
Contributor Author
@ryanofsky ryanofsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated f678151 -> 8bc1602 (pr/ind.4 -> pr/ind.5, compare) updating commit message 8bc1602 to describe some changed behavior and no longer enabling IWYU for now in the new file

if (m_best_block_index == nullptr) {
return false;
bool ok = m_best_block_index != nullptr;
if (ok) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #25494 (comment)

In commit "indexes, refactor: Remove CChainState use in index CommitInternal method" (f678151)

This is still wrong?

Previously the call order was CoinStatsIndex::CustomCommit , then BaseIndex::CommitInternal (which may be skipped).

Now, everything is skipped?

You are referring to the m_best_block_index == nullptr case? If so then yes the whole commit is skipped now when previously only half of the commit was skipped (custom part was not skipped, base part was skipped).

I didn't notice yesterday that only the base half of the commit was skipped, but it's definitely better to skip the whole commit. This m_block_index == nullptr case was only added very recently in 0243907 as part of an attempt to avoid index corruption if state was flushed during startup that would happen because GetLocator can return null on startup. It was an imperfect attempt and according to comments in #24117 (comment), corruption continued to happen after that PR. It's possible this commit might inadvertently fix this by skipping the whole commit instead of half of it, but it's not clear. In any case, it should be better behavior and it makes more sense to skip the whole commit.

Copy link
Member
@maflcko maflcko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 8bc1602 though did not review the last commit 🗡

Show signature

Signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

ACK 8bc1602821202d4863fab05ee1aab3ff3b70a7c0 though did not review the last commit 🗡
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
pUhkpQv9EYKytsqsNKtA9Yte0NOoa/hTFzjqhx34b+kkeB9twTIhnl/8kqLcct6a
1983DDhe835/ZeKxwAGPV9KTnBma+E7Sr9nAaFuY8GJfA2qsFncXzoeRpZRW8e9m
EjZ1Yvi8Yac3AE9RArPayknO0uo7tvlYoL/2mx7k+6V/VvlZSAqrUMgwQY8PvQmJ
c6AoUvbUrMSy6HM4HZHi9TEVveQqwMqYwPwKRqy0A5CBAlwIN75m/NKFSDWcjVTS
p3m88uM5fZQ1gQ4Xu9FTvXSQavIKr883gUeUXOR9yv5FXgXHCRVGjrbpVEey2nmK
jOwdKWHVpVv0+6jI1qzJnkTlKdKvFKcvTdPp9EW/woWW4FdcEAtAG4s0kDfvQrN6
b8ZUUpowDJsM+3Uisuqq6dhq48j56u0Ux0RmXBTGsCT4t8hpPSUbsAqqSZnhpWlJ
SKw4c9amB3i8kcRLIBkpntEsOgtZnXBaNgdlpv3iWUALC8xqBOF8RDi10qMcS0x2
1/7likbu
=LR6p
-----END PGP SIGNATURE-----

@ryanofsky
Copy link
Contributor Author

@fjahr & @mzumsande could you reack? I think only change since previous reviews has been expanding the m_block_index == nullptr case in CommitInternal to keep logging an error and avoid committing anything

…otifications

Add new interfaces::BlockInfo struct to be able to pass extra block
information (file and undo information) to indexes which they are
updated to use high level interfaces::Chain notifications.

This commit does not change behavior in any way.
…to indexes

Passing abstract Chain interface will let indexes run in separate
processes.

This commit does not change behavior in any way.
…ne function

This commit does not change behavior in any way.
Replace overriden index Init() methods that use the best block
CBlockIndex* pointer with pure CustomInit() callbacks that are passed
the block hash and height.

This gets rid of more CBlockIndex* pointer uses so indexes can work
outside the bitcoin-node process. It also simplifies the initialization
call sequence so index implementations are not responsible for
initializing the base class.

There is a slight change in behavior here since now the best block
pointer is loaded and checked before the custom index init functions are
called instead of while they are called.
Replace WriteBlock method with CustomAppend and pass BlockInfo struct
instead of CBlockIndex* pointer

This commit does not change behavior in any way.
Replace Rewind method with CustomRewind and pass block hashes and
heights instead of CBlockIndex* pointers

This commit does not change behavior in any way.
Replace CommitInternal method with CustomCommit and use interfaces::Chain
instead of CChainState to generate block locator.

This commit does not change behavior in any way, except in the
(m_best_block_index == nullptr) case, which was added recently in
bitcoin#24117 as part of an ongoing attempt to
prevent index corruption if bitcoind is interrupted during startup. New
behavior in that case should be slightly better than the old behavior (skipping
the entire custom+base commit now vs only skipping the base commit previously)
and this might avoid more cases of corruption.
@ryanofsky
Copy link
Contributor Author
ryanofsky commented 10000 Jul 18, 2022

Rebased 8bc1602 -> 7878f97 (pr/ind.5 -> pr/ind.6, compare) due to conflicts with #23997 and #25487

@maflcko
Copy link
Member
maflcko commented Jul 19, 2022

ACK 7878f97 though did not review the last commit 🤼

Show signature

Signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

ACK 7878f97bf15b6e7c9b47d1c0d96419b97e1bdcbd though did not review the last commit 🤼
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
pUgZAAwAg2W4ylK74K+MHoP2XiZFNNZe6tzS3wpYnBUZXoP0ee1rOg/lNGl1o2ER
d0zs+tGjHajkpDiQB9DgCqNLQAEq1qOCRmxH1W9nt04gkVqyZPPcOg7GE2da1MNL
n8P8gNTHiqmS8FrRSAs40udE/VuhSr9j/4OLaCEjQiRdhWPKB1J4IduIK0OMQdW0
LQw2wBXRb0rtG4L0ztwmndFXx9IbBLRNZCFRR6a2IW4/2rZN3FmPSr6kDA7PV9S+
bM/+NNPThUga3KTfbPBj5ERivN/4UZXILyOdFVxs+dOcPViZVvtWd8uBLsiH+2QO
5zSuVeXnWPSujZXcNKpWnXbUJv7Pi8ClPkBv2016w5hzGGYOngV/TzLyeZtihn3S
IXQ7afP8zbCPBQvtcVUGB3QLZLvl1Byw88U+6OXSp0Yw/gEXXQLHujyLOI2G2/9P
RwNmi+6qDhhylB5TPist30uVk4D84NEDG/cz269cHe0+M+icnh2cpwysdM7fMO1S
vhWrJkyl
=8s9I
-----END PGP SIGNATURE-----

@mzumsande
Copy link
Contributor

Code Review ACK 7878f97

@fanquake fanquake merged commit 92c8e18 into bitcoin:master Jul 19, 2022
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jul 20, 2022
if (!Init()) return false;

const CBlockIndex* index = m_best_block_index.load();
if (!CustomInit(index ? std::make_optional(interfaces::BlockKey{index->GetBlockHash(), index->nHeight}) : std::nullopt)) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "indexes, refactor: Remove CBlockIndex* uses in index Init methods" (bef4e40)

Calling the CustomInit() function after the Init() function here seems to cause a race condition, noticed by @furszy and described in #27607 (comment). The problem is that Init() can set m_synced to true, which enables BlockConnected callbacks. But if BlockConnected callback runs before CustomInit finishes the index may still be only partially loaded, and the CustomAppend call in BlockConnected could fail

achow101 added a commit to bitcoin-core/gui that referenced this pull request May 31, 2023
…it' prior setting 'synced' flag

3126454 index: prevent race by calling 'CustomInit' prior setting 'synced' flag (furszy)

Pull request description:

  Decoupled from #27607.

  Fixed a potential race condition in master (not possible so far) that could become an actual issue soon.
  Where the index's  `CustomAppend` method could be called (from `BlockConnected`) before its
  `CustomInit` method, causing the index to try to update itself before it is initialized.

  This could happen because we set the index `m_synced` flag (which enables `BlockConnected` events)
  before calling to the child class init function (`CustomInit`). So, for example, the block filter index could
  process a block before initialize the next filter position field and end up overwriting the first stored filter.

  This race was introduced in bitcoin/bitcoin@bef4e40 from bitcoin/bitcoin#25494.

ACKs for top commit:
  achow101:
    ACK 3126454
  mzumsande:
    Code review ACK 3126454
  TheCharlatan:
    Nice, ACK 3126454

Tree-SHA512: 7a53fed1d2035cb4c1f331d6ee9f92d499b6cbb618ea534c6440f5a45ff9b3ac4dcff5fb4b88937f45a0be249e3a9c6dc6c3ac77180f12ae25fc56856ba39735
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request May 31, 2023
…r setting 'synced' flag

3126454 index: prevent race by calling 'CustomInit' prior setting 'synced' flag (furszy)

Pull request description:

  Decoupled from bitcoin#27607.

  Fixed a potential race condition in master (not possible so far) that could become an actual issue soon.
  Where the index's  `CustomAppend` method could be called (from `BlockConnected`) before its
  `CustomInit` method, causing the index to try to update itself before it is initialized.

  This could happen because we set the index `m_synced` flag (which enables `BlockConnected` events)
  before calling to the child class init function (`CustomInit`). So, for example, the block filter index could
  process a block before initialize the next filter position field and end up overwriting the first stored filter.

  This race was introduced in bitcoin@bef4e40 from bitcoin#25494.

ACKs for top commit:
  achow101:
    ACK 3126454
  mzumsande:
    Code review ACK 3126454
  TheCharlatan:
    Nice, ACK 3126454

Tree-SHA512: 7a53fed1d2035cb4c1f331d6ee9f92d499b6cbb618ea534c6440f5a45ff9b3ac4dcff5fb4b88937f45a0be249e3a9c6dc6c3ac77180f12ae25fc56856ba39735
achow101 added a commit that referenced this pull request Mar 20, 2024
f65b0f6 index: Move last_locator_write_time and logging to end of threadsync loop (Fabian Jahr)

Pull request description:

  In the index sync thread, when initializing an index for the first time, stop callng BaseIndex::Commit when m_best_block_index is null, to avoid a spurious "failed to commit" error from that function. This error started happening in commit 7878f97 from #25494 and was reported by pstratem in #26903 with an alternate fix.

ACKs for top commit:
  achow101:
    ACK f65b0f6
  ryanofsky:
    Code review ACK f65b0f6. Just moved log "Syncing" log line since last commit to avoid having to call now() twice.
  furszy:
    ACK f65b0f6
  TheCharlatan:
    ACK f65b0f6

Tree-SHA512: afa8f05786318d36346d167ff53ea0b3bc8abdb0ad04465d199dc3eb91e9f837369e24fcb7e24b5757b02d698ec504e61da6ac365eaf006c874fc07a424a7e20
@bitcoin bitcoin locked and limited conversation to collaborators May 17, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants
0