8000 Active deadlock detection by martinkersner · Pull Request #442 · Bisonai/orakl · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Active deadlock detection #442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 51 commits into from
Mar 27, 2023
Merged

Active deadlock detection #442

merged 51 commits into from
Mar 27, 2023

Conversation

martinkersner
Copy link
Member
@martinkersner martinkersner commented Mar 24, 2023 •
edited
Loading

Description

This PR actively deals with potential heartbeat deadlock. We never want the heartbeat cycle to die. The reason for the heartbeat complications is because the cycle is loosely connected. If any of the following jobs dies, then the cycle is broken.

graph TD;
    eventJob-->aggregatorJob;
    checkHearbeatJob-->submitHeartbeatJob;
    submitHeartbeatJob-->heartbeatJob;
    deviationJob-->reporterJob;
    deviationJob-->submitHeartbeatJob;
    heartbeatJob-->aggregatorJob;
    aggregatorJob-->reporterJob;
    aggregatorJob-->submitHeartbeatJob;
Loading

Other changes

  • deployment and migration files of 11 data feeds for Baobab
  • fix wrong computation of aggregator hash
  • data feed watchman (port 5001 by default) and ephemeral state
  • integration of data feed watchman in Orakl CLI
  • graceful shutdown for data feed services
  • custom retry of failed transactions
  • move submission of new heartbeat round to separate job
  • setup attempts and backoff configs for queue jobs
  • limit calls to OracleRoundState in Aggregator contract
  • fix removal of repeatable job in fetcher
  • tool to flush queues in fetcher

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Checklist before requesting a review

  • I have performed a self-review of my code.

Deployment

  • Should publish npm package
  • Should publish Docker image

@martinkersner martinkersner self-assigned this Mar 24, 2023
@martinkersner martinkersner linked an issue Mar 24, 2023 that may be closed by this pull request
Active deadlock detection #441
Closed
@martinkersner
feat: heartbeatJob aggregator removeOnFail: true
72e6a5a
@martinkersner
feat: Avoid issues due to missing error handlers in datafeed workers
ac4880c
@martinkersner
chore: Unify heartbeat job name
b95ae41
@martinkersner
chore: aggregatorAddress -> oracleAddress
67540cc
@martinkersner
feat: Setup retry mechanism for jobs in heartbeatQueue
f4ad940
@martinkersner
fix: Use more correct variable names for building job IDs
84aed39
@martinkersner
feat: Aggregator queue settings
dd96892
@martinkersner
bugfix: aggregator hash generation & verification
607c711
@martinkersner
fix: Wrong decomposition of oracleAddress param in API
15c3ec6
@martinkersner
feat: Baobab aggregator deployment auxiliary script
6f7a9c8
@martinkersner
chore: Add debug log
64f05e4
@martinkersner
feat: Get reporter address from listener using Orakl API
4a3840e
@martinkersner
fix: Aggregator listing in Orakl Network CLI
0d45607
@martinkersner
wip: Aggregator state + watchman
7a2d43d
@martinkersner
chore: Reporter cleanup
9bd1604
@martinkersner
chore: Reorganize data feed utils
a468125
@martinkersner
feat: Refresh heartbeat job aggregators does not seem as a good idea
c1b70a4
@martinkersner
feat: Access aggregators from local aggregator state
02310f6
@martinkersner
feat: Check whether given aggregatorHash is active in local state
4df6963
@martinkersner
feat: Terminate heartbeat job through change in local aggregator state
df51154
@martinkersner
feat: Connect aggregator state to watchman
c3db4be
@martinkersner
feat: Connect Orakl CLI to Aggregator Watchman
90b3090
@martinkersner
feat: Port 5000 is taken by ControlCenter MacOs app
f324704
@martinkersner
fix: Incomplete code migration
ed5d5cd
@martinkersner
feat: Flush queues in Orakl Fetcher
9f69819
@martinkersner
chore: Add comments
cb661ad
@martinkersner
fix: Aggregator hashes in tests
b9bf54b
@martinkersner
feat: Migration files for aggregator on baobab
2f978d2
@martinkersner
feat: Print error messages within sendTransaction
2c89dcc
@martinkersner
feat: Catch missing response error
e1d48d7
@martinkersner
feat: Catch Transaction failed error
4fa579b
@martinkersner
fix: Remove repeatable job insted of delayed job
0ab7867
@martinkersner
feat: Retry transaction under special circumstances
ea40946
@martinkersner
feat: Resubmit heartbeat by separate submitHeartbeatJob job
1a23c5d
@martinkersner
feat: Data Feed Worker graceful shutdown
407e470
@martinkersner
feat: Data Feed Reporter graceful shutdown
925cad7
@martinkersner
fix: Read aggregatorAddress from top level of deploy config dict
56e2303
@martinkersner
feat: Graceful shutdown of watchman
bec82d1
@martinkersner
feat: Use 5 concurrency BullMQ reporter workers
a595e92
@martinkersner
fix: Remove outdated job back off strategy
df89f0b
@martinkersner
feat: Update property names of IOracleRoundState
3e5ff81
@martinkersner
feat: Launch initial heartbeat jobs through submitHeartbeatQueue
54f20a3
@martinkersner
feat: Limit number of calls for oracleRoundState
4e2b8b1
@martinkersner
feat: Graceful shutdown for data feed listener
50689a6
@martinkersner
fix: eslint
d7b5806
@martinkersner
feat: Pulse check on aggregator heartbeat
86cff6f
@martinkersner
fix: tests
b143ae3
@martinkersner
fix: Rollback localhost aggregator migration
bf3ac49
@martinkersner martinkersner marked this pull request as ready for review March 26, 2023 11:13
@martinkersner martinkersner linked an issue Mar 27, 2023 that may be closed by this pull request
Data Feed Reporter Bottleneck #425
Closed
@martinkersner martinkersner mentioned this pull request Mar 27, 2023
Closed
@martinkersner martinkersner merged commit 545feaa into master Mar 27, 2023
@martinkersner martinkersner deleted the i-441/feat/active-deadlock-detection branch March 27, 2023 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bayram98 bayram98 Awaiting requested review from bayram98

@jo-bisonai jo-bisonai Awaiting requested review from jo-bisonai

@KelvinThai KelvinThai Awaiting requested review from KelvinThai

Assignees

@martinkersner martinkersner

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Active deadlock detection Data Feed Reporter Bottleneck
1 participant
@martinkersner
0