8000 Revert cycle handling. by style95 · Pull Request #5300 · apache/openwhisk · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Revert cycle handling. #5300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Aug 2, 2022
Merged

Revert cycle handling. #5300

merged 2 commits into from
Aug 2, 2022

Conversation

style95
Copy link
Member
@style95 style95 commented Jul 31, 2022 •
edited
Loading

Description

This is to revert the recent change.
#5251 is introduced to handle the cyclic activations storm in a queue manager.
But with this change, I observed severe performance degradation.

This is the performance measurement with the current master.
image

And this is the result of this change.
image

This is because even for normal activations, the queue manager tries to recover a queue.
There is a gap between the time that leader election is done(etcd data is written) and the queue is ready.
If activations come during that time, the queue manager tries to recover a queue and it increases the wait time.

And the result is well aligned with the one I performed for the initial run.
#5194 (comment)

Since the cycle in a critical path is also a big problem, we must handle it.
But until we find a better way to address it I think it would be better to revert this change.

Related issue and scope

  • I opened an issue to propose and discuss this change (#????)

My changes affect the following components

  • API
  • Controller
  • Message Bus (e.g., Kafka)
  • Loadbalancer
  • Scheduler
  • Invoker
  • Intrinsic actions (e.g., sequences, conductors)
  • Data stores (e.g., CouchDB)
  • Tests
  • Deployment
  • CLI
  • General tooling
  • Documentation

Types of changes

  • Bug fix (generally a non-breaking change which closes an issue).
  • Enhancement or new feature (adds new functionality).
  • Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

  • I signed an Apache CLA.
  • I reviewed the style guides and followed the recommendations (Travis CI will check :).
  • I added tests to cover my changes.
  • My changes require further changes to the documentation.
  • I updated the documentation where necessary.

@style95
Copy link
Member Author
style95 commented Jul 31, 2022

I opened an issue for the cycle.
#5301

@codecov-commenter
Copy link
codecov-commenter commented Jul 31, 2022 •
edited
Loading

Codecov Report

Merging #5300 (1b003b9) into master (3014b87) will decrease coverage by 3.63%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #5300      +/-   ##
==========================================
- Coverage   79.97%   76.34%   -3.64%     
==========================================
  Files         238      238              
  Lines       14199    14160      -39     
  Branches      577      601      +24     
==========================================
- Hits        11356    10810     -546     
- Misses       2843     3350     +507
Impacted Files Coverage Δ
...in/scala/org/apache/openwhisk/common/Logging.scala 79.29% <ø> (-7.55%) ⬇️
.../openwhisk/core/scheduler/queue/QueueManager.scala 83.27% <100.00%> (+0.20%) ⬆️
...core/database/cosmosdb/RxObservableImplicits.scala 0.00% <0.00%> (-100.00%) ⬇️
...ore/database/cosmosdb/cache/CacheInvalidator.scala 0.00% <0.00%> (-100.00%) ⬇️
...e/database/cosmosdb/cache/ChangeFeedConsumer.scala 0.00% <0.00%> (-100.00%) ⬇️
...core/database/cosmosdb/CosmosDBArtifactStore.scala 0.00% <0.00%> (-95.85%) ⬇️
...sk/core/database/cosmosdb/CosmosDBViewMapper.scala 0.00% <0.00%> (-93.90%) ⬇️
...tabase/cosmosdb/cache/CacheInvalidatorConfig.scala 0.00% <0.00%> (-92.31%) ⬇️
...enwhisk/connector/kafka/KamonMetricsReporter.scala 0.00% <0.00%> (-83.34%) ⬇️
...e/database/cosmosdb/cache/KafkaEventProducer.scala 0.00% <0.00%> (-78.58%) ⬇️
... and 36 more

Help us with your feedback. Take ten seconds to tell us how you rate us.

@style95 style95 force-pushed the revert-cycle-handling branch from b2a2d78 to 1b003b9 Compare August 1, 2022 01:15
@bdoyle0182
Copy link
Contributor

Do you think this one is related to any of the bugs I've reported to you?

@bdoyle0182
Copy link
Contributor

Also when was this introduced? Has this been on the open source branch since we said the scheduler was functional or was it added in a recent commit?

@style95
Copy link
Member Author
style95 commented Aug 1, 2022

@bdoyle0182
If you haven't seen any huge logs printing the same repeatedly, you haven't hit the issue.
The change I revert here is relatively recently added after I made a change to run the scheduler.
This is the change: #5251

@style95 style95 merged commit 2683ed1 into apache:master Aug 2, 2022
msciabarra pushed a commit to nuvolaris/openwhisk that referenced this pull request Nov 23, 2022
@style95 @michele-sciabarra
Revert cycle handling. (apache#5300)
0a397e1 
* Revert cycle handling.

* Remove the RecoverQueue reference.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jiangpengcheng jiangpengcheng jiangpengcheng approved these changes

@ningyougang ningyougang ningyougang approved these changes

@bdoyle0182 bdoyle0182 bdoyle0182 approved these changes

@KeonHee KeonHee Awaiting requested review from KeonHee

@upgle upgle Awaiting requested review from upgle

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@style95 @codecov-commenter @bdoyle0182 @jiangpengcheng @ningyougang
0