8000 Engine: encourage better garbage collection by josephjclark · Pull Request #932 · OpenFn/kit · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Engine: encourage better garbage collection #932

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
May 6, 2025
Merged

Engine: encourage better garbage collection #932

merged 7 commits into from
May 6, 2025

Conversation

josephjclark
Copy link
Collaborator
@josephjclark josephjclark commented Apr 30, 2025 •
edited
Loading

Short Description

This PR fixes an issue in the engine where state references are not garbage collected, and so memory leaks over time.

This is a probably cause of at least some lost runs.

Fixes #826

Implementation Details

After a heck of a lot of investigation I've worked out that the suspect memory leak flagged by @taylordowns2000 is real. I've also worked out that it comes from the engine, and if you cut out child processes and the runtime and just statically return a large object from each run, the leak still occurs

What seems to be happening is:

  • for every run we build a context object, which tracks a bunch of data for and about that run
  • This is designed to be a temporary object, dropped as soon as the run completes
  • When a run completes, we take the final state object returned by the workflow and write it to this context
  • And for whatever reason, this object does not get unreferenced and so does not get garbage collected

The fix - the dumbest, easiest fix I can find - is to force that object to be unreferenced after the workflow has finished.

I've added a new test:mem script to the engine. This creates an engine with quite a low memory limit, and then infinitely runs a job with a large-ish payload.

Against main the test fails (heap error) on my machine after about 20 runs.

At the time of writing, on this branch, I've just crossed 2000 3000 4000 5000 runs (my poor CPU!)

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

  • Code generation (copilot but not intellisense)
  • Learning or fact checking
  • Strategy / design
  • Optimisation / refactoring
  • Translation / spellchecking / doc gen
  • Other
  • I have not used AI

You can read more details in our Responsible AI Policy

@github-project-automation github-project-automation bot moved this to New Issues in v2 Apr 30, 2025
@josephjclark

This comment was marked as resolved.

Sign in to view
@josephjclark josephjclark force-pushed the garbage-collect-engine-state branch from 8f4e0fc to 71a89c2 Compare April 30, 2025 15:34
@theroinaochieng theroinaochieng moved this from New Issues to DevX Backlog in v2 May 6, 2025
@theroinaochieng theroinaochieng moved this from DevX Backlog to In progress in v2 May 6, 2025
@josephjclark josephjclark merged commit dfc16a9 into main May 6, 2025
10 checks passed
@josephjclark josephjclark deleted the garbage-collect-engine-state branch May 6, 2025 17:59
@github-project-automation github-project-automation bot moved this from In progress to Done in v2 May 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
v2
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

engine: memory leak
1 participant
@josephjclark
0