Description
Platform we're running on : Ubuntu 18.04, Python 3.6.8
Context: We have a long running service that uses watchdog to watch a log file and parse it when it's moved.
Anyways, we've been tracing a memory leak in it with tracemalloc and it appears like watchdog is the culprit, it's the top memory consumer compared to the snapshot taken when starting the app:
/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py:0: size=2470 KiB (+2388 KiB), count=47172 (+47157), average=54 B
some more showing it's one of the top mem incrementors:
/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py:243: size=1280 KiB (+704 KiB), count=2 (+0), average=640 KiB
/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py:462: size=1634 KiB (+184 KiB), count=33095 (+3379), average=51 B"
/opt/app/env/lib/python3.6/site-packages/watchdog/utils/unicode_paths.py:59: size=151 KiB (+151 KiB), count=2016 (+2016), average=77 B
/opt/app/env/lib/python3.6/site-packages/watchdog/events.py:118: size=114 KiB (+112 KiB), count=2558 (+2531), average=46 B
When looking at the tracebacks for this, these are the top memory consuming lines:
81837 memory blocks: 3994.4 KiB
File "/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py", line 462
self._wd = wd
56312 memory blocks: 3342.1 KiB
File "/opt/app/env/lib/python3.6/posixpath.py", line 90
path += b
56313 memory blocks: 3080.0 KiB
File "/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py", line 323
inotify_event = InotifyEvent(wd, mask, cookie, name, src_path)
217 memory blocks: 2573.5 KiB
File "/usr/lib/python3.6/ssl.py", line 651
return self._sslobj.peer_certificate(binary_form)
2 memory blocks: 2560.4 KiB
File "/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py", line 243
self._moved_from_events[event.cookie] = event
So it seems to me that the issue is the InotifyEvent objects that get created are never closed / garbage collected until the Observer object stops / joins.
I've tried issuing a del event after we are finished using it but this has no perceptible effect, something else is holding on to a reference to the event.
How could we "forcibly" close or delete the InotifyEvent or FileSystemEvent when we're done with it?