Description
Hello!
I've been using Nornir to pull data from my network and have been experimenting with ways to improve performance.
I've found that my code is more CPU bound than I/O bound because it does a bunch of normalization (it's basically a bunch of custom napalm getters that feed a postgres database), and because of this leveraging multiprocessing gives me a huge performance boost (as opposed to async which only gives me minor gains).
It was trivial to create my own Multiprocessing runner based on the ThreadedRunner
, but to get it to work I had to tweak the MultiResult
class because __getattr__
is overridden in that class and it breaks the pickling.
Will open a PR with this tweak so you can see what I mean - it's not a particularly elegant solution, but is it something you'd guys consider adding or fixing in some other way? I do get that adding support for Multiprocessing in any form is its own can of worms.