8000 Tweak to MultiResult to allow pickling for multiprocessing runner · Issue #994 · nornir-automation/nornir · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
Tweak to MultiResult to allow pickling for multiprocessing runner #994
Open
@amylieb

Description

@amylieb

Hello!

I've been using Nornir to pull data from my network and have been experimenting with ways to improve performance.

I've found that my code is more CPU bound than I/O bound because it does a bunch of normalization (it's basically a bunch of custom napalm getters that feed a postgres database), and because of this leveraging multiprocessing gives me a huge performance boost (as opposed to async which only gives me minor gains).

It was trivial to create my own Multiprocessing runner based on the ThreadedRunner, but to get it to work I had to tweak the MultiResult class because __getattr__ is overridden in that class and it breaks the pickling.

Will open a PR with this tweak so you can see what I mean - it's not a particularly elegant solution, but is it something you'd guys consider adding or fixing in some other way? I do get that adding support for Multiprocessing in any form is its own can of worms.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      0