8000 Hatchet multi-process problem when handling large graphs · Issue #426 · hatchet/hatchet · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
Hatchet multi-process problem when handling large graphs #426
Open
@Jokeren

Description

@Jokeren

Problem

The following code will be hanging there forever with the default number of procs for g_range.filter. However, if num_procs=1, g_range.filter could return reasonable results.

Code

import hatchet, json
g_range = hatchet.GraphFrame.from_literal(json.load(open('db-range.json', 'r')))
g_range.show_metric_columns()
5E31 g_range_filtered = g_range.filter(lambda x: x['GINS (I)'] > 0.0)
print(g_range_filtered.tree('GINS (E)'))

Tested under hatchet 0283a68, and Python 3.8.5.

Please rename db-range.txt to db-range.json

db-range.txt

@slabasan Could you take a look? It's a database generated by our new hpcprof implementation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: graphframePRs and Issues involving Hatchet's core GraphFrame datastructure and associated classespriority: highHigh priority issues and PRstype: bugIdentifies bugs in issues and identifies bug fixes in PRs

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      0