8000 static graph visualization · Issue #82 · cylc/cylc-ui · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

static graph visualization #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
hjoliver opened this issue May 27, 2019 · 9 comments
Open

static graph visualization #82

hjoliver opened this issue May 27, 2019 · 9 comments
Milestone
Pending

Comments

@hjoliver
Copy link
Member

We probably want to replace cylc graph with web technology too, not just the control GUI graph view.
@hjoliver
Copy link
Member Author

This graph demo (found by @MartinRyan ) allows manual node repositioning (as many do, I think): https://cytoscape.org/cytoscape.js-euler/. Could be useful when automatic layout fails us.

@MartinRyan
Copy link
Contributor
MartinRyan commented May 27, 2019 •
edited
Loading

Cytoscape looks great, it has a dagre extension, and a navigator 'bird's eye view and pan control' extension (mini-map). It consumes JSON though there are convertors for graph objects.

@hjoliver
Copy link
Member Author
hjoliver commented May 27, 2019 •
edited
Loading

Also from @MartinRyan py2cytoscape might be useful for static vis. (Cytoscape itself might be useful for the live control GUI as well)

@oliver-sanders oliver-sanders mentioned this issue May 28, 2019
Closed
14 tasks
@oliver-sanders
Copy link
Member

This is an interesting challenge because:

  • Users typically run cylc graph to visualise a suite whilst developing the working copy.
  • The UI-Server is concerned with the installed copy (in the cylc-run directory).

The UI-Server is only concerned with the installed copy and is not necessarily able to see the working copy.

It is OK to require users to register a suite for cylc graph functionality (as the suite wouldn't appear in the web UI otherwise) but we wouldn't want users to have to repeatedly re-install a suite in order to visualise it.

@hjoliver
Copy link
Member Author

Well, yes and no.

If you're working on the working copy then you are able to install it repeatedly pretty easily, and if the payoff is kick-ass web base viz that seems a small price to pay, And besides you currently have to do that anyway if your suite.rc takes Jinja2 inputs from rose-suite.conf (it won't validate or graph without being installed first).

Also what's the alternative? We don't want to keep a GTK-based static graph GUI. An electron app might allow us to kill both birds with one stone, I guess.

@matthewrmshin
Copy link
Contributor

Assuming that we have done enough logical migration from rose suite-run to Cylc in the coming months, then we should have enough logic in Cylc to deal with the rose-suite.conf stuffs. The web UI can have an local file system browser so users should be able to open a suite from its source location. Otherwise, we can also build support for the UI to search for named suites from various configurabl 8000 e locations on the file system. (E.g. If a suite cannot be found under ~/cylc-run/SUITE/ then look at ~/roses/SUITE/.)

@oliver-sanders
Copy link
Member

If you're working on the working copy then you are able to install it repeatedly pretty easily

But what if the suite is running?

Also this is the situation we have at the moment, I advise people to do this:

$ rose suite-run -l; cylc graph mysuite

Unfortunately this causes confusion and is a barrier to usage. As a result very few people here use essential Cylc utilities such as:

  • cylc graph
  • cylc get-config
  • cylc view --jinja2

I wrote this page to try and break down the barrier but sadly it has made little difference.

I still regularly see users trying is running the entire suite to test changes.

Also what's the alternative?

No nice solution:

  • An electron app if it's quick and simple.
  • The UI Server could attempt to access the working copy (via ~/cylc-run/<suite>/.service/source)
    • If the working copy is not available it could fallback to the installed copy and present a warning.
  • Require the user repeatably re-installs the suite during development.

@hjoliver
Copy link
Member Author

Fair point on barriers to easy use. So let's do better than that if we can. We do need a non-PyGTK static vis tool with cylc-8 though, one way or another.

I still regularly see users trying is running the entire suite to test changes.

Same, and even worse, running an entire beast of a suite repeatedly to test how some cylc feature works, when a 5-line 1-task test suite would do. You can lead a horse to water, but you can't make it change its spots. Or something.

@kinow
Copy link
Member
kinow commented Aug 12, 2019

See comments from (now closed) #81 about graphs in Cylc 8:

@hjoliver

A Vue component to draw directed acyclic graphs in the browser leveraging dagre-d3v4 (https://github.com/smbolton/dagre-d3v4) - https://github.com/vdidonato/vue-dagre-d3v4 - seems to be broken.

#81 (comment)

@oliver-sanders

I'm currently looking at Javascript libraries for graphing, I'll update this issue with conclusions as I investigate. I'll probably put up my experiments as a GitHub repo when more complete.

Graphing consists of two components:

The Layout Engine

The bit which arranges nodes / edges spatially. See the layout engines supported by cytoscape for examples.

  • Directed hierarchical graphs are hard
  • Layout is slow, there isn't a whole lot of choice when it comes to layout engines. Dagre is the standout but is a fairly stagnant project.
  • More arbitrary layouts are easy
  • We might want to automatically swap to a less ideal layout engine for larger graphs.

Requirements

  1. Directed & hierarchical
  2. Must handle edges spanning multiple levels
  3. Width of node considered in graph (i.e. nodes cannot overlap)
  4. Speed!

Results

Engine 1 2 3 4 Supported Engines
Dagre 🆗 🆗 🆗 4.1s cytoscape, d3
Klay 🆗 ⛅ 🚫 6.4s cytoscape, d3
D3-DAG 🆗 🆗 🚫 [2] >> 60s [3] None? [4]

[1] Benchmark using the first cycle of MO global opp suite
[2] Possibly using the separation function though I can't get this to work
[3] Tested "Vert", "Greedy" and "Spread" layout methods
[4] Visualisation constructed by user provided functions, i.e. user provides the engine

The Graph Engine

The bit which renders the graph, provides an interface for manipulating and interacting with it. This is the component we will end up working with / extending to form the graph view.

Requirements

  1. Mobile friendliness
  2. Hackability of the graph:
    • Job status indicators (require more than just text labels)
    • Sub-workflows?
    • Advanced collapsing (e.g. parametrisations using an ellipsis symbol)?
  3. Not a massive RAM hog
  4. Support for the desired layout engine(s)
  5. Node / family collapsing

Results

Engine 1 2 3 4 5
cytoscape 🆗 Dagre, Klay
d3 Dagre, Klay
vis.js
arbor

Hybrids

Systems where the graph engine and layout engine are integrated:

Results

Engine L1 L2 L3 L4 G1 G2 G3 G4 G5
viz.js 🆗 🆗 🆗 🆗

#81 (comment)

@hjoliver

Interactive arbitrary grouping: select a bunch of nodes (individually, or by drawing a rectangle around them) and group them into a single node.
Layout algorithms for visualizing directed acyclic graphs - https://github.com/erikbrinkman/d3-dag - new-ish? demo https://jsfiddle.net/ye2xanf9/12/

#81 (comment)
#81 (comment)
8000

@kinow kinow added this to the 0.2 milestone Sep 14, 2019
@kinow kinow modified the milestones: 0.2, 0.3 Jan 7, 2020
@oliver-sanders oliver-sanders modified the milestones: 0.3, 1.0 Mar 4, 2021
@kinow kinow modified the milestones: 1.0, 2.0 Sep 10, 2021
@renovate renovate bot mentioned this issue May 20, 2022
Closed
1 task
@oliver-sanders oliver-sanders modified the milestones: 2.0.0, Pending Jun 8, 2022
@renovate renovate bot mentioned this issue Oct 26, 2022
Merged
1 task
@oliver-sanders oliver-sanders mentioned this issue Oct 27, 2022
Merged
7 tasks
@renovate renovate bot mentioned this issue May 28, 2023
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Pending
Development

No branches or pull requests
5 participants
@MartinRyan @kinow @matthewrmshin @hjoliver @oliver-sanders
0