Tags: asingh16/dagster
Tags
[RunQueue] RemoteDagitRunQueue
Summary: Adds a `RunQueueClient` that can target a remote dagit running a `RunQueueMaster`
Test Plan:
With instance config
```
run_queue:
module: dagster.core.run_queue
class: LocalComputeQueue
config:
```
run `dagit -p 3333`
In another terminal with DAGSTER_HOME set to instance config
```
run_queue:
module: dagster_graphql.remote.run_queue
class: RemoteDagitRunQueue
config:
address: http://localhost:3333
```
Run
```
dagster-graphql -p enqueuePipelineExecution -v '{"executionParams": {"selector": {"name": "sleepy_pipeline"}, "mode": "default"}}'
```
Observe run happens successfully on terminal 1
Reviewers: #ft
Subscribers:
Allow custom type_check on PythonObjectType Summary: This lets users pass a custom type check in place of the default isinstance check when using the PythonObjectType/as_dagster_type/@as_dagster_type machinery. Stacked on D1273 Test Plan: Unit Reviewers: #ft, alangenfeld, schrockn Differential Revision: https://dagster.phacility.com/D1281
Improve type inference Summary: Previously, it was impossible to create Dagster types directly wrapping built-ins and extensions, because we attempted to monkeypatch the Python type with an annotation pointing to the Dagster type. Additionally, attempting to annotate a solid with a Python type outside of a narrow set would result in an unexpected error, if that type were not explicitly wrapped in as_dagster_type or equivalent. This diff replaces the monkeypatching machinery with an explicit runtime type registry, and also adds a facility for automatically wrapping bare Python types in anonymous Dagster types. Test Plan: Unit Differential Revision: https://dagster.phacility.com/D1273 Nits
Allow custom type_check on PythonObjectType Summary: This lets users pass a custom type check in place of the default isinstance check when using the PythonObjectType/as_dagster_type/@as_dagster_type machinery. Stacked on D1273 Test Plan: Unit Reviewers: #ft, alangenfeld, schrockn Differential Revision: https://dagster.phacility.com/D1281
(intermediate-storage-instance-config-4) Split out DagsterInstance.lo… …cal_temp into local_temp and explicit_local_temp Summary: just looking Test Plan: bk Differential Revision: https://dagster.phacility.com/D1313
(intermediate-storage-instance-config-5) Make context a required argu… …ment in the intermediates layer Summary: todo Test Plan: bk Differential Revision: https://dagster.phacility.com/D1314
(intermediate-storage-instance-config-5) Make context a required argu… …ment in the intermediates layer Summary: todo Test Plan: bk Subscribers:
(intermediate-storage-instance-config-4) Split out DagsterInstance.lo… …cal_temp into local_temp and explicit_local_temp Summary: just looking Test Plan: bk Subscribers:
(intermediate-storage-instance-config-3) Store dict instead of yaml o… …n ConfigPluginData Summary: Seems like we should convert yaml to dict at the periphery so we don't have to convert back and forth Test Plan: BK Reviewers: #ft, max, alangenfeld Differential Revision: https://dagster.phacility.com/D1308
(intermediate-storage-instance-config-2) Substanially restructure Con…
…figurableClass as ConfigPlugin
Summary: First this is a relatively big change in one diff. I'm up for
splitting this up but I do believe it is manageable as is.
I was sorting through the ConfigurableClass stuff and was a bit
confused. It seemed overly complex for what it was trying to accomplish,
so I decided to rethink this a little bit. I think the core problem
was the multiple inheritance, which muddled the clarity of this
subsystem. This is a capability which is really a lot simpler
and is in the province of the config type system, rather than where
it used to live, in serdes.
A config plugin boils down to a config schema and a function that
takes a value validated by that schema and produces an object. It's
very similar to input schema and output schema actually, except
that the config is only evaluated lazily rather than upfront.
With that new framing, we can express this more simply as a decorated
function. Instead of inheriting from ConfigurableClass you just implement
a function decorated by @config_plugin.
An implementation looks like this:
```
@config_plugin(SystemNamedDict('PostgresRunStorageConfig', {'postgres_url': Field(String)}))
def postgres_event_log_storage_config_plugin(plugin_config):
return PostgresEventLogStorage(plugin_config['postgres_url'])
```
To do this I moved the "inst_info" functionality up to the
DagsterInstance, which knows whether or not the objects were
constructed via the plugin system or not. This keeps the
fact there is a plugin system at all out of the core objects,
which is preferable in my view.
I was throughly confused by the arbitary kwargs flowed up and down
this codepath. It was unused, so I eliminated it. I think shuffling
the config values to the __init__ method manually is far more explicit
and clear than relying on kwargs explosion.
This ends up simplifying both the implementation (negative LOC) and
the user-facing API, and is a nice win all-around.
Note that I will have to test the dagster-aws stuff in a follow on diff,
because the best it can is run against master, rather than a local copy.
Test Plan: BK. View instance info in Dagit
Reviewers: max, alangenfeld, #ft
Differential Revision: https://dagster.phacility.com/D1306
PreviousNext