DataLabs comes pre-installed with Composapy, a python package that makes it easy to programmatically interface with Composable. Composapy allows you to interactively author, execute and retrieve results from DataFlows (Directed Acyclic Graphs, DAGs).
Visit our ReadTheDocs page a more detailed look at Composapy.
Composapy looks for the environment variable APPLICATION_URI by default (set by DataLabs). If you are using Composapy outside of the datalabs environment and the APPLICATION_URI
environment variable is not set, you can set it with keyword argument uri. You can create a session with Windows Authentication (if you are in DataLab, this will be the same as the
key on the DataLab edit screen), [string] API Token (can be generated on the composable website), or with a [tuple[str, str]] username/password combination.
from composapy.session import Session
from composapy.auth import AuthMode
session = Session(auth_mode=AuthMode.WINDOWS)
session = Session(auth_mode=AuthMode.TOKEN, credentials="<your-api-token-here>", uri="http://localhost/CompAnalytics/")
session = Session(auth_mode=AuthMode.FORM, credentials=("username", "password"))After creating a session, register it (future operations use this session by default). Optionally, enable the save flag to write to a local config file for automatical loading/registration on next Composapy import.
session.register()
session.register(save=True)You can also call get_session to get the currently registered session.
from composapy.session import get_session
session = get_session()DataFlowObject's can be initialized with the DataFlow API method - create. It takes either a json string (kwarg json) or path to a json file (kwarg file_path) as parameters. Call the save method on an unsaved DataFlowObject to save it in the Composable database. Saving it will make give it an id, making it available for use in the Composable Designer.
from composapy.dataflow.api import DataFlow
dataflow_object = DataFlow.create(file_path="simple-dataflow.json") # DataFlowObject(id=None)
dataflow_object = DataFlow.create(file_path="simple-dataflow.json").save() # DataFlowObject(id=123456)To run a saved DataFlow, you can retrieve the appId in the DataFlow's URL.
dataflow_run = DataFlow.run(444333) # DataFlowRun(id=444333)To run a DataFlow that has external input modules, use the external_inputs kwarg, which is a dictionary with key equal to the external modules name field and the value equal to what you want to pass in.
dataflow_run = DataFlow.run(444333, external_inputs={"a_string_external_input": "foo string"}) # DataFlowRun(id=444333)dataflow_run.modules[0] # Module(name='Calculator', type=Calculator)dataflow_object.modules.filter(name="calc module name")[0].inputs.first() # Input(name=Param1, type=Double, value=1.0)dataflow_run.modules.get(name="string module name").result # Result(name='foo name', type=String, value='foo value')A Composapy Key can be retrieved using the python Key API.
from composapy.key.api import Key
key_object = Key.get(123456) # KeyObject(name='some name', type='StringConnectionSettings')Optionally, if your key name is unique, you can retrieve it by name.
key_object = Key.get(name="a unique name") # KeyObject(name='a unique name', type='SqlConnectionSettings')Similar to Session, KeyObjects can be registered with the save=True flag. This will save the key id in the working directory's "composapy.ini" file, which will then be loaded when the Composapy package is imported.
key_object.register(save=True)You can retrieve the currently registered key_object with get_key_object.
from composapy.key.models import get_key_object
get_key_object() # KeyObject(name='a unique name', type='SqlConnectionSettings')Keys can be searched by name. It returns a list of key objects.
key_objects = Key.search("same name") # [KeyObject(name='same name', type='SqlConnectionSettings'), KeyObject(name='same name', type='StringConnectionSettings')]from composapy.queryview.api import QueryView
driver = QueryView.driver() # you can create a driver and then connect with a key...
driver.connect(key_object)
driver = QueryView.driver(key_object) # ... or create a driver using the key as an argumentIf there is a currently registered key, there is no need to pass the key to the driver object; the driver will be automatically initialized with the database connection.
QueryView.driver()df = driver.run("select * from table_name") # returns a Pandas DataFrame of your queryIn a notebook (IPython) environment, custom sql magic commands are loaded on import with Composapy. Note that both %sql and %%sql require a registered Session and KeyObject. Both can be loaded on import if saved in the "composapy.ini" config file. See register-session and register-key.
%sql select * from some_table --returns the same result as QueryView.driver("select * from some_table")Instead of doing line magic (only takes that line as input), you can alternatively do cell magic (takes entire cell as input).
select
*
from
some_table
where
some_id is not nullAlso, if you want to interface with pythonnet more closely, you can find our csharp documentation here.