Add compiler for Python to procedural SQL translation #2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Using top level logger - Remove unnessesary error handling for saving funccall in exception - Outsource to actual fallback where funccall is extracted by current df
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adding a compiler to translate Python statements into a procedural langauge based on the db-type.
This compiler was developed with the support of @sthagedorn at the Department of Databases and Information Systems at the TU Ilmenau as part of a bachelor thesis.
Changes
_generateCreateFuncfunction in sqlgenerator module.Detailed modifications
Add SQL calls for creating procedural functions and other database specific mappings like datatypes, error types and functions to the grizzly.yml file.
Add exceptions to catch errors while compiling and so enabling fallback and user information.
Change
showfunction and add_fallbackfunction in frame module to enable dataframe fallback.Add fallback and language parameters when creating UDF FuncCall with the map function. Default is
language = 'py'for PL/Python UDF execution andfallback=Falseto prevent fallbacks to PL/Python or local Pandas execution of the UDF.from the last Grizzly DataFrame object without the projection of the udf
Add
to_dffunction to generator.py and relationaldbexecutor.py to execute generated SQL from the last Grizzly DataFrame object without the projection of the udf with the pandasread_sqlfunction to create a DataFrame for local fallback execution.Add an example for udf execution in dbms with procedural functions in example.py.
Add the udfcompiler_test.py file for testing compiler with UDFs from grizzly.udfcompiler.test_udfs module.
Add details about performancetests in grizzly.udfcompiler.speedtest module with queries and times.
Add cx_oracle and psycopg2 dependencies in RelationalExecutor Module to check the given connection type and automatically create the
SQLGeneratorObject with the corresponding profile.If not desired, user has to manually set the db profile when creating the
RelationalExecutorwith theSQLGenerator:grizzly.use(RelationalExecutor(con, SQLGenerator('oracle')).Remove '_' in name of temporary tables, caused trouble in PostgreSQL.