8000 optimize ctime transform statement · Issue #584 · teragrep/pth_10 · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

optimize ctime transform statement #584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
elliVM opened this issue May 14, 2025 · 0 comments · May be fixed by #585
Open

optimize ctime transform statement #584

elliVM opened this issue May 14, 2025 · 0 comments · May be fixed by #585
Assignees
@elliVM

Comments

@elliVM
Copy link
Contributor
elliVM commented May 14, 2025

Description
string conversion to time might be costly operation which could be moved to private final class member field. the same goes for the timeFormat which is unlikely to differ across the calls to the Ctime udf.

Notes:

  • timeformat is final and set in AbstractConvertStep and can be moved to the class
  • Conversion could possibly be done using functions.from_unixtime from spark without the need to use custom function
  • time format is non-standard we should consider switching to use ISO8061
@elliVM elliVM self-assigned this May 14, 2025
@elliVM elliVM changed the title optimize ctime transform statement udf optimize ctime transform statement May 14, 2025
@elliVM elliVM linked a pull request May 14, 2025 that will close this issue
replace custom UDF with spark from_unixtime for ctime transformation #585
Open
@elliVM elliVM mentioned this issue May 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@elliVM elliVM

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

replace custom UDF with spark from_unixtime for ctime transformation elliVM/pth_10
1 participant
@elliVM
0