8000 PR2 for zOS support by v1gnesh · Pull Request #7973 · duckdb/duckdb · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

PR2 for zOS support #7973

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jul 24, 2023
Merged

PR2 for zOS support #7973

merged 10 commits into from
Jul 24, 2023

Conversation

v1gnesh
Copy link
Contributor
@v1gnesh v1gnesh commented Jun 17, 2023

I done goof'd with lightweightsemaphore.h in the first PR.
Have verified that it builds fine with the unix block, so just using that.
My upstream PR to concurrentqueue does this correctly.

There's a situation with jemalloc_internal_types.h.
For example, here's the related command from compiler_commands.json:

{
  "directory": "/v1g/zopen/dev/duckdbport/duckdb/extension/jemalloc/jemalloc",
  "command": "/usr/lpp/IBM/oelcpp/v2r0/bin/clang++ -DDUCKDB_BUILD_LIBRARY -DJEMALLOC_NO_PRIVATE_NAMESPACE -DJEMALLOC_NO_RENAME -I/v1g/zopen/dev/duckdbport/duckdb/src/include -I/v1g/zopen/dev/duckdbport/duckdb/third_party/fsst -I/v1g/zopen/dev/duckdbport/duckdb/third_party/fmt/include -I/v1g/zopen/dev/duckdbport/duckdb/third_party/hyperloglog -I/v1g/zopen/dev/duckdbport/duckdb/third_party/fastpforlib -I/v1g/zopen/dev/duckdbport/duckdb/third_party/fast_float -I/v1g/zopen/dev/duckdbport/duckdb/third_party/re2 -I/v1g/zopen/dev/duckdbport/duckdb/third_party/miniz -I/v1g/zopen/dev/duckdbport/duckdb/third_party/utf8proc/include -I/v1g/zopen/dev/duckdbport/duckdb/third_party/miniparquet -I/v1g/zopen/dev/duckdbport/duckdb/third_party/concurrentqueue -I/v1g/zopen/dev/duckdbport/duckdb/third_party/pcg -I/v1g/zopen/dev/duckdbport/duckdb/third_party/tdigest -I/v1g/zopen/dev/duckdbport/duckdb/third_party/mbedtls/include -I/v1g/zopen/dev/duckdbport/duckdb/third_party/jaro_winkler -I/v1g/zopen/dev/duckdbport/duckdb/extension/jemalloc/include -I/v1g/zopen/dev/duckdbport/duckdb/extension/jemalloc/jemalloc/include -DNSIG=42 -D_XOPEN_SOURCE=600 -D_ALL_SOURCE -D_OPEN_SYS_FILE_EXT=1 -D_AE_BIMODAL=1 -D_ENHANCED_ASCII_EXT=0xFFFFFFFF  -DZOSLIB_OVERRIDE_CLIB=1 -fzos-le-char-mode=ascii -mnocsect -fno-short-enums -Wno-error -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -O3  -fgnu-keywords -I/v1g/zopen/prod/zoslib-zopen/include -O3 -DNDEBUG -O3 -DNDEBUG   -fPIC -w -std=c++11 -o CMakeFiles/jemalloc.dir/src/ctl.cpp.o -c /v1g/zopen/dev/duckdbport/duckdb/extension/jemalloc/jemalloc/src/ctl.cpp",
  "file": "/v1g/zopen/dev/duckdbport/duckdb/extension/jemalloc/jemalloc/src/ctl.cpp"
},

ctl.cpp is being built with clang++, ok.
However, it refers to the internal types header file, which looks for __STDC_VERSION__.
At least on zOS, clang++ doesn't know about __STDC_VERSION__, so it goes down this path of thinking it's < 199901L.

Have addressed this, but wanted to check if this requires any further investigation.

v1gnesh added 3 commits June 17, 2023 09:53
assert.hpp - make MVS take the else block
linenoise.cpp - include strings and sys/time
tsd.cpp - include pthread and use pthread_equal
@Mytherin
Copy link
Collaborator

Maybe just disable jemalloc on zOS instead?

@v1gnesh
Copy link
Contributor Author
v1gnesh commented Jun 20, 2023

With the patches in this PR, it goes through OK.
What's left is to figure out linking errors.. see if static linking is needed.

@github-actions github-actions bot marked this pull request as draft June 26, 2023 04:18
@v1gnesh
Copy link
Contributor Author
v1gnesh commented Jun 26, 2023

@Mytherin Wise words indeed.
After disabling jemalloc extensions and the address sanitizer, it builds & works ok.

@v1gnesh v1gnesh marked this pull request as ready for review June 26, 2023 13:29
@Mytherin Mytherin changed the base branch from feature to master July 4, 2023 13:17
@Mytherin
Copy link
Collaborator
Mytherin commented Jul 6, 2023

Thanks - looks fine to me. Could you just fix the failing CI?

@v1gnesh
Copy link
Contributor Author
v1gnesh commented Jul 6, 2023

CodeQuality - I'll push a commit for the format-check

Windows - I see you were working on the ODBC tests, and the error is probably related to that (?)

LinuxRelease - Fails on Deploy with amalgamation.py unable to find zos-tls.h, which is generally available here.

for x in regex_include_statements:
    included_file = x[1]
    if skip_duckdb_includes and 'duckdb' in included_file:
        continue
+   if ('pg_functions.cpp' or 'src_backend_parser_scansup.cpp' in fpath) and included_file == 'zos-tls.h':
+       continue
    if 'extension_helper.cpp' in fpath and (included_file.endswith('_extension.hpp') or included_file == 'generated_extension_loader.hpp'):
        continue

With the first and then this PR, DuckDB compiles & works on zOS, but some tests from the unittest fail.
Ex: radix sort.
Any pointers on where BE-related review will be required will be super helpful.

Assuming that this is not the end of zOS support work, is the above solution acceptable?

@Mytherin
Copy link
Collaborator
Mytherin commented Jul 6, 2023

For the amalgamation builds the wrong include tags are likely used. #include <...> should be used for system headers to prevent them from being inlined into the amalgamation (and leave the #include statements as-is).

- include zos-tls as a system header
- update test/CMakeLists.txt for format.py
@github-actions github-actions bot marked this pull request as draft July 6, 2023 14:29
@v1gnesh
Copy link
Contributor Author
v1gnesh commented Jul 6, 2023

Awesome, thank you. Bringing this back to 'ready for review'.

@v1gnesh v1gnesh marked this pull request as ready for review July 6, 2023 14:30
@v1gnesh
Copy link
Contributor Author
v1gnesh commented Jul 7, 2023

@Mytherin could you please check if this is ready for pull now.
The failures shown below look unrelated to this PR.

@v1gnesh
Copy link
Contributor Author
v1gnesh commented Jul 21, 2023

@Mytherin
8000
Copy link
Collaborator

Could you merge with master so that the CI can run again?

@github-actions github-actions bot marked this pull request as draft July 21, 2023 12:19
@v1gnesh v1gnesh marked this pull request as ready for review July 21, 2023 12:20
@Mytherin Mytherin merged commit 780db79 into duckdb:master Jul 24, 2023
@Mytherin
Copy link
Collaborator

Thanks!

@v1gnesh v1gnesh deleted the feature branch August 19, 2023 05:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants
0