8000 Improve cmake build to find openssl and fix windows build by Bjoe · Pull Request #2120 · pocoproject/poco · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Improve cmake build to find openssl and fix windows build #2120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 11, 2018

Conversation

Bjoe
Copy link
Contributor
@Bjoe Bjoe commented Jan 28, 2018

Use features from cmake to find and set proper build settings for openssl.

@Bjoe Bjoe force-pushed the fix-cmake-openssl-on-windows branch from 4712a87 to d9d82a8 Compare January 29, 2018 00:57
@aleks-f
Copy link
Member
aleks-f commented Jan 29, 2018

this looks good, there will be a small related change today or tomorrow - i have moved prebuilt binaries dirs to openssl/build (instead of openssl/VS_120), so any VS version openssl binaries will end up there and there will be no need to change any VS settings. also, openssl binaries names will be named properly (with mt, mtd etc suffix, instead of all same libcrypto/libssl) will update cmake as soon as i commit, but there may be a brief period of broken builds /cc @zosrothko

@aleks-f aleks-f requested a review from zosrothko January 29, 2018 01:21
@Bjoe
Copy link
Contributor Author
Bjoe commented Jan 29, 2018

@aleks-f Looks good ... unfortunately not really.
They cannot link libcrypto.lib. I think the reason is that the cmake build system use the headers from openssl/VS_120 ... but the libraries are used from OpenSSL-Win32/lib/VC.

Find openssl package modul from cmake expect following from openssl:
For the headers, they search: openssl/ssl.h with include as path suffix
For the libraries, they search

  • 32-bit crypto debug: libcrypto32MDd libcryptoMDd libcryptod libeay32MDd libeay32d cryptod
  • 32-bit crypto release: libcrypto32MD libcryptoMD libcrypto libeay32MD libeay32 crypto
  • 32-bit ssl debug: libssl32MDd libsslMDd libssld ssleay32MDd ssleay32d ssld
  • 32-bit ssl release: libssl32MD libsslMD libssl ssleay32MD ssleay32 ssl
    As path suffix: lib/VC or VC or lib

In following directories cmake searches for openssl:

  • C:\projects\poco\openssl\VS_120 <---- this I set via OPENSSL_ROOT_DIR
  • [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\OpenSSL (32-bit)_is1;Inno Setup: App Path]
  • [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\OpenSSL (64-bit)_is1;Inno Setup: App Path]
  • C:/Program Files (x86)/OpenSSL
  • C:/Program Files (x86)/OpenSSL-Win32
  • C:/Program Files (x86)/OpenSSL-Win64
  • C:/OpenSSL/
  • C:/OpenSSL-Win32/
  • C:/OpenSSL-Win64/

I see that now the libraries are in openssl/build/win32/lib/debug/ with the names:
libcryptomtd.lib and libsslmtd.lib .... unfortunately find openssl package modul from cmake will not find tihs libraries. May I ask, what means "mt" and "MD" ? I know that boost is using "mt" for Multi-threading build ... but openssl ?

@Bjoe
Copy link
Contributor Author
Bjoe < 8000 /strong> commented Jan 29, 2018

Ok sorry, I found the answer ...
MD for shared ... and MT for static.
I read in the FindOpenSSL documentation that there exists a properties to search for static libs:
Set OPENSSL_MSVC_STATIC_RT set TRUE to choose the MT version of the lib.
I set OPENSSL_MSVC_STATIC_RT to true in our CMakeLists.txt

@aleks-f
Copy link
Member
aleks-f commented Jan 29, 2018

@Bjoe our openssl directory has changed (I committed this morning, develop branch, master not yet) -everything is under openssl/build, vs_version file indicates which MSVC version was used to build (this file is just for informational/troubleshooting purposes, not used by any scripting). So, there is no VS_120 directory anymore.

As for external OpenSSL, we should not guess anything there - if user chooses external openssl, he is entirely on his own and should provide the proper location - it is a perilous process though because names on windows are all over the place - I suggest not to waste your time trying to guess what an external openssl binaries names may be :)

CMakeLists.txt Outdated
@@ -92,11 +92,10 @@ option(DISABLE_INTERNAL_OPENSSL "Disable internal OpensSSL binaries use" OFF)
set(USING_INTERNAL_OPENSSL false)

if(MSVC)
if((EXISTS "${PROJECT_SOURCE_DIR}\\openssl\\VS_120\\") AND NOT DISABLE_INTERNAL_OPENSSL)
if((EXISTS "${PROJECT_SOURCE_DIR}/openssl/VS_120/") AND NOT DISABLE_INTERNAL_OPENSSL)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see my comment

Copy link
Contributor Author
@Bjoe Bjoe Jan 29, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I know ... I merged now the latest develop into my branch and changed the path in CMakeLists.txt.

@aleks-f
Copy link
Member
aleks-f commented Jan 29, 2018

@Bjoe just FYI (in case you are wondering where are they), for our internal build openssl binary names are not needed anywhere in cmake because they are linked automatically.

cmake only has to make sure everything from /openssl/build ends up in the right place in lib(64)/bin(64) directories used by cmake

@aleks-f
Copy link
Member
aleks-f commented Jan 29, 2018

just ported these:
8328785
8b81aea
develop branch should be in order with new layout and binary names now

@Bjoe
Copy link
Contributor Author
Bjoe commented Jan 30, 2018

@aleks-f Hm I see. So we need only to set the path?
Ok, I think I understand the idea:
When we use the internal openssl/build version, we statically linked openssl into libCrypto.lib. So we didn't need to export any libraries dependency in FindPocoCrypto.cmake, but I think we need to export the header path from internal openssl/build.
This is done with:

target_link_libraries( "${LIBNAME}" Foundation OpenSSL::SSL OpenSSL::Crypto )

in Crypto/CMakeLists.txt. OpenSSL::SSL and OpenSSL::Crypto set all the required include-, libraries- paths and compile definitions for the local build system also for the external usage/linking of libPocoCrypto.lib.

Btw. I see that this definitions POCO_EXTERNAL_OPENSSL should also be exported. I will fix this...

@aleks-f
Copy link
Member
aleks-f commented Jan 30, 2018

we statically linked openssl into libCrypto.lib

no, it will statically link directly into any executable that includes Crypto.h; static libCrypto.lib binary will contain only barebone Crypto library code

we need to export the header path from internal openssl/build

correct

@Bjoe
Copy link
Contributor Author
Bjoe commented Jan 31, 2018

Ok, libCrypto.lib means "static" on windows ... .lib on windows is also like .a on Unix/Linux?

Hm Appveyor is not ready yet, but I have a test result from my test. Unfortunately its the same problem.... the openssl libraries are not found in openssl/build/lib/libcrypto32MTd.lib ...

See also the test result

@aleks-f
Copy link
Member
aleks-f commented Jan 31, 2018

libCrypto.lib means "static" on windows

it's different on windows - there is always a .lib file, for static and dynamic builds - for static build, the .lib file contains all the code, but for dynamic builds it is only a stub for linking from an executable (to resolve all linking at build time); at runtime, the stub code automatically loads DLL (equivalent of .so on linux), where the library actual code is.

besides the name suffix MT, you can also distinguish between static and dynamic .lib files by size - the dynamic stub .lib is significantly smaller than the static one

@aleks-f
Copy link
Member
aleks-f commented Jan 31, 2018

here's the full tree, bin directory name means dynamic (so, the *.lib files in there are only stubs for DLLs), lib directory name means static and has full code in *.lib files; pdb are "program database" files (for debugging purposes)

build
│   vs_version
│
├───include
│   └───openssl
│           aes.h
│           applink.c
│           asn1.h
│           ...
├───win32
│   ├───bin
│   │   ├───debug
│   │   │       libcryptod.dll
│   │   │       libcryptod.lib
│   │   │       libcryptod.pdb
│   │   │       libssld.dll
│   │   │       libssld.lib
│   │   │       libssld.pdb
│   │   │
│   │   └───release
│   │           libcrypto.dll
│   │           libcrypto.lib
│   │           libcrypto.pdb
│   │           libssl.dll
│   │           libssl.lib
│   │           libssl.pdb
│   │
│   └───lib
│       ├───debug
│       │       libcryptomtd.lib
│       │       libPreVS2013CRTmtd.lib
│       │       libsslmtd.lib
│       │       ossl_static.pdb
│       │
│       └───release
│               libcryptomt.lib
│               libPreVS2013CRTmt.lib
│               libsslmt.lib
│               ossl_static.pdb
│
└───win64
    ├───bin
    │   ├───debug
    │   │       libcryptod.dll
    │   │       libcryptod.lib
    │   │       libcryptod.pdb
    │   │       libssld.dll
    │   │       libssld.lib
    │   │       libssld.pdb
    │   │
    │   └───release
    │           libcrypto.dll
    │           libcrypto.lib
    │           libcrypto.pdb
    │           libssl.dll
    │           libssl.lib
    │           libssl.pdb
    │
    └───lib
        ├───debug
        │       libcryptomtd.lib
        │       libsslmtd.lib
        │       ossl_static.pdb
        │
        └───release
                libcryptomt.lib
                libsslmt.lib
                ossl_static.pdb

@Bjoe
Copy link
Contributor Author
Bjoe commented Feb 5, 2018

Hi,

cmake is now fixed and found the right openssl libraries:

-- Found OpenSSL: optimized;C:/projects/poco/openssl/build/win32/lib/release/libcryptomt.lib;debug;C:/projects/poco/openssl/build/win32/lib/debug/libcryptomtd.lib (found version "1.1.0g")
-- Using OpenSSL from C:/projects/poco/openssl/build/;C:/projects/poco/openssl/build/win32/lib/debug/;C:/projects/poco/openssl/build/win32/lib/release/

I see also the openssl libraries as parameters for the linker:

C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\link.exe /ERRORREPORT:QUEUE /OUT:"C:\projects\poco\cmake-build\bin\Debug\PocoCryptod.dll" /INCREMENTAL /NOLOGO /LIBPATH:"C:\Tools\vcpkg\installed\x86-windows\debug\lib" /LIBPATH:"C:\Tools\vcpkg\installed\x86-windows\debug\lib\manual-link" ..\lib\Debug\PocoFoundationd.lib ....\openssl\build\win32\lib\debug\libsslmtd.lib ....\openssl\build\win32\lib\debug\libcryptomtd.lib iphlpapi.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /DEBUG /PDB:"C:/projects/poco/cmake-build/bin/Debug/PocoCryptod.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:/projects/poco/cmake-build/lib/Debug/PocoCryptod.lib" /MACHINE:X86 /SAFESEH /machine:X86 /DLL Crypto.dir\Debug\Cipher.obj

But the linker is failing with following error:

LINK : fatal error LNK1104: cannot open file 'libcryptod.lib' [C:\projects\poco\cmake-build\Crypto\Crypto.vcxproj]

Why they search for libcryptod.lib ?
Is that because in Crypto.h these libraries are defined ?

#pragma comment(lib, "libcrypto" POCO_STATIC_SUFFIX POCO_DEBUG_SUFFIX ".lib")
#pragma comment(lib, "libssl" POCO_STATIC_SUFFIX POCO_DEBUG_SUFFIX ".lib")

@aleks-f
Copy link
Member
aleks-f commented Feb 5, 2018

Why they search for libcryptod.lib ?
Is that because in Crypto.h these libraries are defined ?

yes, and that should only be enabled when POCO_INTERNAL_OPENSSL_MSVC_VER is defined; that define in opensslv.h is inserted by our openssl build, so POCO_INTERNAL_OPENSSL_MSVC_VER is defined automatically if internal opensslv.h is included (which should not happen for external OpenSSL and there is a compile time warning if that header is included when POCO_EXTERNAL_OPENSSL is defined).

in other words, when compiling POCO_EXTERNAL_OPENSSL, two conditions should be met

  • external ssl headers location is specified
  • internal ssl headers location is NOT specified

@aleks-f
Copy link
Member
aleks-f commented Feb 5, 2018

Also, when linking with external openssl, linking must be explicit because those pragmas are disabled - we do not know what the library names will be for an external lib

@Bjoe
Copy link
Contributor Author
Bjoe commented Feb 6, 2018

Ok. I see following compiler output:

C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\CL.exe /c /IC:\projects\poco\openssl\build\include /IC:\projects\poco\Crypto\include /IC:\projects\poco\Crypto\src /IC:\projects\poco\Foundation\include /I"C:\Tools\vcpkg\installed\x86-windows\include" /Zi /nologo /W3 /WX- /Od /Ob0 /Oy- /D WIN32 /D _WINDOWS /D UNICODE /D _UNICODE /D LCC /D "CMAKE_INTDIR="Debug"" /D Crypto_EXPORTS /D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /RTC1 /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /GR /Fo"Crypto.dir\Debug\" /Fd"Crypto.dir\Debug\vc140.pdb" /Gd /TP /analyze- /errorReport:queue -std:c++14 C:\projects\poco\Crypto\src\Cipher.cpp C:\projects\poco\Crypto\src\CipherFactory.cpp C:\projects\poco\Crypto\src\CipherImpl.cpp C:\projects\poco\Crypto\src\CipherKey.cpp C:\projects\poco\Crypto\src\CipherKeyImpl.cpp C:\projects\poco\Crypto\src\CryptoException.cpp C:\projects\poco\Crypto\src\CryptoStream.cpp C:\projects\poco\Crypto\src\CryptoTransform.cpp C:\projects\poco\Crypto\src\DigestEngine.cpp C:\projects\poco\Crypto\src\ECDSADigestEngine.cpp C:\projects\poco\Crypto\src\ECKey.cpp C:\projects\poco\Crypto\src\ECKeyImpl.cpp C:\projects\poco\Crypto\src\EVPPKey.cpp C:\projects\poco\Crypto\src\KeyPair.cpp C:\projects\poco\Crypto\src\KeyPairImpl.cpp C:\projects\poco\Crypto\src\OpenSSLInitializer.cpp C:\projects\poco\Crypto\src\PKCS12Container.cpp C:\projects\poco\Crypto\src\RSACipherImpl.cpp C:\projects\poco\Crypto\src\RSADigestEngine.cpp C:\projects\poco\Crypto\src\RSAKey.cpp C:\projects\poco\Crypto\src\RSAKeyImpl.cpp C:\projects\poco\Crypto\src\X509Certificate.cpp

During the compile output I see:

OpenSSLInitializer.cpp
OpenSSL 1.1.0g 2 Nov 2017 (POCO internal build, MSVC version 120)
PKCS12Container.cpp

See also appveyor build line 1522 ...

This means for me POCO_INTERNAL_OPENSSL_MSVC_VER is defined because of \projects\poco\openssl\build\include as include path.

POCO_EXTERNAL_OPENSSL is not set because of DISABLE_INTERNAL_OPENSSL is not set as cmake parameter and \projects\poco\openssl\build\ exists as directory.

Also I didn't see any warnings like:

#pragma warning "External OpenSSL defined but internal headers used - possible mismatch!"

For me it looks like _DLL is defined because this if statement is not getting true:

#if !defined (_DLL)

and the linker is claiming about cannot open file 'libcryptod.lib' and not libcryptomtd.lib ... 'mt' is missing. Who defines _DLL ?

@zosrothko
Copy link
Member
zosrothko commented Feb 6, 2018

See VisualStudio Predefined macros . _DLL is defined by the cl.exe compiler when the linked runtime is conditionned by the /MD or /MDd flags

@aleks-f
Copy link
Member
aleks-f commented Feb 6, 2018

@Bjoe, given all you stated, it sounds like the path to openssl libraries (this, of course, is different from header path, which is obviously known to the compiler) is not specified, so linker can not find the libraries to link with

@Bjoe Bjoe force-pushed the fix-cmake-openssl-on-windows branch from 3c4d849 to 795e496 Compare February 7, 2018 15:57
@Bjoe
Copy link
Contributor Author
Bjoe commented Feb 7, 2018

@zosrothko Thanks, now I understand.
@aleks-f Yes right, because I set in the cmake build system use the static openssl libraries. CMake found the openssl static libraries:

C:/projects/poco/openssl/build/win32/lib/debug/libcryptomtd.lib

and this is also set as parameter for the linker:

....\openssl\build\win32\lib\debug\libcryptomtd.lib

With zosrothko hint, I understand now why _DLL is set. This means for me, we should use static openssl libraries when poco are build as static libraries. When poco is build as shared libraries then we should use the shared openssl libraries. This is what are defined in Crypto.h

#if !defined (_DLL)

May I ask, why we have these "static build dependencies" ?

@aleks-f
Copy link
Member
aleks-f commented Feb 7, 2018

why we have these "static build dependencies" ?

for convenience - otherwise you'd have to explicitly specify the library you're linking with. This way (if you're using poco prebuilt openssl), all you have to do is specify the openssl headers location and linking happens automatically through headers.

as you can see, these dependencies are only in effect when our openssl header is included, otherwise, they are disabled

@Bjoe Bjoe force-pushed the fix-cmake-openssl-on-windows branch 4 times, most recently from 8c3e680 to 83fdfe2 Compare February 10, 2018 11:38
@Bjoe
Copy link
Contributor Author
Bjoe commented Feb 10, 2018

It compile and link now but unfortunately tests are failing with exception code 0xc0000135.

test 8
Start 8: NetSSL
8: Test command: C:\projects\poco\cmake-build\bin\Debug\NetSSL-testrunner.exe "-all"
8: Test timeout computed to be: 1500
8/10 Test #8: NetSSL ...........................***Exception: Exit code 0xc0000135
0.03 sec
test 9
Start 9: Crypto
9: Test command: C:\projects\poco\cmake-build\bin\Debug\Crypto-testrunner.exe "-all"
9: Test timeout computed to be: 1500
9/10 Test #9: Crypto ...........................***Exception: Exit code 0xc0000135
0.02 sec

What is now the problem?

@aleks-f
Copy link
Member
aleks-f commented Feb 10, 2018

most likely missing dll. the location where the DLL binary is must be in the PATH

@Bjoe Bjoe force-pushed the fix-cmake-openssl-on-windows branch from 83fdfe2 to 5dfce53 Compare February 19, 2018 20:15
@Bjoe Bjoe force-pushed the fix-cmake-openssl-on-windows branch from 5dfce53 to a331a7d Compare February 19, 2018 20:19
@Bjoe
Copy link
Contributor Author
Bjoe commented Feb 20, 2018

Ok, test are fixed, I squashed and rebase to latest develop branch. So I'm done, please review.

Copy link
Member
@zosrothko zosrothko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NetSSL_OpenSSL/include/Poco/Net/NetSSL.h can be simplified. Look at this PR #2170. Also I saw that you avoid to add the "d" suffix in nameing the TestLib/TestApp debug shared lib. For what reason did you remove it?

@aleks-f
Copy link
Member
aleks-f commented Feb 21, 2018

@zosrothko, where do you see this?

you avoid to add the "d" suffix in nameing the TestLib/TestApp debug shared lib

Copy link
Contributor Author
@Bjoe Bjoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, we can avoid this. Good solution. Thanks.

#pragma comment(lib, "libcrypto.lib")
#pragma comment(lib, "libssl.lib")
#endif // POCO_EXTERNAL_OPENSSL
#if defined(POCO_INTERNAL_OPENSSL_MSVC_VER)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is "this change" from line 64 to 94 correct? I do not have much windows knowledge. I copied from Crypto.h.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, Crypto.h does all the work.

POCO_INTERNAL_OPENSSL_MSVC_VER is a new define that comes "automatically" from the define injected by poco internal openssl build.

we issue a compile-time warning if user has defined external ssl, but is using internal headers. if user has not specified anything, internal build is used by default.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to know: we also base some hacking (in order to prevent static linking errors) on that define

@Bjoe
Copy link
Contributor Author
Bjoe commented Feb 21, 2018

@zosrothko sorry, I didn't get it too. What you mean with:

you avoid to add the "d" suffix in nameing the TestLib/TestApp debug shared lib

The only think what I see, I remove:

include(OpenSSLInternalLibCopy)

and I seen that this copied the openssl libs to the place where the poco libs are exist.
Is this still needed? Then I should fix this.

@Bjoe
Copy link
Contributor Author
Bjoe commented Feb 21, 2018

@zosrothko Ok think I get it now, I see in Foundation/testsuite/CMakeLists.txt following lines:

set_target_properties( TestApp PROPERTIES DEBUG_POSTFIX "") # don't use DEBUG_POSTFIX ${CMAKE_DEBUG_POSTFIX}

set_target_properties( TestLibrary PROPERTIES PREFIX "" DEBUG_POSTFIX "") # don't use DEBUG_POSTFIX ${CMAKE_DEBUG_POSTFIX}

You mean this lines?
I didn't change this. It's changed in commit id 4027a05
Hm its committed on 12 Dec 2015 from you ...
I'm confused now...

@Bjoe Bjoe requested a review from zosrothko February 22, 2018 10:17
@Bjoe Bjoe force-pushed the fix-cmake-openssl-on-windows branch 2 times, most recently from 2a08756 to 904e332 Compare March 10, 2018 21:55
@Bjoe Bjoe force-pushed the fix-cmake-openssl-on-windows branch from 904e332 to eb63890 Compare March 11, 2018 19:51
@Bjoe Bjoe dismissed zosrothko’s stale review March 11, 2018 21:10

sorry, I didn't get it

@Bjoe Bjoe merged commit d90dca6 into pocoproject:develop Mar 11, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
0