Difference between revisions of "ICS V9.1"
(ICS V9.1 Release Notes) |
|||
(2 intermediate revisions by the same user not shown) | |||
Line 24: | Line 24: | ||
#The OverbyteIcsLIBEAY and OverbyteIcsSSLEAY units no longer support for OpenSSL 1.1.1 which is end of life. The GSSLEAY_DLL_IgnoreOld/New public variables are currently ignored since only 3.x supported. Added public variable GSSLEAY_RES_SUBDIR which defaults to "ICS-OpenSSL", where OpenSSL files will be saved and accessed if linked as a resource in the application, with a sub-directory for each different version, as mentioned earlier. Added GSSL_CERTS_DIR and GSSL_ROOTS_DIR globals where ICS looks for SSL/TLS certificates and bundles. | #The OverbyteIcsLIBEAY and OverbyteIcsSSLEAY units no longer support for OpenSSL 1.1.1 which is end of life. The GSSLEAY_DLL_IgnoreOld/New public variables are currently ignored since only 3.x supported. Added public variable GSSLEAY_RES_SUBDIR which defaults to "ICS-OpenSSL", where OpenSSL files will be saved and accessed if linked as a resource in the application, with a sub-directory for each different version, as mentioned earlier. Added GSSL_CERTS_DIR and GSSL_ROOTS_DIR globals where ICS looks for SSL/TLS certificates and bundles. | ||
#Several new defines are added to the .\Source\Include\OverbyteIcsDefs.inc file to determine how OpenSSL is loaded, all those relevant are: | #Several new defines are added to the .\Source\Include\OverbyteIcsDefs.inc file to determine how OpenSSL is loaded, all those relevant are: | ||
− | ##[$DEFINE USE_SSL} - default enabled, link OpenSSL into all components. | + | ##'''[$DEFINE USE_SSL}''' - default enabled, link OpenSSL into all components. |
− | ##{$DEFINE OpenSSL_Resource_Files} - default enabled, link OpenSSL DLLs as resource file into applications, and extract them to shell path CSIDL_COMMON_APPDATA and sub-directory "ICS-OpenSSL" with a version subdirectory, ie C:\ProgramData\ICS-OpenSSL\3012\ . This happens only once if the files have not already been extracted. Ignored for YuOpenSSL. | + | ##'''{$DEFINE OpenSSL_Resource_Files}''' - default enabled, link OpenSSL DLLs as resource file into applications, and extract them to shell path CSIDL_COMMON_APPDATA and sub-directory "ICS-OpenSSL" with a version subdirectory, ie C:\ProgramData\ICS-OpenSSL\3012\ . This happens only once if the files have not already been extracted. Ignored for YuOpenSSL. |
− | ##{$DEFINE OpenSSL_32} - if OpenSSL_Resource_Files is enabled, determines which major and minor version of OpenSSL is linked into the application, 32 is 3.2.x, or 31 or 30. ICS is currently distributed with OpenSSL 3.0, 3.1 and 3.2, the latest patch of each version so 3.0`13. 3.1.5 and 3.2.1, the resources files are in .\Source\, LibV32OpenSSL32.RES for 3.2 Win32, total six resource files, ICS automatically links Win32 or Win64 RES files. | + | ##'''{$DEFINE OpenSSL_32}''' - if OpenSSL_Resource_Files is enabled, determines which major and minor version of OpenSSL is linked into the application, 32 is 3.2.x, or 31 or 30. ICS is currently distributed with OpenSSL 3.0, 3.1 and 3.2, the latest patch of each version so 3.0`13. 3.1.5 and 3.2.1, the resources files are in .\Source\, LibV32OpenSSL32.RES for 3.2 Win32, total six resource files, ICS automatically links Win32 or Win64 RES files. |
− | ##{$DEFINE OpenSSL_ProgramData} - default enabled, but ignored if OpenSSL_Resource_Files or YuOpenSSL enabled. Causes ICS to load OpenSSL DLLs from C:\ProgramData\ICS-OpenSSL\, an alias for C:\Users\All Users\ICS-OpenSSL. ICS is distributed with Win32 and Win64 DLLs for 3.2.1 in .\ICS-OpenSSL which are copied there when building the IcsCommonXXRun package. Note there is no version sub-directory so no version choice. If enabled, overrides the public variable GSSL_DLL_DIR which some applications set to load OpenSSL from a known directory. | + | ##'''{$DEFINE OpenSSL_ProgramData}''' - default enabled, but ignored if OpenSSL_Resource_Files or YuOpenSSL enabled. Causes ICS to load OpenSSL DLLs from C:\ProgramData\ICS-OpenSSL\, an alias for C:\Users\All Users\ICS-OpenSSL. ICS is distributed with Win32 and Win64 DLLs for 3.2.1 in .\ICS-OpenSSL which are copied there when building the IcsCommonXXRun package. Note there is no version sub-directory so no version choice. If enabled, overrides the public variable GSSL_DLL_DIR which some applications set to load OpenSSL from a known directory. |
− | ##{$DEFINE YuOpenSSL} - default disabled. If enabled, compiles the OpenSSL code as a DCU directly into binaries so the OpenSSL are not needed, YuOpenSSL is a commercial product from https://www.yunqa.de/. OpenSSL 3.0 and 3.2 are available for YuOpenSSL. | + | ##'''{$DEFINE YuOpenSSL}''' - default disabled. If enabled, compiles the OpenSSL code as a DCU directly into binaries so the OpenSSL are not needed, YuOpenSSL is a commercial product from https://www.yunqa.de/. OpenSSL 3.0 and 3.2 are available for YuOpenSSL. |
##NOTE: if defines OpenSSL_Resource_Files, OpenSSL_ProgramData and YuOpenSSL are all disabled or missing, ICS loads OpenSSL from the directory specified in the public variable GSSL_DLL_DIR, which is typically set to the application directory. If blank, Windows will search the path for any OpenSSL 3 DLLs, anywhere. | ##NOTE: if defines OpenSSL_Resource_Files, OpenSSL_ProgramData and YuOpenSSL are all disabled or missing, ICS loads OpenSSL from the directory specified in the public variable GSSL_DLL_DIR, which is typically set to the application directory. If blank, Windows will search the path for any OpenSSL 3 DLLs, anywhere. | ||
− | ##{$DEFINE OpenSSL_CA_Bundle_Small} - default enabled, links a root certificate authority bundle as a resource file into applications, other options are Medium and Large. CA bundles are needed to verify that SSL/TLS certificates are issued by trusted authorities, the resources files are in .\Source\, sslRootCACertsBundle.RES, | + | ##'''{$DEFINE OpenSSL_CA_Bundle_Small}''' - default enabled, links a root certificate authority bundle as a resource file into applications, other options are Medium and Large. CA bundles are needed to verify that SSL/TLS certificates are issued by trusted authorities, the resources files are in .\Source\, sslRootCACertsBundle.RES (OpenSSL_CA_Bundle_Small), TrustedCaBundle.RES (OpenSSL_CA_Bundle_Medium) and RootCaCertsBundle.RES (OpenSSL_CA_Bundle_Large). |
− | ##{$DEFINE OpenSSL_AutoLoad_CA_Bundle} - default enabled. With ICS V9.1 and later, a common IcsSslRootCAStore component is created at application start-up, if this define is enabled OpenSSL will be loaded followed by the root CA bundle RES file according to define OpenSSL_CA_Bundle_Small/Medium/Large. This means OpenSSL is available for all components, without it needing to be loaded again, perhaps repeatedly, and multiple components can share the IcsSslRootCAStore component without needing to load their own CA bundles. If this defined is not enabled, SslRootCAStore.Initialise may be called by the application to load OpenSSL and the CA bundle, which is done automatically by SslContext.InitContext if not done previously. | + | ##{'''$DEFINE OpenSSL_AutoLoad_CA_Bundle}''' - default enabled. With ICS V9.1 and later, a common IcsSslRootCAStore component is created at application start-up, if this define is enabled OpenSSL will be loaded followed by the root CA bundle RES file according to define OpenSSL_CA_Bundle_Small/Medium/Large. This means OpenSSL is available for all components, without it needing to be loaded again, perhaps repeatedly, and multiple components can share the IcsSslRootCAStore component without needing to load their own CA bundles. If this defined is not enabled, SslRootCAStore.Initialise may be called by the application to load OpenSSL and the CA bundle, which is done automatically by SslContext.InitContext if not done previously. |
− | ##{$DEFINE AUTO_X509_CERTS} - default enabled. This define enables automatic SSL/TLS ordering from Let's Encrypt in SocketServer and other servers. Unfortunately this adds a lot of other units, HTTPS REST, Json, OAuth2, etc, increasing the size of server applications, so it may be disabled to make server EXE files smaller if certificates are obtained and installed manually. | + | ##'''{$DEFINE AUTO_X509_CERTS}''' - default enabled. This define enables automatic SSL/TLS ordering from Let's Encrypt in SocketServer and other servers. Unfortunately this adds a lot of other units, HTTPS REST, Json, OAuth2, etc, increasing the size of server applications, so it may be disabled to make server EXE files smaller if certificates are obtained and installed manually. |
##Except when using the OpenSSL_AutoLoad_CA_Bundle define, OpenSSL still needs to be loaded before any SSL/TLS functionality can be used. This is done automatically by TSslContext and some other components that use OpenSSL, but this means SSL errors like the DLLs not being found may not be raised until a web page is accessed, etc. So it is generally better to load OpenSSL early on in your application, when errors are easier to handle There is a function IcsReportOpenSSLVer that returns the OpenSSL version loaded and where it was loaded from, to help debug loading problems. | ##Except when using the OpenSSL_AutoLoad_CA_Bundle define, OpenSSL still needs to be loaded before any SSL/TLS functionality can be used. This is done automatically by TSslContext and some other components that use OpenSSL, but this means SSL errors like the DLLs not being found may not be raised until a web page is accessed, etc. So it is generally better to load OpenSSL early on in your application, when errors are easier to handle There is a function IcsReportOpenSSLVer that returns the OpenSSL version loaded and where it was loaded from, to help debug loading problems. | ||
##When using the OpenSSL_AutoLoad_CA_Bundle define, if the OpenSSL legacy.dll is needed to support old algorithms, which includes most password protected PFX/PCS12 certificates, it must be loaded using LibeayLoadProviders(True, False); since it is not possible to set the GSSLEAY_LOAD_LEGACY global variable early enough. | ##When using the OpenSSL_AutoLoad_CA_Bundle define, if the OpenSSL legacy.dll is needed to support old algorithms, which includes most password protected PFX/PCS12 certificates, it must be loaded using LibeayLoadProviders(True, False); since it is not possible to set the GSSLEAY_LOAD_LEGACY global variable early enough. | ||
Line 54: | Line 54: | ||
#The TIcsTimeClient SNTP component now sends the proper NTP version to the server, we have been sending v6 for 20 years, when v4 is the latest. Added more NTP servers from cloud providers that are more likely to be running than private ntp.org servers. Fixed IcsGetUTCNtpTime always returning midnight due to strange rounding in newer Delphi versions, meant time server sent wrong time. | #The TIcsTimeClient SNTP component now sends the proper NTP version to the server, we have been sending v6 for 20 years, when v4 is the latest. Added more NTP servers from cloud providers that are more likely to be running than private ntp.org servers. Fixed IcsGetUTCNtpTime always returning midnight due to strange rounding in newer Delphi versions, meant time server sent wrong time. | ||
#Historically, ICS has often used AnsiStrings to handle binary data, sometimes custom byte buffers. Modern versions of Delphi now use TBytes (dynamic array of bytes) for binary, so ICS had added many methods and properties using TBytes, mostly with TB added to existing names. There are now TBytes versions of the Jose, hash and digest functions since all input and output is binary: IcsHMACDigestTB, IcsHMACDigestExTB, IcsHMACDigestVerifyTB, IcsHashDigestTB, IcsAsymSignDigestTB, IcsAsymVerifyDigestTB, IcsJoseGetSigTB and IcsJoseCheckSigTB. New utility functions include IcsTBToHex, Base64EncodeTB, Base64DecodeTB, IcsTBytesToString, IcsMoveTBytesToString, IcsTBytesToStringA, IcsStringToTBytes, IcsStringAToTBytes, IcsBase64UrlDecodeTB, IcsBase64UrlDecodeATB, IcsBase64UrlEncodeTB a, IcsBase64UrlEncodeATB, Utf8ToStringTB. Renamed IcsToASCII to IcsPunyToAsci and IcsToUnicode to IcsPunyToUnicode so they don't get used for the wrong purpose. Added IcsFormatHexStr to break long hex string into groups and lines, defaulting to eight chars per group, 64 per line. Added IcsStrRemCntls to replace control codes (< space) in string with ~, optionally leaving line endings, IcsStrRemCntlsA takes an AnsiString or buffer, IcsStrRemCntlsTB is TBytes buffer. Added IcsStrBeakup to break up text into multiple lines of specified length, default 80. Added IcsTimeToZStr to convert DataTime to string hh:mm:ss:zzz. Added IcsResourceGetTB to read TBytes from a named resource. Added IcsResourceSaveFile to save a file from a named resource. Report mobile platforms to IcsBuiltWithEx. Added IcsDataSaveFile and IcsDataLoadFile to save TBytes to a file, and load it from a file, no error reporting. | #Historically, ICS has often used AnsiStrings to handle binary data, sometimes custom byte buffers. Modern versions of Delphi now use TBytes (dynamic array of bytes) for binary, so ICS had added many methods and properties using TBytes, mostly with TB added to existing names. There are now TBytes versions of the Jose, hash and digest functions since all input and output is binary: IcsHMACDigestTB, IcsHMACDigestExTB, IcsHMACDigestVerifyTB, IcsHashDigestTB, IcsAsymSignDigestTB, IcsAsymVerifyDigestTB, IcsJoseGetSigTB and IcsJoseCheckSigTB. New utility functions include IcsTBToHex, Base64EncodeTB, Base64DecodeTB, IcsTBytesToString, IcsMoveTBytesToString, IcsTBytesToStringA, IcsStringToTBytes, IcsStringAToTBytes, IcsBase64UrlDecodeTB, IcsBase64UrlDecodeATB, IcsBase64UrlEncodeTB a, IcsBase64UrlEncodeATB, Utf8ToStringTB. Renamed IcsToASCII to IcsPunyToAsci and IcsToUnicode to IcsPunyToUnicode so they don't get used for the wrong purpose. Added IcsFormatHexStr to break long hex string into groups and lines, defaulting to eight chars per group, 64 per line. Added IcsStrRemCntls to replace control codes (< space) in string with ~, optionally leaving line endings, IcsStrRemCntlsA takes an AnsiString or buffer, IcsStrRemCntlsTB is TBytes buffer. Added IcsStrBeakup to break up text into multiple lines of specified length, default 80. Added IcsTimeToZStr to convert DataTime to string hh:mm:ss:zzz. Added IcsResourceGetTB to read TBytes from a named resource. Added IcsResourceSaveFile to save a file from a named resource. Report mobile platforms to IcsBuiltWithEx. Added IcsDataSaveFile and IcsDataLoadFile to save TBytes to a file, and load it from a file, no error reporting. | ||
+ | |||
+ | |||
+ | '''New Resource Files''' | ||
+ | |||
+ | As mentioned above, ICS now includes several resource files that are linked into applications, to avoid distributing and loading separate files, these includes OpenSSL DLLs, certificate authority bundles, root certificates and network information lists. | ||
+ | |||
+ | It is intended to issue new ICS releases containing the latest OpenSSL DLLs shortly after new versions are released, which is typically every three months unless serious security fixes require more frequent releases. | ||
+ | |||
+ | The OpenSSL resource files included with ICS come from the OpenSSL zip distribution files at https://wiki.overbyte.eu/wiki/index.php/ICS_Download . ICS currently includes three different versions for two platforms, only one is ever linked into applications according to DEFINES, see earlier. These RES files contain all the DLLs, which are extracted once to version specific sub-directories. | ||
+ | |||
+ | LibV30OpenSSL32.RES | ||
+ | LibV30OpenSSL64.RES | ||
+ | LibV31OpenSSL32.RES | ||
+ | LibV31OpenSSL64.RES | ||
+ | LibV32OpenSSL32.RES | ||
+ | LibV32OpenSSL64.RES | ||
+ | |||
+ | ICS contains three root certificate authority bundle files, the latest versions of which can be downloaded from https://www.magsys.co.uk/download/software/ca-root-bundles.zip, one of which is linked into applications according to defines. The source bundle files are located in .\ICS-OpenSSL/ICS-RootCAs\, the RES files contail the P12 files which are smaller than the PEM versions. | ||
+ | |||
+ | RootCaCertsBundle.RES {$DEFINE OpenSSL_CA_Bundle_Large} | ||
+ | sslRootCACertsBundle.RES {$DEFINE OpenSSL_CA_Bundle_Small} | ||
+ | TrustedCaBundle.RES {$DEFINE OpenSSL_CA_Bundle_Medium} | ||
+ | |||
+ | There are other smaller resource files, all build with BuildICSResFiles.cmd for which the source files are part of ICS,. | ||
+ | |||
+ | ICSCerts.RES (contains ICSRootCA.pem and ICS_Intermediate_Short-bundle.pem) | ||
+ | ICSPortList.RES (contains ICSPortList.txt) | ||
+ | nmap-mac-prefixes.RES (contains nmap-mac-prefixes.txt) |
Latest revision as of 11:13, 11 March 2024
Changes in ICS V9.1 include:
- Delphi 10.4, 11, 12 and later now use the same install groups and packages, IcsInstallFmx, IcsInstallVcl and IcsInstallVclFmx, making support a lot easier. Version specific groups remain for Delphi 10.3 and earlier, with new groups D(X)InstallVcl for VCL only replacing the old OverbyteIcs(X) groups, again to simplify support. Dozens of old packages have been removed for this release, so please delete all old groups and packages before installing V9.1, to avoid a mix of old and new packages. Only C++ 10.4, 11, 12 and later are now supported, but untested.
- The old samples directory has gone and many of the older and little used samples have been archived to a separate download. The active samples used to test and demonstrate all ICS components are now split into the following paths, in the ICS root directory. All these samples can now be built for Win32 and Win64 platforms. Beware the sample project files (.dproj) supplied are built with modern compilers, and can not be opened by legacy compilers due to new platforms and features, so you MUST delete the .dproj file before opening samples in legacy compilers so the .dproj file will be automatically recreated from the .dpr project file by Delphi.
- demos-delphi-vcl - 45 VCL samples for Windows.
- demos-delphi-extra - four VCL samples that need third party components to build.
- demos-delphi-fmx - seven FMX samples for Windows, not yet tested on MacOS.
- demos-cpp-vcl - all old C++ samples that have not been tested for 10 years, need help.
- demos-data - data files for samples, such as web pages.
- To ease development, linking and future support, some new units have been added by splitting existing units with multiple components, unfortunately this means many existing projects will need one or more of the new units adding to their uses section. Apologies for the pain, but this should have been done a long time ago. The main change is splitting out much of the SSL/TLS related code from the massive OverbyteIcsWSocket unit to a new unit OverbyteIcsSslBase. Also the OverbyteIcsSslHttpRest.pas has been split with two new units OverbyteIcsDnsHttps.pas and OverbyteIcsSslUtils.pas, to ease linking avoiding circular references. Another new unit OverbyteIcsHtmlUtils.pas now contains functions designed to build HTML pages that were previously split across different units.
- Distribution of the ICS OpenSSL files has changed. Earlier ICS versions required the OpenSSL DLLs to be distributed with applications, and a root CA bundle file to verify SSL/TLS connections, and these needed to be loaded using code. There was little standardisation over where the OpenSSL DLLs were located, applications tended to keep their own copies alongside other executables, leading to multiple DLL copies and needing the public variable GSSL_DLL_DIR set to a specific directory before OpenSSL was loaded. Likewise, root CA bundle directories had to be distributed with applications and loaded with code. ICS V9.1 allows five different ways of loading the OpenSSL DLLs: Which method ICS uses to load OpenSSL depends upon several defines in the .\Source\Include\OverbyteIcsDefs.inc file, please see the readme9.txt file and notes below for details. ICS currently includes resource files for three different OpenSSL releases, 3.0`13. 3.1.5 and 3.2.1, which version is linked is controlled by a define. If the OpenSSL DLLs are linked into the application, they are extracted to a version subdirectory, ie C:\ProgramData\ICS-OpenSSL\3012\ so different applications can use different OpenSSL versions. This happens only once if the files have not already been extracted. When updating existing projects without using any new defines, the ICS old behaviour of methods 3, 4 and 5 above remain with no changes needed.
- DLLs linked into application as resource files
- DLLs loaded from common directory C:\ProgramData\ICS-OpenSSL\
- OpenSSL DCU linked into application using commercial YuOpenSSL
- DLLs loaded from location specified in public variable GSSL_DLL_DIR
- DLLs loaded according to path, may be found anywhere on PC
- A common IcsSslRootCAStore component is now created at application start-up, to avoid different components needing their own CA stores to verify SSL/TLS certificates, and for applications to load those stores. The three different CA stores included with ICS are now supplied as resource files, with a define determining which is linked into applications. Another define causes OpenSSL and this store to be loaded at application startup, so OpenSSL is available for all components, without it needing to be loaded again, perhaps repeatedly. Without new defines, a CA Store can be loaded manually into IcsSslRootCAStore. The ICS servers use CA Stores now use IcsSslRootCAStore and no longer load any files specified.
- All SSL/TLS servers need a certificate and private key to start, even when testing. Previously ICS supplied some self signed certificates for testing, and also created such certificates automatically if they were missing or if the server was about to order a Let's Encrypt certificate. Accessing such servers for testing using browsers raised various warnings. ICS now has it's own SSL root certificate 'ICS Root CA' and two intermediates, 'ICS Intermediate' and 'ICS Intermediate Short', the last of which includes a private key so can be used to automatically sign new certificates by ICS server applications, rather than just self signed certificates as before. If the 'ICS Root CA' certificate is installed in the Window Store and browser stores, it should stop certificate warnings appearing. ICS applications automatically trust the ICS root, so will give no warnings. The short intermediate has a maximum 100 day expiry, so new versions will be issued regularly. There is a single function CreateSelfSignCertEx that created signed certificates, and another IcsInstallIcsRoot that installs the ICS root into the Windows Store, so easy to use. It is possible to replace the ICS root with your own private root certificate and have servers create their own certificates against that root, for internal networks.
- Several client and server components have a new property NoSSL which if set will prevent those components using SSL/TLS for HTTPS or FTPS, even if the application is linked with OpenSSL code. Beware the IcsSslRootCAStore component must not be initialised by the application.
- The large OverbyteIcsWSocket unit has been split, by moving TSslContext, TSslBaseComponent, TX509Base and TX509List to a new unit OverbyteIcsSslBase, with only the SslContext callbacks left here since they need access to it, now set in InitSSLConnection instead of InitContext. No longer supporting defines OPENSSL_USE_DELPHI_MM (never used), OPENSSL_NO_ENGINE (deprecated, never used), OPENSSL_USE_RESOURCE_STRINGS (never used), NO_OSSL_VERSION_CHECK (dangerous), DEFINE OPENSSL_NO_TLSEXT (TLS needed everywhere), and LOADSSL_ERROR_FILE (better debugging now). If a connection fails, don't change State to wsConnected briefly before changing it again to wsClosed. Added TSslWsocket SslAlpnProtocols property to specify a list of protocols for clients to send to servers, instead of a similar SslContext property.
- OverbyteIcsSslBase is a new unit with TSslContext, TSslBaseComponent, T509Base and TX509List from OverbyteIcsWSocket, also function sslRootCACertsBundle moves here from X509Utils. Added property X509PubKeyTB to TX509Base to get the certificate public in DER binary format as TBytes, from where it may be converted to hex or base64, used for Raw Public Key certificate validation. Made more TX509Base functions and variables public so they can be accessed from other units. Added DHE-RSA-CHACHA20-POLY1305 to TLS/1.2 sslCiphersMozillaSrvTLS12. Added IcsReadTBBio, IcsWriteStrBio, IcsWriteTBBio, IcsSslLoadStackFromP12TB which are internal functions for handling TBytes and certificates, to simplify code (we use too many AnsiStrings for binary data). Added function IcsReportOpenSSLVer to centralise version reporting, optionally adding number of CA root certificates loaded. Saving a private key with a PCKS12 file is now optional. Moved BuildCertFName from WSocketS as IcsIcsBuildCertFName. ICSRootCA.pem and ICS_Intermediate_Short-bundle.pem certificates linked as resources, root is added to IcsSslRootCAStore.
- Added new TSslRootCAStore component to OverbyteIcsSslBase derived from TX509List with an Initialise method that loads OpenSSL, then tries to load the internal certificate sslRootCACertsBundle that should be linked into the app, if missing then tries to load DefRootCABundle.pem from C:\ProgramData\ICS-OpenSSL\ or the app path. It also tries to load ExtraRootCABundle.pem which is an optional private root bundle that can be used for private customer or devel roots. Added public IcsSslRootCAStore component created and intialised when this unit is loaded so a common root store is ready for any SslContext or other components. Define OpenSSL_AutoLoad_CA_Bundle can be suppressed to stop OpenSSL and the bundled being automatically loaded, if not needed. SslContext has new property UseSharedCAStore which causes the properties CAFile, CALines and CAPath to be ignored, uses IcsSslRootCAStore instead.
- The TX509List class can now load and save PKCS#12 certificate bundle files, smaller than PEM files, added SaveToP12File, SaveToP12TB, LoadAllFromP12File, LoadAllFromP12TB, intended to load a certificate bundle. LoadAllFromPemFile and LoadAllFromPemTB renamed from LoadAllFromFileEx and AddAllFromFStringEx with new versions handling both PEM and PKCS#12 certificate bundle files. TX509List has new method ListCerts that returns one listing line per cert.
- The OverbyteIcsLIBEAY and OverbyteIcsSSLEAY units no longer support for OpenSSL 1.1.1 which is end of life. The GSSLEAY_DLL_IgnoreOld/New public variables are currently ignored since only 3.x supported. Added public variable GSSLEAY_RES_SUBDIR which defaults to "ICS-OpenSSL", where OpenSSL files will be saved and accessed if linked as a resource in the application, with a sub-directory for each different version, as mentioned earlier. Added GSSL_CERTS_DIR and GSSL_ROOTS_DIR globals where ICS looks for SSL/TLS certificates and bundles.
- Several new defines are added to the .\Source\Include\OverbyteIcsDefs.inc file to determine how OpenSSL is loaded, all those relevant are:
- [$DEFINE USE_SSL} - default enabled, link OpenSSL into all components.
- {$DEFINE OpenSSL_Resource_Files} - default enabled, link OpenSSL DLLs as resource file into applications, and extract them to shell path CSIDL_COMMON_APPDATA and sub-directory "ICS-OpenSSL" with a version subdirectory, ie C:\ProgramData\ICS-OpenSSL\3012\ . This happens only once if the files have not already been extracted. Ignored for YuOpenSSL.
- {$DEFINE OpenSSL_32} - if OpenSSL_Resource_Files is enabled, determines which major and minor version of OpenSSL is linked into the application, 32 is 3.2.x, or 31 or 30. ICS is currently distributed with OpenSSL 3.0, 3.1 and 3.2, the latest patch of each version so 3.0`13. 3.1.5 and 3.2.1, the resources files are in .\Source\, LibV32OpenSSL32.RES for 3.2 Win32, total six resource files, ICS automatically links Win32 or Win64 RES files.
- {$DEFINE OpenSSL_ProgramData} - default enabled, but ignored if OpenSSL_Resource_Files or YuOpenSSL enabled. Causes ICS to load OpenSSL DLLs from C:\ProgramData\ICS-OpenSSL\, an alias for C:\Users\All Users\ICS-OpenSSL. ICS is distributed with Win32 and Win64 DLLs for 3.2.1 in .\ICS-OpenSSL which are copied there when building the IcsCommonXXRun package. Note there is no version sub-directory so no version choice. If enabled, overrides the public variable GSSL_DLL_DIR which some applications set to load OpenSSL from a known directory.
- {$DEFINE YuOpenSSL} - default disabled. If enabled, compiles the OpenSSL code as a DCU directly into binaries so the OpenSSL are not needed, YuOpenSSL is a commercial product from https://www.yunqa.de/. OpenSSL 3.0 and 3.2 are available for YuOpenSSL.
- NOTE: if defines OpenSSL_Resource_Files, OpenSSL_ProgramData and YuOpenSSL are all disabled or missing, ICS loads OpenSSL from the directory specified in the public variable GSSL_DLL_DIR, which is typically set to the application directory. If blank, Windows will search the path for any OpenSSL 3 DLLs, anywhere.
- {$DEFINE OpenSSL_CA_Bundle_Small} - default enabled, links a root certificate authority bundle as a resource file into applications, other options are Medium and Large. CA bundles are needed to verify that SSL/TLS certificates are issued by trusted authorities, the resources files are in .\Source\, sslRootCACertsBundle.RES (OpenSSL_CA_Bundle_Small), TrustedCaBundle.RES (OpenSSL_CA_Bundle_Medium) and RootCaCertsBundle.RES (OpenSSL_CA_Bundle_Large).
- {$DEFINE OpenSSL_AutoLoad_CA_Bundle} - default enabled. With ICS V9.1 and later, a common IcsSslRootCAStore component is created at application start-up, if this define is enabled OpenSSL will be loaded followed by the root CA bundle RES file according to define OpenSSL_CA_Bundle_Small/Medium/Large. This means OpenSSL is available for all components, without it needing to be loaded again, perhaps repeatedly, and multiple components can share the IcsSslRootCAStore component without needing to load their own CA bundles. If this defined is not enabled, SslRootCAStore.Initialise may be called by the application to load OpenSSL and the CA bundle, which is done automatically by SslContext.InitContext if not done previously.
- {$DEFINE AUTO_X509_CERTS} - default enabled. This define enables automatic SSL/TLS ordering from Let's Encrypt in SocketServer and other servers. Unfortunately this adds a lot of other units, HTTPS REST, Json, OAuth2, etc, increasing the size of server applications, so it may be disabled to make server EXE files smaller if certificates are obtained and installed manually.
- Except when using the OpenSSL_AutoLoad_CA_Bundle define, OpenSSL still needs to be loaded before any SSL/TLS functionality can be used. This is done automatically by TSslContext and some other components that use OpenSSL, but this means SSL errors like the DLLs not being found may not be raised until a web page is accessed, etc. So it is generally better to load OpenSSL early on in your application, when errors are easier to handle There is a function IcsReportOpenSSLVer that returns the OpenSSL version loaded and where it was loaded from, to help debug loading problems.
- When using the OpenSSL_AutoLoad_CA_Bundle define, if the OpenSSL legacy.dll is needed to support old algorithms, which includes most password protected PFX/PCS12 certificates, it must be loaded using LibeayLoadProviders(True, False); since it is not possible to set the GSSLEAY_LOAD_LEGACY global variable early enough.
- The TSslWSocketServer class in OverbyteIcsWSocketS has a new property NoSSL that prevents use of SSL/TLS, must be set before server is started. Replaced FX509CAList with public IcsSslRootCAStore. When creating a local SSL/TLS certificate to allow a server to start, ICS now creates a certificate with the IcsHosts.Hosts names signed by an internal ICS intermediate 'ICS Intermediate Short' signed by 'ICS Root CA' which if installed in Windows and browsers will stop certificate warnings appearing. Previously ICS only created self signed certificates. The global GSSL_INTER_FILE may be changed to an alternate intermediate bundle. The ICS bundle has the password 'password' and a maximum 100 day life, so new intermediates will be required regularly, to prevent misuse. Use the function IcsInstallIcsRoot to install the ICS root certificate into the Windows Root Store, needs admin rights for the Local Machine store. Added property ListenAny returns true if any sockets are listening, ie server is running.
- The TSslHttpRest component now allows TRestParams to be created as content type 'Form-Data Body' to create MIME multipart/form-data parameters that may include new TParamType of RPTypeFile that specifies a file name whose binary content will be added to the parameters as a file upload, allowing multiple files and extra parameters. File uploading with HttpUploadSimple can now use TRestParams. TRestParams are now built into a TStream rather than a string to allow larger parameter sizes. Added new property MaxLogParams to TSslHttpRest defaulting to 4,096 to restrict the length of params logged before requests with DebugLevel is DebugParams or better, there may be megabytes. Params are now line broken and binary stripped. Added progress information for file uploading, that may take a while, uploads tested to 7GB, beware preparing the form-data content stream may take a few minutes without progress information. Added new property SharedSslCtx which allows an external TSslContext component to be set to the SslContext property (just as with TSslHttpCli) rather than using the internal RestSslCtx automatically. This will be more efficient on memory when using multiple TSslHttpRest components in parallel Added new property NoSSL to TSslHttpRest that prevents use of HTTPS, must be set before any requests. HTTP redirected to HTTPS will fail.
- Redesigned TRestParams to build parameters into ParamStream using GetParamStream, to allow parameters including very large files and since the HTTP component needs a post stream rather than a string, mainly for multipart/form-data parameters, see below, GetParams still returns an AnsiString while GetParametersTB returns TBytes. Added new TRestParams content type of PContNone to make them easier to disable, beware ordial values have changed if this saved rather than a literal. Added new TRestParams content type PContFormData to create multipart/form-data parameters, according to RFC7578 which may include multiple binary files and _charset_ part. The TRestParams AddItem method has a new optional ContentType argument, currently used for PContFormData only. Added TParamType of RPTypeFile for binary file content. Added new TRestParams AddItemFile method that takes a full binary file name with optional file size and ContentType, the latter two will be looked up if not supplied, content from file extension and a MIME table. Added new TRestParams FormDataUtf8 property that if true will add a FormData _charset_ part with utf-8 and send all textual content as utf-8 without UrlEncoding. Added GetEstParamSize that returns Int64 estimated size of the parameters, to allow the application to allocate a TFileStream instead of TMemoryStream if massive files are included, typically more than 50MB. Added IcsPercentEncode and IcsPercentDecode to percent encode and decode any non 7-bit characters, ignore charsets. Similar to UrlEncode but does not change spaces or special chars, except %. ExtractURLEncodedParamList has new optional Values parameter than adds all values to the strings as name=value. The ResultSet2Json method of TRestParamsSrv has a new optional query parameter that is added to the Json to assist processing.
- The TSslHttpCli component now only call SetSslAlpnProtocols if using Https. If the Location property is cleared during the OnLocationChange event, relocation is stopped, can be used stop relocation from http to Https. When sending proxy CONNNECT request, add ALPN: header (RFC7639) which will be forwarded to target by some proxies, needed for Acme protocol.
- The TSslHttpAppSrv application web server has improvements for processing POST data. Added properties PostedDataTB and PostedDataStr to return posted data in easier to use types than an PAnsiChar buffer. Added MaxUploadMB defaults to 200 MBbyte to restrict maximum size of POST or PUT requests. Added MaxStreamMB defaults to 50 MBbyte as the maximum TMemoryStream size before a TFileStream is instead used with a temporary file name. Added PostedDataStream to which POST and PUT content is written which is what TFormDataAnalyser needs, PostedData pointer now points to the stream memory rather than a stack buffer. PUT requests now save uploaded data similarly to POST. These changes allows file uploads larger than memory, up to MaxUploadSize. Added new property NoSSL that prevents use of HTTPS, must be set before server is started. The INI file reads NoSSl, MaxUploadMB and MaxStreamMB. The web server samples have a new postinfo.html page that decodes and displays any parameters passed.
- The Web Socket Client class TSslWebSocketCli has a new property WSFullHdrs which when true causes all HTTP request headers to be sent when upgrading a connection to WebSocket, normally only the important headers are sent. Fixed a problem where multiple or partial frames might arrive together, ensure they are corrected assembled. Added new frame state wsfsIncompleteHeader when this happens.
- The Web Socket Server class THttpWSSrvConn now skips websocket upgrade if authentication is needed. Fixed a problem where multiple or partial frames might arrive together, ensure they are corrected assembled.
- In the OverbyteIcsPemTool sample, when displaying an X509 certificate, show Raw Public Key in base64, should match that of a PEM file with a public key. Allow to save PKCS12 without a private key. Added Basic Constraints 'Root Certificate Authority' tick box that ignores pathlen, 'Certificate Authority' box is now renamed 'Self Signed or Intermediate' and sets pathlen=0 to restrict signing to top level. Removed creating DH Params, not used nowadays with modern ciphers. Added Create Quick Certificates, allows self signed or CA signed certificates to be created with a single button using function CreateSelfSignCertEx. Only uses CommonName, Alt DNS Names, key type and password, and a root CA bundle if the certificate is CA signed, ICS includes a bundle with the file name in GSSL_INTER_FILE. Always creates a PEM bundle with key and intermediate. When installing certificate into the Windows Store, only install key and inter if supplied, and allow all store types, previously always MyStore. Added button 'Install ICS Root in Windows Store' to he Quick box which calls the function IcsInstallIcsRoot to install the ICS root certificate into the Windows Root Store, needs admin rights for the Local Machine store.
- The TRestOAuth class has a new OAuthOption of OAopAuthBasic which means use Basic Authentication with client id and secret instead of sending them as parameters.
- In the TIcsRestEmail class, Microsoft 365 Rest Email now supports EmailFmtRaw for both GetEmail and SendEmail to receive and send RFC822 SMTP format messages (like GMail) prepared by the TSslHtmlSmtpCli component with HTML content and attachments, and received message can be decoded using TMimeDecodeW, tested using the OverbyteIcsHttpRestTst sample. New TRestEmailType of RestEmailNone where we don't want REST email, beware ordial values changed if saved instead of literals, default now None.
- The TIcsFtpMulti, TIcsHttpMulti and TIcsMailQueue components have a new property NoSSL that prevents use of SSL/TLS, must be set before any requests. SslContext now uses the public IcsSslRootCAStore and ignores root bundle.
- The TIcsHttpProxy component now supports the CONNECT ALPN: header (RFC7659), to forward ALPN to target. If source sends SSL ALPN, forward it to target. Perhaps optional or at least remove h2 and h3 which we don't support.
- With the TSslX509Certs component, made sure certificate extensions are set for server certificate before creating certificate request so international domain name with accents gets processed. Validation now uses public IcsSslRootCAStore and ignores root bundle. The OwnCASign method to sign our own certificates has a new optional OwnCA that creates an intermediate certificate that can sign certificate requests.
- In the OverbyteIcsSslX509Utils unit, the function CreateSelfSignCertEx has an extra argument for the file name of a root CA signing bundle, usually an intermediate bundle, that is used to create a CA signed certificate instead of self signed. Password for CA must be same as certificate. Designed for use with public variable GSSL_INTER_FILE which defaults to an ICS signed intermediate allowing servers to issue their own certificates. The SslCertTools class has a new CaCertLines property which returns CA PEM lines, used to create bundle with intermediate. When creating certificates, if BasicPathLen=-1 leave out Basic Constraints pathlen so root certificates can sign intermediates.
- The OverbyteIcsHttpRestTst sample has a new 'No SSL/HTTPS' tick box to disable SSL and HTTPS requests, and a new 'Rest Content' type of 'Form-Data Body' to create MIME multipart/form-data parameters that may include new TParamType of RPTypeFile that specifies a file name whose binary content will be added to the parameters as a file upload, in a similar way to the existing 'Upload File' as 'Form-Data' except allowing multiple files and extra parameters. Added 'Form-Data UTF-8 Charset' tick box so form parameters are encoded as UTF-8 rather than HTML characters. TRestParams are now into a TStream rather than an AnsiString to allow larger sizes, tested up to 8GB. Websocket testing will now parse Json if returned, added Send Multi Lines to send two or more lines of text in a single message or as multiple separate messages.
- The TMsCertTools class method SaveToStorePfx has a new argument MsCertStore to allow loading into Windows roots store as well as MyStore. Added function IcsInstallIcsRoot to install the ICS Root CA from linked resource into the Windows Trust Store. Also a new method GetOneCert by SHA1 Digest.
- In the OverbyteIcsIpUtils unit, IcsLoadMacPrefixes now tries to load a MAC list file from a resource file nmap-mac-prefixes.RES if linked into application, otherwise loads file nmap-mac-prefixes.txt. Likewise the common port list is loaded from a resource file icsportlist.RES if linked, otherwise loads file icsportlist.txt. These changes avoid needing to distribute the files separately with applications, beware they are now automatically linked, in case not needed.
- The TIcsTimeClient SNTP component now sends the proper NTP version to the server, we have been sending v6 for 20 years, when v4 is the latest. Added more NTP servers from cloud providers that are more likely to be running than private ntp.org servers. Fixed IcsGetUTCNtpTime always returning midnight due to strange rounding in newer Delphi versions, meant time server sent wrong time.
- Historically, ICS has often used AnsiStrings to handle binary data, sometimes custom byte buffers. Modern versions of Delphi now use TBytes (dynamic array of bytes) for binary, so ICS had added many methods and properties using TBytes, mostly with TB added to existing names. There are now TBytes versions of the Jose, hash and digest functions since all input and output is binary: IcsHMACDigestTB, IcsHMACDigestExTB, IcsHMACDigestVerifyTB, IcsHashDigestTB, IcsAsymSignDigestTB, IcsAsymVerifyDigestTB, IcsJoseGetSigTB and IcsJoseCheckSigTB. New utility functions include IcsTBToHex, Base64EncodeTB, Base64DecodeTB, IcsTBytesToString, IcsMoveTBytesToString, IcsTBytesToStringA, IcsStringToTBytes, IcsStringAToTBytes, IcsBase64UrlDecodeTB, IcsBase64UrlDecodeATB, IcsBase64UrlEncodeTB a, IcsBase64UrlEncodeATB, Utf8ToStringTB. Renamed IcsToASCII to IcsPunyToAsci and IcsToUnicode to IcsPunyToUnicode so they don't get used for the wrong purpose. Added IcsFormatHexStr to break long hex string into groups and lines, defaulting to eight chars per group, 64 per line. Added IcsStrRemCntls to replace control codes (< space) in string with ~, optionally leaving line endings, IcsStrRemCntlsA takes an AnsiString or buffer, IcsStrRemCntlsTB is TBytes buffer. Added IcsStrBeakup to break up text into multiple lines of specified length, default 80. Added IcsTimeToZStr to convert DataTime to string hh:mm:ss:zzz. Added IcsResourceGetTB to read TBytes from a named resource. Added IcsResourceSaveFile to save a file from a named resource. Report mobile platforms to IcsBuiltWithEx. Added IcsDataSaveFile and IcsDataLoadFile to save TBytes to a file, and load it from a file, no error reporting.
New Resource Files
As mentioned above, ICS now includes several resource files that are linked into applications, to avoid distributing and loading separate files, these includes OpenSSL DLLs, certificate authority bundles, root certificates and network information lists.
It is intended to issue new ICS releases containing the latest OpenSSL DLLs shortly after new versions are released, which is typically every three months unless serious security fixes require more frequent releases.
The OpenSSL resource files included with ICS come from the OpenSSL zip distribution files at https://wiki.overbyte.eu/wiki/index.php/ICS_Download . ICS currently includes three different versions for two platforms, only one is ever linked into applications according to DEFINES, see earlier. These RES files contain all the DLLs, which are extracted once to version specific sub-directories.
LibV30OpenSSL32.RES LibV30OpenSSL64.RES LibV31OpenSSL32.RES LibV31OpenSSL64.RES LibV32OpenSSL32.RES LibV32OpenSSL64.RES
ICS contains three root certificate authority bundle files, the latest versions of which can be downloaded from https://www.magsys.co.uk/download/software/ca-root-bundles.zip, one of which is linked into applications according to defines. The source bundle files are located in .\ICS-OpenSSL/ICS-RootCAs\, the RES files contail the P12 files which are smaller than the PEM versions.
RootCaCertsBundle.RES {$DEFINE OpenSSL_CA_Bundle_Large} sslRootCACertsBundle.RES {$DEFINE OpenSSL_CA_Bundle_Small} TrustedCaBundle.RES {$DEFINE OpenSSL_CA_Bundle_Medium}
There are other smaller resource files, all build with BuildICSResFiles.cmd for which the source files are part of ICS,.
ICSCerts.RES (contains ICSRootCA.pem and ICS_Intermediate_Short-bundle.pem) ICSPortList.RES (contains ICSPortList.txt) nmap-mac-prefixes.RES (contains nmap-mac-prefixes.txt)