Difference between revisions of "FAQ SSL/TLS Certificate Authority Root Stores"

From Overbyte
Jump to navigation Jump to search
 
(8 intermediate revisions by the same user not shown)
Line 32: Line 32:
 
But it's not easy to create root bundles from CCADB and another developer
 
But it's not easy to create root bundles from CCADB and another developer
 
got frustrated with updating roots, and created a Trust Stores Observatory
 
got frustrated with updating roots, and created a Trust Stores Observatory
Git repository:
+
Git repository: https://github.com/nabla-c0d3/trust_stores_observatory which contains over 680 root certificates and lists of which trust store contain which roots by different operating systems. But even this does not contain certificates in a form easily used by OpenSSL, so Magenta Systems Ltd has written a small tool that converts the YAML files from TSO into PEM and PKCS7/12 bundle files, one each for the different operating systems.
 
 
https://github.com/nabla-c0d3/trust_stores_observatory
 
 
 
which contains about 600 root certificates and lists of which trust store
 
contain which roots by different operating systems. But even this does
 
not contain certificates in a form easily used by OpenSSL, so Magenta
 
Systems Ltd has written a small tool that converts the YAML files from
 
TSO into PEM bundle files, one each for the different operating systems.
 
  
 
== New PEM Bundle CA Trusted Store Files ==  
 
== New PEM Bundle CA Trusted Store Files ==  
  
 
There are six different PEM CA bundle files, built from the Trust Stores
 
There are six different PEM CA bundle files, built from the Trust Stores
Observatory Git repository:
+
Observatory Git repository in February 2024:
  
  apple.pem - 166 Certificates
+
  apple.pem - 169 Certificates
  google_aosp.pem - 129 Certificates
+
  google_aosp.pem - 134 Certificates
  microsoft_windows.pem - 331 Certificates
+
  microsoft_windows.pem - 282 Certificates
  mozilla_nss.pem - 133 Certificates
+
  mozilla_nss.pem - 147 Certificates
 
  openjdk.pem - 89 Certificates
 
  openjdk.pem - 89 Certificates
 
  oracle_java.pem - 93 Certificates
 
  oracle_java.pem - 93 Certificates
  
Each certificate is prefixed by it's description, issuer fields, expiry, public key type and SHA256 hash, so the bundles are self documenting rather than being just cryptic base64 blocks. These PEM bundles may be loaded into an OpenSSL context as a root store.  Magenta Systems Ltd will periodically update these bundles, as needed. The files are all UTF-8 with a BOM. While the certificates are base64 encoded, the aded comments may include Unicode characters for non-English issuers.  
+
Each certificate is prefixed by it's description, issuer fields, expiry, public key type and SHA256 hash, so the bundles are self documenting rather than being just cryptic base64 blocks. These PEM bundles may be loaded into an OpenSSL context as a root store.  Magenta Systems Ltd will periodically update these bundles, as needed. The files are all UTF-8 with a BOM. While the certificates are base64 encoded, the added comments may include Unicode characters for non-English issuers.  
  
The zip file contains two versions of each bundle, the name above and one ending with -clean.pem which omits all the added textual comments so is smaller and less likely to cause problems with non-English characters.  There are also -titles.txt and fprints.txt files which are one line per certificate listing the main details, and fingerprint in the latter file.  There are also changes files for the Microsoft Windows bundlle that indicates which certificates were removed or added with each update.
+
The zip file contains three versions of each bundle, the name above, one ending with -clean.pem which omits all the added textual comments so is smaller and less likely to cause problems with non-English characters, and a third PKCS7/12 version with extension P12 which is smaller than PEMs.  There are also -titles.txt and -fprints.txt files which are one line per certificate listing the main details, and fingerprint in the latter file.  There are also changes files for the Microsoft Windows bundle that indicates which certificates were removed or added with each update.  
  
 
These bundles may be downloaded at:
 
These bundles may be downloaded at:
Line 74: Line 66:
 
certificate store and instead downloads new certificates as needed by
 
certificate store and instead downloads new certificates as needed by
 
Windows browsers.  So with ICS V8.63 and later, it is now the same as the new
 
Windows browsers.  So with ICS V8.63 and later, it is now the same as the new
microsoft_windows.pem bundle mentioned above.  It currently contains 331
+
microsoft_windows.pem bundle mentioned above.  It currently contains 282
certificates and is 675 Kbytes in size and may be found in the
+
certificates and is 570 Kbytes in size for the PEM version.
Samples/Delphi/SslInternet/ directory.
 
  
 
2 - TrustedCABundle.pem is a smaller file, with certificate for major
 
2 - TrustedCABundle.pem is a smaller file, with certificate for major
 
commercial issuers manually updated as newer sites are found to have
 
commercial issuers manually updated as newer sites are found to have
 
missing root certificates.  But this file is more dynamic than
 
missing root certificates.  But this file is more dynamic than
RootCaCertsBundle.pem.  It currently contains 97 certificates and is
+
RootCaCertsBundle.pem.  It currently contains 109 certificates and is
179 Kbytes in size and may be found in the Samples/Delphi/SslInternet/
+
201 Kbytes in size for the PEM version.
directory.
 
  
 
3 - To avoid distributing bundle files and as a fail safe if a file can
 
3 - To avoid distributing bundle files and as a fail safe if a file can
not be found, ICS includes 59 built-in hard coded certificates in
+
not be found, ICS V8.32 to V9.0 include built-in hard coded certificates in
 
OverbyteIcsSslX509Utils.pas which can be returned as a string by the
 
OverbyteIcsSslX509Utils.pas which can be returned as a string by the
function sslRootCACertsBundle. Again this unit may be dynamic with new
+
function sslRootCACertsBundle. Note only the TSslHttpRest, TIcsIpStrmLog,
certificates added as needed.  Note only the TSslHttpRest, TIcsIpStrmLog,
 
 
TIcsFtpMulti, TIcsHttpMulti and TIcsMailQueue components use the built-in
 
TIcsFtpMulti, TIcsHttpMulti and TIcsMailQueue components use the built-in
 
bundle by default, other components need to add it manually to avoid the
 
bundle by default, other components need to add it manually to avoid the
extra program code involved.
+
extra program code involved. With V9.1, all the bundles can be optionally liniked
 +
into applications as resource files and loaded automatically on start-up,
 +
so this bundle is no longer needed. 
  
 
4 - ICS also includes a component TMsCertChainEngine in the unit
 
4 - ICS also includes a component TMsCertChainEngine in the unit
Line 108: Line 99:
 
or even longer.
 
or even longer.
  
Either of the certificate bundle files may be loaded into an SslContext
+
With ICS versions V9.0 and earlier, the certificate bundle files may be found in the
 +
Samples/Delphi/SslInternet/ directory  and may be loaded into an SslContext
 
by using the SslCAFile property.  The built in bundle may be specified
 
by using the SslCAFile property.  The built in bundle may be specified
 
before the SslContext is initialised using SslCALines.Text property, or
 
before the SslContext is initialised using SslCALines.Text property, or
Line 116: Line 108:
 
'unable to get local issuer certificate' if a trusted certificate is not
 
'unable to get local issuer certificate' if a trusted certificate is not
 
found in the store.
 
found in the store.
 +
 +
With ICS version V9.1 and later, the bundles files have moved to a new common directory
 +
'C:\ProgramData\ICS-OpenSSL\ICS-Certs\' and are available as three resource files
 +
RootCaCertsBundle.RES, TrustedCaBundle.RES and sslRootCACertsBundle.RES, one of
 +
which may be linked into applications using {.$DEFINE OpenSSL_CA_Bundle_Large},
 +
{.$DEFINE OpenSSL_CA_Bundle_Medium} or {$DEFINE OpenSSL_CA_Bundle_Small} in the
 +
OverbyteIcsDefs.inc file.  A common IcsSslRootCAStore component is created at start-up,
 +
optionally loading OpenSSL and the root CA bundle linked as a resource. 
  
 
The contents of the three certificates bundles are listed at [[FAQ_ICS_SSL/TLS_CA_Trusted_Store_Contents | FAQ - ICS SSL/TLS CA Trusted Store Contents]]
 
The contents of the three certificates bundles are listed at [[FAQ_ICS_SSL/TLS_CA_Trusted_Store_Contents | FAQ - ICS SSL/TLS CA Trusted Store Contents]]

Latest revision as of 19:22, 16 February 2024

CA Trusted Store Background

The whole SSL/TLS certificate verification process depends upon finding a trusted root certificate that signed the next certificate up the chain, and so on. But who chooses those trusted roots?

The answer is the author of the application that accepts an SSL/TLS certificate, or maybe the operating system the application is running on, or the SSL library used to build the application.

The bad news is that root certificates come and go, old roots stop being accepted for various reasons, and new roots are introduced for new security standards or new businesses.

Historically most roots were RSA Sha1 digest signed and most still are, despite Sha1 being banned for new certificates. Increasingly Sha256 and ECC root certificates are being introduced and required by new intermediate certificates. Typically, active certificate authorities have at least four roots, with 2,048 and 4,096 bit RSA keys and 256 and 394 bit ECDSA keys (which are much smaller than RSA).


Sources of CA Trusted Stores

SSL/TLS trusted root certificate bundles and always changing, annually perhaps for major changes, although Microsoft officially Windows roots every two months. The Common CA Database (CCADB) https://www.ccadb.org/ is a repository of information about Certificate Authorities (CAs), and is used by a number of different root store operators to manage their root stores.

But it's not easy to create root bundles from CCADB and another developer got frustrated with updating roots, and created a Trust Stores Observatory Git repository: https://github.com/nabla-c0d3/trust_stores_observatory which contains over 680 root certificates and lists of which trust store contain which roots by different operating systems. But even this does not contain certificates in a form easily used by OpenSSL, so Magenta Systems Ltd has written a small tool that converts the YAML files from TSO into PEM and PKCS7/12 bundle files, one each for the different operating systems.

New PEM Bundle CA Trusted Store Files

There are six different PEM CA bundle files, built from the Trust Stores Observatory Git repository in February 2024:

apple.pem - 169 Certificates
google_aosp.pem - 134 Certificates
microsoft_windows.pem - 282 Certificates
mozilla_nss.pem - 147 Certificates
openjdk.pem - 89 Certificates
oracle_java.pem - 93 Certificates

Each certificate is prefixed by it's description, issuer fields, expiry, public key type and SHA256 hash, so the bundles are self documenting rather than being just cryptic base64 blocks. These PEM bundles may be loaded into an OpenSSL context as a root store. Magenta Systems Ltd will periodically update these bundles, as needed. The files are all UTF-8 with a BOM. While the certificates are base64 encoded, the added comments may include Unicode characters for non-English issuers.

The zip file contains three versions of each bundle, the name above, one ending with -clean.pem which omits all the added textual comments so is smaller and less likely to cause problems with non-English characters, and a third PKCS7/12 version with extension P12 which is smaller than PEMs. There are also -titles.txt and -fprints.txt files which are one line per certificate listing the main details, and fingerprint in the latter file. There are also changes files for the Microsoft Windows bundle that indicates which certificates were removed or added with each update.

These bundles may be downloaded at:

https://www.magsys.co.uk/download/software/ca-root-bundles.zip

Magenta Systems Ltd will periodically update these bundles, as needed.

ICS CA Trusted Stores

ICS includes three CA Trusted Store, two as PEM bundle files, one in a source unit, and access to the Window Certificate Store directly:

1 - RootCaCertsBundle.pem is a large file that was originally created 15 years ago by exporting the Windows certificate store using the OverbyteIcsPemTool sample. But Windows 10 no longer has a complete local certificate store and instead downloads new certificates as needed by Windows browsers. So with ICS V8.63 and later, it is now the same as the new microsoft_windows.pem bundle mentioned above. It currently contains 282 certificates and is 570 Kbytes in size for the PEM version.

2 - TrustedCABundle.pem is a smaller file, with certificate for major commercial issuers manually updated as newer sites are found to have missing root certificates. But this file is more dynamic than RootCaCertsBundle.pem. It currently contains 109 certificates and is 201 Kbytes in size for the PEM version.

3 - To avoid distributing bundle files and as a fail safe if a file can not be found, ICS V8.32 to V9.0 include built-in hard coded certificates in OverbyteIcsSslX509Utils.pas which can be returned as a string by the function sslRootCACertsBundle. Note only the TSslHttpRest, TIcsIpStrmLog, TIcsFtpMulti, TIcsHttpMulti and TIcsMailQueue components use the built-in bundle by default, other components need to add it manually to avoid the extra program code involved. With V9.1, all the bundles can be optionally liniked into applications as resource files and loaded automatically on start-up, so this bundle is no longer needed.

4 - ICS also includes a component TMsCertChainEngine in the unit OverbyteIcsMsSslUtils.pas which allows applications to avoid using bundle files and instead access the Windows Certificate Store directly to validate certificates. There is a very slight overhead as the store is opened and Windows may need to download missing root certificates. Only the TSslHttpRest, TIcsIpStrmLog, TIcsFtpMulti, TIcsHttpMulti and TIcsMailQueue components includes TMsCertChainEngine by default, with the CertVerMethod property selecting CertVerNone, CertVerBundle or CertVerWinStore. TMsCertChainEngine does include one extra optional feature to check if any certificates in the chain have been revoked by their issuer, perhaps for fraudulent use, beware revoke checks involve contacting each issuer and can slow down chain validation by a few seconds or even longer.

With ICS versions V9.0 and earlier, the certificate bundle files may be found in the Samples/Delphi/SslInternet/ directory and may be loaded into an SslContext by using the SslCAFile property. The built in bundle may be specified before the SslContext is initialised using SslCALines.Text property, or the LoadCAFromString method after initialisation.

If checking a certificate chain, OpenSSL will issue the error message 'unable to get local issuer certificate' if a trusted certificate is not found in the store.

With ICS version V9.1 and later, the bundles files have moved to a new common directory 'C:\ProgramData\ICS-OpenSSL\ICS-Certs\' and are available as three resource files RootCaCertsBundle.RES, TrustedCaBundle.RES and sslRootCACertsBundle.RES, one of which may be linked into applications using {.$DEFINE OpenSSL_CA_Bundle_Large}, {.$DEFINE OpenSSL_CA_Bundle_Medium} or {$DEFINE OpenSSL_CA_Bundle_Small} in the OverbyteIcsDefs.inc file. A common IcsSslRootCAStore component is created at start-up, optionally loading OpenSSL and the root CA bundle linked as a resource.

The contents of the three certificates bundles are listed at FAQ - ICS SSL/TLS CA Trusted Store Contents