ICS V8.69

From Overbyte
Revision as of 19:21, 27 May 2022 by Magsys (talk | contribs) (Created page with "Changes in '''ICS V8.69''' include: #V8.69 has a lot of minor improvements, more HTTP client and server features, better built-in authentication, and added built in file uplo...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Changes in ICS V8.69 include:

  1. V8.69 has a lot of minor improvements, more HTTP client and server features, better built-in authentication, and added built in file uploading to the REST client component. Also continuing improving all the HTTP samples so they now hopefully test all the features of the ICS HTTP server, specifically adding POST and PUT file uploading, and a new client login window for interactive authentication with servers. Also added OCSP (Online Certificate Status Protocol) to ICS, which is used to check SSL/TLS certificates are legitimately issued and not revoked.
  2. In the HTTP client, added new methods RequestAsync and RequestSync which start a specified request by parameter, useful to repeat the last request from an event (such as should be trust the certificate and repeat). When using a proxy, make sure Path is not blank which will break proxies, change to / as for non-proxy requests. GetRequestDoneErrorStr now handles TWsocket errors. Fixed NTLM authentication which got broken in V8.61. Added a new THttpAuthType of httpAuthDigest2 for Digest with the modern SHA-256 algorithm instead of MD5 used by httpAuthDigest, only with USE_SSL. Added new property WWWAuthInfos array filled after 401/407 failure by parsing AuthorizationRequest headers for AuthType and Realm that may be presented to select an authentication option, get login and repeat request. Remove # fragment anchor from the URL unless the new Option httpoAllowAnchor is set.
  3. In the HTTP REST client, added built in file uploading using POST or PUT. The file name is specified in HttpUploadFile using type HttpUploadStrat of HttpUploadSimple with parameters in the URL or HttpUploadMIME for multipart with parameters in the first MIME part. Note the applications needs to supply parameters like FileName so the server knows what to do with the file. This may be tested against the ICS web server samples. Upon request completion, ReasonPhrase now also has status, so OK becomes 200 OK, etc. Previously the SslRevocation property was only effective when checking the windows certificate store, now it also works with bundle files using the new TOcspHttp component and OCSP stapling if available.
  4. In the HTTP server component, now converting FLastModified to UTC/GMT time for the response header. Added new authentication type atDigestSha2 with a SHA256 hash instead of MD5 with atDigest only supported with USE_SSL. Note: Mozilla Firefox supports Digest SHA-256, Chrome and Edge do not. Added a new client connection OnHttpRespHdr event to allow response headers to be logged, previously only request headers could be logged. Cleaned up AnswerStream functions so separate lines don't go into send buffer. Added OcspSrvStapling property which should be set to enables OCSP checks and stapling only with AUTO_X509_CERTS define since it adds extra HTTP client code. A revoked certificate will be auto ordered.
  5. In TWSocket, added new method SendTB(const Data: TBytes; Len: Integer=-1) where Len is optional, also similar SendToTB and SendToTB6 with TBytes, as an alternative to casting such types to use Send(). Restored the TlsExtension_cb callback for client debugging since ClientHelloCallback only works with servers.
  6. Various improvement to TLS/SSL certificate handling. In TX509Base, added UrlOcsp and UrlIssuer properties that parse the certificate authority extension and get useful URLs. In TX509List, added FindSubject and IndexOfSubj to find certificate by SubjectOneLine, to find issuer certificate in bundles, SaveToStack which saves all the list certificates to a Stack to pass to OpenSSL functions, and SetX509Store which sets the property X509Store with all the list certificates for use with OpenSSL chain verification functions. TX509List has various new OCSP functions but these are only used internally by the TOcspHttp component.
  7. In TSslWSocketServer added OCSP (Online Certificate Status Protocol) support with IcsHosts using the TOcspHttp component to confirm server SSL/TLS certificates are legitimate and not revoked for security reasons. The certificate OCSP response is also stapled to the initial SSL/TLS HELO handshake and sent to the client to avoid it needing to lookup OCSP using HTTP itself. OCSP responses are cached and saved to a file for reloading later, but are refreshed every time the certificate is validated, at least once a day. The new server property OcspSrvStapling enables OCSP checks and stapling only with AUTO_X509_CERTS define since it adds extra HTTP client code. A revoked certificate will be auto ordered. OCSP checking is done in LoadOneCert and the stapled response sent in TriggerSslServerName when checking SNI. The same OCSP support is available in all ICS servers that use IcsHosts, including FTP, HTTP and TIcsIpStrmLog.
  8. In the IP Streaming Log component TIcsIpStrmLog, RetryAttempts -1 now means no retry attempts, 0 mean try for ever. After failure to connect, change the State to Stopping, then None since it never really started. Changed FSendBuffer to TBytes, use SendTB methods with TBytes.
  9. In the FTP server, fixed a bug in V8.65 that meant Implicit SSL only worked when using IcsHosts so was not backward compatible with old apps. Added OCSP Stapling support, similarly to HTTP.
  10. All three main HTTP SSL client sample applications FrameBrowserIcs, OverbyteIcsHttpsTst and OverbyteIcsHttpRestTst now support interactive authentication with a new Login window that displays the different methods the server will accept (from the new property WWWAuthInfos array) allowing one to be selected from Basic, Digest MD5, Digest SHA256 and NTLM logins. This window is displayed after an 401 error and the request then repeated.
  11. The OverbyteIcsHttpsTst SSL sample now has all the missing features from the non-SSL samples, Content Encoding Gzip tick box to support compression, persistent cookie support, POST/PUT support to either send simple data or upload files in various ways (from OverbyteIcsHttpPost1 sample), may be tested against the ICS web server samples.
  12. The OverbyteIcsHttpRestTst SSL sample tests file uploading using the new upload strategies supported by the REST component. # The OverbyteIcsDDWebService SSL web server sample now builds on unicode compilers. Added authentication for POST requests and new 'Password protected page (POST)' button on the demo menu to test authentication using POST. Fixed web logging to log correct multiple listener. Builds on unicode compilers. Added Digest SHA-256 authentication page DemoDigest2Auth.html and DemoDigestsAll.html that does both digests. DemoAuthAll.html no longer does NTLM, use the separate page. Now displays server response headers if box ticked, only displayed request headers before. File Upload Form and Email Form pages now work without exceptions. The OverbyteIcsSslMultiWebServ SSL web sample has similar new authentication features and logging fixes.
  13. There is a new project group SslSpecials.groupproj which includes the two samples OverbyteIcsDDWebService and FrameBrowserIcs that require extra components to be installed so are not in normal project groups, but which were often missed during routine compatibility testing.
  14. The TRestOAuth, TSimpleWebSrv, TIcsTwitter and IcsRestEmail components have moved to a new unit OverbyteIcsSslHttpOAuth, which should be added to uses automatically if the components were dropped on a form.
  15. Various fixes so ICS builds on MacOS and without SSL.
  16. Updated OpenSSL to 3.0.3 and ZLIB (HTTP compression) to 1.2.12, sorry for long delay in updating zlib, now including some important bug fixes, although never saw any issues with the minimal use ICS makes of it.
  17. The OverbyteIcsPemtool SSL sample now does OCSP checks when examining certificate files. There is also a new Test Host Certificates tab that tests SSL/TLS handshake and certificates (using TIcsIpStrmLog) for a list of host names and ports, building a list of host and intermediate certificates that may be further checked and saved as files or added to an intermediate bundle.This new tester should prove useful for debugging sites that return SSL or certificate errors.
  18. Updated the various ICS CA Trusted Stores. Microsoft has removed 81 expired and duplicate certificates that have been re-issued. Added a new file InterCaCertsBundle.pem that initially includes 28 intermediate certificates built by OverbyteIcsPemtool, which is used by that tool when checking single certificate files. In the longer term, it will be used by ICS where servers fail to send an intermediate due to misconfiguration. The industry would like to stop sending intermediate certificates with every new SSL connection to improve efficiency, if not cached they can be download as required (TOcspHttp has a method to download intermediates), so this may be necessary in ICS sooner or later.
  19. A little background to OCSP. OCSP (Online Certificate Status Protocol) is used to check SSL/TLS certificates are legitimately issued. OCSP replaces CRL (Certificate Revocation Lists, never supported by ICS) as the way to confirm SSL/TLS certificates are legitimate and not revoked for security or other reasons. The OCSP status of a certificate is found by contacting an HTTP server at the certificate issuer (URL in the certificate). OCSP responses are cached to avoid repeated downloads, and optionally saved to a file for reloading later or sharing with other applications, they generally have a seven day refresh life, but ideally need to be checked more often. OCSP responses are usually signed by the same intermediate issuer as the certificate, but sometimes the response includes a different signer certificate, both are checked against the root store as being a valid OCSP response.
  20. For efficiency, most SSL servers will send the OCSP response for their certificates during the initial SSL handshake, this is OCSP Stapling, the server should refresh this response at least daily. If there is no stapled response, the client needs to contact the OCSP URL in the certificate itself. To avoid delaying new connections while waiting for an HTTP response, OCSP is initially ignored and the now cached response checked for the next connection to that host. OCSP revocation checking is soft fail due to uncertainties in the internet, the OCSP server may be offline or not accessible via firewalls or proxies. But if a revoked status is received, subsequent connections will be blocked.
  21. OCSP is handled by a new TOcspHttp component (in OverbyteIcsSslHttpRest.pas) using new OpenSSL OCSP functions in TX509List. For clients, OCSP checking is part of the certificate chain checking during the initial handshake and is handled in the onHandshakeDone event where it can prevent a connection if revoke checking is enabled. All ICS client components that include certificate chain checking and a SslRevocation property now automatically handle OCSP, specifically TSslHttpRest, TIcsIpStrmLog, TIcsMailQueue, TIcsFtpMulti and TIcsHttpMulti. The OverbyteIcsHttpsTst and OverbyteIcsSslWebServ samples illustrates how OCSP support can be added for lower level components.
  22. For servers, OCSP is built into the certificate verification functions of IcsHosts in TSslWSocketServer so is available in all servers using IcsHosts. OCSP is used before the server starts to check that none of the supplied SSL/TLS server certificates have been revoked, if so the server will attempt to order new certificates (if enabled). The OCSP response for the certificate is then stapled to the initial SSL/TLS handshake (if OCSP status is requested) and sent to the client. On 28th January 2022, Let's Encrypt revoked several million certificates with two day's notice because some of them might have been issued using TLS/1 and not TLS/1.2, which caused a lot of manual certificate updating including with some ICS servers. Implementing OCSP means revoked certificates will be renewed automatically. The OCSP Stapling feature is also important, since it means browsers do not need to make an HTTP request to an OCSP server.
  23. ICS clients with OCSP support expose the OcspHttp property, servers OcspSrvHttp. CacheFName is the optional file name to save the cache, OcspCache.recs (or variations) in the samples which is flushed to disk after CacheFlushMins, including stapled responses if CacheStapled=True. OcspStapleOnly=True means only used stapled responses and don't make HTTP requests, OcspMaxDays is expiry of a status response, CacheRefrDays is how often to refresh responses since certificates may be revoked with only a couple of days notice in serious cases. OcspHttpProxy allows a proxy URL to be specified. There are two main responses from the OCSP server, OcspRespStatus is whether the OCSP returned information about the certificate and OcspCertStatus is that result. Note that when using low level components like TSslHttpCli, one TOcspHttp component and cache can be shared between multiple components, but not for the higher level components with OCSP built-in. A means to share cache files is planned.