ICS V9.0

Changes in ICS V9.0 include:

1. ICS V9 is planned as a long term support release with no new components or major features added, just bug fixes as needed, major changes will be for ICS V10. It uses OpenSSL 3.0 which is supported until September 2026 so applications should have a good life.
2. To ease introduction to ICS and for existing users looking for new features, the samples have been re-arranged with a new ActiveDemos project group for modern compilers only, that includes about 50 sample projects that between them illustrate and test all the components that make up ICS. All these samples are also available pre-compiled from the wiki pages so they can be easily tested without needing to build them. There is also a new sample OverbyteIcsSnippets that contains small samples of codes for HTTP REST, upload and download, WebSockets, FTP, simple sockets and send email. The unit includes several almost self contained methods each implementing a single functions, which are hopefully easier to follow than the normal samples used to develop ICS and are heavily documented to try and explain usage.
3. For OAuth2 authentication, TRestOAuth now supports both embedded and standard browsers, the embedded browser gives a better user experience with the window closing automatically once authentication is complete and not needing a local web server. Beware it may not be supported by Windows or end points. Launching a web page into the standard browser may replace a page being viewed, there may be firewall or other problems connecting to the localhost web server and the browser window remains open upon completion. So the end user should ideally be given a choice of which browser to use. There is a new TOAuthBrowser component and TOAuthLoginForm window that uses TEdgeBrowser (Delphi 10.4 and later) or TWebBrowser (no longer supported by Google) to display the login web pages. The LoginHint property is display in the login window and copied to the clipboard so it may be pasted into the login account field. Any applications using OAuth2 and requiring the embedded browser will need updating to add TOAuthBrowser. Only available for Delphi 2007 and later. The embedded browser can be tested with the sample OverbyteIcsHttpRestTst.
4. Added new WebSocket client and server components. WebSocket is a full duplex TCP protocol for web servers to support interactive web pages, typically dynamic updating such as chat sessions, spell checkers as you type, search hints, etc, using ws:// or wss:// URLs. The client is TSslWebSocketCli which descends from TSslHttpRest with new methods to connect, send text, bytes, a nary stream or pings, and events for new connections, received or sent frames. The WebSocket server uses the ICS web server with a new class THttpWSSrvConn which overrides THttpAppSrvConnection and provides the same WebSocket methods and events as the client, it's a duplex protocol so client and server can send data. The WebSocket client is tested with the sample OverbyteIcsHttpRestTst and the server with the sample OverbyteIcsSslMultiWebServ which has Echo, EchoPing and Chat demonstration URLs.
5. Added new MQTT protocol client and server components. MQ Telemetry Transport is a lightweight, publish-subscribe, machine to machine network protocol for message queue/message queuing service. The MQTT protocol defines two types of network entities: a message broker and a number of clients. An MQTT broker is a server that receives all messages from the clients and then routes the messages to the appropriate destination clients. An MQTT client is any device (from a micro controller up to a fully-fledged server) that runs an MQTT library and connects to an MQTT broker over a network. TIcsMQTTServer and TIcsMQTTClient handle the MQTT protocol, tested by sample OverbyteIcsMQTTst which has both client and server.
6. Made many improvements to the TDnsQuery component, to make it easier to use in other components and applications, adding synchronous methods and support for trying lists of DNS servers where one or more don't respond. It is also easier to access the arrays of different DNS answers. Added AAAALookup method for IPv6, similar to ALookup, sets array of IPv6 addresses. Allow lookup using multiple DNS servers if one or more fail, from a supplied list or internal public server list including Google, Cloudfare, OpenDNS and others, only works with the new sync methods. Added OnLogEvent primarily for debugging multiple requests and servers, may mostly be ignored.
7. Added new components TIcsDomainNameCache and TIcsDomNameCacheHttps designed to simplify forward domain name and reverse IP address lookup in applications, to avoid needing use of TWSocket or TDnsQuery components often several to support parallel lookups. There are synchronous methods that wait until a response is received and asynchronous methods that return immediately with an event called when the response is available. Lookups may be performed using uses winsock so results come from the operating system cache, using the TDnsQuery component to make requests to specific servers or public DNS servers, or using DNS over HTTPs for secure lookups. Up to 100 parallel lookups are supported although defaults to five, with waiting lookups queued and performed first in, first out. TIcsDomainNameCache is mainly for use with diagnostic components but also for servers logging remote access. Used in the OverbyteIcsNetTools, OverbyteIcsNetMon, OverbyteIcsBatchDnsLookup, OverbyteIcsDDWebService and OverbyteIcsSslMultiWebServ sample applications. Could potentially be used in other high level applications to avoid using the operating system DNS cache.
8. ICS servers and clients can now use SSL/TLS certificates from the Windows Store instead of PEM and PFX disk files, with some limitations. PEM and PFX ICS has long being able to read certificates from the Windows Store, but reading some private keys failed due to Windows APIs issues, for which a workaround has now been implemented so LoadFromStore is now able to read EC keys as well as RSA. TMsX509List allows all certificates and private keys in a store to be loaded, with the new FindBest method finding a certificate matching a search string, checking common name, part friendly name or any alternate domain name, selecting the one with the latest expiry if more than one. If the certificate name is a wildcard (*), this will be matched with any first node For clients, this is illustrated in the sample OverbyteIcsHttpRestTst to select a client certificate, for servers, IcsHosts has a new property SslLoadSource which can be set to CertWinStoreUser or CertWinStoreMachine. with former causing the store to be searched for the host name. Note the application will need administrator rights to load certificates from the Local Machine store, it will also not read keys from external hardware devices, and probably not in Trusted Platform Modules either. The sample OverbyteIcsPemTool can be used to view, select and extract certificates from the Windows Store.
9. There are a lot of general stability improvements in various low level components, protecting inherited class destroy methods from exceptions at higher levels to avoid memory leaks, such as closing a socket left open before destroy, particularly with Win64 applications. Generally free objects istead of destroying them. IcsX509VerifyErrorToStr now checks OpenSSL is loaded to avoid an exception when it's called. If neither OpenSSL DLL can be found, report both names instead of just oldest. Ensure OpenSSL is loaded before using hash functions.
10. For ICS servers, increased the default IcsHosts security level to sslSrvSecHigh for TLS/1.2 minimum, with certificate key size 2,049 bits and SHA-256 digest. Added CliCertMethod to IcsHosts to allow specific hosts to request a client SSL/TLS certificate rather than all hosts.
11. HTTP clients have a number of improvements. Some servers allow the GET and DELETE requests to have content similarly to PUT so allow this if new Options httpoGetContent is specified. Beware to set SendStream to nil if no content is intended. Try to prevent the header and content from being sent as separate TCP packets which may confuse some middleware. Always check if SendStream exists before accessing it. Added OnSyncRequestWait event called while waiting for sync operations to finish, so it can be stopped before timeout expires. Improved EHttpException messages with more detail. In TSslHttpRest, when a client certificate is requested, check it has a private key and log some information about it, better logging if no certificate. The sample OverbyteIcsHttpRestTst has a new Client SSL Certificate drop down box with options to load the certificate from a PEM/PFX file, or from the Windows Current User Store or Windows Local Machine Store (admin rights needed). A real application could offer a selection of which certificate to use as browsers do. Added ics-client-test.pem as default Client SSL Certificate, for testing against ICS servers.
12. FTP clients have a number of improvements. Added NoopAsync method sends NOOP no operation command to try and keep the control connection alive during long transfers, note there is no sync version and the response is ignored (but logged). Beware some older FTP servers may treat NOOP as illegal and fail the transfer (including ICS FTP server V8.70 and earlier). NOOP is sent by TIcsHttpMulti defaulting to 10 minutes. Without this change, FTP transfers to some public servers are failing after two or more hours due to the control connection having been closed by a router or firewall somewhere for inactivity, now tested OK with 50GB uploads (VM images). Added UTF8_ON and UTF8_ONAsync commands as an alternate to using Opts command with an argument. Increased sync timeout to 30 seconds and make sure reset with Progress more often so requests don't timeout. Fixed unicode compiler bug parsing Cmlsd/XCmlsd command response. Don't compress iso, pdf, vhd, vhdx files by default. Disable MD5/CRC32 by default, connections are now reliable and they are slow.
13. The FTP server has a fix to allow the NOOP keep-alive command to be sent while a file transfer command is being processed without causing it to fail. Added exception handling generating directory listings due to strange directory inputs by hackers causing a problem. Give msgNoPortPsv response if PORT, EPRT, EPSV or PASSIVE command has not been sent when a directory command requiring a data channel is received.
14. When ordering SSL/TLS certificates from Let's Encrypt, the local web server can now listen on both IPv4 and IPv6 addresses for domains with both. Removed OAuth2 authentication code from the component and get the tokens using and event. Simplified checking DNS challenges with the new TDnsQuery sync methods. Added MsCertLoc property to specify which Windows Certificate Store to save certificates if OutFmtWinStore is specified.
15. When sending SMTP HTML emails, the EMailImages property may be used to add streams as well as files, instead of ImageStream and StreamArray which never worked properly. Use EMailImages.AddObject(filename, TStream) where file name will be used instead of opening the file (the file need not exist).
16. The proxy server component has a new OnSrvSslHandshakeDone event called when new client connects to proxy server allowing application to check for a client certificate and abort the connection, needs CliCertMethod to be set in IcsHosts to sslCliCertRequire or sslCliCertOption, so a client certificate is requested. Fixed auto certificate ordering reading well-known file on unicode compilers.
17. OverbyteIcsWSocket includes various new utility functions. WSocketSockAddrToStr converts TSockAddrIn6 with IPv4 or PIv6 address to a string, WSocketIPAddrToSocAddr convert a string IPv4 or IPv6 address into TSockAddrIn6, WSocketFamilyToAF to find family for Windows APIs from TSocketFamily, WSocketIPv6Same to compare two TIcsIPv6Address. Added an overloaded ReceiveTB that returns a TByte instead of the received size, it also no longer fails if the buffer is not initialised.
18. OverbyteIcsUtils corrects RFC3339_DateToStr to add colon to time zone, RFC3359 requires +00:00, ISO also accepts +0000. Added StringToUtf8TB convert string to TBytes, IcsTextOnStart case insensitive text at start of line, and IcsTBytesToString to convert TBytes to unicode string. IcsWcToMb and IcsMbToWc now use cross platform RTL functions instead of OverbyteIcsIconv and USE_ICONV which have been removed. IcsIconvNameFromCodePage is now POSIX instead of USE_ICONV.
19. Added ICS Internet Packet Monitoring Components which display internet packets using raw sockets or Npcap NDIS driver, similarly to Wireshark. They are based on Magenta Systems Internet Packet Monitoring Components but updated with IPv6 and new filtering by protocol and IP address to restrict the amount of data being captured. TIcsMonSocket in OverbyteIcsMonSock provides internet packet monitoring using raw sockets. TIcsMonPcap in OverbyteIcsMonPcap provides internet monitoring using the Npcap NDIS driver. The class TIcsMonFilterClass filters captured packets. There is a new sample OverbyteIcsNetMon that is a simplified Wireshark, displaying captured packets or just totaling traffic. Packets may be captured to a textual log for easy saving or to a grid for improved display including examining each packet separately. Filters include all local IPs including broadcast and multicast, or specific protocols or services, in all case either accepting or rejecting packets according to the filter settings, dynamically during capture. This makes it easy to ignore a lot of local LAN traffic from appliances that can obscure higher level traffic.
20. Added Internet Protocol Helper Component for Windows, updated from the Magenta Systems units with full IPv6 support and new components. TIcsIpChanges in OverbyteIcsIpHlpApi monitors IP address changes and calls an event for new IPs configured or old ones removed, useful for servers where the listening address suddenly disappears. TIcsNeighbDevices in OverbyteIcsIpHlpApi builds a historic LAN MAC device and IPv4 and IPv6 address table using ARP, neighbourhood and IP range scanning with reverse host lookup. Both are tested with sample OverbyteIcsNetTools. There are many new IpHlp functions including IpHlpAdaptersInfo, IpHlpNetworkParams, IpHlpConnsTable, IpHlpTCPStatistics, IpHlpIfTable2, IpHlpAdaptersAddr, IpHlpIpAddrTable, IpHlpIpNeighbTable, IpHlpIPForwardTable, IpHlpGetRouteProtocol and IpHlpIpPathTable, all of which return various Windows networking tables and information that is useful for diagnosing network problems, all illustrated by the sample OverbyteIcsNetTools. For completeness, the sample also includes other ICS components, for Whois, NsLook, Ping and Trace Route. Other new functions include IpHlpGetDnsServers to get a list of DNS servers for this PC, IcsGetMacVendor which uses the Organizationally Unique Network Interface Identifier nmap-mac-prefixes.txt file to get the MAC adaptor vendor, very useful for identifying strange IoT devices on a LAN.
21. Updated the OpenSSL DLLs included with ICS to 1.1.1v and 3.1.2. Updated all the certificate root stores. Also supports 3.0.10 which can be downloaded from the wiki pages.
22. Added two new FMX samples, IcsHttpRestTstFmx and IcsSslMultiWebServ, converted from the VCL versions, rather messy since no proper TRadioGroup in FMX so changed to TlistBox without a caption, TListView exists in FMX but without columns so they become TStringGrid, TGrid would be better except it uses virtual data (which the help fails to mention) which needs a lot of rewriting. A few more FMX samples using new ICS components will be added for the next release.