ICS V8.60

Major changes in ICS V8.60 include:

1. Added several new components and sample applications created by Magenta Systems Ltd over the past 17 years and previously distributed and installed separately to the ICS distribution. Bundling them with ICS makes installation and updating easier, and allows existing ICS samples to make use of many the new components, such as UTF-8 file logging. All the components have new names so existing applications using the originally distributed versions will still work, but it's recommended updating existing applications for the new ICS versions. The added components include IP stream logging, SMTP Mail Queue, Time Server and Client, Whois client, blacklisting of malicious IP addresses, file logging functions, file indexing, copying and deleting, FTP indexing, multiple file uploading and downloading, HTTP page parsing and URL downloading.
2. In THttpCli, only follow relocation for 3xx response codes, not 201 Created, but keep Location property for 201 which is often response to a POST and may be needed by the application.
3. When starting TWSocket Connect, the IP address chosen for DNS lookup is now saved in ASCII as AddrResolvedStr which is useful in connect OK or failed events to see whether an IPV4 or IPv6 address was chosen, and which was used if DNS offered multiple IPs. AddrResolvedStr is exposed as a property in TWSocket, THttpCli, TSslHttpRest and TFFtpClient and reports in failed connection events. Other clients will be added soon.
4. Various samples have been updated to ease testing of IPv6 and to save the diagnostic window activity to a disk log file, OverbyteIcsHttpsTst, OverbyteIcsHttpRestTst, OverbyteIcsTimeTst, OverbyteIcsX509CertsTst.
5. Added round robin DNS lookup if DNSLookup returns multiple IP addresses, so they are used in turn after a failure when a component is called repeatedly without being freed. This is implemented in THttpCli, TSslHttpRest TFFtpClient and TIcsTimeClient. Other clients will be added soon. There is a new OnSelectDns event to override round robin lookup and make your own choice. By default, the DNS lookup in ICS components ignores IPv6 addresses and always use the first IPv4 address offered, when there is more than one. This is usually implemented in the OnDnsLookupDone event in the application or high level component. So if that first address does not respond, the application never tries any other addresses. This has become more of a problem when enabling applications for IPv6, by changing SocketFamily from the default sfIPv4, to sfAny, sfAnyIPv4 (prefer IPv4), sfAnyIPv6 (prefer IPV6) or sfIPv6 (only IPv6), when IPv6 addresses may also be returned as well as IPv4. Due to routing or firewall issues, IPv4 and IPv6 might not both be available and so connection will fail if that address is chosen first. Previously it was necessary to restrict SocketFamily so only the working family is attempted. The DNS round robin implementation relies on keeping the last successful connected IP address, so it can be re-used for subsequent connections, but looping through any alternative addresses if the last connection failed, for subsequent connection attempts. Most existing applications use the OnDnsDone event to select the connection IP address so the round robin code is added there. Newer applications making use of TWSocket ComponentOptions wsoAsyncDnsLookup should added code in OnDNSLookupDone and update the DnsResult property which is then used by Connect.
6. IcsHosts supports two new TSslSrvSecurity server security levels, sslSrvSecTls12Less and sslSrvSecTls13Only, the former disables TLS1.3 in ICS servers if TLSv1.3 fails (perhaps a bad client implementation) while the second only supports TLSv1.3.
7. Up to date C++ packages are included for 10.2 Tokyo and 10.3 Rio. Information on installing ICS for C++ 2007 may be found at: https://en.delphipraxis.net/topic/844-use-ics-with-c-builder-2007/ . Sorry, currently don't believe it's easy to update the ICS source code to avoid the changes needed for C++ 2007.

New Units

OverbyteIcsIpStreamLog.pas - TIcsIpStrmLog
IP stream logging, using TCP Client or Server, UDP Client or Server, sends simple text.

OverbyteIcsMailQueue.pas - TIcsMailQueue
SMTP Mail Queue with extended retries, multiple SMTP servers or MX look up.

OverbyteIcsSntp.pas -TIcsTimeServer, IcsTimeClient
Time server and client supporting SNTP time protocol.

OverbyteIcsWhoisCli.pas - TIcsWhoisCli
Whois protocol client with server directory.

OverbyteIcsBlacklist.pas - TIcsBlackList, TIcsStringBuild, TIcsBuffLogStream
Blacklisting of malicious IP addresses, file logging functions.

OverbyteIcsFileCopy.pas - TIcsFileCopy
Indexing, copying and deleting of multiple file directories, optionally only transferring different files, it will copy entire volumes.

OverbyteIcsFtpMulti.pas - TIcsFtpMulti
FTP client that indexes, uploads or downloads multiple files in multiple sub-directories optionally only transferring different files.

OverbyteIcsHttpMulti.pas - TIcsHttpMulti
HTTP client that downloads multiple files from a list or by parsing web links.

OverbyteIcsHtmlPars.pas - TIcsHTMLParser
HTML web page parser.

OverbyteIcsFileCopyW.pas - TIcsFileCopyW
Same as OverbyteIcsFileCopy but Unicode for Delphi 2007.

OverbyteIcsFtpMultiW.pas - TIcsFtpMulti
Same as OverbyteIcsFtpMulti but Unicode for Delphi 2007.

OverbyteIcsHttpMultiW.pas - TIcsHttpMulti
Same as OverbyteIcsHttpMulti but Unicode for Delphi 2007.

OverbyteIcsFtpCliW.pas - TFtpClientW, TSslFtpClientW
Same as OverbyteIcsFtpCli but Unicode for Delphi 2007.

OverbyteIcsFtpSrvW.pas - TFtpServerW, TSslFtpServerW
Same as OverbyteIcsFtpSrvW but Unicode for Delphi 2007.

OverbyteIcsFtpSrvWT.pas
Same as OverbyteIcsFtpSrvWT but Unicode for Delphi 2007.

New Sample Applications

OverbyteIcsMailQuTst.dpr
Simple mailing list tool using Mail Queue using SSL.

OverbyteIcsIpStmLogTst.dpr
Test IP stream logging, sending streams as client or server using SSL.

OverbyteIcsWhoisCliTst.dpr
Test Whois protocol, looks up servers automatically.

OverbyteIcsTimeTst.dpr 
Test SNTP time protocol as client or server.

OverbyteIcsXferTst.dpr
File transfer testing, file copying, FTP upload and download, HTTP download, and file listing, with SSL.

OverbyteIcsXferTstW.dpr
Same as OverbyteIcsXferTst but Unicode for Delphi 2007.

OverbyteIcsIpStreamLog.pas

TIcsIpStrmLog is designed for IP stream logging, using TCP Client, TCP Server, UDP Client or UDP Server protocols, sending simple text lines across a network so they may be displayed or written to disk remotely. The component allows two way communication with TCP and UDP, so may also be used for simple protocols such as communication between two applications. The component supports multiple client sockets so may be used to send data to two or more different remote servers at the same time. Supports SSL including ordering X509 SSL certificates.

Note, this component is not fully working yet, pending ordering certificates.

Use of TIcsIpStrmLog is illustrated in the new OverbyteIcsIpStmLogTst sample project.

OverbyteIcsMailQueue.pas

TIcsMailQueue component has two main benefits over a simple TSslSmtpCli component: it supports extended retries over many hours or days, and supports multiple SMTP relay servers or looks up MX servers, while alleviating the need for the application to handle retries. Mail is queued to disk, so retries will continue if the application is restarted.

TIcsMailQueue is designed to prepare, queue and send email. Preparing the email is done using the the ICS THtmlSmtpCli component so it may be plain text or HTML email with one or more file attachments.

Once the mail properties in QuHtmlSmtp have been specified, it is queued using the QueueMail method which saves it to an EML formatted spool file.

The component runs a thread which checks the queue for new EML spool files, and attempts to forward them to one or more SMTP Mail Servers using TSslSmtpCli, optionally with SSL. If mail delivery succeeds, the spool file may be deleted or moved to an archive folder. If mail delivery fails, the spool file remains in the queue and further attempts are made separated by the times in minutes listed in the RetryList list. If all delivery attempts fail, the spool file may be deleted or moved to a badmail folder.

Use of TIcsMailQueue is illustrated in the new OverbyteIcsMailQuTst sample GUI project, and in the OverbyteIcsSslMultiWebServ sample project as a server. The OverbyteIcsMailQuTst sample includes a window that displays the content of the mail queue, to see what has yet to be sent and allowing it to be cancelled, this window may be copied or modified for use in end user applications.

OverbyteIcsSntp.pas

TIcsTimeServer and TIcsTimeClient, time server and client, supporting time protocol with UDP and TCP portions of RFC868, and SNTP v4 (Simple Network Time Protocol) to RFC2030. Note that full NTP is not supported.

The TIcsTimeClient includes round robin DNS lookup attempts if there are multiple DNS addresses for the host, trying each alternate in case of failure. Multiple IPs are common for pool.ntp.org hosts. Ideally this round robin DNS concept should be incorporated into other high level components that currently choose the first IP address offered, or even TWSocket itself.

Support is planned for Roughtime, a new secure time protocol designed by Google. also supported by Cloudfare.

Use of TIcsTimeServer and TIcsTimeClient are illustrated in the new OverbyteIcsTimeTst sample GUI project.

OverbyteIcsWhoisCli.pas

TIcsWhoisCli is a Whois protocol client using TWSocket. Conforms to RFC- 954. Looks up host names and IP addresses.

The component has an internal list of over 200 top level domains available through the WhoisServers property with their whois servers, if none is found the IANA registry is checked instead.

Some queries will result in a secondary query to another whois server for more detail, the result of just the last query is available as property LastResp. Most COM queries are two stage due to multiple registries, likewise IP address queries are first to ARIN then often to the regional registry. Alternatively, the StartQuery method does a single query to the server specified in the Host property.

Use of TIcsWhoisCli is illustrated in the new OverbyteIcsWhoisTst sample project. Unfortunately, Whois is less useful now than for the past 20 years since the owners of most web sites are now hidden for privacy, when they were previously useful identifying malicious and spam sites.

OverbyteIcsBlacklist.pas

IcsBlackList supports blackisting or block listing of IP addresses that attempt repeated failed access to TCP/IP servers. It maintains a list of IP addresses or Values that have previously exceeded a specific number of failed attempts, against which new attempts may be checked.

TIcsStringBuild to efficiently build Ansi or Unicode strings on all versions of Delphi.

TIcsBuffLogStream buffered log stream designed to write large log files, flushing regularly to disk by opening, writing and closing each time the buffer fills or after a timeout of X seconds. The file name is date/time mask format, typically for one log file per day. Writes files in ANSI, UTF- 8 or UTF-16, with a BOM.

IcsSimpleLogging write text to end of old or new file, opening and closing file, ignores any errors, not designed for continual updating! The file name is date/time mask format, typically for one log file per day.

Use of these functions is illustrated in the OverbyteIcsSslMultiWebServ sample project. TIcsBuffLogStream is also used on other samples for logging.

OverbyteIcsFileCopy.pas

TIcsFileCopy allows indexing, copying and deleting of multiple file directories, using a single function call. The component handles listing source and destination files, and comparing them to avoid unnecessary transfers, selection using a file mask, deletion of old files, delete after copy, safe copy (to avoid partial files), and zip before copying or unzip after copy. A progress event provides various levels of information for logging or display, depending upon application requirements, and allows copying to be cancelled. The component is equally useful for copying single files or entire volumes with hundreds of thousands of files, and provides 'sync files' functionality making it ideal for backups, only copying changed files and deleting old target files. OverbyteIcsFileCopyW.pas is similar, but Unicode for Delphi 2007.

OverbyteIcsFtpMulti.pas

TIcsFtpMulti allows uploading or downloading of multiple files from or to an FTP server, from a single function call. The component handles listing local and remote files, including subdirectories, and comparing them to avoid unnecessary transfers, selection using a file mask, deletion of old files, resuming failed FTP downloads, unzipping of downloaded files, or zipping before upload, deletion after uploading or downloading, archive copy after upload. A progress event provides various levels of information for logging or display, depending upon application requirements, and allows transfers to be cancelled. TIcsFtpMulti descends from ICS TFtpClient and publishes all it's logon and proxy properties and events. OverbyteIcsHttpMultiW.pas is similar, but Unicode for Delphi 2007.

OverbyteIcsHttpMulti.pas

TIcsHttpMulti allows downloading of multiple files from an HTTP server using full URLs, or listed by parsing links from a web page, using a single function call. The component handles listing local and remote files, and comparing them to avoid unnecessary transfers, selection using a file mask, unzipping of downloaded files. A progress event provides various levels of information for logging or display, depending upon application requirements, and allows downloads to be cancelled. TIcsHttpMulti descends from ICS THttpCli, and publishes all it's properties and events. OverbyteIcsHttpMultiW.pas is similar, but Unicode for Delphi 2007.

OverbyteIcsHtmlPars.pas

TIcsHTMLParser is an HTML web page parser.