ICS Getting Started

Introduction

ICS has a large number of sample application whose primary purpose is to test all the components and to learn about using those components and how to use them in your own applications. There are often several samples for a single protocol with different purposes, so this section should help get you started choosing the components and samples for your internet project.

ICS often offers low and high level versions of components, the former allow your application to send the various commands used by the protocol but you need to send those commands in the correct order often dependent upon the result from earlier commands, so you need to understand the protocol, but have control over the commands. The high level components are quicker and easier to implement because they hide most of the protocol and offer complex methods instead such as download a file, they often include extra functionality.

Historically, most ICS components are available on non-SSL and SSL versions, these notes assume you are using SSL/TLS components which are often essential today. Note most low level component need SSL/TLS adding using an SslContext and need SSL certificate chain checking added to applications, while the higher level components mostly already include the SslContext and chain checking and hide much of the SSL/TLS complexity making them faster to implement and easier to maintain as SSL changes.

Note that any ICS applications using SSL/TLS need to redistribute two or three OpenSSL DLLs unless you license YuOpenSSL from https://www.yunqa.de/ who supply compiled DCUs to replace the DLLs, easing distribution.

ICS Code Snippets

The SSL sample OverbyteIcsSnippets contains small samples of codes for FTP, HTTP, sockets and email. The unit includes several almost self contained methods each implementing a single functions, which are hopefully easier to follow than the normal samples used to develop ICS components and which often become very complicated due to all the different functionality supported. The snippets are heavily documented to try and explain usage.

Most of the snippets access Magenta Systems Ltd public ICS web and FTP servers and should just work without change, except for FTP uploading where you will need to request an account by emailing delphi@magsys.co.uk. Snippets available include:

Snippet: View Local Directories - print a directory file listing.

Snippet: File Copy One File - copy a single file.

Snippet: File Copy Multiple Files - copy multiple files.

Snippet: FTP View Directories - print a remote directory listing from an FTP site.

Snippet: FTP Download One File - download a single file from an FTP site.

Snippet: FTP Download Multiple Files - downloads multiple files from an FTP site.

Snippet: FTP Upload One File - upload a single file to an FTP site.

Snippet: FTP Upload Multiple Files - upload multiple files to an FTP site.

Snippet: HTTP Download List of Files - downloads a list of files from a web site.

Snippet: HTTP Download Linked Files - downloads multiple files from a web site by parsing HTML pages for links.

Snippet: HTTP REST Json Request - makes an HTTP GET request to a REST server receiving a Json response data.

Snippet: HTTP REST Download - makes an HTTP GET request to download a file, with optional resume of partial download.

Snippet: HTTP POST Upload File - makes a HTTP POST request to upload a file to a special upload web page.

Snippet: Local Socket Traffic - Send simple text traffic between two sockets on the same PC, using client server concepts.

Snippet: Remote Socket Traffic - Receive simple text traffic from a remote TCP Server.

Snippet: WebSocket Client - Connect to a remote WebSocket server to send and receive data.

Snippet: Send Email using Mail Queue - Runs a mail queue to send multiple emails with extended retries over many hours or days.

World Wide Web, HTTP Client

There are four types of HTTP component, with many extra components used to extend their capabilities.

TSslHttpCli in unit OverbyteIcsHttpProt is the low level HTTP protocol client that is tested using sample OverbyteIcsHttpsTst. It has buttons for GET and HEAD commands and allows numerous SSL parameters to be specified. POST requests are tested with samples OverbyteIcsHttpPost and OverbyteIcsHttpPg. Other units containing components assisting HTTP include OverbyteIcsHttpCCodZLib, OverbyteIcsHttpContCod, OverbyteIcsCookies, OverbyteIcsMimeUtils, OverbyteIcsFormDataDecoder, OverbyteIcsCharsetUtils, OverbyteIcsMsSslUtils, MIME with sample OverbyteIcsMimeDemo, SSL certificate chains with sample OverbyteIcsMsVerify. Note TSslHttpCli requires an SslContext for SSL configuration. Note HTTP clients do not need SSL/TLS certificates, but generally should check the certificate chains received from HTTPS servers to ensure they are talking to the correct servers.

TSslHttpRest in unit OverbyteIcsSslHttpRest is the high level HTTP protocol client that has additional methods and properties for making GET, POST, PUT and HEAD REST (REpresentional State Transfer) client requests, but can still do everything TSslHttpCli does. It includes a TRestParams class to build and encode GET/PUT/POST parameter strings. It also includes SSL configuration and certificate validation with a root bundle, SSL session caching, content compression, content code page decoding, persistent cookies, Json handling, logging and client SSL certificate support. There some REST examples TDnsQueryHttps, TIcsSms, TIcsTwitter and TIcsRestEmail. All tested using sample OverbyteIcsHttpRestTst.

TRestOAuth in unit OverbyteIcsSslHttpOAuth handles OAuth1/2 authentication using either embedded EdgeBrowser or TWebBrowser to display the logins web pages, or they can be viewed using an external browser. The unit supports various Microsoft User Authorities for corporate accounts. Note OAuth1/2 use requires a developer application account at Google or Microsoft, or other providers which includes Ids and secrets that need to be securely stored.

TIcsHttpMulti in unit OverbyteIcsHttpMulti is another high level HTTP client that allows downloading of multiple files from an HTTP server using full URLs, or listed by parsing links from a web page, using a single function call. It also includes SSL configuration and certificate validation with a root bundle. Tested using sample OverbyteIcsXferTst.

TSslWebSocketCli in unit OverbyteIcsWebSocketCli is WebSocket client component that descends from TSslHttpRest so most of it's properties and events are common, but there are new methods and events to access WebSocket servers using ws:// or wss:// URLs. WebSocket is a full duplex TCP protocol for web servers to support interactive web pages, typically dynamic updating such as chat sessions, spell checkers as you type, search hints, etc.

ICS has a visual web browser sample FrameBrowserIcs which needs the HtmlViewer component to be installed, which will view simple web pages that don't need Javascript, it logs both HTTP and HTML protocol and can be very useful for debugging.

There are two SSL samples OverbyteIcsHttpsTst and OverbyteIcsHttpRestTst that illustrate HTTP GET and POST requests, authentication including OAuth2, file uploading and downloading, cookies, certificate chain verification, content encoding and decoding and WebSockets with a chat demo.

There are some older non-SSL demos for console and DLL and threads, see samples OverbyteIcsConHttp, OverbyteIcsHttpAsp and OverbyteIcsHttpThrd. Another sample OverbyteIcsJoseTst can be used to test Json Object Signing (Jose) functions often used for REST requests, URL encoding and decoding and display of Json and XML data.

World Wide Web, HTTP Server

There are five different HTTP web servers, which are based on TSslWSocketServer.

TSslHttpServer in unit OverbyteIcsHttpSrv is the main web server, tested with sample OverbyteIcsSslWebServr, while TSslHttpAppSrv in unit OverbyteIcsHttpAppServer adds session support and page handlers for creating dynamic page web applications tested with sample OverbyteIcsSslWebAppServer. These servers only listen on one IP address and port, but you use multiple components for multiple listeners sharing the same events. Note TSslHttpServer and TSslHttpAppSrv require an SslContext for SSL configuration. The samples are full web servers with a lot of SSL configuration options for an SSL/TLS certificate, note HTTPS servers require an SSL certificate and will not start without one. Both samples include a number of dynamic web pages to illustrate basic web server facilities, including a contact form that sends email.

The WebSockets protocol is supported using the THttpWSSrvConn client class instead of THttpAppSrvConnection for only normal HTTP.

There is a third more advanced HTTP sample OverbyteIcsSslMultiWebServ which configures TSslHttpAppSrv differently using collections of IcsHosts properties. This allows the web server to listen on multiple IP addresses and ports at the same time, and to support multiple hosts on the same IP address serving different page content (as do most web servers). IcsHosts allow different SSL/TLS certificates to be specified for each host using built-in SslContexts, will automatically create self signed SSL/TLS certificates so the server can start, and will them order free SSL/TLS certificates from Let's Encrypt (provided running on the public internet), and re-order them every three months before they expire. IcsHosts can accept server certificates as PEM or PFX files, or from the Windows Certificate Store (but not from USB dongles). The sample OverbyteIcsSslMultiWebServr is different to most ICS samples in having a minimal GUI and being entirely configured using an INI file, it is really designed to be built as a Windows service application to run unattended in background. It includes a separate web log for each host, and will send emails when it starts and stops. It also includes some anti-hacking tests and will block abusive IP addresses. The sample is based on a commercial web server.

Sample OverbyteIcsSslMultiWebServ also includes WebSockets support with simple echo servers and a chat server.

Sample OverbyteIcsDDWebService is very similar to OverbyteIcsSslMultiWebServ but designed as a Windows service, although it will also run as a GUI for debugging. It requires DDService service framework to be installed. It also includes a REST server with simple lookup responses from a SQL database, which requires DISQLite3 to be installed.

The fifth web server is TSimpleWebSrv in unit OverbyteIcsSslHttpOAuth which is a lightweight server with minimal functionality designed for embedding in applications needing OAuth2 or SSL/TLS certificate ordering that require access to web server to check a host exists. It has a single event that presents a request and returns a response. It supports SSL with IcsHosts. There is no sample, but it is used by other ICS components.

File Transfer Protocol, FTP Client

There are two types of FTP components for file transfers.

TSslFtpClient in unit OverbyteIcsFtpCli is the low level FTP client that is tested with sample OverbyteIcsSslFtpTst. It has about 50 buttons the test the various FTP commands in various ways, and allows numerous SSL parameters to be specified. Note TSslFtpClient requires an SslContext for SSL configuration. Other older FTP samples include OverbyteIcsBasFtp, OverbyteIcsConFtp, OverbyteIcsFtpAsy and OverbyteIcsFtpMulti.

TIcsFtpMulti in unit OverbyteIcsFtpMulti is a high level FTP client that indexes, uploads or downloads single or multiple files automatically, without needing to understand most FTP commands. One function indexes files and directories on an FTP server building a list compatible with the TIcsFileCopy component that indexes Windows directories, allowing local and remote directories to be compared and files FTP uploaded or downloaded so they match. It also includes SSL configuration and certificate validation with a root bundle, SSL session caching and logging. Use the sample OverbyteIcsXferTst to test TIcsFtpMulti.

File Transfer Protocol, FTP Server

The FTP server is based on TSslWSocketServer.

The FTP server is based on TSslWSocketServer.

TSslFtpServer in unit OverbyteIcsFtpSrv is the FTP server, tested using sample OverbyteIcsSslFtpServ. The FTP server only listens on one IP address and port, but you use multiple components for multiple listeners sharing the same events. Note TSslFtpServer usually requires an SslContext for SSL configuration. The sample is a full FTP server for file uploads and downloads, with a lot of SSL configuration options for the SSL/TLS certificate and will not start without one.

There is a more advanced FTP server sample OverbyteIcsSslMultiFtpServ which configures TSslFtpServer differently using collections of IcsHosts properties. This allows the FTP server to listen on multiple IP addresses and ports at the same time, and to support multiple hosts on the same IP address. IcsHosts allow different SSL/TLS certificates to be specified for each host using built-in SslContexts, will automatically create self signed SSL/TLS certificates so the server can start, and will them order free SSL/TLS certificates from Let's Encrypt (provided running on the public internet), and re-order them every three months before they expire. OverbyteIcsSslMultiFtpServ is different to most ICS samples in having a minimal GUI and being entirely configured using an INI file, it is really designed to be built as a Windows service application to run unattended in background. The sample is based on a commercial FTP server.

Sending Email, SMTP Client

There are three types of components for sending email using the SMTP protocol or HTTP REST protocol.

TSslSmtpCli in unit OverbyteIcsSmtpProt is the low level SMTP client that is tested with sample OverbyteIcsSslMailSnd1. It has about 16 buttons to test various SMTP commands and allow an email to be sent with attachments. Note TSslSmtpCli requires an SslContext for SSL configuration. Other older test samples include OverbyteIcsConSmtp, OverbyteIcsMailHtml and OverbyteIcsMailSndAsync.

TIcsMailQueue in unit OverbyteIcsMailQueue is the high level SMTP client, tested by sample OverbyteIcsMailQuTst. It supports extended retries over many hours or days, and supports multiple SMTP relay servers or looks up MX servers using DNS, while alleviating the need for the application to handle retries. It spools emails as EML files, and can send them as well. It includes SSL configuration and certificate validation with a root bundle and logging.

TIcsRestEmail in unit OverbyteIcsSslHttpRest is alternative means of sending email using HTTP REST requests to Google and Microsoft, instead of using SMTP. Tested using sample OverbyteIcsHttpRestTst. This component also adds XOAuth2 authentication to the other SMTP components.

All the mail components can use TRestOAuth in unit OverbyteIcsSslHttpOAuth to handle OAuth2 authentication using either embedded EdgeBrowser or TWebBrowser to display the logins web pages, or they can be viewed using an external browser. The unit supports various Microsoft User Authorities for corporate accounts. Note OAuth2 use requires a developer application account at Google or Microsoft, or other providers which includes Ids and secrets that need to be securely stored.

While mailboxes requiring OAuth2 require an initial interactive login, once that completes successfully a refresh token is returned which can be securely stored and treated like a password for future access without requiring another login. The refresh token can also be used by other applications provided they uses the same account Ids and secrets, allowing non-interactive applications like the ICS web, FTP and proxy servers to use GMail using TIcsMailQueue.

Receiving Email, POP3 Client

There are two types of components for receiving email using the POP3 protocol or HTTP REST protocol.

TSslPop3Cli in unit OverbyteIcsPop3Prot is the low level POP3 client that is tested with sample OverbyteIcsSslMailSnd1. It has about 22 buttons to test various POP3 commands and allow emails to be retrieved from a mailbox. The unit OverbyteIcsMimeDec contains functions for decoding MIME encoded emails, tested with sample OverbyteIcsMimeDemo. AnOther older test sample is OverbyteIcsConPop3. Note TSslPop3Cli requires an SslContext for SSL configuration.

TIcsRestEmail in unit OverbyteIcsSslHttpRest is alternative means of receiving email using HTTP REST requests to Google and Microsoft, instead of using POP3. Tested using sample OverbyteIcsHttpRestTst. This component also adds XOAuth2 authentication to the POP3 component. TRestOAuth in unit OverbyteIcsSslHttpOAuth handles OAuth2 authentication, see above.

Forwarding Email, SMTP Server

TSslSmtpServer in unit OverbyteIcsSmtpSrv is an SMTP server that accepts emails from a client, making some checks and adding headers, which is tested by sample OverbyteIcsSslSmtpServ which writes emails to an EML spool file. Note neither component or sample support POP3 access, nor do they do anything with the EML file. The TIcsMailQueue component could be used to forward EML files. Note TSslSmtpServer requires an SslContext for SSL configuration and SSL/TLS certificate, it does not yet support IcsHosts.

Simple TCP Socket Client

TSslWSocket in unit OverbyteIcsWSocket is the root of most other ICS components opening a socket to either connect to a remote server, or to listen for connections from a remote server. The component always opens a socket by IP address, but will look-up that IP address from a host name if required, or provide a reverse look-up of host or domain name from an IP address. TSslWSocket sends or receives a stream of 8-bit binary characters, but does have methods to send and receive lines by checking or sending a CRLF line ending, which is the Telnet protocol, used for the headers all most other high level protocols like HTTP, FTP, SMTP, etc. TSslWSocket can use TCP or UDP transmission, most protocols use TCP, except DNS and SNMP. TSslWSocket can be tested using samples OverbyteIcsSimpleSslCli, OverbyteIcsCliDemo, OverbyteIcsBinCliDemo, OverbyteIcsUdpLstn, OverbyteIcsUdpSend and many others. Note TSslWSocket requires an SslContext for SSL configuration.

TIcsIpStrmLog in unit OverbyteIcsIpStreamLog is a higher level version of TSslWSocket, originally designed for IP stream logging with minimal events and extra coding, including an SslContext and full SSL/TLS certificate chain checking, with better line handling, multiple connection attempts and retries on failure or loss of connection. TIcsIpStrmLog can be configured a client or server, TCP or UDP, and is tested by sample OverbyteIcsIpStmLogTst which can run as client and server at the same time, sending data to itself.

Simple TCP Socket Server

TSslWSocketServer in unit OverbyteIcsWSocketS is the main socket server accepting a few thousand remote clients using multiple IP addresses and ports, and separately allowing data to be sent and received from those remote clients, all in a single thread. Applications need to derive a client from TSslWSocketClient into which the required functionality is added. TSslWSocketServer supports using collections of IcsHosts properties. This allows the server to listen on multiple IP addresses and ports at the same time with different SSL/TLS certificates for each host using built-in SslContexts, will automatically create self signed SSL/TLS certificates so the server can start, and will them order free SSL/TLS certificates from Let's Encrypt (provided running on the public internet), and re-order them every three months before they expire.

TSslWSocketServer is mostly tested using the ICS HTTP and FTP servers, but there are other samples, OverbyteIcsSimpleSslServer, OverbyteIcsTcpSrv, OverbyteIcsTcpSrvIPV6, OverbyteIcsThrdSrv, OverbyteIcsThrdSrvV2, etc.

TIcsIpStrmLog mentioned just above uses TSslWSocketServer for simpler server applications with a small number of remote clients.

There is also a threaded version TSslWSocketThrdServer in unit OverbyteIcsWSocketTS where each client is created with a separate thread to avoid blocking on high load servers. Beware this server does not yet support IcsHosts and multiple IP addresses, nor is there a web server using it. It is tested using sample OverbyteIcsThrdSrvV3.

Forward or Reverse Proxy Server

TIcsProxy and TIcsHttpProxy in unit OverbyteIcsProxy are designed for forward or reverse socket proxying and are tested by sample OverbyteIcsProxySslServer. Despite the component names, these components support SSL using IcsHosts with all the usual functions. TIcsProxy is protocol agnostic and may be used to proxy any TCP protocol, the sample includes SMTP, POP3, NNTP and telnet. TIcsHttpProxy is a full forward and reverse HTTP/HTTPS proxy with header and body parsing and processing host names and URLs to match the source and destination. Note the sample has a minimal GUI and is configuring using an INI file.

Websockets Client and Server

See HTTP Client and HTTP Server above.

MQ Telemetry Transport

TIcsMQTTServer and TIcsMQTTClient in OverbyteIcsMQTT handle the MQ Telemetry Transport message queuing service, tested by sample OverbyteIcsMQTTst which has both client and server,

Telnet Client

TTnCnx in unit OverbyteIcsTnCnx implements the TCP/IP telnet protocol including some options negotiations, tested by sample OverbyteIcsTnDemo. TTnEmulVT in unit OverbyteIcsTnEmulVT offers ANSI terminal emulation tested using sample OverbyteIcsTelnetClient.

Network News Reader, NNTP Client

TNntpCli in unit OverbyteIcsNntpCli is a NNTP client, tested by OverbyteIcsNewsReader.dpr with 28 buttons for the various commands,

Create, Order or Review SSL/TLS Certificates

ICS contains many functions for processing SSL/TLS X509 certificates and private keys. TX509Base in unit OverbyteIcsWSocket may contain and server or client certificate, private key and one of more intermediate certificates, and has properties to display most of the certificate elements, all tested by sample OverbyteIcsPemtool. TX509List contains multiple certificates, typically a root store loaded from a PEM file. TMsX509List will load certificates from any Windows Certificate Store including private keys. TSslCertTools in unit OverbyteIcsSslX509Utils can read, create and save certificates, private keys, certificate requests and sign requests as a certificate authority.

The OverbyteIcsPemtool sample can be used to create certificates and private keys in various RSA, EC and other formats, create certificate requests and sign requests as a certificate authority, and convert certificate files between different formats, by reading as one and saving as another, also combining keys and certificates in a file. Root stores and single certificates may be to be viewed. Certificates and private keys in Windows Stores may be viewed and deleted, and a PEM or PFX certificate bundle installed into any Windows Store.

TSslX509Certs in unit OverbyteIcsSslX509Certs, tested by sample OverbyteIcsX509CertsTst which automatically downloads SSL/TLS X509 certificates from various issuers, including free certificates from Let's Encrypt, and commercial certificates from CertCentre AG. Supports ACME V2 protocol, and REST protocols for specific vendors. Domain and DNS validated certificates should generally be issued without intervention, other commercial certificates may take days to be approved. This unit may be added to ICS server applications using IcsHosts for automatic ordering, while the sample may be separately used to order certificates manually, including DNS validated wildcard certificates from Let'S Encrypt. All orders are kept in a database to allow automatic or manual re-ordering before expiry.

Lookup Domain Names, DNS

Simple DNS host look-ups using the DNS servers configured for Windows are done using the DnsLookup method in TSslWSocket and also the ReverseDnsLookup method, both fire an event with potentially multiple results, tested by sample OverbyteIcsDnsLook. These methods are used by almost all ICS components, although they currently only use the first IPv4 or IPv6 result returned, if more than one.

TDnsQuery in unit OverbyteIcsDnsQuery allows more complex DNS requests to be made to specific DNS servers to get all DNS records such as MX or TXT, tested using sample OverbyteIcsNsLookup. It includes a list of public DNS servers including Google, Cloudfare, OpenDNS and others, and will access these sequentially if one does not respond. TDnsQueryHttps in unit OverbyteIcsSslHttpRest adds DNS over HTTPS for secure lookups. Also tested with sample OverbyteIcsNetTools.

TIcsDomNameCache and TIcsDomNameCacheHttps cache forward and reverse DNS lookup requests, mainly for diagnostic components but also for servers logging remote access. May be configured to use Windows lookup, UDP/TCP using TDnsQuery or HTTPS, testing using samples OverbyteIcsBatchDnsLookup and OverbyteIcsNetTools.

Unit OverbyteIcsWmi contains a number of functions for accessing a Windows DNS Server (Windows Server 2012 and later) to list DNS zones and zone records, and to add zone records, tested by sample OverbyteIcsWmiTst. The functions are also used by sample OverbyteIcsX509CertsTst to add DNS records for the ACME DNS challenge.

Network Diagnostic Tools

TPing and TPingThread in unit OverbyteIcsPing is used to ping any host to see if it's available on the internet, note some hosts may deliberately not reply, tested by samples OverbyteIcsPingTst and OverbyteIcsNetTools which both include trace route.

TIcsWhoisCli in unit OverbyteIcsWhoisCli makes Whois requests to get details for the registrations of domain names and IP address ranges, tested by samples OverbyteIcsWhoisCliTst and OverbyteIcsNetTools. The component has a large list of Whois servers for various countries around the world.

TIcsMonSocket in OverbyteIcsMonSock provides internet packet monitoring using raw sockets. TIcsMonPcap in OverbyteIcsMonPcap provides internet monitoring using the Npcap NDIS driver. There are both tested using sample OverbyteIcsNetMon which is similar to the WireShark diagnostic tool and can be used to monitor internet packets on a LAN, with filtering using TIcsMonFilterClass to include or exclude IPs, port or protocols.

TIcsIpChanges in OverbyteIcsIpHlpApi monitors IP address changes and calls an event for new IPs configured or old ones removed. TIcsNeighbDevices in OverbyteIcsIpHlpApi builds a historic LAN MAC device and IPv4 and IPv6 address table using ARP, neighbourhood and IP range scanning with reverse host lookup. Both are tested with sample OverbyteIcsNetTools which also uses several other IpHlp functions including IP Connections list, Network Adaptors and Interfaces, IP Routing and Path tables, ARP tables and Network Statistics.

TSnmpCli in unit OverbyteIcsSnmpCli does SNMP (simple network management protocol), tested by sample OverbyteIcsSnmpCliTst.

TSysLogClient in unit OverbyteIcsSysLogClient send syslog packets, tested by sample OverbyteIcsSysLogClientDemo.

TSysLogServer in unit OverbyteIcsSysLogServer receives syslog packets, tested by sample OverbyteIcsSysLogServerDemo.

TIcsTimeClient and TIcsTimeServer in unit OverbyteIcsSntp support SNTP for getting and setting the correct time over the internet, tested using sample OverbyteIcsTimeTst.