THTTPCli
Last Updated: 01/27/2007

Frequently Asked Questions for THTTPCli:

     
  1. Checking last-modified date and time 
  2. ContentLength returns -1 
  3. Cookies, how to handle them 
  4. Cookies 
  5. Determine filesize before downloading 
  6. Download several files at the same time 
  7. Get the remote file size 
  8. How to submit a Web Form with HTTPCli? 
  9. How to use Cookies with HttpCli 
  10. HTTP/1.0 and HTTP/1.1 difference 
  11. HTTPCli cookies support 
  12. Why does HttpCli connects faster after the second time? 
  13. Http proxy error without an exception? 
  14. Is HTTP Resume implemented in HttpCli? 
  15. HTTP Status Codes 
  16. OnDocData triggered several times 
  17. Parsing cookies 
  18. Redirection and OnChangeLocation 
  19. Resuming a file download 
  20. Resuming a file partially downloaded 
  21. Retrieve several documents simultaneously 
  22. Tutorial for using cookies 

[Back to Main]













  
Checking last-modified date and time
J. Merrill jvm_cop+ics@spamcop.net    
11/07/2003

My application is downloading Web Cam images using the HTTP Client.  I would like to be able to check the last-modified date and time to see if the image is updated, before downloading it

Whether this will work will depend on the quality of the HTTP server you're talking to.

If you use the Head (or HeadAsync) method, instead of Get[Async], the response should have the date/time of the last update of the document you're referencing.  That's probably the best you can do.

 
[Return to Top]

  
ContentLength returns -1
Angus Robertson - Magenta    
06/06/2001

> Have a fixed html page that am endeavouring to 
> download from the web and am using a progressbar
> to provide visual display of progress of the process,
> HttpCli.ContentLength of the page returns correctly 
> sometimes but other times get -1 for the same page,


The HTTP Content-Length: header is optional, dynamic web pages often  leave it as zero.  So your program must allow progress when there is no known download size.

 
[Return to Top]

  
Cookies, how to handle them
DZ-Jay dz@caribe.net    
15/10/2004

Before sending the HTTP request (GET/POST) to server...
- open up the cookie file stored on local harddrive and read the info
- See if any of the cookies stored belong to that particular domain, server and URL path, and if they haven't expired. (If this is your first request to the server, there will be no cookies for it.)
- Submit your cookie(s) to the server by including them in the HTTP headers of your request (GET or POST).

When the server responds...
- retrieve header from WWW server to retrieve the cookie (by handling the OnCookie event)
- Store cookies on local cookie file.
- The next request you make to the WWW server, continue with step 1.

In short:
1. You send cookies specific to a domain/server/path
2. Server sends cookies back
3. You store new cookies on local file
4. continue with step 1.

 
[Return to Top]

  
Cookies
From the archive    
06/05/2001

The general cookie management guidelines are contained in the document RFC2109 which can be found in ftp://ftp.internic.net/rfc/ and there are some other places for RFCs .The cookies are parts of HTTP requests header described in the documentationon HTTP:

RFC1945 (version 1.0) and RFC2068 (version 1.1). These documents are available in abovementioned ftp.Cookies are implemented in ICS but poorly documented. One way to get cookies is to use OnCookie event of THttpCli. The eventhandler has 'Data' parameter which is the received cookie value. Since the HTTP header can contain several cookie the event may be fired sequentiallyfor several times. If you do not like this you can extract all cookies at once from the request header instead. The header will contain lines like:

Set-cookie:cookie1_dataSet-cookie:cookie2_data...

'Set-cookie' keyword is case insensitive but cookies themselves are case sensitive. For sending cookies back to web-server application you must assign them to Cookie property of HttpCli. The property is the single AnsiString (in BCB)or String (in Delphi). If you need to send several cookies then you merge them into single AnsiString separating with commas:

AnsiString AllCookies =
     cookie1_data + AnsiString (",") + cookie2_data;

HttpCli1->Cookie = AllCookies;
...
HttpCli1->Get (); // or so

The cookies are intended for internal use by web-server application. It obtains each cookie from client as whole string, with all data fields.In general you don't need to analyze any cookie data in client application.But if you wish save cookie to disk the cookie expiration time field will beinteresting for you (if exists).If you are receiving several cookies for single request then you should sendthem back to server at once. To do this you need to merge cookies intosingle string using commas.  Two cookies are used most often: one for useraccount and next for session account. The cookie data for both are createdby web-server application. So if you wish to connect to server as old userthen you send the previously saved 'user account cookie' to server in thebeginning of session. Then you receive two cookies: the 'user accountcookie' and the 'session account cookie'. Afterwards you should send themboth. Some thinking to decide which cookie is for user accound may berequired. In this scenario you don't need for analisys of cookie data. Ingeneral case it is useful to  create a simple client application wich willconnect with your web server application and show all cookies received upon HTTP request. Then you will be able to learn which cookies are used by webserver application.The conclusion: ICS can receive and send cookies for you but you must thinkyouself what should be done with them.

Cookies (if they merged into the single string) must be separated by commas(',') as it stated in the document RFC2109 and shown in my example.In fact, I had not need for advanced cookie management until now but after some thinking I am suggesting a complicated but general scenario for cookie management.1. You save long-term cookies to disk and send them at once to server-side application at the beginning of session. Long-term cookies are recognized by'expires' field in the body. You get expiration date/time from there andcontinue to send that cookie until expares. To do this you need to compare expiration time with current time. The current time is not your local timebut server time. It is contained in the field 'Date' (case insensitive keyword) of received HTTP header. Remember what expiration time can bechanged by server in any moment so you should check it every time when youreceive cookie.2. Short-term cookie have to be send to server 'as is' from first receivinguntil end of session. Short-term cookie is the cookie without 'expires'field in the body or with expiration period of few minutes. You don't save them to disk anddon't send them to server at the beginning of session.3. The body of each cookie can be changed by server-side application in anymoment. So you should have a table containing received cookie names and their last values.Only last values should be sent to server. Particularilly, the expirationtime of the cookie can be changed too. Cookie name is the part of cookie until first '='Here the slightly adopted example of received HTTP header with two cookies(first of them is long-term cookie intended for user account) which are received by my program from a server-side application. You can see abovementioned fields 'expires' and 'Date' in them. Header is RcvdHeaderproperty of HttpCli object which have the type TStrings.

 
[Return to Top]

  
Determine filesize before downloading
Mihail Belanov  belanov@galaktika.ru    
09/03/2001

> How can i determine the size of a file (e.g.
> http://www.comtecAustria.at/news.htm) _before_ downloading?

Answer1:
To do this you should send HEAD request to that address using Head() method of ThttpCli. Then search the received header (the RcvdHeader property which is TStrings) for the string like:
     Content-length: XXXX
Where 'Content-length' is case insensitive. (It is convenient to uppercase the string before searching). If such string exists then XXXX will be the document length in bytes. If the string beginning from 'Content-length' does not exists then the size cannot be known. For the address http://www.comtecAustria.at/news.htm , I have received the header showing 7184 bytes as the doc length.

Answer2:
you may do the same thing but... the other way. Just do the GET request, but create your own event handler to OnHeaderDone event of HttpCli. This way you may get the Content-Length field value, and then just stop the HttpCli by using Abort method.

 
[Return to Top]

  
Download several files at the same time
Steve Williams SMTP:tinyduck@bigpond.net.au    
27/12/2001

With several HTTP components.  ;)

 
[Return to Top]

  
Get the remote file size
Alexandre Leclerc alexandre_leclerc@eduglobe.ca    
01/04/2003

I have more difficult one: does anyone know if I can get the size of a remote HTTP file without downloading it ? Just a connect > get size, no get.

You need HTTP 1.0 RFC 1945 (HTTP 1.1 is 2068) tells this:

HTTP 1.0:  8.2  HEAD

    The HEAD method is identical to GET except that the server must not
    return any Entity-Body in the response. The metainformation contained
    in the HTTP headers in response to a HEAD request should be identical
    to the information sent in response to a GET request. This method can
    be used for obtaining metainformation about the resource identified
    by the Request-URI without transferring the Entity-Body itself. This
    method is often used for testing hypertext links for validity,
    accessibility, and recent modification.

So you could get the size, if a normal GET command would have return it.

 
[Return to Top]

  
How to submit a Web Form with HTTPCli?
Francois PIETTE francois.piette@overbyte.be    
24/03/2002

> I want to fill a web form and submit it. I wonder if this would be possible
> with the HTTPCli
component.


Yes, it is possible.

> Is there any documentation, the help file is empty on HTTPCli.
> Where can I find documentation or some source for my problem ?


Using your browser, look at the source code for the HTML form and get:
- Form ACTION: it is the URL which need to be used by THttpCli
- Form METHOD: Get or Post which is what has to be used by THttpCli
- Form ENCODING (default is URL encode): you need to format data accordingly
- Form input field NAMES: you need them to construct your data in Delphi

You also need to know if there are some cookies involved. You can know that using your browser by disabling cookies and see if the form still works with the browser. You can also know the cookies by looking in your browser cookie directory.

Once you have all those informations, you can build your data, either as a document (Post method) or as query string in a URL (Get method). Then use THttpCli Get or Post methods.

 
[Return to Top]

  
How to use Cookies with HttpCli
Francois PIETTE francois.piette@overbyte.be    
19/06/2002

Have a look at the Cookie Specs:  http://wp.netscape.com/newsref/std/cookie_spec.html

 
[Return to Top]

  
HTTP/1.0 and HTTP/1.1 difference
Angus Robertson - Magenta Systems Ltd angus@magsys.co.uk    
22/12/2002

When I use the HttpCli to get some webpages, its acquiescence uses HTTP/1.0. May I change refering into the HTTP/1.1 directly

No, not unless you modify the client component to support HTTP/1.1, which effectively means leaving a connection open and receiving and parsing multiple objects from a single stream.

 
[Return to Top]

  
HTTPCli cookies support
Francois PIETTE francois.piette@overbyte.be    
14/01/2006

You have to write a cookie manager (and maybe give it to the ICS community). Not very difficult to write. A cookie is a quartet: name, value, path and expiration date. The name is the key.

 To implement a cookie manager, create a new component (TCookieManager), embbed a TStringList to stroe the cookies names. A TStringList can have an object associated with each string so it is easy to create a small object (TCookieObject) with value, path and expiration date data, and probably a few methods to handle them.

 TCookieManager could have methods such as procedure TCookieManager.AddCookie(const Name, Value, Path : String; Expiration : TDateTime); function GetCookieValue(const Path : String) : String; procedure SaveToStream; procedure LoadFromStream; procedure Clear; procedure RemoveTimedout; and maybe some other method you could think of...

 Once the CookieManager is operating, and you have a wroking application using it thru the HTTP component events, it will be easier to add a property CookieManager into the HTTP component and make it's use as transparent as possible.

 
[Return to Top]

  
Why does HttpCli connects faster after the second time?
David Colliver david.colliver@revilloc.com    
20/12/2001


> I'm using HttpCli in an Array and fire off 4 .GetAsync methods at once.  I capture
> the results in the OnRequestDone event.  I  notice that if I run this same task
> again, the "200 OK
" results come back much faster than the first time - is there
> some kind of caching going on here with the second .GetAsync calls being made?
>
> If so, is there anyway to force the HttpCli connection to get the document from the
> actual web server and not use the cache?


Could be a number of things. First off would be the DNS lookup. If you are looking at a web site name, then that lookup is cached, so the next time, it won't need to do it.

Also, you could be behind a proxy, wether local or at your ISP. I am not sure the settings to force new page. I believe you can use a combination of code in the page, as well as the client.

 
[Return to Top]

  
Http proxy error without an exception?
Francois PIETTE francois.piette@overbyte.be    
18/06/2002


> I am using ICS HttpCli component to access a web site
> through a proxy server. Sometimes the proxy returns an
> error but I don't get an exception so I'm not aware of
> the error. The return html is as below. Not sure if there
> is a way to find out the error? I know I can parse the page :-)
>
> I'm trying quite a few proxy servers and this looks like
> a common message among some proxy servers.
>
> <TITLE>400 Bad Request</TITLE>
> </HEAD><BODY>
> <H1>Bad Request</H1>
> Your browser sent a request that this server could not understand.<P>
> Invalid URI in request GET  HTTP/1.1<P>
> </BODY></HTML>


Some proxies (and webservers by the way) do not return an HTTP error when a page is not found, but instead return a valid document stating that the requested URL is invalid. As this is not an HTTP error, nothing can be done at the component level! You must parse the received document and try to find out if it is a fake error or not.

 
[Return to Top]

  
Is HTTP Resume implemented in HttpCli?
Francois PIETTE francois.piette@overbyte.be    
20/02/2002

Yes, it is. See ContentRangeBegin and ContentRangeEnd properties. They need to be set before calling the Get method.

 
[Return to Top]

  
HTTP Status Codes
Dennis Poon dennis.poon@i-cable.com    
22/03/2002

I believe the official definition only ranges from 100 and up.

Here is what Delphi Help has:
StatusCode ReasonString
100 Continue
101 Switching Protocols
200 OK
201 Created
202 Accepted
203 Non-Authoritative Information
204 No Content
205 Reset Content
206 Partial Content
300 Multiple Choices
301 Moved Permanently
302 Moved Temporarily
303 See Other
304 Not Modified
305 Use Proxy
400 Bad Request
401 Unauthorized
402 Payment Required
403 Forbidden
404 Not Found
405 Method Not Allowed
406 Not Acceptable
407 Proxy Authentication Required
408 Request Time-out
409 Conflict
410 Gone
411 Length Required
412 Precondition Failed
413 Request Entity Too Large
414 Request-URI Too Large
415 Unsupported Media Type
500 Internal Server Error
501 Not Implemented
502 Bad Gateway
503 Service Unavailable
504 Gateway Time-out
505 HTTP Version not supported


 
[Return to Top]

  
OnDocData triggered several times
Maurizio Lotauro Lotauro.Maurizio@dnet.it    
11/07/2005

When I execute a GET, on some sites OnDocData is fired several times.

The component now can handle the NTLM authentication. In the previous version it can handle only the Basic. Conseguently it doesn't send anymore the credential with the first request (and what kind of authentication should it use?). If the server answer with a 401 code (or a proxy with 407) then the component repeat automatically the request using the credential and the strongest authentication that the server accpet and the component can hadle. This is why the event is triggered two times (for Basic, otherwise for NTLM is three times).

FYI, browser handle the authentication in the same way.

In the next release (or ATM using the "fixed" version) you can force the component to start the authentication with the first request. In any case for NTLM the event will triggered two times.

 
[Return to Top]

  
Parsing cookies
Joe Pearson joe.pearson@icon.co.za    
01/07/2003

If I request a page using an httpcli, can I parse out the cookies that were sent in the RequestDone handler or do I have to create an OnCookie handler?

The cookies are being sent in the format SetCookie: cookiename=value. Is that what all sites do?

Returning the cookies appears to be simply the process of appending the cookies into one long string separated by '; ' and assigning to the httpcli.cookie property.  Am I correct?


There's an OnCookie event  triggered when a SetCookie header is received, but you can do it all in OnHeaderData (I would say this is a better place than OnRequestDone). The newer standards define a SetCookie2 header as well, which httpcli doesn't recognise at the moment - meaning it doesn't generate an event for it - but you still get it in OnHeaderData.

The cookie protocol is not as simple as you would think from looking at one site because there are two or three levels of it, and different servers and clients/browsers implement different capabilities. That said, I've only ever seen servers using the original "Netscape" format.

The essence is that you send all cookies relevant to the site and path of the page you are requesting in a single Cookie: request header, separated by semi-colons, and the web site *may* send you multiple Set-Cookie: response headers, each of which has a name=value field and then more fields, separated by semi-colons, defined by the protocol, such as expiry times, paths, etc.

You *should* deal with expiring cookies, how many cookies you save, and other matters.

References:
cookie_spec.html from netscape
RFC2109, 1997
RFC2965, 2000

 
[Return to Top]

  
Redirection and OnChangeLocation
Wilfried Mestdagh wilfried@mestdagh.biz    
17/03/2001

> How THttpCli handle redirection and What OnChangeLocation event handler is use for

It is automatic. In the OnChangeLocation you can eventually trap a redirection.

 
[Return to Top]

  
Resuming a file download
Jon Robertson Jon.Robertson@pobox.com    
15/10/2003

Use ContentRangeBegin & ContentRangeEnd to implement "HTTP resumed downloads"

For example, if you've already download 10,500 bytes of the file, use ContentRangeBegin := 10500;  If you don't set ContentRangeBegin, the beginning of the file is assumed.  If you don't set ContentRangeEnd, the end of the file is assumed.

 
[Return to Top]

  
Resuming a file partially downloaded
Robert Chafer rob@silverfrost.com    
21/04/203

Is it possible to resume a file partially downloaded with HTTPCli.Get

Yes for a 1.1 server set ContentRangeBegin and ContentRangeEnd.

 
[Return to Top]

  
Retrieve several documents simultaneously
Francois Piette  francois.piette@overbyte.be    
21.09.1998

Use several HTTP or FTP components and use the xxxAsync commands to startthe request, which will occur in the background.  The OnRequestDone tells you when the document or file is received.

 
[Return to Top]

  
Tutorial for using cookies
Francois Piette francois.piette@overbyte.be    
11/10/2003

http://wp.netscape.com/newsref/std/cookie_spec.html [Return to Top]
   

The ICS FAQ is created and maintained by the ICS VIP Documentation Effort Group.
For more information on how to join our efforts, send email to:
[ICS FAQ]