Chunked transfers in HTTP/1.1

June 11, 2000

Delphi 5 Professional & Enterprise

The Hypertext Transfer Protocol is probably the most widely used protocol on the Internet today. For example, this document you are currently reading was probably transmitted using the HTTP protocol version 1.1. This new version of the HTTP protocol allows you to transfer content in chunks, which is a technique you should learn if you are to create professional applications over HTTP.

The version 1.1 of the HTTP protocol, or HTTP/1.1 as it's officially called, is defined in RFC 2068 and later updated by RFC 2616. In section 3.6.1 of the latter, the Standards Track document defines a transmission coding method called the Chunked Transfer Coding.

At first, using this method might not sound very practical, as using this method requires some overhead and also support from the user's browser. However, chunked transfers are important when it comes to uplifting the end-user's experience in HTTP applications.

Is the web server processing my request?

Have you ever surfed the Internet and noticed that some operations take forever to complete? Maybe you just submitted a form and are waiting for the result. But is it just taking a while on the server or is the problem your connection to the Internet? Well, you are not alone with your feeling.

When using the Internet, end-user experience is important. In ordinary Windows-based applications it is easy to provide feedback to the user. For example, if you are creating a database application with Delphi, you might display a progress bar to the user. This progress bar would then allow the user to see how long the operation is going to take.

In HTTP, you cannot normally display such progress bars because operations are atomic. When user requests a resource from a server, the web server responds with the resource. However, let's assume you are creating a dynamic web application, say an ISAPI application with Delphi. Further, assume this application is quite slow due to the amount of data it must process.

In this case, how can the user know whether the server is processing the query or not? Of course, the easy solution is to first send some HTML code to the browser, and then proceed with the operation. The browser might then display a "Please wait" message and then continue to wait for more information.

However, the problem is that most browsers don't behave like that. Instead, they wait for the whole transfer to complete, and only then display the results. Surely, the case is different when the document is long, but for short documents the data won't be displayed until it is transferred completely.

Chunked transfers solve the problem

How could this problem be solved, then? The solution lies in the HTTP protocol itself. By using chunked transfers, you indicate that the data you send will be transmitted in parts. Whenever one of such parts arrives at the browser, the browser knows it can safely display the data even if it is not completely transmitted.

Technically, using chunked transfers requires a new browser (at least Internet Explorer 4.0) and a special field in the HTTP response header, as follows:

Transfer-Encoding: chunked

That is, the complete HTTP header might look like this:

HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Date: Sun, 11 Jun 2000 14:05:53 GMT
Transfer-Encoding: chunked
Content-Type: text/html

It is not necessary to go into the details how the chunked Transfer-Encoding works, but you should understand that using it allows you to display text on the browser while you are processing rest of the request. Let's see how this might work in practice.

Using chunked transfers in ISAPI applications

In the example program described, we'll assume you are using Delphi 5 to build an application that need to create a ZIP file. For example, the application might issue a query to a database, fetch the results, and then ZIP them for faster transfer. Also, assume this operation takes a while, about 15 seconds.

To keep the user happy while waiting for the operation to finish, the ISAPI application displays a "progress bar" to the user that looks like this (IE5 only):

(click here to stop/restart the animation)

To display the progress bar from the server-side, the example application uses the following code:

begin 
  Response.SetCustomHeader('Transfer-Encoding','chunked');
  Response.SendResponse;
  SendChunk(PleaseWaitChunk);
  ID := CreateZip;
  SendChunk(Format(RedirectChunk,[ID]));
  SendChunk(''); { the final, empty chunk to mark end of the transfer }
end;

Here, the CreateZip function is used to do the real processing. In this case, it is simply:

Function CreateZip : String;
Var F : File;
Begin
  Result := IntToStr(Random(10000));
  AssignFile(F,'f:\inetpub\wwwroot\'+Result+'.zip');
  ReWrite(F,1);
  BlockWrite(F,F,SizeOf(F));
  CloseFile(F);
  Sleep(10000); { assume operation takes 10 seconds }
End;

Because the PleaseWaitChunk HTML code is sent to the browser before the CreateZip function is called, the user's browser will display the above progress bar while the ISAPI DLL is processing the result.

Please note that creating a file in the WWWRoot directory through an ISAPI application will require the appropriate file access rights (Write and Modify) if the directory is on a NTFS partition.

Redirecting to a new page

After the ZIP file has been successfully created, the second HTML chunk redirects the browser to a page that displays the message "Operation finished". This is achieved using the following JavaScript code:

<SCRIPT LANGUAGE="JavaScript">
<!--
active = 0;

function doRedirect() {
  document.location = "/scripts/chunked.dll/ready?id=%s";
}

setTimeout(doRedirect,1000);
// -->
</SCRIPT>

The final page will then redirect the browser once more, this time to the final ZIP file. This is done using the following HTML metatag:

<META HTTP-EQUIV="Refresh" CONTENT="1; URL=/%s.zip">

This will make the browser to display the familiar File Download dialog box. Of course, "%s" will be replaced by the name of the ZIP file.

Download the example code

Download chunkedtransfers.zip (178 kB) which contains the sample ISAPI application "Chunked". Please note that the sample application will require Delphi 5 Professional or Enterprise as well as Internet Information Server (IIS) or compatible.

* * *

Need help developing advanced HTTP applications with Delphi? Contact us!