HTTP Request/Response Paths

In this section, we provide an overview of each of the components in the architecture by means of an example of an HTTP request/response path. We then describe the functionality of each component in the system.

The numeric steps below refer the the steps shown in Figure 1:

1.

When the user requests a document, the browser issues the request to the local proxy server.

2.

This request first goes through an HTTP request filter. The request may be immediately satisfied (e.g. if the request is for a site that is blocked out, such as advertisements), may be modified (e.g. header compression [13]), or may be passed through without modification. Determination of filterable requests is based on substring matching of URLs to key strings and running corresponding scripts defined in a configuration file.

3.

If a response was not generated immediately, the request is logged by the local profile manager and the user profile is updated.

4.

The request is then passed on to the cache manager.

5.

The profile manager will create a pre-fetch list based on the usage profile and send it to the pre-fetcher.

6.

Requests which change the profile, specifically URLs which point to HTML pages, are sent to the backbone profile engine to enable backbone aggressive pre-fetches and to update the backbone profiles. Note that the use of an explicit connection to send the profile updates is mainly for ease of implementation. A more efficient mechanism would be to piggy-back such data on HTTP requests that gets transmitted from the local proxy server to the backbone proxy server.

7.

Periodically, the backbone profile engine returns a list of recommended pages to pre-fetch based on group profiles. This can occur when many users of a particular group visit a particular page. Similar to above, such information can be piggy-backed onto HTTP responses in a more efficient implementation.

8.

The recommended URLs are operated on by a function in the HTTP request filter to eliminate URLs that would be filtered (i.e., we do not want to pre-fetch items that we will filter). This new list is submitted to the pre-fetcher.

9.

The pre-fetcher collates the pre-fetch list and group document pre-fetch recommendations that are found to be not filterable. It then amortizes the pre-fetch requests to the cache manager.

10.

If the cache has a fresh copy of the document originally requested, the request is satisfied immediately.

11.

Otherwise, the request is forwarded to the backbone proxy server.

12.

The normal HTTP transaction occurs between the backbone cache manager and the WWW server.

13.

After retrieval, the document is passed through the backbone HTTP response filter.

14.

The response is sent back to the local cache manager, who will cache the document if it is a cacheable item. It is then sent back to the browser (10).

15.

The backbone profile manager maintains individual as well as group profiles. Periodically, it creates a list of recommended group documents and sends it to the local proxy server (7) of each member of the group. As profile updates arrive, it creates a list of documents to pre-fetch based on individual and group usage profiles. The only difference from the local pre-fetch list is that the backbone list is longer (i.e., we do more aggressive pre-fetches on the backbone). This list is then submitted to the backbone pre-fetch engine.

16.

The backbone pre-fetcher will issue the necessary pre-fetch requests.