I recently attended a seminar hosted by Microsoft at Pune, India unveiling the potential of new standards (html 5, css3) and innovations (IE9) that are going to make the entire web experience different (for better). The discussion, no doubt, offered good amount of food for thought. What I intend to discuss here is not comprehensive sets of features present in the trio but the challenges we are likely to face and a few questions raised in the seminar that shouldn’t had been answered but weren’t.
video:Introduction to Remoting
Here is a quick video blog on Remoting Architecture. This has been well received and appreciated in the inner circle. The video training discusses the working and architectural overview of how a remoting system works. The discussion doesn’t talk about any particular remoting framework and the same idea, in principle, govern the working of technologies such as Java RMI, .Net Remoting and WCF.
Video: Introduction to Http
I have already devoted several posts on HTTP Protocol, its significance in web application development. Here is a short video talk on Http.
Internet Internationalized
Finally Internet is changing. And its not a change that we come across every other day. It is the internationalization of Internet and we stand to witness the history getting created. The ICANN chairman Peter Dengate declares the event as the
biggest technical change to the Internet since it was created four decades ago
Asp.Net Http Handlers (Part 1)
In the last instalment we discussed the need and motivation behind http handlers. We also discussed the creation and deployment of http handlers using ISAPI specification. In this section we will discuss how we can implement Http Handler using .net framework.
Before we talk about ‘how’ lets first talk about ‘what’ and ‘why’. Although we have already discussed about Http Handlers let us re-define from a different perspective.
Http handlers are programs that are invoked by a web server to get particular job done. period.
Now this job can be as big as implementing a whole new programming infrastructure or as small as printing a hello world message.
Let us also now discuss the possible meaning of an asp.net http handler. By extending the above definition we can easily deduce:
.Net Http Handlers are .net programs that will be invoked by the web server to get some job done.
But then, we already have facility to write a .net application (.aspx page) to get job done. So why do we need a new application? Why not just use a .aspx page? Let us try to understand this part with a sample application.
Why .net HTTP Handlers?
Let us try to answer this question by discussing a scenario:
Job Description: We need to create a image generator. The Image Generator can generate a given text into an image on the fly. Such images can be used for generating CAPTCHA or creating graphs based on the data present in database or creating an image of your email. The possibilities are endless.
Now that we know our job, let us try to find out what are the alternatives:
Create a .aspx page
Let us say we can create an imagegen.aspx to handle the situation. User can generate the image by invoking it with right set of parameters example:
http://dev.vnc.in/imagegen.aspx?type=text2img&text=dev@vnc.in&font-size=12&color:blue
This approach will work. However, there are several problems:
- The page need to be incorporated in every website. A .aspx page is typically not considered as a component module. While same .aspx page can be included in several different website; it is not a very elegant pattern.
- Although not explicitly stated, .aspx page is typically used for generating .html output. It is evident from the fact that a .aspx page is designed as a web page. Code behind is an optional entity. While it can generate an image; it is certainly not a very clean solution.
- Implementing a complicated set of steps in a single .aspx page goes contrary to the best coding practices. Over a period of time such pages will become unmanageable.
- Because a .aspx page can’t serve all sorts of purpose .net came up with other extensions such as .asmx.
While a .aspx page can be used for writing any kind of logic; it is neither the design goal nor a good practice us a .aspx page for roles other than generating a web output. This is clear from the fact that Microsoft itself came up with other extensions such as .asmx for web service.
It appears we have two options – wait for Microsoft to come up with a solution for my problem or we come up with our own customized solution.
I am sure you are not going to wait for Microsoft to come up with some solution. Are you?
If you are thinking in terms of our customized solution; it implies you are thinking about a HTTP Handler
Ok So we need a HTTP handler. But then we already have an infrastructure to create and deploy and HTTP Handler – ISAPI HTTP Handler. We can use it. Isn’t it?
Why not an ISAPI Http Handler?
Sure. ISAPI seem to be an appropriate solution for creating an HTTP Handler. After all it exists with the sole purpose of creating an Http Handler. But then it is not really a .net based solution. Is it?
ISAPI is a win 32 dll and the handlers are typically written in C/C++ or other programming languages. The ISAPI don’t support extensive .net API. And a .Net developer need to master a new language to write a Handler.
Of the two choices – ISAPI vs .aspx ; developer often choose easier and not so good approach – writing an .aspx page.
Having discarded the two alternatives, we are left with the only obvious choice – .net based HTTP Handlers. Let us try to understand why .net http handlers makes sense:
An asp.net http handlers are http handlers written in .net. It utilizes the full capacity of .net API and is an easier solution for a .net developer. It uses the best of both the worlds – ease of Handler coupled with convenience of .Net.
Implementing HTTP Handler in .Net
Having convinced ourselves about the benefit of a .Net Http Handler, let us understand the steps needed for implementing a .Net based Http Handler.
Step 1 – Creating The Handler
Creating the Handler is a simplicity in itself. All We need is to create a class that implements System.Web.IHttpHandler interface. The Interface has got just two methods. The following class diagram represents our ImageHandler and its relation with IHttpHandler.
Typically an Http Handler will be created as a class library. We also need to add a reference to System.Web assembly.
Let us create a class library. Name it vnc.web.ImageGenerator –
Now Add the necessary System.web assembly.
Change the project property to make the default namespace to vnc.web instead of vnc.web.ImageGenerator
Now delete the default class1.cs and add a new class – ImageGenerator.cs. Next implement interface System.Web.IHttpHandler. Implement the Interface.
using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Web; namespace vnc.web { public class ImageGenerator: IHttpHandler { #region IHttpHandler Members public bool IsReusable { get { throw new NotImplementedException(); } } public void ProcessRequest(HttpContext context) { throw new NotImplementedException(); } #endregion } } |
http://dev.vnc.in/app-name/red/black/georgia/12/vivek.jpg.image
Now let us go ahead and write our business logic. The business logic for the current ImageGenerator can be downloaded from the project and I am not including it here.
Compile the project to generate the necessary assembly. The Creation of HTTP Handler is complete.
Deploying the HTTP Handler
Now that the HTTP Handler is ready, let us focus on the deployment steps. It includes two steps:
Configuring IIS Handler Mapping
Since our HTTP Handler is not an ISAPI Handler, it is not supposed to be registered directly in the IIS Handler Mapping as we discussed in the previous episode – Http Handlers (Although the latest IIS does have an option). What we need to do is to map the new URL Pattern to asp.net engine. Now IIS will pass the request for new resource to aspnet_isapi.dll . (Ofcourse asp.net engine doesn’t know how to process this request. But we will handle this in a separate step)
Now all request for the .image extension will be diverted to aspnet_isapi.dll – The Asp.net engine. However, predictably, asp.net has got no idea as to how to handle this request or whom to pass this request. This configuration will be done in web.config or mechine.config depending on the requirement.
Registering Our Handler with asp.net
To configure asp.net engine to pass request for .image resources to our handler we need to register our handler in web.config or machine.config. We need following steps:
- Add a Test website to our solution
- Add a reference to the website project. Select the vnc.web.ImageGenerator.dll assembly from the list.
- Add handler reference to web.config. Locate handler section and add the code as mentioned below. Look up for a Handler section and add following entry. The code has been modified for clarity
<httpHandlers> <remove verb="*" path="*.asmx"/> <add verb="*" path="*.asmx" validate="false" … > |
The newly added entry has been underlined. Now we are done and we can test our application safely.
try out running following url
http://localhost:47877/HandlerDemoSite/red/black/georgia/22/Vivek%20Dutta%20Mishra.jpeg.image
And you should get a jpeg image generated for you with your choice. The given path doesn’t exist physically and yet it will return the result.
In our next instalment, we will discuss some other advanced features of asp.net http handler.
Download File – ImageGeneratorSolution
Http Handlers
This is a continuation of our series on HTTP. This Articles discusses the motivations and design concerns related to creation of Handler. The articles covers the technical details of what we discussed in the preceding article Http Hacked.
Before we attempt to understand the concept of handlers we need to understand the working of HTTP. I would recommend going through the previous articles for this purpose:
- story of www – between http and html – This is the main article that discusses the evolution of www, http protocol. Its motivation and the road map.
- Http Hacked – The most important article to understand the current discussion. This is the second instalment on the running story on http. The episode mainly discusses how http was hacked for web programming.
- Http Through Ages – This article is a technical overview of the changes between various versions of http and you may like to go through it for the completion of discussion.
Now lets quickly recap how an Http Server works.
Scenario 1. Http Server is requested to Serve a static resource
By static resource we mean a html document or an image or a downloadable content that always exists (or can exist) on the server, and all a server does is to pick it and pass it to the client. The following diagrams represents the scenario:
Any and every Http Server is designed to serve this kind of request. As such the request is handled in strict accordance with http protocol.
Scenario 2. Http Server is requested a Dynamic Resource
The situation get trickier when you need to serve a resource that doesn’t exist; rather it is created. More accurately – that is computed. An example could be – requesting a list of customers. The information exists in the database. It needs to be extracted and presented in a tabulated format. So we need a program that –
- Accepts users request and relevant parameters
- Parameter may tell to get a list of priority customer
- Filter customers based on geographical location and so on
- Executes relevant queries on to the database and extracts the necessary records
- Creates an HTML page with the extracted records in a presentable format.
- Sends the newly created html page to the user.
So we need a program. But how does this program relate to the web server. Web server need to delegate its work to this special program which is designed to handle the request. The overall interaction process is represented in following diagram:
The diagrams brings out following points clearly:
- Web server really doesn’t know how exactly to use the dynamic resource (in our case customer.aspx)
- It relies on an external application or extension to get the computation done. These extension are typically termed as http handlers.
- A web server typically can delegate the computation to one of the many handlers.
- Each Handler is supposed to process a particular kind of request which can be identified by url format.
- A url format may mean
- In most of the cases a particular extension – *.asp, *.aspx, *.php, *.jsp and so on
- It may also mean request associated with a particular folder. Eg. /cgi-bin/*
- It may be a particular absolute path – Eg /log/log.app
- The web server checks the Url and then decides whom to delegate the request.
Developing The Handlers
As we already discussed in our previous article Http Hacked, that the need of such an extension was felt clearly. We needed a mechanism to extend the server in different and unpredictable ways. For this reason different vendors proposed their solution. The solutions had implementation differences, however, philosophically they had same working principle:
Define a set of guidelines and API that will be used for creating a handler. The Handler will expose a certain well defined set of functionality that will be called by the web server to get the request processed.
Microsoft proposed an API layer and termed it as ISAPI – Internet Server API
Netscape, for example, proposed NSAPI – Netscape Server API.
Servers and handlers are supposed to be compliant with one of the available standards. For example IIS is an ISAPI compliant server and can work with any ISAPI compliant handler. All ISAPI compliant handlers are supposed to work with every ISAPI compliant servers.
An ISAPI handler is typically written as a win32 dll and is designed in C/C++ language. However, other language choices are also available.
Deploying the Handler
Once a Http Handler is created, it needs to be registered with the web server and mapped with an url pattern. Following screenshots shows different ISAPI handlers registration with IIS 7.0.
Step 1: Start Internet Service Manager from Control Panel –>Administrative Tools
Step 2: Navigate to Http Handler Mapping section
Step 3: We will have a list of existing mapping. We can modify them or add new entries
Step 4: This is a sample mapping of .aspx files to aspnet_isapi.dll
Summary
In this section we looked at the roles of Http handlers and their deployment. In the next episode we will be checking deployment of http handler using asp.net.
SDDL
Security Descriptor Definition language (SDDL) is a language used for defining the NT Security Descriptor attribute and has been traditionally used to define ACL in windows registry and other NTFS Files. Most of the time the one is not required to understand the cryptic syntax of SDDL as it is manipulated using MMC interface. The syntax was little productive for the power users who may like to manipulate the windows registry using text editor.
Http Hacked
www was born to be just another route to internet; fate made it the internet.
In its inception it was meant to be for publishing electronic documents and the model was quite similar to print industry. concentration was on formatting the documents and documents changed infrequently. Access may require authentication but authentication was simple and need for a real time data non-existent.
As dependence on internet grew more and more complicated, need for a smarter communication was clear. There were seemingly two options:
- Create a new protocol.
- Hack and Existing Protocol.
And if your choice was second, your choice had to be HTTP.
Why hack HTTP and not just create a new protocol?
Here we will be getting into the guessing work. But the guesses are likely to be as perfect as fact itself. But why guess work? Because, hacks are rarely documented and never systematic.
First reason, that favours use of any existing protocol, and not just http, is the availability of large infrastructure which needs to be created across the globe, otherwise.
Http protocol emphasised on presenting formatted information, something which was the common denominator of all the programming requirement. Using http as the starting point made sense. The other important reasons that favoured http are:
- Presence of customizable request and response headers.
- Output in presentable format that can support layouts, links, tables etc.
- Availability of forms to get user input.
- HTTP anyway needed a solution for such a dynamic requirement.
As we have already talked, http soon felt the need of a search engine and the first proposed solution to it was to modify the http server itself. While the solution worked, it was clearly less than a desirable design and very far from an ideal solution. Not only it required modification to the server, it was apparent that this kind of solution would necessitate a lot of change in the server for so many different reasons.
What we really needed was what we term as open-close principle. A design that can allow other applications to run and support server thus extending its functionality without modifying the server for every business requirement.
Roadmap to http hack
Before we understand the solution, let us quickly picture how a simple http communication works. We will look at four different scenario of http communication.
Scenario 1: Requesting an html page
Client sends following request to the server.
GET HTTP/1.1 /profile/vivek.html [cr][lf]
CAN-ACCEPT: */*[cr][lf]
[… more headers …]
[cr][lf]
The server responds to client with a response header and the content.
HTTP/1.1 200 OK [cr][lf]
CONTENT-TYPE: text/html[cr][lf]
CONTENT-LENGTH: 2012[cr][lf]
[… more headers …]
[cr][lf]
[… actual content follows … ][content ends; connection closes]
Notice, the request was for an html page and the server responds with status code of 200, indicating that request was successful. Next it sends CONTENT-TYPE as text/html. It indicates that the generic type of output is text and the format of document is html. The header gives sufficient information so that client can save the output as a .html file and render it as an html document.
Scenario 2: Requesting an image
A typical request for image will be no different from the previous one.
GET HTTP/1.1 /profile/vivek.jpg [cr][lf]
CAN-ACCEPT: */* [cr][lf]
[… more headers …]
[cr][lf]
The server once again checks for the document and sends a response:
HTTP/1.1 200 Ok [cr][lf]
CONTENT-TYPE: image/jpeg [cr][lf]
CONTENT-LENGTH: 42085 [cr] [lf]
[… more headers …]
[cr][lf]
[… byte by byte content of image … ]
Notice this time server reports generic content type as a image file. The specification further reveals that the image is of type jpeg. Thus the client saves the image in a file with extension .jpg. The content is once again rendered as a image and appropriate viewer is used.
Scenario 3: Requesting an executable
Once again the request remains same.
GET HTTP/1.1 /downloads/time-piece.exe [cr] [lf]
CAN-ACCEPT: */* [cr][lf]
[… more headers …]
[cr][lf]
Here we have placed a request for an executable called time-piece.exe. Server checks for the same and responds:
HTTP/1.1 200 Ok [cr][lf]
CONTENT-TYPE: application/octet-stream [cr][lf]
CONTENT-LENGTH: 47200 [cr][lf]
[… more headers …]
[cr][lf][… byte by byte content of the executable …]
As we can see this time the mime type reported is application/octet-stream. This is a indication to the server that the response is an application and the content is not supposed to be displayed on the browser. It is supposed to be downloaded. The browsers typically present a save option.
Let us look at a fourth scenario. Here the request is made for an image. However server sends header such that the image is downloaded rather than displayed in the browser.
Scenario 4: Request that allows you to download the image
GET HTTP/1.1 /misc/india-map.jpg [cr][lf]
CAN-ACCEPT:*/* [cr][lf]
[… more headers …]
[cr][lf]
This time server sends the response in the following manner.
HTTP/1.1 200 Ok [cr][lf]
CONTENT-TYPE: application/octet-stream [cr][lf]
CONTENT-LENGTH: 48029 [cr][lf]
[… more headers …]
[cr][lf]
[… actual image content …]
Notice the request and response is proceeds in the same way as in scenario 2, except for one change. This time server describes content-type as application/octet-stream. This is a signal to client to save the output rather than to display on the browser.
So what is the bottom line?
Clients can place the request for a resource. All requests are placed in the same way.
Server Response typically include a CONTENT-TYPE. Clients reaction to response will typically be governed by the CONTENT-TYPE specified in the response header and NOT on the request headers. That also implies that the things can change between the request and response.
CGI – The http Hack
Since the modification to the server was neither desirable nor practical, NSCA developed Common Gateway Interface (CGI) specification. CGI soon became an standard way in which applications can interact with the server and generating dynamic content.
CGI is a specification and not an application or a particular programming language. The specification is designed for an application that reads (http request) from STDIN and presents the output on STDOUT. As long as this requirement is met, any application compiled or interpreted or scripts can act as CGI.
So how does it typically work?
- The client will request for the CGI application. Let us say a binary executable file.
- This time server instead of providing the binary executable for download will actually execute the application on server.
- The CGI application will generate the output to STDOUT
- The output from the CGI application will be piped to HTTP response stream.
- Thus the client will get the output of executable rather than executable itself.
Let us understand the whole scenario in http request response format:
GET HTTP/1.1 /whats-the-time.exe [cr][lf]
CAN-ACCEPT:*/* [cr][lf]
[…other headers…]
[cr][lf]
You will notice that the request header remains unchanged. However this will work differently this time. Server will actually execute the application whats-the-time.exe on the server. This application will check the current time on the server and print it in html format on the STDOUT. Server will respond to client as:
HTTP/1.1 200 ok [cr][lf]
CONTENT-TYPE:text/html [cr][lf]
CONTENT-LENGHT:1280[cr][lf]
[… other headers …]
[cr][lf]
[… the STDOUT output of the application …]
Notice this time we requested for a .exe. However, the content-type in response indicate a text/html output. The application executes on the server and generates dynamic information which is sent to the client.
But how will the server know whether to execute the executable as an CGI or to allow it as a download? To simply the issue, a directory was designated to store all the CGI application. A request from the designated directory will be treated as CGI, any other requested will be treated differently.
The First CGI application was written in C language. The CGI directory was accordingly named as cgi-bin as the CGI was a binary executable. Ever since it is a convention to name the CGI directory as cgi-bin.
This kind of application opened a Pandora of new possibilities including:
- User registration and authentication.
- online data manipulation, query and reporting. This is the most important aspect which is the back bone of all e-commerce, b2b and b2c applications.
- Accessing the network resources using dynamic interface
Beyond CGI
A host of technology evolved over time. Many claiming to be superior and more efficient than the original CGI. However, in philosophy they remain exactly similar to the original idea of CGI:
- All these technology essentially mapped a request to some external application.
- The application processed the request, interacted with database or other server resources and presented the output.
- The output is then sent to the client.
They however, differed with the CGI in some subtle aspects:
- Most of the newer technology used scripts rather than an executable. For efficiency, these scripts can be compiled to an intermediate code.
- Use of multiple thread rather than multiple process.
- The respective handlers are typically mapped to a particular extension rather than a particular folder.
The alternative technologies that work on these principles are asp and asp.net from Microsoft, jsp and servlets from Java world, php and host of other open source technology. CGI scripts are still written mostly in languages like perl.
Client side hacks
Although, initially the server side hacks well served its purpose, as the load on server begin to increase problems with this approach became more than apparent.
- Every decision needs to be taken on the server (html doesn’t have decision making capabilities)
- This caused un-due load on the server.
- Round the trip communication between client and server were costly both in terms of time and the bandwidth.
- If technology could support, many a decision can actually be taken on client side and it even makes more sense.
- A client side technology, however, can never replace the need of server side technology as several information are available only on the server end.
The first client side technology which revolutionized the scenario was java applet. Java applet is essentially an application written in java which is embedded in a web page in a manner quite similar to a picture. However, it is an intelligent application written in java. Also the browser need to have special plug-in support to understand and execute an applet in addition to html.
Soon a number of new technologies offered similar approach:
Client side technologies requires special browser addons. These intelligent objects are downloaded off the browser using standard HTTP request and are stored in local machine in a temporary cache. Once loaded these applications are executed with the help of their respective addons. Support for these technology typically depends on browser and their capacity to expand.
Popular client side technology include:
- Java applet – being the first of its kind
- Scripts – Java script being most popular. other scripts include vbscript, jscript and so on
- Adobe Flash – Orginally macromedia flash. Rich graphical experience. Continued to rule its world
- Microsoft Silverlight – Comparatively new entrant; Offers functionality similar to flash. However, it is backed by .Net programming language which makes it more charming. However, being a Microsoft technology, chances of full hearted support from industry looks bleak.
Ajax
A wonderful hack which comprises support from both client side and server side and gets the best of both the worlds. A complete discussion on Ajax is beyond scope of current discussion but certainly merits on on itself.
Hack and hell
Believe it or not, the two words often go hand in hand. A typical web application is a mix of so many different technology:
- HTML Document
- A server side script merged in the HTML document. These scriptlets will be processed by the server side applications to produce clean HTML document.
- Client side scripts. These scripts will be executed by browser addons.
- Client side objects. These objects will be embedded within the page and displayed. However, they are not HTML and they remain together physically.
A web application is typically made up of different components developed in different technology and they execute in different environment and machine. Their development requires varied set of expertise and negotiating between them is often complicated. Deployment of these technologies on different machines and configuration is another herculean task.
There are real difference in how different browsers interpret various client side components such as html, css, javascript etc.
All these things create a real hell for web developer and confusion for end users.
We still dream to see a more perfect web world
story of www – between http and html
Long Long ago (well, just two decades to be precise) when Google didn’t exist, hotmail was not thought of and domains were free (yes free!!!), there were few first class citizens of the web – Archie, Veronica and Jughead who played around Gopher and FTP playgrounds (networks). People exchanged emails and finger pointed at the online people and machines. They called it Internet. It was a simple world and every body had just one role. People used internet using ftp and telnet, exchanged mails using pop and smtp and found the status information using finger. (if you want to check more about this fascinating era have a look at Internet Life before www)
Then came the www. It was destined not just to change the internet for ever; but also to push most of the older protocol to the books of history. Gopher was forgotten; Archie, Veronica, Jughead returned back to the pages of comic. Other protocols such as ftp, pop, smtp remained active but became an alternative approach of what was actually their primary roles. Steadily http redefined internet as the world wide web. It all started with one server and one browser.
The first http server was named httpd. It was developed by CERN. Ever since, many web servers name their main executable as httpd. Later, it was taken over by software development group of NCSA (National Centre of Supercomputing Application). NCSA also developed Mosaic, the browser that revolutionized the internet. Later the same team that developed Mosaic went on writing Netscape.
www started for publishing hyperlinked information to the web so that many people can access the information at one central location rather than exchanging using e-mails.
There were three core elements that redefined internet as www:
Core Elements of WWW
1. HTML
The first public document on the HTML specification was titled HTML Tags in late 1991. In this document, Tim Berners-Lee described 20 elements (tags) for describing a simple html document. Thirteen of those 20 elements still exists in HTML. The idea of a hypertext system for internet was an direct adaption of Tim’s own 1980 prototype called ENQUIRE implemented for CERN researchers for sharing documents. Later in year 1989, Robert Cailliau, another CERN Engineer also proposed a hypertext system for similar functionality. In year 1990, they collaborated on a joint project – World Wide Web (W3) which was accepted by CERN. In 1993, HTML was officially declared as an application of SGML and necessary DTD were created. Over years HTML would undergo a lot more change such as acknowledging custom tag to display inline image with the text. Versions of HTML and specification would come. However all those additions concentrated on formatting the document. Ironically the Hypertext Mark up Language lacked on of the most fundamental elements of a language – ability to express and decide. It has no concept of variables, loops conditions and so on…
2. A Browser
The credit to popularise the idea of WWW goes to a large extent to Mosaic, often credited as the first graphical browser (at least the first real popular one). It was Mosaic that proposed the idea of displaying image inline with text. The browser was released in the year 1993 and was officially discontinued in the year 1997. However, the road was set for newer browsers that will continue to struggle for their supremacy.
3. HTTP
HTTP started as a humble transmission protocol to transmit the HTML page across the web. Client would simply connect to internet and send a simple command –
GET /welcome.html
And server would respond with an HTML content. Disconnects. Period. There is no other option in request; no other possibility of a response. It was meant to pull textual (html) content from web pages which will be hyperlinked together. No images, no other content. Just HTML. But it was supposed to change soon.
Because pictures have been integral part of literature and content, soon HTTP grew more flexible and more complex. HTML became just one of the contents that can be sent. More Verbs, more control. It underwent several changes from its inception (version 0.9) to its current release (version 1.1). A detailed discussion on HTTP through ages is available here. However, a brief account of what is available in the current release of HTTP is as follows:
- Allows a range of requests apart from the original GET. The other allowed include POST, HEAD,PUT,DELETE etc.
- Request and Response can be contents other than HTML page. In fact it can be almost anything that can pass over the wire.
- Connection can be persistent making HTTP faster.
- Transfer can be in chunks and byte range; allowing resuming downloads.
- Better support for proxies.
www – Is it really the next generation ready?
Its important to realize that neither www nor http were designed to carry out the task future had in store for them.
It appears that the whole idea was designed for a publishing industry. The idea was to publish hypertext documents which would typically be an electronic version of the text books. Just like the real books the chances of revisions are likely to be far and wide and when they come a new version can either replace the old one or perhaps can be published at a new location. The content would be mostly read only and you really don’t need much of programming in it. Another advantage of such publication was to reduce transfer of information to every body using emails that seem to be only alternative pre www.
This idea is clear from the design of both HTML and HTTP. Let us have a look at the design once again from a different perspective.
- HTML is no programming language. It doesn’t have the ability to decide or loop. And the new world changes more frequently than a text book. Users interaction often needed validations, conditions. Strangely all those things are completely unavailable in HTML.
- HTTP is connection less. In simple terms it means.
- It doesn’t have any built in ability to remember what it served last.
- There is no direct way to co-relate between different requests made by an user agent.
- It can’t distinguish weather the requests are made by the same user agent or different ones.
- There is no required sequence in which requests are supposed to me made.
However, world needed to change. Perhaps the very first dynamic requirement was for searching the web.
When the need of a search engine was felt; the first proposed solution was to modify the web server to add this functionality. Before late it was clear that such a solution is not only troublesome but also doesn’t fit in the big picture. As there could just be so many other requirement that will necessitate a similar change to server. A good solution would be to allow separate applications to run an assist server.
So there we stand. We have a technology that is supposed to bear the load of future generation of programming. We stood on the threshold of new generation of programming system – Web Programming. Desktop seem to be the way of past. And our best bet is www. And the two main components of www, namely html and http, are just not fit enough. Unfortunately its too late now to turn back or consider alternatives. To conclude:
Web programming is all about making www the platform of future generation of programming system. HTTP and HTML are not just ready to shoulder the responsibility and yet there is no real alternative other than to perhaps hack into it.
We will look at the story of how www transformed from a publishing protocol to almighty web application platform in our next instalment titled HTTP Hacked
Http Through Ages
Originally, I sat down to write down an article titled:
story of www – between http and html
When the story finally moved into the domain of http, it started to grow bigger. I enjoyed documenting about http. However, in doing so I was deviating badly from what I originally intended to do with the story. So I decided to split the story in parts. In this episode, we shall talk about the evolution of HTTP through ages.
The Hyper Text Transfer Protocol is the protocol for WWW. It was developed in collaboration of W3C (WWW Custodian) and Internet Engineering Task Force for distributed, collaborative, hypermedia information systems. It was essentially designed to retrieve linked resources. Understanding HTTP is perhaps the most crucial step in understanding how www works. So let’s have a detailed look at HTTP.
The HTTP Protocol
HTTP Protocol had a real humble beginning as a application layer protocol in TCP Suite for transmission of an HTML File. Period. The Protocol underwent several changes over a period of time –
HTTP/0.9
The protocol was a simplicity in itself. Client would connect to the server and issue a GET request for retrieving an HTML document. The overall process of a request-response cycle is as follows-
- Client connects to server on port 80
- Client sends a get request to retrieve a HTML document in the following format
GET /hello.html [cr][lf] - Server responds with the content of requested HTML file. The response will invariably be an HTML document.
- Once document is transferred to the Client; Server disconnects the connection. Client may disconnect the connection while transmission is in process, however server wont register it as an error case.
- If another document is required the connection process is re-initiated.
There were to verbs other than GET, no request or response header. It lacked all the fancy that future of HTTP was to see. While most servers are still capable for handling HTTP/0.9, there is no good reason why it should still be used.
HTTP/1.0
If www was to become the synonym of internet itself, a lot more was needed than just HTML output. HTTP must change first. Version 1.0 was a major change. The official RFC RFC1945 for next major HTTP/1.0 came out as late as 1996. However, browsers and servers had already moved up with newer ideas long before. HTTP/1.0 merely seem to be consolidation of ideas already in practice. The major Highlights of the version 1.0 included:
- New Verbs: Whereas HTTP/0.9 had just one request format, the new version included several verbs including POST, HEAD, CONNECT. These commands (verbs) in turn extended the ways in which client can connect to server and use its functionality. Not only we get read from server but also can add new content to server (PUT), delete the contents from server (DELETE) and run debug (TRACE) and other test while reducing traffic (HEAD). more of these verbs later.
- Request Header: Now the request is no more limited to a verb line. Request can also carry a lot of key value pair along with request. Each key-value pair will be send on a separate line (ending with a [cr][lf] ). The verb line and the key value pairs are together described as Request Header. A Request header terminates with an empty line. The empty line indicates end of request header and a signal to server to start its processing. While the quite few request headers are standardized the basic idea is supply extra information to the server so that server can more effectively serve the request. Examples of this approach could be:
- A user agent specification can help server send different version of document depending on who is requesting.
- We can specify the date of cached version of page and server may send new content only if there is change since the last cached version.
- HTTP Response: Once server is ready to with a response, good or bad, it sends it response which is is a three part response:
- Status Code: Indicates a numeric code that sums of server’s response.
- 200 Ok: The most favoured response. Meaning granted. And server is going to send a proper response to the request.
- 1xx : It is a series of response which are informal. That may mean processing. Or may need more input. It is far from completing the request
- 2xx : Indicates that the request is received, successfully understood and accepted as valid response. 200 Ok granted. 201 indicates request to create certain request succeeded (recall PUT)
- 3xx: Redirection. 301 indicates resources permanently moved to a new location or 302 may indicate temporary redirection, 304 indicates that content not modified since last request (date need to be sent in request).
- 4xx: Request Error. Request has a bad format, resource not found or an authorized request fall in this category.
- 5xx: Server Side Error: This may be caused due to some problem with the server such as configuration issue, server busy or server unavailable.
- Response Header: Response header typically is a meta information about the actual content that is supposed to follow. It typically includes date when content was changed, length of content, type of content (html, image, zip etc.). The response header ends with an empty line. The empty line acts as a separator between the Response Header and actual content.
- Actual Content: Server sends the actual content soon after Response Header. They are typically collected and saved.
- Status Code: Indicates a numeric code that sums of server’s response.
How HTTP/1.0 Interaction works
- Client connects to server on port 80 (typically)
- Client Sends the Request Header including the verb.
- Client sends an empty line to indicate end of request.
- Server checks the request.
- Server sends status code
- Server sends Response Header
- Server sends an empty line
- Server sends the actual content
- Server terminates the connection.
- Any related or connected resource need to be requested by starting a new connection.
HTTP/1.1
The latest version of HTTP is 1.1 was released under RFC 2068 in 1997 and later improved under RFC 2616 in 1999. The latest HTTP which is still almost a decade old added certain new provisions to already stable HTTP/1.0
- New Verbs: OPTIONS, TRACE, DELETE, PUT
- Host name Identification: HTTP/1.1 allowed the identification of Host. Prior of this version, server never new the host name. This provision paved way for sharing a single IP among various host because now server can now send different content depending on Host provided.
- Content Negotiation: HTTP/1.1 now allows server to maintain different versions of a single resource. For example, a document might be available in English as well as Hindi or may be available as a pdf document or word document format or a desktop version of document or pda version of it. Further HTTP allows the this negotiation to be either server driver or agent driven (client or browser)
- Server Driven Negotiation: Here the server decides or guesses the right version depending on the OS, Browser versions, country info based on the client IP and other information provided in the request Header
- Agent Driven Negotiation: Here the browser directly requests for the resource version or type it requires using suitable request header and server doesn’t decide.
- Persistent Connections: One of the major issues with the previous versions of HTTP was the fact that the connection was closed by the server as soon as response is sent. As it is often with the case to display a web page browser needs more than just the html page. For every other connected document (.css, .js, images, applet etc) it needs to create a new connection to the server. Such connection appear to be quite time taking. With HTTP/1.1 you can now request server to keep the connection alive. Earlier this was implemented by sending a special request header in this format:
connection:keep-alive [cr][lf]However, with the official documentation out, the persistent is the default behaviour for HTTP server and now the connection is closed either on time out or by sending an explicit header request to do so.
connection:close [cr][lf] - Chunk Transfer: This allows server to send chunks of data followed by additional header and more data. This is a good solution in case the content is dynamic and its size may not be known before hand.
- Byte Range: Byte range allows user to request documents in parts. This seem to be one of the biggest advantage in the sense it allows us to resume broken downloads and also allows us to have faster download by virtue of multi-threading. (Check out this link for an example of Byte Range download).
- Other Changes: There are several other changes done to make HTTP/1.1 a better protocol.
- Proxy and cache management: Gives better control on who access the document and how.
- Digest Authentication
- More Response Codes
- New Headers : It introduced new Headers such as Retry-After and Max-Forwards.
- New Media type: message/http, multipart/byteranges
So Where do we stand today
Let us sum up where we stand today and what is the significance of the latest innovations of HTTP protocol.
- Non-IP virtual Hosts
Virtual hosts can be used without needing additional IP addresses. - Content Negotiation means more content types and better selection
Using content negotiation means that resources can be stored in various formats, and the browser automatically gets the ‘best’ one (e.g. the correct language). If a best match cannot be determined, the browser or server can offer a list of choices to the user. - Faster Response
Persistent connections will mean that accessing pages with inline or embedded documents should be quicker. - Better handling of interrupted downloads
The ability to request byte ranges will let browsers continue interrupted downloads. - Better Behaviour and Performance from Caches
Caches will be able to use persistent connections to increase performance both when talking to browsers and servers. Use of conditionals and content negotiation will mean caches can identify responses quicker.