Comments on: High Availability Principle : Request Queueing

By: Joey

Joey — Thu, 04 Mar 2010 16:16:01 +0000

Guys

I’ve just come across this thread googling around on ideas to implement a ‘queing concept’ for web sites. What I want to understand is this:

Say that a web site copes with 1000 concurrent requests at any time. The web site is database driven.

And for a specific hour, you expect the concurrent requests to possibly hit 5 fold (ie 5000 concurrent requests) due to a timed activity say advertised on MSN.

Is there a way to allow the first 1000 users into the web site, and let them visit the 4-5 pages required to complete a shopping basket transaction, and hold the remaining 4000 queing them in.

I’m not an expert at all in this subject, but curious as to how to implement a solution to queue up requests without having to spend loads on additional web servers to handle more traffic.

From what I understand on web requests, each user request is separate to their previous request. So if a user is one of the first 1000 to get in, the next time they traverse to another page on the site, their request will be new and therefore queued with the remaining 4000, which is not what I want. I want this user to complete their journey on the web site pages.

thanks

Joey

By: Rag

Rag — Mon, 01 Mar 2010 18:51:22 +0000

The problem with today’s web servers is that requested are served synchronously. Typically, the request processing involves talking to a database, file system, or similar long lasting operation, which holds up the connection. A new slew of web servers explore handling requests asynchronously. Node.js (http://nodejs.org/), Tornado (http://www.tornadoweb.org/), etc. are some of the non-blocking servers that might help with concurrency issues. Queuing is required eventually, but async handling at least delays queuing requirement and helps supporting a larger concurrency.

By: Ashish Soni

Ashish Soni — Fri, 26 Feb 2010 02:10:55 +0000

Thanks Martin,
You make a great point that is worth highlighting : “You need headroom between your usual traffic and the degeneration point in order to recover from spikes.” So in the example above with the degeneration point being at 500 requests/second – one’s steady system state should be below that.
As an example : With the usual traffic being 50% below the degeneration point(in the example at 500) following would be the response time profile for a system handling a spike (while getting the usual traffic):
Spike equal to 1x usual traffic : No response time degradation
Spike equal to 2x usual traffic : Response times doubled.
Spike equal to 3x usual traffic : Response times tripled.
and so on.

By: Martin Waite

Martin Waite — Thu, 25 Feb 2010 11:45:48 +0000

Your example neglects to mention that new requests are coming even while entries are being removed from the queue. If new entries come in at 500 requests per second, from the point of the spike onwards, your response time will always be 3 seconds. Another 1500 spike next week will push the response time to 6 seconds, and so forth.

You need headroom between your usual traffic and the degeneration point in order to recover from spikes.

By: Willie Wheeler

Willie Wheeler — Wed, 24 Feb 2010 20:51:55 +0000

Yup.

Also, I was a bit unclear in one part of my explanation, so I want to clarify. We would fail fast on just that one panel of the page (leave the panel blank or else have a error message for that panel), but deliver the overall page itself within SLA.

By: Ashish Soni

Ashish Soni — Wed, 24 Feb 2010 18:52:25 +0000

Thanks for the comment Willie.
I too am a believer in the “Fail Fast” paradigm. Request Queueing along with some reasonably set queue timeouts can achieve linear degradation along with the ability to Fail Fast.

By: Willie Wheeler

Willie Wheeler — Wed, 24 Feb 2010 15:58:30 +0000

This is a great article. Thanks for posting it.

I agree with the points you make but I want to add one more. In some cases it is preferable to simply reject the request rather than respond slowly, even with linear degradation from the end user’s perspective. Suppose for example that you have a portal home page that makes server-side web service calls to grab the various pieces of the page. If some of those pieces are less important to the user, it’s probably better for the web service to simply reject the request than to have the web service latency cascade into whole-page latency.

Anyway great article.