Comments on: The most common flaw in software performance testing

By: Dustin Headd

Dustin Headd — Wed, 13 Jan 2010 05:50:00 +0000

Thank you for the article. I enjoyed reading it. You have a very good site.

By: Tal Broda

Tal Broda — Wed, 30 Dec 2009 04:56:21 +0000

Yes, testing with production-level dataset, on the real production hardware, and really across the internet is the only way to know that your site will hold when you need it to. This is what we do here at SOASTA: massive-scale load testing, from the cloud!

By: Artur Ejsmont

Artur Ejsmont — Tue, 29 Dec 2009 11:13:13 +0000

Another problem is to reproduce real user activity. Hitting same pages over and over will acomplish nothing. Do it with or without sessions, going through geoip recognition or not, sending tracking cookies or not, being logged in or not, submitting error forms etc etc etc.

Having production alike DB size is important but another important factor is to be able to reproduce real world activity.

By: Kalpesh Patel

Kalpesh Patel — Sun, 20 Dec 2009 03:10:49 +0000

In my previous company we used to run our performance test on production snapshot data but one problem I found with OLTP systems is that the data becomes obsolete soon. There were lots of screens that were showing yesteday to future data and being an OLTP system most of the data was in the current week only. After running the original tests if the offshore team comes with all the fixes, by the time the test were ran again the data was obsolete. The point I want to make is just even if you get productions snapshot you need to tweak it to simulate real live scenario.

By: Ashish Soni

Ashish Soni — Fri, 18 Dec 2009 16:20:11 +0000

Hi Michal,
What you say about Recording real production traffic and to replaying it on a test system is a great way to get as close to a production profile as possible. In the past we were also thinking of strategies to record front end traffic and trying to replay the traffic at the front end layer of the test systems. That theoretically should simulate exact production traffic and get many more layers than just the database involved. We made some progress but not as much as I would have liked.

By: Ashish Soni

Ashish Soni — Fri, 18 Dec 2009 16:07:51 +0000

I wonder if you were able to do the same test on the old hardware. That is, to retrieve random documents. In your case you basically bypassed the cache layer. Would have been interesting to see the result delta between old hardware and new hardware in this case. Since it is indeed difficult to simulate traffic loads that exactly mimic production traffic the ‘trend’ of the test results is sometimes useful. For example in your test one could theoretically state that the “The new hardware is X% faster/scalable than the old hardware when random id’s are retrieved”. May not be exactly the production simulation but at least gives a relative benchmark.

By: Bobby

Bobby — Fri, 18 Dec 2009 15:07:17 +0000

I was asked to do a benchmark of a document server (get document by ID, basically). I was told to pick random ids and just blast it.

I begged them not to have me do this. I told them this would prove absolutely nothing and would be worthless data and a waste of time. The top request rate I would get would be maybe 10% of what our barely loaded live servers got, and therefore incredibly obviously incorrect. The real live requests are very very cache-friendly.

They said “Maybe. Do it anyway.”.

My report ended up saying that the new hardware was benchmarked to top out at 10% of what the old hardware was doing already. We rightly ignored it and put it in production anyway. Worked like a charm, 5% load if that.

One of the reasons I quit working there.

By: Owen Garrett

Owen Garrett — Thu, 17 Dec 2009 11:16:03 +0000

Another reason for the disconnect between in-the-lab stress tests and real-world deployments is the nature of the traffic that the front-end of your application has to manage.

In the lab, it’s common to run tests over a fast, local network, often with heavy use of HTTP keepalives and simple client settings to help tune the application to get the best possible performance. You might get 2,000 transactions-per-second, 100% CPU and be happy that you’ve configured their app to the best of its capability.

You’ll get a shock when you put the application live and the latency is terrible, TPS never exceeds 10% of what you got in the lab and the CPU never exceeds 20%. What gives – why is the performance so poor?

On real-world networks, connections are slow, keepalives are held idle for many seconds and jitter and packet loss further increases the duration of each connection. As connections take longer to complete, the number of concurrent connections grows, and with thread or processed based applications, concurrency is a real killer.

One solution is to deploy a front-end proxy to convert these many slow connections with inefficient use of keepalives and HTTP options into a small number of fast, local connections, putting your application back into the environment it works best. That’s one of the things that products called ‘ADCs’ (‘Application Delivery Controllers’) do.

By: Michal Kuratczyk

Michal Kuratczyk — Thu, 17 Dec 2009 10:12:57 +0000

> Snapshots of production data can be forbidden for data protection reasons

If you’re on Oracle DB, you can use Data Masking option for data protection. It takes the production data and scrambles so that they are still representative but not real anymore.

> the main reason we see differences between performance metrics on real
> production versus test runs is due to the difference in usage scenarios.

If you’re on Oracle DB, that’s what Real Application Testing does. You can record real database “traffic” on production and replay it on the test system. A great way to test whenever you change something (DB upgrade/patching, hardware replacement, schema change, etc).

Hope you will find that knowledge useful,

DISCLAIMER: yes, I work for Oracle

By: Ashish Soni

Ashish Soni — Thu, 17 Dec 2009 09:11:07 +0000

Thanks for the comment Mark.
I understand that sometimes it becomes impractical to copy complete production data sets to the test systems. Its good to see that in your case that ‘production-like’ data sets are generated. This is certainly a reasonable next best approach if indeed due to internal policies and logistics the ability to get snapshots of production data is not present.