25 Sep The Four Categories of Performance Testing Metrics
Over the years, a set of four performance metric categories have become very valuable to me.
Without the categories of 1) Workload, 2) Work, 3) Throughput, and 4) Experience, all the metrics that are available become a jumble. For example, somebody throws Average Response Time, Bytes per Second, Memory Pressure, Connection Pool Utilization at you, and you’re like “Yeah? So?”. But, if we have some meaningful categories to bring order to them, well then, now we’re talking.
AT THE GROCERY STORE
To get to know how these four categories fit together, let’s consider how they would make sense in a very familiar context, the grocery store. We’ll focus particularly on the checkout process. (This analogy hits close to home since my youngest daughter works as a cashier and bagger at our local market.)
Workload
How many people are on line at the registers?
How many items do they each have in their carts? (If it’s the express line, do they have more than 10 items? A challenging situation for cashiers managing other customers’ expectations)
Work
Cashiers scan the items or type the codes.
The bagger bags your items (Hopefully separating them properly. We don’t want our veggies and cleaning products in the same bag. Ick!)
Throughput
How many items are scanned per minute?
How many customers complete paying for their purchases per hour?
How many customers are extreme couponers that will hold up the line?
Experience
How long did each customer have to wait?
What kinds of errors/issues did they have?
With all this in mind, can you see the cause and effect relationship as we moved through the checkout process? I figured you would.
Workload → Work → Throughput → Experience
IN COMPUTER SYSTEMS
Now that we can see the categories and how they fit together, let’s apply this to any computer system or application. Below are just a few classic metrics per category for software applications.
Workload
Users present
Requests per Second
Work
CPU Utilization
Database Concurrent Connections
Throughput
Transactions per second
Bytes per second
Experience
Average Response Time
Errors per second
Once these categories make sense, you can use the following questions to help you write and read performance test reports:
What category does a metric fit in?
Do I have information from each of the four categories?
Does this data tell the whole story of application performance?
Are there reliable cause and effect associations shown by this data?
Did we articulate our performance requirements properly and completely?
So when you’re in line at the market during the Thanksgiving rush, or online on Cyber Monday, this blog will give you something to think about. You’re welcome. 😉
P.S. If you would like to take these four categories out for a test drive, check out Andreas Grabner’s posts here and here. Use the questions above with each of the metrics he discusses and consider how they work together to provide clearer insights into systems.
NOTE WELL: We can use the four categories for each subsystem as well as the entire system.
