Robi-Wan's Collosumus.NET Blog : Windows performance

Calculating single disk performance for random I/O

robertvv — Tue, 20 Dec 2005 05:36:00 GMT

I have been doing a lot of performance analysis lately. Nine out of ten times the bad performance is caused by a not optimized storage solution. Why? When data doesn't fit into the memory of a server, it has to come from disk sooner or later. In my experience it's usually later (due to non optimized solutions), whereas sooner was to be expected. So, it's time to write down some disk performance basics.

I used a Seagate ST-373307FC 10.000 Rotations Per Minute (RPM) disk with a Fibre Channel (FC) interface for my calculations. You can find its specification sheet here. All information that is used for calculating RAW disk performance are obtained from this spec sheet.

                             Random I/O (R/W)
Average Seek (mSec)               4.7/5.3
Average latency (mSec)           2.99/2.99
Command and data transfer (mSec) 0.2/0.2
------------------------------------------- +
Total access time (mSec)          7.9/8.5
These numbers were obtained from the spec sheet

So the total number of I/O Operations Per Second, better known as IOP's can be calculated by dividing "1 second / total access time". Using the numbers from the table above: 1/0.0079 = ~ 126 IOP's for read operations and 1/0.0085 = ~ 117 IOP's for write operations.

Now we know how to calculate maximum random I/O performance for a single disk. Why is it so important to be able to calculate random I/O performance and not sequential I/O performance? The answer is quite simple. The more clients/processes that are to be served by a server, the bigger the chance its data access pattern will be random. Do you really think that when 4000 concurrent users connected to a file server generate sequential I/O and that the server will be able to predict the next piece of data to be requested by a client? No, it won't. And the fact it won't be able to predict it, limits the positive performance impact of caching algorithms in the process.

To make things even more complicated, it doesn't matter when a disk has to read one sector of 512 bytes or 32 sectors (16 KB block). Its platters rotate with 10.000 RPM per minute (in case of our Seagate disk), remember. So it reads 32 sectors almost as fast as one sector. Only when track boundary is reached, the performance drops a bit, due to the time it needs to seek to the adjacent track.

Knowing this, let's calculate the maximum random I/O performance for reading 8 KB blocks (16 sectors) and 16 KB blocks (32 sectors).

The formula is quite simple: Total numbers of IOP's x block size
Max random I/O (8 KB): 126 x 8 KB = 1008 KB per second!!!
Max random I/O (16 KB): 126 x 16 KB = 2016 KB per second!!!
Max random I/O (32 KB): 126 x 32 KB = 4032 KB per second
Max random I/O (64 KB): 126 x 64 KB = 8064 KB per second

You might want to increase block sizes, segment sizes, stripe size etc. in order to increase the performance... Please, consider the impact of RAID-levels on this and choose the right one. For more information on the impact of RAID-levels, read this blog.

As a baseline, use the following numbers for your calculations:
for a 10.000 RPM disk: 125 IOP's.
for a 15.000 RPM disk: 175 IOP's.

What about latency then? The killing factor for disk performance is the actual time needed to perform an I/O operation. Most disks run optimally with 2-3 outstanding I/O's in their queue. Using 7.9 mSec random latency for read operations, this gives us 7.9 mSec x 3 = 23.7 mSec maximum latency. For write operations this is 8.5 mSec x 3 = 25.5 mSec maximum latency (still using our Seagate disk as reference). As a rule of thumb, 21 mSec is considered the maximum, any average latency above that is considered a bottleneck. Any reoccurring spike far above that value is also considered a bottleneck.

Next time, I will talk about sequential disk performance.

To be or not to be... Anti Virus solutions on a server

robertvv — Mon, 19 Dec 2005 20:32:00 GMT

Not to be... (at least not always)

Why? Simple, try to image, you are a real time mail server, doing 5000 I/O's per second and you're interrupted everytime by Anti Virus (AV) software when you try to write, or even worse try to read to and from a storage system. The attempt made by the AV solution fails after a number of retries, since you have a non shared lock on the file you are accessing. In one of my previous posts, I talked a bit about drivers and DPC's and how Windows schedule system threads/drivers using the dispatcher (effectively doing cooperative multi-tasking on kernel level). Imagine the serialization this is causing, not to mention the fact virusses only infect files when performing a write operation, not when doing a read operation.

To all people out there in the world, please look closely at your standard AV configs deploying on servers. Make sure scan policies are set to scan write operations only, when this is not enough, schedule as many manual scan tasks as you see fit and please exclude any files that are exclusive in use by an application or operating system (database files, cluster configuration files immediately pop-up in my mind).

Don't say I didn't warn you...

P.S.: Yes I understand the necessity of having an AV solution, I don't say never to install one, just be careful with its configuration, because AV solutions can be as dangerous for systems stability as virusses can be for security.

The sense and nonsense of scalability by adding CPU's

robertvv — Mon, 19 Dec 2005 19:43:00 GMT

I'm often faced with the following customer situation: "I added additional CPU's to the system because the average CPU utilization exceeded 80%, but the application doesn't run any smoother. Why is that?". Well, let me try to explain.

To determine a CPU is busy in terms of the amount of work (threads) that is waiting to be executed, take a closer look at the following Performance Monitor counter: "System\Processor queue length". If this one exceeds 10 per CPU installed, you definitely need more CPU's. But... take a close look at the following two Task Manager screenshots. I used the Task Manager, because it's easier to visualize my point.

A two CPU system at approx 80%.

A quad CPU (the same) system at approx. 40% (with extra memory added).

What do you see there? You'll expect to see what percentage of the total 100% of CPU time the CPU is spending on executing threads... Wrong. You see the amount of time the Windows "idle thread" runs subtracted from 100 (it counts the amount the times the "Idle Thread" is called). Why is this important, because it shows you one important thing. Two CPU's at approx 80% is roughly the same as four CPU's at 40%. 80 x 2 = 40 x 4. Simple mathematics, right?. Right. So, this tells you one thing. The application doesn't run any smoother or "faster" then a system equipped with two CPU's. This tells you a lot about the application, not about the system. It tells you the application doesn't scale from two CPU's to four CPU's. So, one should have looked at the "System\Processor queue length" counter. If that one indicates more then 20 threads (more then 10 per processor), the application generates enough work to take advantage of extra CPU's. If the "System\Processor queue length" counter is far away from 20, don't bother to add additional CPU's, the chances are that the application won't take advantage of them anyway.

So, the moral of the story... Looking at CPU utilization and one doubles the amount of CPU's and the utilization droppes by 50%, run away as fast as you can... You just encountered one of many bad scaling applications and they blame Windows or the system for not scaling.

Windows Server Performance considerations - introduction

robertvv — Thu, 15 Dec 2005 05:31:00 GMT

The last months I am more and more engaged in accounts who often experience serious performance issues. By the time I am called on to the scene, they status of the account is usually at a critical level. After I have done my research, I often write a report on what is causing the performance issues and what to do in order to solve it and hot to prevent it from happening again. I do this, what I call, "one man's squad team" kind of job for several years now. The environments I am engaged in, are getting bigger and bigger. I am talking here about Storage Area Networks (SAN's), ususally with Tera Bytes (TB) connected to tens and even hunderds of systems with tens of thousands of users. Analyzing these kinds of environments is more than just specialize on e.g. Windows, or storage, or network. It's the area that connects these technologies where problems arise. This is 999 out of a 1000 times, caused by lack of experience or to be more precise, understanding of the involved technologies when designing and implementing these environments. To give an example, consider the following:

You drive a truck, loaded with 2000 crates of pineapples and you need to transport them from point A to point B over a distance of 400 km with an average speed of 80 kph. You need to make a decision. A) you drive the truck via the beach (on the sand) or B) via the highway. Which one would you choose? It's obvious, right? option "B". Now image the following environment. You have a fully loaded file server with 1000 concurrent users and you need to choose between a file server connected to a storage solution with A) 3x 300 GB SCSI disks or b) 12x 75 GB SCSI disks. Which one would you choose? The answer depends who makes the decision. The person who owns the budget or the specialist who happens to know a bit about performance. In the field option "A" is chosen most of the time, because of the costing factor involved. Why is that important you think? The answer is as simple as it is obvious. You don't expect a truck to do 80 kph driving of the beach, but you do expect the truck to make 80 kph on a highway. Why do you expect 3x 300 GB SCSI disks to perform as well as 12x 75 GB SCSI disks? The point I am trying to make, is that most performance problems are caused by lack of proper insight on the technologies used (truck on the beach vs. a truck on the highway) results in degraded performance. Using our truck as example, the truck on the beach doesn't make 80 kph, so people want to buy a bigger one, or one with a bigger engine capacity, but most of people forget to consider, a beach is simply not the right environment for the truck to operate in. A truck loaded with 2000 crates of pineapples has other requirements than a buggy loaded with just one crate of pineapples. A buggy happens to thrive well on a beach, the truck needs a proper infrastructure in order to complete its task. A decent highway, preferably without traffic jams. The same applies to servers who need to serve a lot of clients at the same time. A truck is made to transport many things at the same time instead of just one thing very fast. if you want that, buy a Ferrari instead.

A server is designed to serve many clients at the same time instead of completing one task very fast. Then why don't you give it the means to do so?

In my next post on this topic, I will discuss some Windows performance considerations on a technical level.

Why RAID-5 is a potential disk performance killer

robertvv — Sun, 03 Jul 2005 16:15:00 GMT

I often come across environments where databases perform poorly. When I do analyses of those environments (one needs to analyze an entire E2E chain instead of just one component before drawing any conclusions), I often run into poorly configured systems. These systems can host SQL, Oracle, DB2 or any other RDBMS. When I’m lucky people took the effort to split up database and transaction log files, that is, they partitioned the disk in several partitions and assigned the basic concept of splitting database transaction log files from database files. At least they figured out they had to do it… But, instead of creating separate disk arrays, one disk array with one logical driver was created which was partitioned under Windows. Or creating small Logical Units Numbers (LUN) on a Storage Area Network (SAN, effecient in terms of not waisting any disk space on a SAN), then people use volume management software to combine them into one Windows volume, which is partitioned again to separate database file from database transaction log files. Yes, I know, a normal person won't even consider doing that, let alone carry it out. But guess what, they do... I will talk about sizing, configuring a disk sub system and the importance of doing it right in a future blog.
In this blog I talk a bit about the myths surrounding the all holy RAID-5. Yes, also the first RAID level I heard and learned about during my Windows NT 3.5 Server training… ;-) Oh and I am not going to explain the differences in RAID levels, you can read all about that here.

One can use the following RAID Ratio of performance for comparing RAID strategies:
%Reads * (Physical Read Ops) + %Writes * (Physical Write Ops)

RAID-0, JBOD:
One physical disk write per logical write request is required
I/O Performance = % Read * (1) + % Write * (1)

RAID-10, RAID-1, RAID 0+1, RAID 1+0:
Two physical disk writes per logical write request are required
I/O Performance = % Read * (1) + % Write * (2)

RAID-5:
Four physical disk I/O operations per logical random write request are required (two reads and two writes)
I/O Performance = % Read * (1) + % Write * (4)

Since mirroring technologies need to copy the data to an alternate location, it makes sense that every application generated I/O needs to be mirrored to another disk. Therefore write operations on RAID-1x configurations require 2 physical I/O's per application generated I/O.

But then why are there 4 physical I/O’s needed for RAID-5 when performing an application write request? Well, most modern RAID controllers use a XOR algorithm for RAID-5 arrays with 3 disks and a read/modify write algorithm for RAID-5 arrays with 4 or more disks to write/change data. The read/modify write algorithm is more effecient for RAID-5 arrays with 4 or more disks then the XOR algorithm. It uses the following sequence to write/change blocks of data:

Read old data in stripe (can be any block)
Read old checksum in stripe
Calculate the new checksum from old data, new data and old checksum
Write new data in stripe (can be any block, depends on one to be modified)
Write new checksum in stripe

So, if for every requested write action, 4 physical I/O operations are required, why would one use RAID-5 everywhere? Because, very little people know how the various RAID levels really work and what their impact is on system performance.

What about Write Back (WB) cache? With cache the controller can do step 1 to 5 in memory and doesn't need to wait for for the actual disks to complete the I/O operations, so it can send an I/O completion to the requesting host almost immediately. However, this is only true when:

Cache is not satured.
Data is to be found in cache.

In short, when the demand for I/O is higher than the efficiency of the cache, the cache becomes a bottleneck. This is a reason why modern cached controllers use mechanisms like fly-by-cache. This actually disables the caching fuctions when a certain type of data stream is detected by the controller logic. This is especially true for streaming write operations, where the amount of data being generated is simply to large to be effeciently processed by the controller logic.

So, is RAID-5 bad? Yes and no. Yes it is, when the data stream to be served (mostly) consists of write requests, like database transaction logging. No, when the type of data stream to be served is (mostly) read. It doesn't matter whether it's sequential read or random read, because all RAID levels do one I/O for single block read operations.

Can RAID-5 serve high performance transaction logging databases? Yes, just keep in mind you need twice as many I/O's to be performed as with RAID-1x solutions, so double the amount of disks to equal the performance.

Did you know there is a community against using RAID-5 in production environments? It's called: Battle Against Any Raid Five.