Understanding I/O - What about IOPS, Throughput, Latency and I/O Size

This blog article is about one of my most favourite topics with just two letters: I/O. Why? Because really everything around virtualization and storage is about how I/O works and there is not much fun in life without Input/Output. And yes there are many blog articles about I/O but not so much from the I/O size perspective. So hopefully this will help you with some confusion. Let’s start very simple and define this four keywords:

IOPS

IOPS is a number of I/O per second. This can be a number of IOPS delivered by particular Storage System or consumed by a single Virtual Machine or physical computer with local storage, IOPS of a single disk in Windows Performance Monitor etc. Every storage vendor will always throw a maximum IOPS number at you but it does not mean too much because the pure number of IOPS is not as important as the behaviour of other factors like I/O size, Read or Write, bustiness, transfer delay etc.!

I/O Size (Block Size)

I/O size or block size is the size of an Input/Output operation. You can also think of them as the “payload”. If you want to compare it to a highway it is the size of the cars and trucks on this highway. Some of them are really small like a “Smart Car”, while others are huge trailer trucks. The I/O size is measured in kB. The following figure shows you the difference from a relatively small I/O size of 4 kB to a big I/O size of 512 kB. Just think about the fact that a 512 kB block has 128 times the payload of a 4 kB I/O.

Figure 1: I/O Size

Most of today’s storage or virtualization systems will only show you the average I/O size. Let’s take an example. Your application is doing 950 IOPS x 4 kB and 50 IOPS x 512 kB. You agree with me that the 95% of your IOPS are 4 kB and just 5 % are 512 KB correct? But your average I/O size is:

Throughput: 950 IOPS x 4 kB + 50 x 512 kB = 3,800 kB + 25,600 kB = 29,400 kB

Average I/O size: 29,400 / 1,000 = 29.4 kB

Is that fair? Yes from an average I/O size perspective it is absolutely but how do you know now that 95% were 4 kB I/O’s and just 5% 512 kB I/O’s? You usually won’t.

Throughput

Throughput is the size measured in MB/s or GB/s: Throughput = IOPS x I/O Size

Let’s take an example of 1000 IOPS with the I/O size of 4 kB and 512 kB respectively. You would agree that though each workload delivers 1000 IOPS, the behaviour depends extremely on the I/O size.

1,000 IOPS x 4 kB = 4,000 kB

1,000 IOPS x 512 kB = 512,000 kB

A very simple rule is that the higher the I/O size, the higher the Throughput will be.

Latency

Latency is the time an I/O takes from request to completion. Latency is measured in ms (milliseconds) or msec (microseconds). As you can imagine, the above example the latency will be different for a 4 kB I/O vs. a 512 kB I/O. In my opinion most vendors talk way too much about IOPS, Throughput and Latency for a very specific use case. And most of the time just about 4 kB reads which are relatively easy to archive or a mix of 70/30 Read/Write Ratio. The problem is that a workload of a system is changing permanently and every customer environment is different.

Real World Example

In the real world example I use PernixData’s product Architect to visualize. This article is clearly not about PernixData Architect but the fact that it is important to understand what your systems are doing and Architect is very good to archive this. The following figure shows you how the Read/Write workload changes over time.

Figure 2: PernixData Architect Workload Read/Write

The next figure shows you the same workload but the change for the I/O size over time. You clearly see different I/O sizes at different time and this workload is even very similar over time. There are many examples from my experience where the I/O size over time changes completely.

Figure 3: PernixData Architect Workload BlockSize

Coming back to my earlier example of 95% x 4 kB read and 5% x 512 kB read I used a simple Iometer workload pattern. As you see in the following figure we have 76,964 x 4 kB (307,856 kB) IOPS and 4,056 x 512 kB (2,076,672 kB). So calculating the average I/O size would be 2,384,528 kB / 81,020 = 29.43 kB.

Figure 4: PernixData Architect IOPS

The next figure shows the same from the latency perspective and shows that a 512 kB I/O has a much higher latency compared to a 4 kB block. Just as a comparison:

76,964 x 4 kB IOPS = 0.332 ms

4,056 x 512 kB = 2.037 ms

Figure 5: PernixData Architect Latency

Going back to my Iometer screen I see the workload and no  I/O size just the average I/O size I could calculate dividing Throughput / IOPS and the average latency of this workload which is in the below figure 0.7894 ms. This shows that it is very important to understand the I/O size in combination with your I/O (read & write).

Figure 6: Iometer

Conclusion

The conclusion of this article is that it is very important to understand your I/O. This is not easily done with many of today's tools. The problem with synthetic benchmark tools like Iometer is that testing a simple 100% 4 kB workload is not what your systems will do. It will make all storage systems shine. But what happens when the application owner calls and says that the application is slow at certain times a day? If you don’t have the data to know what happened it is mostly guessing. Because it could be that the 5% 512 kB I/O’s are causing the storage a headache and high latencies for other applications. Fixing issues based on assumptions is never easy.

As always if you have any questions, recommendations, concerns please don’t be hesitate to contact me or comment.