Red Hat backported I/O accounting to RHEL5

Submitted by dag on Thu, 2009/05/07 - 19:24

One of the things I do on a weekly basis is follow the kernel development that Red Hat undertakes for their future RHEL5 kernels. This is very interesting because you can check the changelog for fixes, new hardware support, backported features (eg. kvm) and newly supported stuff (eg. fuse, xfs) that is coming in RHEL 5.4.

We discovered xfs was coming to RHEL, new ath5k fixes prove helpful on a friend's laptop, and I was waiting for I/O accounting, kvm and fuse to hit these releases too.

Yesterday I got an early birthday present as Red Hat backported I/O accounting in their 2.6.18-144.el5 kernel and it works really well.

Dstat can show I/O accounting information and the topio and topbio dstat modules show you the biggest I/O consumer both virtually as well as physically.

Virtually in the sense that it includes the I/O that is going to and coming from the pagecache. Physically is what comes and goes from the real disk.

In the past during presentations I always had to demonstrate this from a Fedora image in Virtualbox, but being able to show that on a running laptop is much more convenient because it represents the real workload of a common CentOS-5 desktop system. This is what you would see with dstat on a 2.6.18-144.el5 kernel:

[root@moria ~]# dstat -M topio -d -M topbio
----most-expensive---- -dsk/total- ----most-expensive----
     i/o process      | read  writ|  block i/o process   
rhythmbox    26k:  16k|   0     0 |                      
rhythmbox    30k:  16k|   0    20k|kjournald     0 :4096B
rhythmbox    25k:  16k|   0     0 |                      
rhythmbox    26k:  16k|   0     0 |                      
rhythmbox    30k:  16k| 128k    0 |rhythmbox   128k:   0 
rhythmbox    26k:  16k|   0  8192B|                      
rhythmbox    26k:  16k|   0     0 |                      
rhythmbox    30k:  16k|   0   104k|kjournald     0 :  28k
firefox     225B:  73k|   0   104k|firefox       0 :  80k
rhythmbox    30k:  16k| 128k 4096B|rhythmbox   128k:   0 
gconfd-2      0 :  92k|   0    92k|gconfd-2      0 :  96k
rhythmbox    26k:  16k|   0     0 |                      
rhythmbox    30k:  16k|   0   124k|kjournald     0 :  48k
rhythmbox    26k:  16k|   0     0 |                      
rhythmbox    26k:  16k| 128k    0 |rhythmbox   128k:   0 
rhythmbox    22k:  16k|   0     0 |                      

Showing the biggest normal I/O consumer (application level, virtual), the total disk statistics, and the biggest real I/O consumer (disk level, physical). rhythmbox is playing music and reading 26k/sec, but in reality it loads about 128k every 5 seconds. kjournald is invoked after something is being written. Firefox is writing to its cache.

And what gconfd-2 is repeatedly doing, I would really like to know because it is my biggest mystery (writing!) consumer :-/

PS: I also need to do something to my Drupal theme, since it is not properly showing consecutive spaces in <code> blocks... Fixed!


This is great news!

so when in CentOS?

I assume this will show up in CentOS at some point, following the upstream releases. Is it going to be a kernel update to 5.3, or are we going to have to wait for 5.4?

Red Hat is not doing this

Red Hat is not doing this work for an update RHEL 5.3 kernel, this is targeted for RHEL 5.4 only.

But you can take these "development" RHEL kernels directly to CentOS right now. CentOS will only ship them some weeks after RHEL 5.4 becomes Generally Available (GA).

Beware that if you find any problem with these kernels, you should report them in Bugzilla so that any flaws can still be fixed in the final RHEL 5.4 (and CentOS-5.4) kernel. (You know when to hurry when the RHEL 5.4 Beta is announced on this blog ;-))

development status

Can you tell us, please, where we can follow the status of rhel kernel (or entire system) development?

Thank you!

PS: This is great news!

Very nice! Interesting that

Very nice! Interesting that you prefer Virtualbox on EL5 (or CentOS-5).

I'd love to see a post on the details of your choice considering Xen is included and KVM is available. :)

The big feature I'm looking forward to is OpenGL support for scientific applications. Here's my notes on the subject:

"VMGL brings 3-D effects to VMs"
--awkward install and no support (Ph.D. dissertation)
--modified VNC
--openGL with Linux guests over Linux hosts
--based on rendering on clusters
2009-02-20 16:48:07 by frank
"Work is going on."
12-28-2008, 02:45 PM
"We are working in that direction, but please note, I am not able to provide you exact ETA "
Feb 23, 2009 3:36 PM
"Linux apps are valid data points as well. At this point, I really wanted to just get a sense what people are using."

I actually use both Qemu and

I actually use both Qemu and VirtualBox a lot.

Qemu is my tool for testing bootloader development as it is easy to drive from the commandline with options that fit your purpose. (Quickly add another disk, boot an ISO, or bootstrap a kernel/initrd)

VirtualBox I like because it is solid and easy to use even for advanced purposes. And it integrates well with RHEL/CentOS as it offers both 32bit and 64bit RPM packages (they could be advertised as CentOS compatible from the download section though). I mostly use VirtualBox for LiveCD's and other distributions.

Xen is too complex on my laptop and interferes with some of the things I do on the desktop. (hardware monitoring for one)

KVM is currently not up to speed on CentOS-5 (or RHEL5) because the KVM module currently available is old. This might change with RHEL 5.4 when Red Hat is finished with the recent KVM backport to RHEL5. By then I might replace Qemu by KVM.


rythmbox is probably converting the music stream to hex, and saving the last ten seconds in the registry, ehrm, gconf I mean, in case you want to re-hear it at some later date. Or maybe gconf is saving all of your window and mouse positions in realtime so they can go right back to where they were next time you open gnome.

url for RH kernel changelog?

Hi Dag,

where can we find the actual resource (URL/RSS/...?) you use to keep up with the rh kernel developements?

You have to download the

You have to download the package and look at the changelog of the RPM. I simply install the package and do:

rpm -q --changelog kernel-2.6.18-144.el5

If you download it you can do:

rpm -qp --changelog kernel-2.6.18-144.el5.x86_64.rpm

If you are only wanting to

If you are only wanting to monitor the changelog, then it is easier to look directly in the spec file - 650KB instead of 15MB.

The kernels are found in subdirectories under:

Kernel repos

How about using PRE tags instead of CODE

PRE tags preserve whitespace, line breaks. Simply copy-paste your monospaced text in a single PRE tag and be happy :-)

Yup, that's what I eventually

Yup, that's what I eventually did, however I don't understand why that is not the default behaviour of <code> in Drupal. And why <pre> is not part of the default input format list.

For most purposes that seems to be useful (if <code> is not doing what one should expect).

Same problem when your content is being used in RSS feeds, the <code> blocks are not represented correctly. I guess I have different expectations...

new version use: dstat

new version use:
dstat --top-io -d

topio and topbio not in RHEL 5.7

Dag, it appears the version of dstat I see in RHEL 5 repos as of 10 Aug 2011 is 0.6.6 from Nov 2009. The backport level is 3.el5_4.1. I have updated to the latest RHEL packages for 5.7 and do not see the topio or topbio listed when I do the list command (which strangely in this version is 'dstat -M list', implying the list is a module?). Do you know if RH simply moved back to another version before your post here (as you don't specify a version)? Do you have any influence on which version is in RHEL 5? I am sort of stuck by config management rules to certain packages, mainly those provided by RH repos, and I really want the topio and topbio modules - they would solve so many arguments.

hi, if you could kindly


if you could kindly advice me how i do track these development ? Any specific website ?

"One of the things I do on a weekly basis is follow the kernel development that Red Hat undertakes for their future RHEL5 kernels."

many thanks

Future RHEL kernels becoming extinct

For RHEL5, you can find newer kernels at:

However, for RHEL6, Red Hat stopped providing future kernels because of the competition from Oracle and Canonical. Red Hat invests a lot of money in backporting drivers and functionality to their Enterprise kernels.

The RHEL6 kernels were available previously at:

But being able to track those versions would make it easier to isolate individual patches :-/ I regret this decision, even though I understand Red Hat to protect their business. In the past this helped me bisect problems before they would be released and ship in production.