Red Hat backported I/O accounting to RHEL5
One of the things I do on a weekly basis is follow the kernel development that Red Hat undertakes for their future RHEL5 kernels. This is very interesting because you can check the changelog for fixes, new hardware support, backported features (eg. kvm) and newly supported stuff (eg. fuse, xfs) that is coming in RHEL 5.4.
We discovered xfs was coming to RHEL, new ath5k fixes prove helpful on a friend's laptop, and I was waiting for I/O accounting, kvm and fuse to hit these releases too.
Yesterday I got an early birthday present as Red Hat backported I/O accounting in their 2.6.18-144.el5 kernel and it works really well.
Dstat can show I/O accounting information and the topio and topbio dstat modules show you the biggest I/O consumer both virtually as well as physically.
Virtually in the sense that it includes the I/O that is going to and coming from the pagecache. Physically is what comes and goes from the real disk.
In the past during presentations I always had to demonstrate this from a Fedora image in Virtualbox, but being able to show that on a running laptop is much more convenient because it represents the real workload of a common CentOS-5 desktop system. This is what you would see with dstat on a 2.6.18-144.el5 kernel:
[root@moria ~]# dstat -M topio -d -M topbio ----most-expensive---- -dsk/total- ----most-expensive---- i/o process | read writ| block i/o process rhythmbox 26k: 16k| 0 0 | rhythmbox 30k: 16k| 0 20k|kjournald 0 :4096B rhythmbox 25k: 16k| 0 0 | rhythmbox 26k: 16k| 0 0 | rhythmbox 30k: 16k| 128k 0 |rhythmbox 128k: 0 rhythmbox 26k: 16k| 0 8192B| rhythmbox 26k: 16k| 0 0 | rhythmbox 30k: 16k| 0 104k|kjournald 0 : 28k firefox 225B: 73k| 0 104k|firefox 0 : 80k rhythmbox 30k: 16k| 128k 4096B|rhythmbox 128k: 0 gconfd-2 0 : 92k| 0 92k|gconfd-2 0 : 96k rhythmbox 26k: 16k| 0 0 | rhythmbox 30k: 16k| 0 124k|kjournald 0 : 48k rhythmbox 26k: 16k| 0 0 | rhythmbox 26k: 16k| 128k 0 |rhythmbox 128k: 0 rhythmbox 22k: 16k| 0 0 |
Showing the biggest normal I/O consumer (application level, virtual), the total disk statistics, and the biggest real I/O consumer (disk level, physical). rhythmbox is playing music and reading 26k/sec, but in reality it loads about 128k every 5 seconds. kjournald is invoked after something is being written. Firefox is writing to its cache.
And what gconfd-2 is repeatedly doing, I would really like to know because it is my biggest mystery (writing!) consumer :-/
PS: I also need to do something to my Drupal theme, since it is not properly showing consecutive spaces in <code> blocks...