Simulating a slow disk with nbd

[Michael Foord]

Hey all,

I've been diagnosing some replicaset issues that particularly show up on 
systems with slow disks, particularly our CI infrastructure. To simulate 
a slow disk I've been using nbd (Network block device [1]), with 
trickle. This provides a "remote mounted" disk (actually local served 
over the loopback) that is rate limited. I then run the tests inside an 
lxc container with its filesystem on the rate limited disk.

Getting this working was tricky, so I thought I'd share the information. 
In particular I couldn't get nbd-server to read a configuration file, so 
I force it to skip loading the config file and specify the parameters at 
the command line. This is deprecated, but works fine.

The sequence of commands to create the nbd drive and create / start the 
lxc container are as follows:

sudo apt-get install trickle nbd-server nbd-client

# create a 10GB file to act as the disk.
dd if=/dev/zero of=/path/to/some/file bs=1024 count=10000000
# create a file system on it
mke2fs /path/to/some/file

# start nbd-server under trickle
trickle -d 2000 -u 2000 nbd-server -C " " 1234 /path/to/some/file

# start the client
sudo nbd-client localhost 1234 /dev/nbd0

# create mount point
sudo mkdir /mnt/nbd
# mount the nbd device
sudo mount /dev/nbd0 /mnt/nbd

# create the lxc container
sudo lxc-create -t ubuntu -n nbd --dir=/mnt/nbd

# start the container
sudo lxc-start --name nbd


The "-d 2000 -u 2000" parameters to trickle tell it to rate limit access 
to nbd-server to 2000 kB/s. Adjust this appropriately for a 
faster/slower system.

Something worth noting is that when shutting down it matters what order 
you do things. If you shut down things in the wrong order you can end up 
with an lxc container that you can't restart or a device that you can't 
unmount / remount.

The right order to (un)do things is:

Shut down lxc container
Unmount device
Kill nbd-client / server

I hope this is helpful to anyone else who may want to simulate a system 
with slow i/o performance.

All the best,

Michael Foord

[1] http://nbd.sourceforge.net/