qcow2 format state in Lucid
Nikolai K. Bochev
n.bochev at grandstarco.com
Tue Mar 23 05:37:16 UTC 2010
----- "Dustin Kirkland" <kirkland at canonical.com> wrote:
> On Mon, Mar 22, 2010 at 12:39 PM, Nikolai K. Bochev
> I'm using KVM + qcow2 + virtio with Lucid host and Lucid guest quite
>
> extensively.
>
I've encountered this when i updated from jaunty to karmic on hosts. The guests are different, but it's mostly either hardy or karmic.
Same slowdown no matter what guest.
>
>
> Can you tell me more about your setup, as I'd like to see if I can
>
> reproduce the problem you're seeing?
One of my hosts is like this ( sorry if it's long ) :
system X8DT3
/0 bus X8DT3
/0/0 memory 64KiB BIOS
/0/4 processor Intel(R) Xeon(R) CPU E5520 @ 2.27GHz
/0/4/5 memory 128KiB L1 cache
/0/4/6 memory 1MiB L2 cache
/0/4/7 memory 8MiB L3 cache
/0/8 processor Intel(R) Xeon(R) CPU E5520 @ 2.27GHz
/0/8/9 memory 128KiB L1 cache
/0/8/a memory 1MiB L2 cache
/0/8/b memory 8MiB L3 cache
/0/39 memory System Memory
/0/39/0 memory 2GiB DIMM 1333 MHz (0.8 ns)
/0/39/1 memory DIMM 1333 MHz (0.8 ns) [empty]
/0/39/2 memory 2GiB DIMM 1333 MHz (0.8 ns)
/0/39/3 memory DIMM 1333 MHz (0.8 ns) [empty]
/0/39/4 memory 2GiB DIMM 1333 MHz (0.8 ns)
/0/39/5 memory DIMM 1333 MHz (0.8 ns) [empty]
/0/47 memory System Memory
/0/47/0 memory 2GiB DIMM 1333 MHz (0.8 ns)
/0/47/1 memory DIMM 1333 MHz (0.8 ns) [empty]
/0/47/2 memory 2GiB DIMM 1333 MHz (0.8 ns)
/0/47/3 memory DIMM 1333 MHz (0.8 ns) [empty]
/0/47/4 memory 2GiB DIMM 1333 MHz (0.8 ns)
/0/47/5 memory DIMM 1333 MHz (0.8 ns) [empty]
/0/55 memory Flash Memory
/0/55/0 memory 4MiB FLASH Non-volatile 33 MHz (30.3 ns)
/0/1 memory
/0/2 memory
/0/100 bridge 5520 I/O Hub to ESI Port
/0/100/1 bridge 5520/5500/X58 I/O Hub PCI Express Root Port 1
/0/100/1/0 eth0 network 82576 Gigabit Network Connection
/0/100/1/0.1 eth1 network 82576 Gigabit Network Connection
/0/100/3 bridge 5520/5500/X58 I/O Hub PCI Express Root Port 3
/0/100/3/0 scsi0 storage MegaRAID SAS 1078
/0/100/3/0/2.0.0 /dev/sda disk 997GB MegaRAID SAS RMB
/0/100/3/0/2.0.0/1 /dev/sda1 volume 1906MiB EXT4 volume
/0/100/3/0/2.0.0/2 /dev/sda2 volume 927GiB Extended partition
/0/100/3/0/2.0.0/2/5 /dev/sda5 volume 927GiB Linux LVM Physical Volume partition
/0/100/3/0/2.1.0 /dev/sdb disk 498GB MegaRAID SAS RMB
/0/100/3/0/2.1.0/1 /dev/sdb1 volume 464GiB Linux LVM Physical Volume partition
/0/100/5 bridge 5520/X58 I/O Hub PCI Express Root Port 5
/0/100/7 bridge 5520/5500/X58 I/O Hub PCI Express Root Port 7
/0/100/8 bridge 5520/5500/X58 I/O Hub PCI Express Root Port 8
/0/100/9 bridge 5520/5500/X58 I/O Hub PCI Express Root Port 9
/0/100/14 generic 5520/5500/X58 I/O Hub System Management Registers
/0/100/14.1 generic 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers
/0/100/14.2 generic 5520/5500/X58 I/O Hub Control Status and RAS Registers
/0/100/14.3 generic 5520/5500/X58 I/O Hub Throttle Registers
/0/100/16 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/16.1 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/16.2 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/16.3 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/16.4 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/16.5 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/16.6 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/16.7 generic 5520/5500/X58 Chipset QuickData Technology Device
/0/100/1a bus 82801JI (ICH10 Family) USB UHCI Controller #4
/0/100/1a.1 bus 82801JI (ICH10 Family) USB UHCI Controller #5
/0/100/1a.2 bus 82801JI (ICH10 Family) USB UHCI Controller #6
/0/100/1a.7 bus 82801JI (ICH10 Family) USB2 EHCI Controller #2
/0/100/1d bus 82801JI (ICH10 Family) USB UHCI Controller #1
/0/100/1d.1 bus 82801JI (ICH10 Family) USB UHCI Controller #2
/0/100/1d.2 bus 82801JI (ICH10 Family) USB UHCI Controller #3
/0/100/1d.7 bus 82801JI (ICH10 Family) USB2 EHCI Controller #1
/0/100/1e bridge 82801 PCI Bridge
/0/100/1e/3 display MGA G200eW WPCM450
/0/100/1f bridge 82801JIR (ICH10R) LPC Interface Controller
/0/100/1f.3 bus 82801JI (ICH10 Family) SMBus Controller
/0/101 bridge Intel Corporation
/0/102 bridge Intel Corporation
/0/103 bridge Intel Corporation
/0/104 bridge Intel Corporation
/0/105 bridge 5520/5500/X58 Physical Layer Port 0
/0/106 bridge 5520/5500 Physical Layer Port 1
/0/107 bridge Intel Corporation
/0/108 bridge Intel Corporation
/0/109 bridge Intel Corporation
/0/10a bridge Intel Corporation
/0/10b bridge Intel Corporation
/1 vnet0 network Ethernet interface
/2 vnet1 network Ethernet interface
/3 vnet2 network Ethernet interface
/4 vnet3 network Ethernet interface
/5 vnet4 network Ethernet interface
/6 vnet5 network Ethernet interface
/7 vnet6 network Ethernet interface
/8 vnet7 network Ethernet interface
/9 vnet8 network Ethernet interface
/a vnet9 network Ethernet interface
/b vnet10 network Ethernet interface
>
>
>
> Specifically, what is your kvm command line when you're seeing the
>
> problem, and what are you using to measure your performance or disk
>
> throughput?
I have around 7 or 8 hosts, most of them are using mdadm though, but that just reflects the overall speeds difference, i've encountered slowdowns no matter if i use hw raid or mdadm.
We're running mostly raid 5's and 10's but there's the occasional 6 and slowdowns are seen on all of them.
On the above configuration i have the following running :
2053 ? Sl 181:54 /usr/bin/kvm -S -M pc-0.11 -m 512 -smp 2 -name dev -uuid f0366094-81f3-22d9-ec9e-e93d30855a74 -monitor unix:/var/run/libvirt/qemu/dev.monitor,server,nowait -boot c -drive file=/srv/storage/virt/ubuntu-9.10-server-amd64.iso,if=ide,media=cdrom,index=2,format= -drive file=/srv/storage/virt/dev-sda.img,if=virtio,index=0,boot=on,format= -net nic,macaddr=52:54:00:67:39:db,vlan=0,model=e1000,name=e1000.0 -net tap,fd=15,vlan=0,name=tap.0 -net nic,macaddr=52:54:00:21:6b:1d,vlan=1,model=e1000,name=e1000.1 -net tap,fd=16,vlan=1,name=tap.1 -serial pty -parallel none -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus
2274 ? Sl 960:12 /usr/bin/kvm -S -M pc-0.11 -m 2048 -smp 2 -name purplewing -uuid 2020a1fb-bf5e-6d7b-ef14-b6cb5b0d7c2e -monitor unix:/var/run/libvirt/qemu/purplewing.monitor,server,nowait -boot c -drive file=,if=ide,media=cdrom,index=2,format= -drive file=/srv/storage/virt/purplewing-sda.img,if=scsi,index=0,boot=on,format= -net nic,macaddr=52:54:00:0d:2c:c6,vlan=0,model=e1000,name=e1000.0 -net tap,fd=16,vlan=0,name=tap.0 -serial pty -parallel none -usb -vnc 127.0.0.1:1 -k en-us -vga cirrus
2458 ? Sl 148:40 /usr/bin/kvm -S -M pc-0.11 -m 2048 -smp 2 -name bluewing -uuid 20709a00-bdcb-d561-de00-b396c531643d -monitor unix:/var/run/libvirt/qemu/bluewing.monitor,server,nowait -boot c -drive file=/srv/storage/virt/bluewing-sda.qcow2,if=ide,index=0,boot=on,format=raw -drive file=,if=ide,media=cdrom,index=2,format= -net nic,macaddr=52:54:00:39:6e:31,vlan=0,model=e1000,name=e1000.0 -net tap,fd=17,vlan=0,name=tap.0 -net nic,macaddr=52:54:00:0f:37:07,vlan=1,model=e1000,name=e1000.1 -net tap,fd=18,vlan=1,name=tap.1 -serial pty -parallel none -usb -vnc 127.0.0.1:2 -k en-us -vga cirrus
2469 ? Sl 595:54 /usr/bin/kvm -S -M pc-0.11 -m 1024 -smp 2 -name redwing -uuid ae675ba2-dae6-46e1-cb94-56302696f596 -monitor unix:/var/run/libvirt/qemu/redwing.monitor,server,nowait -boot c -drive file=/srv/storage/virt/redwing-sda.img,if=scsi,index=0,boot=on,format= -net nic,macaddr=52:54:00:0a:ea:74,vlan=0,model=e1000,name=e1000.0 -net tap,fd=18,vlan=0,name=tap.0 -net nic,macaddr=52:54:00:09:9d:f2,vlan=1,name=nic.0 -net tap,fd=19,vlan=1,name=tap.1 -serial pty -parallel none -usb -vnc 127.0.0.1:3 -k en-us -vga cirrus
2480 ? Rl 34:38 /usr/bin/kvm -S -M pc-0.11 -m 1024 -smp 1 -name whitewing -uuid 62b51468-b022-7a3c-642a-88ca4218ee74 -monitor unix:/var/run/libvirt/qemu/whitewing.monitor,server,nowait -boot c -drive file=/srv/storage/virt/whitewing-sda.qcow2,if=ide,index=0,boot=on,format=raw -drive file=/srv/storage/virt/ubuntu-9.04-server-amd64.iso,if=ide,media=cdrom,index=2,format= -net nic,macaddr=52:54:00:6e:29:3e,vlan=0,model=e1000,name=e1000.0 -net tap,fd=19,vlan=0,name=tap.0 -net nic,macaddr=52:54:00:61:5a:11,vlan=1,model=e1000,name=e1000.1 -net tap,fd=20,vlan=1,name=tap.1 -serial pty -parallel none -usb -vnc 127.0.0.1:4 -k en-us -vga cirrus
2486 ? Rl 1893:04 /usr/bin/kvm -S -M pc-0.11 -m 2048 -smp 2 -name yellowwing -uuid ca1df004-80dd-646d-6fd1-36e1c112da2a -monitor unix:/var/run/libvirt/qemu/yellowwing.monitor,server,nowait -boot c -drive file=/srv/storage/virt/yellowwing-sda.qcow2,if=ide,index=0,boot=on,format=raw -drive file=/srv/storage/virt/ubuntu-8.04.3-server-amd64.iso,if=ide,media=cdrom,index=2,format= -net nic,macaddr=52:54:00:7e:d8:5d,vlan=0,model=e1000,name=e1000.0 -net tap,fd=20,vlan=0,name=tap.0 -net nic,macaddr=52:54:00:62:17:5e,vlan=1,model=e1000,name=e1000.1 -net tap,fd=21,vlan=1,name=tap.1 -serial pty -parallel none -usb -vnc 127.0.0.1:5 -k en-us -vga cirrus
Do not be fooled by the filenames of the disks tho as i have experimented and converted the between formats quite a few.
I've changed hdd drivers from virtio to scsi or even ata a few times, and networking drivers from virtio to e1000 or rtl8139 and there's mostly no difference.
I encountered this when i was preparing a host with just 2 guests on it - one web server and one file server ( samba ). Everything went fine until i started to rsync files from the network to the samba guest. I will try and get you exact measures of speed slowdowns later today ( as it is already in production and i can't experiment with it right now ), but we're talking something among the lines of :
host disk speeds - around 60MB/s ( reads ) and 30MB/s writes ( ext4 fs ). We got a gigabit network at the office.
guest disk speeds - around 2-7MB/s both reads and writes.
I'm using dstat and iotop to see disk speeds. rsync is :
rsync -avz source target
Now the speeds above would be acceptable if the guest was acting normally, but unfortunately once you start a longer copy ( say around a 2GB file ) TO the guest, it hits the guest CPU pretty hard ( CPU at 100%, loads around 5 or so ) and the machine becomes very slow and unresponsive. The host is acting normally.
All of this happens only with qcow2 format of the disks. If switch to raw , the disk speeds increase by 1/4 or so, and the cpu usage goes back to normal.
At first we thought it's the networking, so i ran iperf between two guests, and it did not have the same effect. Then we tried bonnie++ on the guests and we saw the slowdowns.
I will try to provide exact speeds and measures later today.
My only concern is if this behavior persist in Lucid. At the moment i have no free hardware to test Lucid and i am considering converting all qcow2 formats to raw, but i wanted to keep qcow2 for the snapshot ability.
>
>
>
> :-Dustin
--
Nikolai K. Bochev
System Administrator
Website : GrandstarCO | http://www.grandstarco.com
More information about the ubuntu-server
mailing list