[Bug 997695] Re: severe performance issue using mdadm raid5 on top of LUKS
Sven Reinecke
sreinecke at reinsn.de
Wed Sep 26 21:11:09 UTC 2012
I am having a similar issue using Ubuntu 12.04 x86_64.
mdadm RAID5, 3x 2TB Hard drives.
dm-crypt on /dev/md0 (aes-xts-plain -s 256, but cipher does NOT matter)
System performance is very bad on 12.04:
root at chimera:~# time sh -c "dd if=/dev/zero of=/dev/mapper/md0_crypt bs=8k count=2000000 && sync"
2000000+0 Datensätze ein
2000000+0 Datensätze aus
16384000000 Bytes (16 GB) kopiert, 271,431 s, 60,4 MB/s
real 4m31.558s
user 0m0.352s
sys 0m11.829s
On 10.04 bootet from flash drive:
root at ubuntu:~# time sh -c "dd if=/dev/zero of=/dev/mapper/md0_crypt bs=8k count=2000000 && sync"
2000000+0 Datensätze ein
2000000+0 Datensätze aus
16384000000 Bytes (16 GB) kopiert, 178.911 s, 91.6 MB/s
real 2m59.269s
user 0m0.290s
sys 0m11.680s
System load on 12.04 goes up to 10, on 10.04 it is only ~1.
This is "top" while dd'ing on 12.04:
top - 20:53:42 up 14:06, 3 users, load average: 4.63, 2.62, 1.67
Tasks: 119 total, 2 running, 117 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 20.6%sy, 0.0%ni, 59.9%id, 19.3%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 8087012k total, 7873560k used, 213452k free, 7297544k buffers
Swap: 8298492k total, 0k used, 8298492k free, 99324k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5137 root 20 0 0 0 0 D 21 0.0 0:21.24 kworker/3:1
2126 root 20 0 0 0 0 R 21 0.0 0:34.29 kworker/1:2
5140 root 20 0 0 0 0 D 18 0.0 0:03.54 kworker/2:3
2912 root 20 0 0 0 0 D 17 0.0 0:55.45 kworker/0:1
2436 root 20 0 0 0 0 S 12 0.0 69:22.81 md0_raid5
5134 root 20 0 12912 664 544 D 4 0.0 0:09.65 dd
35 root 20 0 0 0 0 S 1 0.0 0:02.37 kswapd0
718 root 20 0 0 0 0 S 0 0.0 0:05.05 kworker/3:2
4585 root 20 0 0 0 0 S 0 0.0 0:48.45 kworker/2:2
5135 root 20 0 0 0 0 S 0 0.0 0:04.76 flush-252:0
5138 root 20 0 0 0 0 S 0 0.0 0:00.50 kworker/1:1
5339 root 20 0 17332 1292 960 R 0 0.0 0:00.02 top
1 root 20 0 24324 2300 1372 S 0 0.0 0:00.80 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:00.21 ksoftirqd/0
6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
7 root RT 0 0 0 0 S 0 0.0 0:00.10 watchdog/0
Please just tell me what further information you need.
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to cryptsetup in Ubuntu.
https://bugs.launchpad.net/bugs/997695
Title:
severe performance issue using mdadm raid5 on top of LUKS
Status in “cryptsetup” package in Ubuntu:
Invalid
Status in “linux” package in Ubuntu:
Confirmed
Bug description:
I've upgraded to 12.04 a few days ago and since this, the performance
of my crypto-drive is limited by CPU.
I'm running an MDADM RAID5 on top of 7 independendtly encrypted drives
(Seagate Barracuda XT 7200.12 2TB each encrypted using crypsetup /
LUKS).
The problem seems to be, that in 12.04 all encryption is done using
only a single CPU core. In 10.04 each crypt-device used its own thread
(kcrypt), that was the reason to encrypt each device independently and
build a raid on top of the crypto devices.
I'll attach a screenshot of munins iostat and CPU graph to illustrate the difference before and after the upgrade
---
AlsaDevices:
total 0
crw-rw---T 1 root audio 116, 1 Mai 1 15:52 seq
crw-rw---T 1 root audio 116, 33 Mai 1 15:52 timer
AplayDevices: aplay: device_list:252: no soundcards found...
ApportVersion: 2.0.1-0ubuntu7
Architecture: amd64
ArecordDevices: arecord: device_list:252: no soundcards found...
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
DistroRelease: Ubuntu 12.04
HibernationDevice: RESUME=
IwConfig:
lo no wireless extensions.
virbr0 no wireless extensions.
eth0 no wireless extensions.
Lsusb:
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 002: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
MachineType: To Be Filled By O.E.M. To Be Filled By O.E.M.
Package: linux (not installed)
ProcEnviron:
LANGUAGE=de_DE:de:en_US:en
TERM=xterm
LANG=de_DE.UTF-8
SHELL=/bin/bash
ProcFB: 0 nouveaufb
ProcKernelCmdLine: root=/dev/mapper/md1_crypt ro quiet splash
ProcVersionSignature: Ubuntu 3.2.0-24.37-generic 3.2.14
RelatedPackageVersions:
linux-restricted-modules-3.2.0-24-generic N/A
linux-backports-modules-3.2.0-24-generic N/A
linux-firmware 1.79
RfKill: Error: [Errno 2] No such file or directory
StagingDrivers: cxd2099
Tags: precise staging
Uname: Linux 3.2.0-24-generic x86_64
UpgradeStatus: Upgraded to precise on 2012-04-30 (10 days ago)
UserGroups: mythtv
dmi.bios.date: 04/30/2010
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: P2.30
dmi.board.name: K10N78FullHD-hSLI..
dmi.board.vendor: ASRock
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrP2.30:bd04/30/2010:svnToBeFilledByO.E.M.:pnToBeFilledByO.E.M.:pvrToBeFilledByO.E.M.:rvnASRock:rnK10N78FullHD-hSLI..:rvr:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: To Be Filled By O.E.M.
dmi.product.version: To Be Filled By O.E.M.
dmi.sys.vendor: To Be Filled By O.E.M.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/997695/+subscriptions
More information about the foundations-bugs
mailing list