<div>
Hello Eduardo,
</div><div><br></div><div>I've rebooted the server and now it's running memtest, 40% done and no errors so far.</div><div><br></div><div>We've openstack (essex) configured on two servers, first one (this server) was an all-in-one complete install, then I've setup the second compute node adding nova-network in multi-host mode. Each server has a public internet interface and a crossover cable between them on an additional NIC.</div><div><br></div><div>Each server's cpu is an Intel i7-2600 with 16GB non ECC ram, 2 x 3TB HDD with software raid1 (mdadm).</div><div><br></div><div>So far I've about 8 KVM machines running on it and yes, I've rabbitMQ running on this node too, also with MySql server, nova api, keystone and glance.</div><div>Kernel version is 3.2.0-37-generic. Openstack packages has been updated to latest version 1 week ago.</div><div><br></div><div>For the first 5-10 days it works fine, no processes in D state except for some seconds (which I think is normal)</div><div><br></div><div>When the memtest finished I'll run the kernel traces you've asked for.</div><div><br></div><div>By the way, I've started encountering this problem some months ago due a byobu-status processes that got run lot of times by byobu and driven the load to more then 200. Then I've disabled byobu and got the issue on other processes too (even an ls -la hangs sometimes)</div><div><br></div><div>I'll let you know of the results.</div><div><br></div><div>Thank you very much for helping.</div><div><br></div><div>Best</div><div><br></div>
<div>
--<br>
<br>
Alessandro Tagliapietra
<br>
<a href="http://www.alexfu.it">alexfu.it</a>
</div>
<p style="color: #A0A0A8;">Il giorno lunedì 25 febbraio 2013, alle ore 15:39, Eduardo Damato ha scritto: </p>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span><div><div>
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
<div>Hi Alessandro,<br>
<br>
What's the node you're having problems with? Is this a compute
node? Can you give more information on the layout of your nova
installation? I can see that qemu and rabbit-mq are running on the
same node. Do you use the compute node as an MQ node as well?<br>
<br>
The problem here seems more to be related to the kernel, since
many many tasks are stuck in the same W_CHAN. <br>
<br>
Ideally It would be good to have the output of sysrq-t from this
system, but this can cause the system to hang or crash depending
on what the status is, specially because we already know that
there are many task_structs blocked in the same place. <br>
<br>
you could do:<br>
<br>
# echo t > /proc/sysrq-trigger<br>
(wait 5 s)<br>
# echo t > /proc/sysrq-trigger<br>
(wait 5 s)<br>
# echo t > /proc/sysrq-trigger<br>
<br>
And then we can have a look at the traces and see if they're
moving or not.<br>
<br>
lsof is blocked reading the memory maps of process 1227. This
could lead to more information on the problem, but at the same
time because there are so many blocked processes it could be just
another sign of the problem and not a hint to the reason why this
is happening. <br>
<br>
Without kernel traces (sysrq-t) or a vmcore it would be
complicated to understand what's happening. It doesn't seem to be
IO related.<br>
<br>
Cheers,<br>
Eduardo.<br>
<br>
On 25/02/13 12:10, Alessandro Tagliapietra wrote:<br>
</div><blockquote type="cite"><div>
<div>After an strace of lsof I've seen it hangs on</div>
<div><br>
</div>
<div>
<pre style="word-wrap: break-word; ">stat("/proc/1227/", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/proc/1227/stat", O_RDONLY) = 4
read(4, "1227 (nova-dhcpbridge) D 1224 25"..., 4096) = 242
close(4) = 0
readlink("/proc/1227/cwd", "/"..., 4096) = 1
stat("/proc/1227/cwd", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
readlink("/proc/1227/root", "/", 4096) = 1
stat("/proc/1227/root", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
readlink("/proc/1227/exe", "/usr/bin/python2.7"..., 4096) = 18
stat("/proc/1227/exe", {st_mode=S_IFREG|0755, st_size=2989480, ...}) = 0
open("/proc/1227/maps", O_RDONLY) = 4
read(4,</pre>
<pre style="word-wrap: break-word; "><font face="Helvetica">Could it be a memory issue?</font></pre>
<pre style="word-wrap: break-word; "><font face="Helvetica">Actually I cannot run the memory test, maybe tomorrow. Just to know if someone else had the same issue.</font></pre>
<pre style="word-wrap: break-word; "><font face="Helvetica">Thanks in advance</font></pre>
</div>
<div>
--<br>
<br>
Alessandro Tagliapietra
<br>
<a moz-do-not-send="true" href="http://www.alexfu.it">alexfu.it</a>
</div>
<p style="color: #A0A0A8;">Il giorno lunedì 25 febbraio 2013, alle
ore 12:29, Alessandro Tagliapietra ha scritto: </p><blockquote type="cite"><div>
<span>
<div>
<div>
<div> Hello guys, </div>
<div><br>
</div>
<div>at work we've the openstack controller that since
some months started to increase its load after some days
of uptime.</div>
<div><br>
</div>
<div>I've seen that the cause is that processes sometimes
hangs and remain in D state.</div>
<div><br>
</div>
<div>I've used some combination of ps args to get these
outputs:</div>
<div><br>
</div>
<div><a moz-do-not-send="true" href="http://pastebin.com/raw.php?i=LGGzGrWu">http://pastebin.com/raw.php?i=LGGzGrWu</a></div>
<div><a moz-do-not-send="true" href="http://pastie.org/pastes/6332964/text">http://pastie.org/pastes/6332964/text</a></div>
<div><a moz-do-not-send="true" href="http://pastie.org/pastes/6332979/text">http://pastie.org/pastes/6332979/text</a></div>
<div><br>
</div>
<div>The hdd is a soft-raid1 over 2 disks, which SMART
values are fine.</div>
<div><br>
</div>
<div>Commands like lsof, strace on a D process doesn't
return.</div>
<div><br>
</div>
<div>Any idea on what could be the cause?</div>
<div><br>
</div>
<div>Thanks in advance</div>
<div><br>
</div>
<div>
--<br>
<br>
Alessandro Tagliapietra
<br>
<a moz-do-not-send="true" href="http://www.alexfu.it">alexfu.it</a>
</div>
</div>
</div>
</span> </div></blockquote><div> <br>
</div>
<br>
<fieldset></fieldset>
<br>
</div></blockquote><br>
</div><div><div>-- </div><div>ubuntu-server mailing list</div><div><a href="mailto:ubuntu-server@lists.ubuntu.com">ubuntu-server@lists.ubuntu.com</a></div><div><a href="https://lists.ubuntu.com/mailman/listinfo/ubuntu-server">https://lists.ubuntu.com/mailman/listinfo/ubuntu-server</a></div><div>More info: <a href="https://wiki.ubuntu.com/ServerTeam">https://wiki.ubuntu.com/ServerTeam</a></div></div></div></span>
</blockquote>
<div>
<br>
</div>