Server increasing load due increasing processes in D state

[Peter M. Petrakis]

summary: You're massively over provisioned, +100%

resolution:
1. Reconfigured your VMs to use a single vcpu and cap your available vcpus to 8,
    even less considering that more than half of your applications are IO bound.
2. the software raid can be a choke point for all application progress,
    virtual or otherwise; it needs CPU time too.

remarks:
you have a test setup, treat it as such, 19 vcpus is overboard.

Peter

On 02/25/2013 12:17 PM, Alessandro Tagliapietra wrote:
> Sorry for disturbing again,
>
> after the restart I've seen that I'm unable to ssh to VM since on login they run byobu which now hangs (never did on VM).
>
> I managed to ctrl-c fast enough to don't start byobu and an strace on it gave me this:
>
> http://pastebin.com/raw.php?i=KYMbsxKV
>
> Thanks
>
> Best Regards
>
> --
>
> Alessandro Tagliapietra
> alexfu.it (http://www.alexfu.it)
>
> Il giorno lunedì 25 febbraio 2013, alle ore 17:51, Alessandro Tagliapietra ha scritto:
>
>> Hi Eduardo
>>
>> Thank you for the tips.
>>
>> I'll wait a few days and let you know when this happens again.
>>
>> About the load, system cpu wasn't more then 10% used from top, io wait was at 2% most of the time.
>>
>> We've 4 x 2 (HT) cores on the server and a total number of 19 vcpu allocated on VM running on that host.
>>
>> Vm runs mostly nginx+php-fpm+mysql, one runs also rabbitMQ and a python rabbitMQ consumer.
>>
>> I'll let you know later then.
>>
>> Thanks again!
>>
>> Best
>>
>> --
>>
>> Alessandro Tagliapietra
>> alexfu.it (http://www.alexfu.it)
>>
>> Il giorno lunedì 25 febbraio 2013, alle ore 16:44, Eduardo Damato ha scritto:
>>
>>>
>>> Hi Alessandro,
>>>
>>> Thanks for the information.
>>>
>>> The sysrq-t that I requested is *only* useful during the problem. Please
>>> do that when you encounter the problem again.
>>>
>>> It may be that you are overcommitting cpus on your system by having many
>>> virtual machines running on the nova controller node. This is a
>>> completely wild guess, but I would recommend you to look at how many
>>> cpus you have and how many virtual machines and if you have any
>>> processes in real time or sched FIFO.
>>>
>>> Cheers,
>>> Eduardo.
>>>
>>>
>>>
>>
>>
>
>
>
>