IO Wait very high and real IO speed is very low

Tue Mar 4 00:59:27 UTC 2014

Hi Marty,

     There is no more "Call Trace" from /var/log/kern.log from 2nd Mar.
     And there is no warning/error found from fsck.
     I just upgrade the kernel as you recommended and also upgrade the 
Raid Controller firmware to the latest, but seems the problem still exists.
     Is there anything I can do avoid driving the processes to D stats?

Best Regards
Tomas

On 2014/3/3 16:35, Marty Sweet wrote:
> Hi Tomas,
>
> Are you still receiving Call Traces in /var/log/kern.log? I would of
> expected there to be something wrong with fsck with those messages.
> It might be worth upgrading to Linux 3.5 and see if the messages continue.
>
> $ apt-get install linux-image-3.5.0-34-generic
>
> Thanks,
> Marty
>
> On 3 March 2014 07:43, Tang Jianwei <myhnet at gmail.com> wrote:
>> Hi Marty,
>>
>>      I am right now using 3.2.0-48
>>
>>      Linux 3.2.0-48-generic #74-Ubuntu SMP Thu Jun 6 19:43:26 UTC 2013 x86_64
>> x86_64 x86_64 GNU/Linux
>>
>>
>> Best Regards
>> Tomas
>>
>>
>> On 2014/3/3 14:55, Marty Sweet wrote:
>>
>> Hi Tomas,
>>
>> What kernel version are you currently running?
>>
>> $ uname -a
>>
>> Thanks,
>> Marty
>>
>>
>>
>> -------- Original message --------
>> From: Tang Jianwei
>> Date:03/03/2014 03:36 (GMT+00:00)
>> To: Marty Sweet ,ubuntu-users at lists.ubuntu.com
>> Subject: Re: IO Wait very high and real IO speed is very low
>>
>> Hi Marty,
>>
>> Thank you very much, please see my comments:
>>
>> that machine is my backup server and the following is how my backup task
>> working procedure:
>> 1. make a local copy of the current backup with cp -al (hard link)
>> 2. use rsync to sync remote files if there is change/new/delete data
>>
>> So, all the data should be write to local disk, and it won't send data out.
>>
>> I understand that the write speed might be low for the reason that there
>> is not too much data to write due to the hard link, but the problem is:
>>
>> when there is no backup tasks running(low load), the command "dd
>> if=/dev/zero of=/backup/test10G bs=1M count=10240" write data in
>> 500MBps, and it reduce to around 10-20MBps when the backup tasks
>> running(high load).
>>
>> and the problem still exist after I run fsck as you suggested.
>>
>> Best Regards
>> Tomas
>>
>> On 2014/3/2 20:32, Marty Sweet wrote:
>>> Hi Tomas,
>>>
>>> So it appears you are reading from the disk but not putting it
>>> anywhere, for example, backing it up via a network. A utility which
>>> will probably help drill this down is $ iotop, here you will be able
>>> to see what processes are using what disk %. You may also like to try
>>> $ nmon, then press d for disk information.
>>>
>>> I would imagine it will be one of your rsync processes although those
>>> should of been going onto the network.
>>>
>>> When you say 'the write speed can reach 500MB/s when there is no other
>>> processes running', what size file are you testing this with?
>>> Something reasonably small will be saved to cache (Write-Back) before
>>> actually being committed to disk. As for the read speeds with those
>>> Wait % something is definitely odd looking at your hardware.
>>>
>>> Just to be on the safe side, I would recommend running fsck on your
>>> data volume, this will take the volume offline for some time.
>>> umount /dev/sda2
>>> fsck -f /dev/sda2
>>> mount /dev/sda2
>>>
>>> Marty
>>>
>>> On 2 March 2014 11:10, Tang Jianwei <myhnet at gmail.com> wrote:
>>>> Hi Marty,
>>>>
>>>>       Thank you very much for your time and help, here is the output of
>>>> $dstat
>>>>       http://pastebin.com/kiHijEKY
>>>>
>>>> Best Regards
>>>> Tomas
>>>>
>>>> On 2014/3/1 23:52, Marty Sweet wrote:
>>>>> Hi Tomas,
>>>>>
>>>>> Re-reading your original messages: If the system is experiencing lots
>>>>> of IO it will appear as wait %, this can also be seen by processes
>>>>> excessively in D state.
>>>>> It appears (looking at kern.log) that processes are getting blocked
>>>>> from disk, which obviously isn't ideal.
>>>>>
>>>>> Does this happen at set intervals?
>>>>>
>>>>> I have greped out the D state processes here:
>>>>> http://pastebin.com/b04bgHbL
>>>>>
>>>>> Could you run $ dstat and send a 10 lines when the issue occurs. This
>>>>> will allow us to tell the traffic being written to disk / read from
>>>>> network and the loads of the system.
>>>>>
>>>>> Marty
>>>>>
>>>>> On Sat, Mar 1, 2014 at 4:16 AM, Tang Jianwei <myhnet at gmail.com> wrote:
>>>>>> Hi Marty,
>>>>>>
>>>>>> thank you very much for your reply and the following is the files you
>>>>>> are
>>>>>> asking
>>>>>>
>>>>>> /var/log/kern.log (should be the same as what you were asking)
>>>>>> http://pastebin.com/3fC1qxrq
>>>>>>
>>>>>> /var/log/dmesg
>>>>>> http://pastebin.com/G059cxXN
>>>>>>
>>>>>> output of the commands
>>>>>> http://pastebin.com/6faPXt43
>>>>>>
>>>>>> And thank you very much in advance for your great help.
>>>>>>
>>>>>>
>>>>>> Best Regards
>>>>>> Tomas
>>>>>>
>>>>>> On 2014/2/28 20:00, ubuntu-users-request at lists.ubuntu.com wrote:
>>>>>>> Message: 3
>>>>>>> Date: Fri, 28 Feb 2014 08:00:31 +0000
>>>>>>> From: Marty Sweet <msweet.dev at gmail.com>
>>>>>>>
>>>>>>> To: "Ubuntu user technical support, not for general discussions"
>>>>>>>            <ubuntu-users at lists.ubuntu.com>
>>>>>>> Subject: Re: IO Wait very high and real IO speed is very low
>>>>>>> Message-ID:
>>>>>>>
>>>>>>> <CALoaWF5ACbWc=wwEiv4WorQZu0gpSEpk0UUSLMX6QhqnO1-m+w at mail.gmail.com>
>>>>>>> Content-Type: text/plain; charset=ISO-8859-1
>>>>>>>
>>>>>>>
>>>>>>> Hi Tomas,
>>>>>>>
>>>>>>> Could you provide the output of /var/log/kernel.log and /var/log/dmesg
>>>>>>> and $ df -h and $ fdisk -l ?
>>>>>>> When the problem occurs, can you also run $ ps aux ?
>>>>>>>
>>>>>>> Once you have done this please post it on pastebin and send the link,
>>>>>>> as there will be a few hundred lines.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Marty
>>>>>>>
>>>>>>> On Fri, Feb 28, 2014 at 6:00 AM, Tang Jianwei <myhnet at gmail.com>
>>>>>>> wrote:
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>>         recently I get a very weird problem from my Ubuntu Server 1204
>>>>>>>> LTS
>>>>>>>> 64bit:
>>>>>>>>         1. the write speed can reach 500MB/s when there is no other
>>>>>>>> processes
>>>>>>>> running
>>>>>>>>         2. the write speed down to around 10MB/s when there are many
>>>>>>>> other
>>>>>>>> processes and meanwhile:
>>>>>>>>                 a. only very high usage from io wait
>>>>>>>>                 b. sys, user and all the other cpu usages are very low
>>>>>>>>                 c. the system load average can reach to 15 and even
>>>>>>>> more
>>>>>>>> higher
>>>>>>>>
>>>>>>>>         can any one give me some advice what is wrong with my system,
>>>>>>>> and
>>>>>>>> what I
>>>>>>>> should to adjust in hardware/software?
>>>>>>>>
>>>>>>>>         the platform is:
>>>>>>>>         MB: Intel S3240GPLC
>>>>>>>>         CPU: I3-540
>>>>>>>>         RAM: Kingston 4G ECC
>>>>>>>>         Storage: LSI MegaRaid 9270-8i, Raid 5 with 5 drives + 1 spare,
>>>>>>>> ForcedWB
>>>>>>>> with no BBU
>>>>>>>>         Drive: WD 3T
>>>>>>>>
>>>>>>>>
>>>>>>>>         thank you very much.
>>>>>>>>
>>>>>>>>
>>>>>>>> Best Regards
>>>>>>>> Tomas
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> ubuntu-users mailing list
>>>>>>>> ubuntu-users at lists.ubuntu.com
>>>>>>>> Modify settings or unsubscribe at:
>>>>>>>> https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
>>>>>>>
>>