IO Wait very high and real IO speed is very low
Marty Sweet
msweet.dev at gmail.com
Mon Mar 3 06:55:04 UTC 2014
Hi Tomas,
What kernel version are you currently running?
$ uname -a
Thanks,
Marty
-------- Original message --------
From: Tang Jianwei <myhnet at gmail.com>
Date:03/03/2014 03:36 (GMT+00:00)
To: Marty Sweet <msweet.dev at gmail.com>,ubuntu-users at lists.ubuntu.com
Subject: Re: IO Wait very high and real IO speed is very low
Hi Marty,
Thank you very much, please see my comments:
that machine is my backup server and the following is how my backup task
working procedure:
1. make a local copy of the current backup with cp -al (hard link)
2. use rsync to sync remote files if there is change/new/delete data
So, all the data should be write to local disk, and it won't send data out.
I understand that the write speed might be low for the reason that there
is not too much data to write due to the hard link, but the problem is:
when there is no backup tasks running(low load), the command "dd
if=/dev/zero of=/backup/test10G bs=1M count=10240" write data in
500MBps, and it reduce to around 10-20MBps when the backup tasks
running(high load).
and the problem still exist after I run fsck as you suggested.
Best Regards
Tomas
On 2014/3/2 20:32, Marty Sweet wrote:
> Hi Tomas,
>
> So it appears you are reading from the disk but not putting it
> anywhere, for example, backing it up via a network. A utility which
> will probably help drill this down is $ iotop, here you will be able
> to see what processes are using what disk %. You may also like to try
> $ nmon, then press d for disk information.
>
> I would imagine it will be one of your rsync processes although those
> should of been going onto the network.
>
> When you say 'the write speed can reach 500MB/s when there is no other
> processes running', what size file are you testing this with?
> Something reasonably small will be saved to cache (Write-Back) before
> actually being committed to disk. As for the read speeds with those
> Wait % something is definitely odd looking at your hardware.
>
> Just to be on the safe side, I would recommend running fsck on your
> data volume, this will take the volume offline for some time.
> umount /dev/sda2
> fsck -f /dev/sda2
> mount /dev/sda2
>
> Marty
>
> On 2 March 2014 11:10, Tang Jianwei <myhnet at gmail.com> wrote:
>> Hi Marty,
>>
>> Thank you very much for your time and help, here is the output of $dstat
>> http://pastebin.com/kiHijEKY
>>
>> Best Regards
>> Tomas
>>
>> On 2014/3/1 23:52, Marty Sweet wrote:
>>> Hi Tomas,
>>>
>>> Re-reading your original messages: If the system is experiencing lots
>>> of IO it will appear as wait %, this can also be seen by processes
>>> excessively in D state.
>>> It appears (looking at kern.log) that processes are getting blocked
>>> from disk, which obviously isn't ideal.
>>>
>>> Does this happen at set intervals?
>>>
>>> I have greped out the D state processes here: http://pastebin.com/b04bgHbL
>>>
>>> Could you run $ dstat and send a 10 lines when the issue occurs. This
>>> will allow us to tell the traffic being written to disk / read from
>>> network and the loads of the system.
>>>
>>> Marty
>>>
>>> On Sat, Mar 1, 2014 at 4:16 AM, Tang Jianwei <myhnet at gmail.com> wrote:
>>>> Hi Marty,
>>>>
>>>> thank you very much for your reply and the following is the files you are
>>>> asking
>>>>
>>>> /var/log/kern.log (should be the same as what you were asking)
>>>> http://pastebin.com/3fC1qxrq
>>>>
>>>> /var/log/dmesg
>>>> http://pastebin.com/G059cxXN
>>>>
>>>> output of the commands
>>>> http://pastebin.com/6faPXt43
>>>>
>>>> And thank you very much in advance for your great help.
>>>>
>>>>
>>>> Best Regards
>>>> Tomas
>>>>
>>>> On 2014/2/28 20:00, ubuntu-users-request at lists.ubuntu.com wrote:
>>>>> Message: 3
>>>>> Date: Fri, 28 Feb 2014 08:00:31 +0000
>>>>> From: Marty Sweet <msweet.dev at gmail.com>
>>>>>
>>>>> To: "Ubuntu user technical support, not for general discussions"
>>>>> <ubuntu-users at lists.ubuntu.com>
>>>>> Subject: Re: IO Wait very high and real IO speed is very low
>>>>> Message-ID:
>>>>>
>>>>> <CALoaWF5ACbWc=wwEiv4WorQZu0gpSEpk0UUSLMX6QhqnO1-m+w at mail.gmail.com>
>>>>> Content-Type: text/plain; charset=ISO-8859-1
>>>>>
>>>>>
>>>>> Hi Tomas,
>>>>>
>>>>> Could you provide the output of /var/log/kernel.log and /var/log/dmesg
>>>>> and $ df -h and $ fdisk -l ?
>>>>> When the problem occurs, can you also run $ ps aux ?
>>>>>
>>>>> Once you have done this please post it on pastebin and send the link,
>>>>> as there will be a few hundred lines.
>>>>>
>>>>> Thanks,
>>>>> Marty
>>>>>
>>>>> On Fri, Feb 28, 2014 at 6:00 AM, Tang Jianwei <myhnet at gmail.com> wrote:
>>>>>> Hi All,
>>>>>>
>>>>>> recently I get a very weird problem from my Ubuntu Server 1204
>>>>>> LTS
>>>>>> 64bit:
>>>>>> 1. the write speed can reach 500MB/s when there is no other
>>>>>> processes
>>>>>> running
>>>>>> 2. the write speed down to around 10MB/s when there are many
>>>>>> other
>>>>>> processes and meanwhile:
>>>>>> a. only very high usage from io wait
>>>>>> b. sys, user and all the other cpu usages are very low
>>>>>> c. the system load average can reach to 15 and even more
>>>>>> higher
>>>>>>
>>>>>> can any one give me some advice what is wrong with my system, and
>>>>>> what I
>>>>>> should to adjust in hardware/software?
>>>>>>
>>>>>> the platform is:
>>>>>> MB: Intel S3240GPLC
>>>>>> CPU: I3-540
>>>>>> RAM: Kingston 4G ECC
>>>>>> Storage: LSI MegaRaid 9270-8i, Raid 5 with 5 drives + 1 spare,
>>>>>> ForcedWB
>>>>>> with no BBU
>>>>>> Drive: WD 3T
>>>>>>
>>>>>>
>>>>>> thank you very much.
>>>>>>
>>>>>>
>>>>>> Best Regards
>>>>>> Tomas
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> ubuntu-users mailing list
>>>>>> ubuntu-users at lists.ubuntu.com
>>>>>> Modify settings or unsubscribe at:
>>>>>> https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
>>>>>
>>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-users/attachments/20140303/660a1a3b/attachment.html>
More information about the ubuntu-users
mailing list