Fwd: IO Wait very high and real IO speed is very low
Marty Sweet
msweet.dev at gmail.com
Tue Mar 4 07:47:24 UTC 2014
Hi Tomas,
D state processes are perfectly normal for a system interacting disk.
You mentioned you had no BBU, which might be causing an issue.
I am currently testing a 28-disk backup server with an LSI raid card
and no BBU and speeds aren't really up to standard.
We will be adding a BBU and forcing Write-Back in /etc/fstab later on
today. I will let you know if there is any major improvement.
Looking around online it seems a common solution with LSI raid cards,
even if Write-Back is enabled in the first place (We currently do on
the raid sets).
Thanks,
Marty
On 4 March 2014 00:59, Tang Jianwei <myhnet at gmail.com> wrote:
> Hi Marty,
>
> There is no more "Call Trace" from /var/log/kern.log from 2nd Mar.
> And there is no warning/error found from fsck.
> I just upgrade the kernel as you recommended and also upgrade the Raid
> Controller firmware to the latest, but seems the problem still exists.
> Is there anything I can do avoid driving the processes to D stats?
>
> Best Regards
> Tomas
>
>
> On 2014/3/3 16:35, Marty Sweet wrote:
>>
>> Hi Tomas,
>>
>> Are you still receiving Call Traces in /var/log/kern.log? I would of
>> expected there to be something wrong with fsck with those messages.
>> It might be worth upgrading to Linux 3.5 and see if the messages continue.
>>
>> $ apt-get install linux-image-3.5.0-34-generic
>>
>> Thanks,
>> Marty
>>
>> On 3 March 2014 07:43, Tang Jianwei <myhnet at gmail.com> wrote:
>>>
>>> Hi Marty,
>>>
>>> I am right now using 3.2.0-48
>>>
>>> Linux 3.2.0-48-generic #74-Ubuntu SMP Thu Jun 6 19:43:26 UTC 2013
>>> x86_64
>>> x86_64 x86_64 GNU/Linux
>>>
>>>
>>> Best Regards
>>> Tomas
>>>
>>>
>>> On 2014/3/3 14:55, Marty Sweet wrote:
>>>
>>> Hi Tomas,
>>>
>>> What kernel version are you currently running?
>>>
>>> $ uname -a
>>>
>>> Thanks,
>>> Marty
>>>
>>>
>>>
>>> -------- Original message --------
>>> From: Tang Jianwei
>>> Date:03/03/2014 03:36 (GMT+00:00)
>>> To: Marty Sweet ,ubuntu-users at lists.ubuntu.com
>>> Subject: Re: IO Wait very high and real IO speed is very low
>>>
>>> Hi Marty,
>>>
>>> Thank you very much, please see my comments:
>>>
>>> that machine is my backup server and the following is how my backup task
>>> working procedure:
>>> 1. make a local copy of the current backup with cp -al (hard link)
>>> 2. use rsync to sync remote files if there is change/new/delete data
>>>
>>> So, all the data should be write to local disk, and it won't send data
>>> out.
>>>
>>> I understand that the write speed might be low for the reason that there
>>> is not too much data to write due to the hard link, but the problem is:
>>>
>>> when there is no backup tasks running(low load), the command "dd
>>> if=/dev/zero of=/backup/test10G bs=1M count=10240" write data in
>>> 500MBps, and it reduce to around 10-20MBps when the backup tasks
>>> running(high load).
>>>
>>> and the problem still exist after I run fsck as you suggested.
>>>
>>> Best Regards
>>> Tomas
>>>
>>> On 2014/3/2 20:32, Marty Sweet wrote:
>>>>
>>>> Hi Tomas,
>>>>
>>>> So it appears you are reading from the disk but not putting it
>>>> anywhere, for example, backing it up via a network. A utility which
>>>> will probably help drill this down is $ iotop, here you will be able
>>>> to see what processes are using what disk %. You may also like to try
>>>> $ nmon, then press d for disk information.
>>>>
>>>> I would imagine it will be one of your rsync processes although those
>>>> should of been going onto the network.
>>>>
>>>> When you say 'the write speed can reach 500MB/s when there is no other
>>>> processes running', what size file are you testing this with?
>>>> Something reasonably small will be saved to cache (Write-Back) before
>>>> actually being committed to disk. As for the read speeds with those
>>>> Wait % something is definitely odd looking at your hardware.
>>>>
>>>> Just to be on the safe side, I would recommend running fsck on your
>>>> data volume, this will take the volume offline for some time.
>>>> umount /dev/sda2
>>>> fsck -f /dev/sda2
>>>> mount /dev/sda2
>>>>
>>>> Marty
>>>>
>>>> On 2 March 2014 11:10, Tang Jianwei <myhnet at gmail.com> wrote:
>>>>>
>>>>> Hi Marty,
>>>>>
>>>>> Thank you very much for your time and help, here is the output of
>>>>> $dstat
>>>>> http://pastebin.com/kiHijEKY
>>>>>
>>>>> Best Regards
>>>>> Tomas
>>>>>
>>>>> On 2014/3/1 23:52, Marty Sweet wrote:
>>>>>>
>>>>>> Hi Tomas,
>>>>>>
>>>>>> Re-reading your original messages: If the system is experiencing lots
>>>>>> of IO it will appear as wait %, this can also be seen by processes
>>>>>> excessively in D state.
>>>>>> It appears (looking at kern.log) that processes are getting blocked
>>>>>> from disk, which obviously isn't ideal.
>>>>>>
>>>>>> Does this happen at set intervals?
>>>>>>
>>>>>> I have greped out the D state processes here:
>>>>>> http://pastebin.com/b04bgHbL
>>>>>>
>>>>>> Could you run $ dstat and send a 10 lines when the issue occurs. This
>>>>>> will allow us to tell the traffic being written to disk / read from
>>>>>> network and the loads of the system.
>>>>>>
>>>>>> Marty
>>>>>>
>>>>>> On Sat, Mar 1, 2014 at 4:16 AM, Tang Jianwei <myhnet at gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Marty,
>>>>>>>
>>>>>>> thank you very much for your reply and the following is the files you
>>>>>>> are
>>>>>>> asking
>>>>>>>
>>>>>>> /var/log/kern.log (should be the same as what you were asking)
>>>>>>> http://pastebin.com/3fC1qxrq
>>>>>>>
>>>>>>> /var/log/dmesg
>>>>>>> http://pastebin.com/G059cxXN
>>>>>>>
>>>>>>> output of the commands
>>>>>>> http://pastebin.com/6faPXt43
>>>>>>>
>>>>>>> And thank you very much in advance for your great help.
>>>>>>>
>>>>>>>
>>>>>>> Best Regards
>>>>>>> Tomas
>>>>>>>
>>>>>>> On 2014/2/28 20:00, ubuntu-users-request at lists.ubuntu.com wrote:
>>>>>>>>
>>>>>>>> Message: 3
>>>>>>>> Date: Fri, 28 Feb 2014 08:00:31 +0000
>>>>>>>> From: Marty Sweet <msweet.dev at gmail.com>
>>>>>>>>
>>>>>>>> To: "Ubuntu user technical support, not for general discussions"
>>>>>>>> <ubuntu-users at lists.ubuntu.com>
>>>>>>>> Subject: Re: IO Wait very high and real IO speed is very low
>>>>>>>> Message-ID:
>>>>>>>>
>>>>>>>> <CALoaWF5ACbWc=wwEiv4WorQZu0gpSEpk0UUSLMX6QhqnO1-m+w at mail.gmail.com>
>>>>>>>> Content-Type: text/plain; charset=ISO-8859-1
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Tomas,
>>>>>>>>
>>>>>>>> Could you provide the output of /var/log/kernel.log and
>>>>>>>> /var/log/dmesg
>>>>>>>> and $ df -h and $ fdisk -l ?
>>>>>>>> When the problem occurs, can you also run $ ps aux ?
>>>>>>>>
>>>>>>>> Once you have done this please post it on pastebin and send the
>>>>>>>> link,
>>>>>>>> as there will be a few hundred lines.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Marty
>>>>>>>>
>>>>>>>> On Fri, Feb 28, 2014 at 6:00 AM, Tang Jianwei <myhnet at gmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> recently I get a very weird problem from my Ubuntu Server
>>>>>>>>> 1204
>>>>>>>>> LTS
>>>>>>>>> 64bit:
>>>>>>>>> 1. the write speed can reach 500MB/s when there is no other
>>>>>>>>> processes
>>>>>>>>> running
>>>>>>>>> 2. the write speed down to around 10MB/s when there are
>>>>>>>>> many
>>>>>>>>> other
>>>>>>>>> processes and meanwhile:
>>>>>>>>> a. only very high usage from io wait
>>>>>>>>> b. sys, user and all the other cpu usages are very
>>>>>>>>> low
>>>>>>>>> c. the system load average can reach to 15 and even
>>>>>>>>> more
>>>>>>>>> higher
>>>>>>>>>
>>>>>>>>> can any one give me some advice what is wrong with my
>>>>>>>>> system,
>>>>>>>>> and
>>>>>>>>> what I
>>>>>>>>> should to adjust in hardware/software?
>>>>>>>>>
>>>>>>>>> the platform is:
>>>>>>>>> MB: Intel S3240GPLC
>>>>>>>>> CPU: I3-540
>>>>>>>>> RAM: Kingston 4G ECC
>>>>>>>>> Storage: LSI MegaRaid 9270-8i, Raid 5 with 5 drives + 1
>>>>>>>>> spare,
>>>>>>>>> ForcedWB
>>>>>>>>> with no BBU
>>>>>>>>> Drive: WD 3T
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> thank you very much.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best Regards
>>>>>>>>> Tomas
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> ubuntu-users mailing list
>>>>>>>>> ubuntu-users at lists.ubuntu.com
>>>>>>>>> Modify settings or unsubscribe at:
>>>>>>>>> https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
>>>>>>>>
>>>>>>>>
>>>
>
More information about the ubuntu-users
mailing list