Responsiveness testing through dpkg

[Chase Douglas]

On Tue, Apr 27, 2010 at 3:46 AM, Chow Loong Jin <hyperair at ubuntu.com> wrote:
> On Tuesday 27,April,2010 03:33 PM, Tim Gardner wrote:
>> On 04/26/2010 10:45 PM, Chase Douglas wrote:
>>> Hello all,
>>>
>>> I'm interested in doing some desktop responsiveness testing, and I've
>>> found that when I have a large number of packages to install/update it
>>> kills my desktop responsiveness not only during installation, but also
>>> for quite a bit of time afterwards. Desktop responsiveness is a rather
>>> nebulous issue, one that occurs in different situations on different
>>> hardware in different ways. However, apt-get/dpkg seem to do a good
>>> job of causing desktop slowdowns across many configurations :).
>>>
>>> What I would like to do is find a good test case I can run on bare
>>> hardware (not in a VM with snapshot capabilities) that is
>>> reproducible. I think doing a reinstallation of a set of packages, say
>>> ubuntu-desktop, would work well. Is there a safe way to do this?
>>>
>>> Thanks
>>>
>>> -- Chase
>>>
>>
>> I think dpkg is triggering the fsync problem. Andy and Surbhi have some
>> information on that.
>
> That would be during the installation. But Chase did mention that it affected
> responsiveness for some time afterwards as well. I believe this could be due to
> large amounts of memory being swapped out to cache the large amounts of reads
> and writes that dpkg makes.

Yes, this makes sense up to a point. I would understand if my machine
were a little slow to respond to events when I start using it again
after dpkg runs due to the need to swap pages back in. However, the
slowness can last for up to an hour, and it still occurs when
switching between applications I've already pulled up. The pages
should have already swapped in once, yet the hard disk is still
churning doing something.

> Another thing worth noting is dpkg's memory usage. It seems to take anywhere
> between 100M and 300M of memory on my system, for *any* operation at all. That
> is almost on par with the memory usage of a virtual machine capable of running
> Windows XP. Truncating /var/lib/dpkg/available to 0 before any dpkg runs brings
> the memory usage closer to 100M than 300M. In fact, I'm thinking of replacing
> dpkg with a script that truncates this file to 0 before running the real dpkg.
>
>> Have you considered generating some kind of steady state background load
>> using bonnie or stress? I'd think those kind of loads are more likely in
>> the real world, whereas the dpkg scenario is relatively rare for most folks.
>
> That doesn't stop dpkg from messing up system responsiveness big time. I think
> this issue should be looked into as well.

My point here is that I'm seeing responsiveness issues that are
triggered by dpkg. Maybe dpkg should be optimized somehow, but I've
seen other people report similar issues when using other software (bug
566841). I think a dpkg test case would be a somewhat uniform and easy
to reproduce scenario that causes the issue. My goal is to tackle one
responsiveness issue at a time, so I want to focus on this specific
reproduction scenario unless it's just not possible to reproduce
easily.

Thanks,

-- Chase