[Bug 1089195] Re: linux-headers will eat your inodes on LTS.

Mon Nov 18 15:48:03 UTC 2013

I also have just run into this problem on an installation of Ubuntu
Lucid, owned by a non-technical user.

The machine had become unusable - many programs could not run, including
the package manager. Symptoms were totally baffling for the user:
although some errors appear claiming that there is not enough space, a
check of the filesystem capacity using GUI tools shows there is in fact
plenty of space.

Indeed, this foxed me for a while, as even when it was evident that the
limit was the number of inodes rather than storage space, there seems to
be no easy way to find what is using up inodes except by running shell
commands which count files, a slow and painful process, especially when
directing a user over the phone.

The issue: about 20 kernel package upgrades using up all the available
inodes on the root filesystem. Uninstalling these freed 55% of a total
of 650000 inodes.  Entirely simple, and with a bit of polish, could be
made totally avoidable with a suitable option.  There's no need to keep
so many kernel packages around.

Removing old kernels does require more effort than it would appear to
deserve (list the packages, grep, pipe into apt-get remove).  It would
make sense to me if the automatic updates manager could be told to
handle removing kernels too.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to update-manager in Ubuntu.
https://bugs.launchpad.net/bugs/1089195

Title:
  linux-headers will eat your inodes on LTS.

Status in “update-manager” package in Ubuntu:
  Confirmed

Bug description:
  Hello all,

  Summary
  -----------------
  Both linux-image-* and linux-headers-* are installed every time you upgrade the kernel. However, they are never removed by any maintenance process.

  Every linux-headers-*-generic-pae package has approx. 6,700 files on
  it. Regular headers packages have even more files: around 11,700 files
  each.

  Although these packages' files won't occupy much space, after some
  years (think LTS installation), they will "eat" all the inodes on your
  root partition.

  The first effect you'll encounter will be the you are unable to
  upgrade your system.

  This is a situation that will affect all users, however it will be a
  greater problem regular non-technical user.

  There's no simple, high-level tool to solve this problem.

  Case Study
  ---------------
  I have a 2-year-and-8-months old 10.04 installation. I have a ~10GB root partition, which is double of the minimum recommend (5GB).

      $ df -h --type=ext4
      Filesystem            Size  Used Avail Use% Mounted on
      /dev/sda5             9.4G  6.2G  2.8G  70% /
      /dev/sda6              94G   45G   45G  51% /home

  It looks that I had plenty of space left to upgrade, but I got this
  error while upgrading:

      unable to create `/usr/src/linux-headers-2.6.32-45/arch/s390/include/asm/nmi.h.dpkg-new'
      (while processing `./usr/src/linux-headers-2.6.32-45/arch/s390/include/asm/nmi.h'): No space left on device

  It was because of I was running out of inodes:

      $ df -i /
      Filesystem            Inodes   IUsed   IFree IUse% Mounted on
      /dev/sda5             625856  618015    7841   99% /

  Why? Because the huge amount of linux-headers files:

      $ find /usr/src -type f | wc -l
      355112

  That is more than three hundred and fifty thousand files!

  These are the packages that I removed:

  Purg linux-headers-2.6.32-21 [2.6.32-21.32]
  Purg linux-headers-2.6.32-22-generic-pae [2.6.32-22.36]
  Purg linux-headers-2.6.32-22 [2.6.32-22.36]
  Purg linux-headers-2.6.32-23-generic-pae [2.6.32-23.37]
  Purg linux-headers-2.6.32-23 [2.6.32-23.37]
  Purg linux-headers-2.6.32-24-generic-pae [2.6.32-24.43]
  Purg linux-headers-2.6.32-24 [2.6.32-24.43]
  Purg linux-headers-2.6.32-25-generic-pae [2.6.32-25.45]
  Purg linux-headers-2.6.32-25 [2.6.32-25.45]
  Purg linux-headers-2.6.32-26-generic-pae [2.6.32-26.48]
  Purg linux-headers-2.6.32-26 [2.6.32-26.48]
  Purg linux-headers-2.6.32-27-generic-pae [2.6.32-27.49]
  Purg linux-headers-2.6.32-27 [2.6.32-27.49]
  Purg linux-headers-2.6.32-28-generic-pae [2.6.32-28.55]
  Purg linux-headers-2.6.32-28 [2.6.32-28.55]
  Purg linux-headers-2.6.32-29-generic-pae [2.6.32-29.58]
  Purg linux-headers-2.6.32-29 [2.6.32-29.58]
  Purg linux-headers-2.6.32-30-generic-pae [2.6.32-30.59]
  Purg linux-headers-2.6.32-30 [2.6.32-30.59]
  Purg linux-headers-2.6.32-31-generic-pae [2.6.32-31.61]
  Purg linux-headers-2.6.32-31 [2.6.32-31.61]
  Purg linux-headers-2.6.32-32-generic-pae [2.6.32-32.62]
  Purg linux-headers-2.6.32-32 [2.6.32-32.62]
  Purg linux-headers-2.6.32-33-generic-pae [2.6.32-33.72]
  Purg linux-headers-2.6.32-33 [2.6.32-33.72]
  Purg linux-headers-2.6.32-34-generic-pae [2.6.32-34.77]
  Purg linux-headers-2.6.32-34 [2.6.32-34.77]
  Purg linux-headers-2.6.32-35-generic-pae [2.6.32-35.78]
  Purg linux-headers-2.6.32-35 [2.6.32-35.78]
  Purg linux-headers-2.6.32-36-generic-pae [2.6.32-36.79]
  Purg linux-headers-2.6.32-36 [2.6.32-36.79]
  Purg linux-headers-2.6.32-37-generic-pae [2.6.32-37.81]
  Purg linux-headers-2.6.32-37 [2.6.32-37.81]
  Purg linux-headers-2.6.32-38-generic-pae [2.6.32-38.83]
  Purg linux-headers-2.6.32-38 [2.6.32-38.83]
  Purg linux-headers-2.6.32-39-generic-pae [2.6.32-39.86]
  Purg linux-headers-2.6.32-39 [2.6.32-39.86]
  Purg linux-headers-2.6.32-40-generic-pae [2.6.32-40.87]
  Purg linux-headers-2.6.32-40 [2.6.32-40.87]
  Purg linux-headers-2.6.32-41-generic-pae [2.6.32-41.94]
  Purg linux-headers-2.6.32-41 [2.6.32-41.94]
  Purg linux-headers-2.6.32-42-generic-pae [2.6.32-42.96]
  Purg linux-headers-2.6.32-42 [2.6.32-42.96]
  Purg linux-headers-2.6.32-43-generic-pae [2.6.32-43.97]
  Purg linux-headers-2.6.32-43 [2.6.32-43.97]
  Purg linux-headers-generic-pae [2.6.32.45.52]
  Purg linux-image-2.6.32-42-generic-pae [2.6.32-42.96]
  Purg linux-image-2.6.32-43-generic-pae [2.6.32-43.97]

  Then, problem solved:

      $ find /usr/src/ -type f | wc -l
      28276

      $ ls /usr/src/
      linux-headers-2.6.32-44              linux-headers-2.6.32-45-generic-pae
      linux-headers-2.6.32-44-generic-pae  nvidia-current-195.36.24
      linux-headers-2.6.32-45              virtualbox-ose-3.1.6

      $ df -i --type=ext4
      Filesystem            Inodes   IUsed   IFree IUse% Mounted on
      /dev/sda5             625856  192891  432965   31% /
      /dev/sda6            6225920   95025 6130895    2% /home

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/1089195/+subscriptions