[Bug 771372] Re: procps runs too early in the boot process

[James Hunt]

= Short Answer =

The immediate fix (for the majority of cases) is to modify
/etc/init/procps.conf so that it's "start on" condition becomes:

  start on virtual-filesystems or started networking

= Long Answer (brace yourselves! :-) =

This is an interesting issue. The ideal is a generic solution to the
problem but unfortunately this cannot be provided at this point in time.
The following attempts to explain why...

== Summary of Current Behaviour ==

The sysctl(8) facility allows *kernel parameters* to be set at any time.
The current procps.conf Upstart job is started as early as possible to
achieve this. The job calls sysctl which sets the specific kernel
parameters by reading from /etc/sysctl.conf and /etc/sysctl.d/* and
writing these values to /proc/sys/*.

Therefore, /proc must be mounted read-write and /etc must be mounted
atleast read-only. These requirements are satisfied by the "start on"
condition specified in procps.conf.

This works for all "built-in" kernel functionality, that is all functionality that isn't provided by kernel modules. It should be
pointed out that some settings *must* be applied at this the earliest point, for example "kernel.printk" (aka /proc/sys/kernel/printk). So, procps.conf is behaving correctly.

== The Two Types of Parameters ==

However, some kernel modules provide new kernel parameters. Note that
there is a distinction between *kernel parameters* and *module
parameters*:

- kernel parameters are set via sysctl(8) and sysctl can be called any
  number of times to change these values at any time after the
  functionality is available in the kernel.

- module parameters are set *once*: when the kernel module is loaded
  (generally via modprobe(8)). One they are set you cannot change them
  until you reload the module.

But the picture isn't as clear-cut as that since some modules link their
module parameters with kernel parameters such that when you load a
module, you can specify values for the parameters, but you can *also*
change these same parameters via sysctl after the module has been
loaded. The "sunrpc" module is a good example of this:

  sunrpc.udp_slot_table_entries = 128
  sunrpc.tcp_slot_table_entries = 128

Further, lets summarise the available methods for setting *kernel
parameters* and *module parameters*:

=== Methods for Setting Parameters ===

== Kernel Parameters ==

(1) The "procps.conf" Upstart job (already covered above).

(2) Running sysctl(8) manually.

== Module Parameters ==

(1) The "module-init-tools.conf" Upstart job.

    This is available when you install the bridge-utils package, which is
    not installed by default. This job job reads a list of modules and
    optional parameters from "/etc/modules" and loads those modules by
    calling modprobe(8) directly.

(2) The "/etc/modprobe.conf" and "/etc/modprobe.d/*" files.

    As specified in modprobe.conf(5), these files can specify some quite
    clever options. The crucial point though is that these files can
    specify modules to load along with parameters for those modules
    using the "options" keyword.

    Note that there is *no* Upstart job that uses these files: there
    doesn't need to be since any invocation of modprobe(8) will read
    these files and apply the correct settings automatically.

(3) Running modprobe(8)/insmod(8) manually.

== Definition of the Actual Problem ==

So the *real* problem is:

  We need a way to set a loadable modules *kernel parameters* as soon
  after module load time as possible for those modules that do not
  expose such parameters via *module parameters*.

This generates another question:

- How can we establish if a loadable module exposes its module
  parameters as *kernel parameters*? I think the answer is:

    /sys/module/<module>/parameters/

NOTE: no mention has been made of network/bridge modules at this point:
they are just the most common manifestation of the problem but do not
encompass the entire problem space.

== Suggestions ==

(1) Change "/etc/sysctl.d/README", sysctl(5) and sysctl(8) to suggest
    that users *only* use sysctl for built-in kernel functionality or
    for those modules that do not expose such settings as *module
    parameters*.

(2) Encourage users to utilise /etc/modprobe.d/* for modules that expose
    their settings as both *kernel parameters* and *module parameters*.

(3) Decide upon a strategy for /etc/modules.

    Do we deprecate its use in favour of /etc/modprobe.d/*? Maybe we
    could suggest it is only to be used for modules for which users do
    not wish to provide parameters for?

(4) Provide a script that will ease the burden of understanding all
    these details and allow users to set parameters in a sane manner using
    the "DRY" principle.

    Such a tool would provide a set of feature such as:

    - when given a kernel module, return a list of parameters denoting
      whether each parameter is:
      - a *module parameter*
      - a *kernel parameter*

      If a parameter is both, that will be made clear.

    - when given a sysctl config name or partial name, such as
      "udp_slot_table_entries", the tool will return details of the
      module that provides this value.

    - the ability to set a parameter and have the tool take the
      appropriate action:

      - if the parameter is only a module parameter, add it to
        /etc/modprobe.d/*.
      - if the parameter is only a kernel parameter, add it to
        /etc/sysctl.d/*.

      - if the parameter is both a module parameter and a kernel
        parameter, the tool witll update /etc/modprobe.d/* unless the
        user somehow marks the setting as "volatile" (denoting that they
        might want to change it post-module-load( whereupon it would get
        added to /etc/sysctl.d/*.

We should obviously discuss with Debian prior to making any changes like
this though.

== Compromise Solution ==

The compromise solution is to have procps run a second time after all
network devices and bridges are up. This *should* resolve most users
problems, but note again that we are here only addressing the networking
modules pat of the problem namespace.

We may need to run the procps job at other points in the boot but we
cannot know exactly when as we've only had input from users with network
module issues.

== Ideal Solution for the Immediate Problem ==

Ideally, modprobe(8) itself would be modified to apply any sysctl
settings immediately after the module in question had been loaded. This
needs pursuing with upstream.

== Related Bugs ==

There are actually two fixes for bug 771372:

  (1) Use an updated procps.conf job.
  (2) Use modprobe.d/*.

If you do not intend to change the values of the "slot_table_entries"
variables, solution (2) is the "most correct".

However, until this problem is more widely understood by users, the
safest approach is to ship a modified procps.conf that essentially runs
twice:

1) as early as possible for printk ("virtual-filesystems")

2) after all network interfaces are available ("started networking")

This is accomplished by changing procps.conf's start on condition from:

     start on virtual-filesystems

   ... to:

     start on virtual-filesystems or started networking

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to procps in Ubuntu.
https://bugs.launchpad.net/bugs/771372

Title:
  procps runs too early in the boot process

Status in “procps” package in Ubuntu:
  Triaged
Status in “procps” source package in Lucid:
  Triaged
Status in “procps” source package in Maverick:
  Triaged
Status in “procps” source package in Natty:
  Triaged
Status in “procps” source package in Oneiric:
  Triaged
Status in “procps” source package in Precise:
  Triaged

Bug description:
  Binary package hint: upstart

  The start on criteria is for procps.conf is:
  start on virtual-filesystems

  This runs before some kernel modules are loaded, and procps applies
  the settings before they "exist", this is most noticed with network
  and network-related jobs (nfs, bridge).

  This bug may be considered a duplicate of LP Bug #690433.  I am
  opening a new one anyway, however because I think it's worth
  considering a more robust solution that would work for any possible
  kernel module.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/procps/+bug/771372/+subscriptions