Some extensive system health monitoring

John Richard Moser nigelenki at
Mon Mar 7 22:10:53 CST 2005

Hash: SHA1

So I was considering the smartmon thread about making an icon to
indicate health, and system recovery tools, and whatnot, and came up
with a bigger idea than just "Your hard drive will fail soon."
Potentially, "You need more CPU," "Memory is low," "Security regressions
detected," "Users passwords cracked," etc.

Basically, I want a pluggable system to be created for the smartmontools
if that venue will be followed, rather than just an icon to warn about
imminent disk failure.  This system would then be expanded as time goes
on so that update-manager would be integrated into it (then depricated
as a separate utility), and other tasks would be added.

Other tasks could include simple things such as a johnd/cracklib based
plug-in which notifies the user when it discovers passwords can be
cracked; or potentially a monitor to watch drive usage and warn when 95%
of a drive is used, especially when / has less than 256MiB free.

More advanced plug-ins would include monitors to watch memory usage and
make suggestions to optimize disk cache by tuning swappiness or adding
more RAM, and warn about the imminent threat of the OOM killer when swap
and RAM get too full.  A CPU analyzer to notice when X11 apps (which
should be interactive realtime tasks) are spending way too much time
cranking 100% CPU could suggest a faster CPU, or possibly more RAM if
the disk cache is very very small.

Some other things could include security regression tests, such as
periodic paxtest and grsecurity regression tests for use if Ubuntu ever
adopts those systems.  Periodically testing during system run would be
important in case the internal kernel state was destroyed by a bug,
allowing some security system (SELinux, PaX, GrSecurity) to pass checks
it should fail or otherwise eb bypassed.

A firewall utility (see below for ideas) wouldn't be bad either for
this, as a nice plug-in in the future.  Using proper GPG signatures and
firewall "modules," it should be possible to supply a flexible,
maintained firewall and allow the user to also configure NAT, DHCP, and
IP connection tracking.

In the end, a single, central utility for the monitor and maintenance of
system health could be created to monitor hardware (CPU, disk failure,
disk usage, memory, even drivers losing track of hardware that WAS there
last boot); keep software up to date; handle security regression tests
on any future security enhancements and for existing systems like
passwords (john the ripper); apply firewall rules; and handle sysetm
configuration backup and restoration.

Aside from simply flashing the icon on health issues and security
hazards, the interface should allow for categorical separation of the
context menu for the task tray program to allow for scailability in the
predicted volume of plug-ins.  Potentially:

Health      > Disk failure (S.M.A.R.T.)
              Disk usage
              Network availability
              Software updates
Performance > CPU usage statistics
              Memory usage statistics
              Disk cache size
              Network throughput
Backup      > System configuration backup
Security    > Security updates
              Password cracking
              PaX regressions
              GrSecurity regressions
              SeLinux regressions
              Stack Smash Protection
              PIE and relocation statistics
              Firewall managment

You get the idea.  The major concern is doing all this "in one place"
without clutter.  Most of this is just using existing tools (regression
tests, John the Ripper, smartmontools etc) and gathering data from /proc
(memory, CPU, disk usage, network throughput) with a pretty interface
(which must of course be coded during many laboreous hours of hacking).

System Health Indicator Terminal
(get me a better name)

Monitors should be individual plug-ins; you'll notice some of them
are specific for certain systems, like i.e. PaX tests!

Note that tests run "at boot" mean run the tests FOR THE BOOT.  Don't
bother init with actually waiting; just make init start the background
daemon, which will go about its business running tests in idle time
etc etc etc.  Passive tests are also PASSIVE, in the background at
idle time when we really really know nothing else is going on!  We
want to monitor system health and enhance the user's experience, not
extend boot time and lag the system.

 - Task tray icon for easy access
 - Warnings when system is unhealthy
 - Pluggable monitors for easy expansion and paced development

 + System Configuration Recovery
  - Check system configuration in /etc at each successful boot and
    make a backup tarball
  - Allow separate backup tarball of /etc on demand, isolated from
    automated backups
  - Provide for restoration of backup tarball during install and at
    any arbitrary point
   - Recovery should allow for four types of recovery:
    - Replace existing files in /etc with matching files from backup
    - Restore or augment entire /etc structure with backup
    - Restore individual /etc functions such as init.d scripts, rc.d
      runlevels, SQUID configuration, profile, authentication (passwd,
      shadow, group), apt, "Unknown configurations," etc
    - Restore individual selected files
 + S.M.A.R.T. monitoring using smartmontools
  - Warn when disks are faulty and will fail soon
 + Memory size
  - Warn when too much swap is used
   - Subtract off total size of files on tmpfs mounts in calculation
  - Warn when disk cache drops below X% (possibly 25%) of memory
   - Suggest more RAM
   - Suggest increasing swappiness if swap is relatively unused
  - Warn when too much total available memory is used
   - Combine swap and ram for total available memory
   - Ignore any disk cache over 25% of physical RAM in usage
   - Warn when 95% limit reached
    - Warning message should briefly explain OOM killer:
      "If memory usage reaches 100%, tasks the OS thinks are unimportant
       will be terminated automatically.  No opportunity to save your
       work will be given!"
   - Suggest terminating tasks with large RSS
   - Suggest more memory
    - Note that more swap, swapfiles, and swapd may be used; but that
      these solutions may cause excessive system slowdown
  - Warn about particular tasks utilizing a great percentage more
    memory than when they started (difficult!  Tasks all have different
 + CPU audit
  - Audit lengths of high CPU usage bursts
  - Allow tracking of which programs use large amounts of CPU for
    extended periods, and how long
  - When many programs (i.e. not just SETI or gcc, but esp. anything
    linked to Xlibs, if through GTK+ or Qt or whatnot) cause
    particularly long CPU spikes, i.e. >10S, suggest faster CPU
   - Again, this is a difficult task, as some programs should use lots
     of CPU
 + Security
  - Password testing
   - Have John hack passwords passively
   - Warn about weak passwords being cracked
   - Information about exact user account is privileged information!
     Only show to root!
  - PaX tests
   - PaX test once at boot as root and as normal user
   - PaX test once every 24 hours again to detect abnormal kernel states
   - Warn about abnormal failures
    - Possibly use ProPolice with paxtest to finish off tests PaX does
      NOT cover (don't warn about those tests, i.e. ret2library)
   - Passively scan system at idle time for libraries and executables
     with relocations
    - Allow review in the console
    - Allow specific active scanning of system or individual binaries
   - Passively scan system at idle time for ELF ET_EXEC executables
    - Allow review in console
    - Allow specific active scans
  - ProPolice tests
   - Test a ProPolice regression test suite once at boot
   - Warn about failure
   - Passively scan system at idle time for libraries and executables
     without reference to __guard and __stack_smash_handler
    - Allow review in console
    - Allow specific active scans
  - GrSecurity regression tests
   - Use a GrSecurity regression suite to do tests at each boot
   - Warn user of abnormal failures
   - Run tests that need root as root
   - Run tests that can be tested as user as user AND root
  - Security related updates
   - Warn when security related software updates are available (how I
     don't know)
   - Allow running of update manager
  - Firewall
   - Allow remote firewall rule "modules" to be fetched to construct a
     firewall of stock options (REQUIRE SIGNATURE)
   - Allow on-site configuration of IP masquerading, routing, port
     forwarding, and IP connection tracking
   - Notify when firewall rule modules are updated and ask the user if
     he wishes to update the firewall
 + Software managment
  - Update notifier functionality becomes integrated
  - Easy access to synaptic, or simply integrate Synaptic

- --
All content of all messages exchanged herein are left in the
Public Domain, unless otherwise explicitly stated.

    Creative brains are a valuable, limited resource. They shouldn't be
    wasted on re-inventing the wheel when there are so many fascinating
    new problems waiting out there.
                                                 -- Eric Steven Raymond
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird -


More information about the ubuntu-devel mailing list