Framebuffers, plymouth, upstart and server installs.

[Mark - Syminet]

On Jan 8, 2013, at 1:57 PM, James Hunt <james.hunt at ubuntu.com> wrote:  

[…] 

>> 
>>> The worst one here is the fact that during bootup fsck's, there is no
>>> longer any progress bar showing status.  No big deal for a small
>>> single disk system but If your server has a multi-terabyte drive
>>> array, fscks can take hours to complete
> Can you provide more details on this? Plymouth should display fsck progress
> which are passed to it via mountall. There is even a special "fsck" message in
> the Ubuntu theme to handle displaying such data [1]. If this is no longer
> working for your particular scenario, please raise a bug so we can look at it.
> 

AFAICT there is no progress bar for anyone during bootup fscks since Karmic.  
Bugs have already been filed, some of them ongoing for over three years now: 

https://bugs.launchpad.net/ubuntu/+source/plymouth/+bug/540645 

https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/446596 

Googling "fsck progress ubuntu" gives tons of threads too - this one sums it 
up well: http://ubuntu.5.n6.nabble.com/oneiric-progress-status-during-boot-time-fsck-td4832484.html 

> I'm sure you understand that 'hacking plymouth out of your system' is not a
> recommended course of action as Plymouth is _the_ boot and shutdown splash
> facility used by Ubuntu (and other distributions of course!) 
> 

Yes, understood (and why) - but it appears upstart's backward compatibility 
with sysv has turned out to be a bit deceptive for many folks.  People are blaming 
plymouth, "that graphical crap" without realizing the *real* cause is that 
sysv init has been completely removed from ubuntu since Karmic.  Nobody 
running ubuntu is running sysv init anymore.  Things *look* business as usual 
due to upstart's backward compatibility layer, but they are using upstart and the 
sysvinit package was deleted from ubuntu back in 2009.  

I know upstart is used in Fedora and a few others, but notably *not* in Debian,
which is still on sysvinit and gently pushing people from sysv to dependency based 
boot for now (via warnings during upgrade and pointers to documentation how to 
cleanly migrate *when you are ready*).  This (for now) does seem to be a much 
more careful/server friendly approach.  

>>>  any solution is going to
>>> have to work best practice with upstart/plymouth, with no way to get
>>> back old behavior.
> What do you mean exactly by 'old behaviour'? Upstart has been used in Ubuntu
> since Edge (6.10).
> 

I mean pre-upstart behavior: plain old ugly VGA 80x25 text console, the usual sysv 
output we had before upstart (including our precious fsck progress bar).  Also we 
used to be able to ctrl+c out of say, apache's startup during boot - we will be able 
to do this when apache gets converted to an upstart script?  

Upstart may have been optionally available since Edge, but it's my understanding 
it became mandatory with Karmic when the sysvinit package was removed from 
Ubuntu.  So really, most people weren't running upstart until Karmic (and even 
then, many weren't even aware that sysv init was gone).  

>> Since 'old style' init scripts can still talk to the text console, why
>> wouldn't upstart be able to do so?
> It can. simply specify 'console output' to have a job output data direct to the
> console (via Plymouth) [2]. This is how you see the SysV output currently - take
> a look at /etc/init/rc.conf and you'll see 'console output' specified.
> 

Plymouth's details theme output has improved a lot since Karmic, but it appears 
that mountall makes a special call to fsck (since the -C option takes a file descriptor), 
so its progress bar isn't getting through. Looking at /etc/init/mountall.conf we 
see this: 

# temporary, until we have progress indication
# and output capture (next week :p)
console output 

…this works for fsck output, *except* for the progress bar.  

A couple of years ago I went ahead and downloaded the mountall package 
sources and changed this line in mountall.c : 

arg = NIH_MUST (nih_sprintf (NULL, "-C%d", fds[1]));

…to this: 

arg = NIH_MUST (nih_sprintf (NULL, "-C", fds[1]));

…this brought back the fsck progress bars.  I extended it a bit to implement a 
"FSCKCONSOLE=yes" option to /etc/default/rcS so that it would be easy 
to switch between behaviors, and packaged it up as a ppa.  This was mentioned 
somewhere in the bugreports but someone pointed out that unfortunately this 
won't work, because it is sending output to the console - someone needs to 
update things to talk to the plymouth details plugin.  

Is this not true?  Will the above change work?  I did this on a vanilla lucid 
system; plymouth was not tinkered with in any way and it showed up.   

> As of version 1.4, Upstart will, by default, log all job data to files under
> /var/log/upstart/. 
> 

This is good, but what people are complaining about are boot-time processes; 
if we can login to look at logs, then it is already up.  What some refer to as "admin time" -  
that is, the time between when you enter the reboot command … and the server 
comes back up on the network.  Since the servers can be thousands of miles away, 
only a sysadmin can attest to that feeling while you are waiting for it to come back up.  
With a server dist, the expectation has to be that there is no one there to provide any 
input; it should expect to come up unattended no matter what, as quickly as possible.  
Bad fstab lines etc. it should just ignore so long as the rootfs is accessible.  

> Meantime, since few jobs actually explicity specify the 'console' stanza, what
> you can do is add '--default-console=output' to your kernel command-line which
> will send all job output to the console for those jobs which have not explicitly
> specified 'console'.
> 

Sounds like this should be the default for any server dist.  

Best - 

-- 
Mark