L-o-n-g delay for rc.local in systemd on Ubuntu.

[Ken D'Ambrosio]

OK.  Turned out that the problem with my rc.local wasn't *with* my 
rc.local, per se, but rather with the scripts it was invoking.  Which 
were written in Ruby -- a language I love.  When I finally got around to 
really digging in, I saw that it was hanging not on network stuff the 
script was doing (my first expectation, once I realized that the script 
was at fault), but the initial invocation of the interpreter.  E.g., if 
I logged in to the machine quickly, and typed this:
ruby -e 'print 1'
it would take *three to five minutes* to execute.  Using "strace" helped 
me figure out that it was blocking on getrandom().  Apparently, as 
someone on my LUG mailing list noted, Ruby does "eager RNG 
initialization" (as opposed to Python and Perl, which do the "lazy" 
variant).  And a newly launched VM -- with no video card, audio card, 
keyboard, or mouse for entropy sources -- has to largely rely on network 
traffic... of which there's very little.  (See "newly launched.")  
Catch-22.  (Note that this was probably a conscious design decision on 
the part of the Ruby crew; there are downsides and upsides to both 
approaches.)

Turns out, however, that there's a program to solve this, too: haveged.  
It's a daemon that executes various code and looks for deltas in 
execution times, using that as a source of entropy.  My initial 
execution script execution times went from three-to-five minutes to 
about a second.  (Subsequent execution times had always been fast -- the 
random number generators had been seeded with entropy by then.)

Muuuuuuuuuuch better.

-Ken

P.S.  Lest this e-mail beget a /dev/random vs. /dev/urandom debate, 
apparently, getrandom() (as per its manpage) goes for /dev/urandom -- 
which generally doesn't block, except when it hasn't yet been 
initialized.  (As opposed to /dev/random, which blocks when its entropy 
pool empties.)


On 2017-08-09 06:39, Karl Auer wrote:
> On Wed, 2017-08-09 at 12:21 +0200, Ralf Mardorf wrote:
>> On Wed, 09 Aug 2017 10:22:06 +0200, Xen wrote:
>> > I don't know why something would take forever just because it is
>> > in rc.local.
>> There are two more likely possibilities.
>> 
>> 1. A race conditions related to the startup process
>> 2. An issue that isn't related to the startup process at all
> 
> In my experience (unscientific) loooong delays in startup are almost
> always caused by one of two things - a dying disk, or a network
> problem.
> 
> Regards, K.
> 
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Karl Auer (kauer at biplane.com.au)
> http://www.biplane.com.au/kauer
> http://twitter.com/kauer389
> 
> GPG fingerprint: A52E F6B9 708B 51C4 85E6 1634 0571 ADF9 3C1C 6A3A
> Old fingerprint: E00D 64ED 9C6A 8605 21E0 0ED0 EE64 2BEE CBCB C38B