<div dir="ltr"><div>Hmm. Answering my own question: no, the webapps stuff occurs also on other days that do not have a system failure. I still haven't found anything that sticks out as happening at the start of a freeze-up. And the first lockup message is so close to the restart that I doubt it's coincidence. Very hard to say, because when the system has failed, all the screens are blank, and won't come alive until reboot.<br><br></div>About all I can see to do is to stop the screens from blanking and hope there's some clue on the workspace I happen to be on when it fails.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 16, 2017 at 8:55 AM, Kevin O'Gorman <span dir="ltr"><<a href="mailto:kogorman@gmail.com" target="_blank">kogorman@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>The problem is it's kind of overwhelming. At 4MB the log is over 1000x the normal size. The previous day's log is 10x normal. The failure happened unattended, so I'm not even sure when it happened.<br></div>It would sure help if I knew in advance what the first stages of a reboot looks like.<br></div><div><br></div><div>Just making some guesses, I grepped for "lockup" on unzipped logs: "grep -i lockup saved-*" and got 662 lines<br></div><div>The first looked the most severe:<br><div style="margin-left:40px">saved.syslog.1:Feb 15 16:48:51 plato-x kernel: [1377271.765054] NMI watchdog: Watchdog detected hard LOCKUP on cpu 1<br></div>Oddly, it was followed by the rest of those 662 that looked more less like the next one and the last one:<br><div style="margin-left:40px">saved.syslog.1:Feb 15 16:55:57 plato-x kernel: [1377749.441524] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 23s! [systemd-timesyn:1338]<br></div><div style="margin-left:40px">saved.syslog.1:Feb 15 18:18:05 plato-x kernel: [1382677.646422] NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [chromium-browse:19520]<br></div>with a considerable variety of daemons implicated, none of them the 32 processes that are the real workload, and which run at near 100% CPU according to top(1)<br><br></div>But then, I think this may have happened during the bootup, not the failure. That's because after silence from 18:18:05 to 18:25:51 there's<br><div style="margin-left:40px">Feb 15 18:25:51 plato-x rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="1409" x-info="<a href="http://www.rsyslog.com" target="_blank">http://www.rsyslog.com</a><wbr>"] start<br></div>and very soon there's a log entry that looks very much like startup:<br></div><div style="margin-left:40px">Feb 15 18:25:51 plato-x systemd[1]: Activating swap Swap Partition...<br></div>It's surrounded by a lot of stuff I don't understand, including many occurrences of the phrase "failed with exit code" which really makes me wonder<br><br></div>Anyway. before the first lockup line, things looked very quiet, as you might expect for an unattended system, except for some oddities that went on for a long time but ended about 10 hours earlier, here's the complete sequence from the last few of 09:33:59 up to the first lockup message:<br><div style="margin-left:40px">Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-livemail<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-mail-ru<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-newsblur<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-pandora<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-qml-autopilot<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** (process:22931): WARNING **: Found newline before all of met<br>adata, skipping app<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-qq-mail<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-reddit<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-tumblr<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-twitter<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-vkcom<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-wordpress<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-wordpress-com<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-yahoomail<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-yahoonews<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-yandex-music<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-yandexmail<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-yandexmusic<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-yandexnews<br>Feb 15 09:33:59 plato-x com.canonical.Unity.Webapps.<wbr>Service[2265]: ** Message: Investigating webapp: unity-webapps-youtube<br>Feb 15 09:43:34 plato-x systemd[1]: snapd.refresh.timer: Adding 56min 56.771453s random time.<br>Feb 15 10:10:17 plato-x dhclient[30629]: DHCPREQUEST of 192.168.1.172 on enp6s0 to 192.168.1.1 port 67 (xid=0x6d95b0a3)<br>Feb 15 10:10:17 plato-x dhclient[30629]: DHCPACK of 192.168.1.172 from 192.168.1.1<br>Feb 15 10:10:17 plato-x NetworkManager[1493]: <info> [1487182217.6309] address 192.168.1.172<br>Feb 15 10:10:17 plato-x NetworkManager[1493]: <info> [1487182217.6310] plen 24 (255.255.255.0)<br>Feb 15 10:10:17 plato-x NetworkManager[1493]: <info> [1487182217.6310] gateway 192.168.1.1<br>Feb 15 10:10:17 plato-x NetworkManager[1493]: <info> [1487182217.6310] server identifier 192.168.1.1<br>Feb 15 10:10:17 plato-x dbus[1459]: [system] Activating via systemd: service name='org.freedesktop.nm_<wbr>dispatcher' unit='dbus-org.f<br>reedesktop.nm-dispatcher.<wbr>service'<br>Feb 15 10:10:17 plato-x NetworkManager[1493]: <info> [1487182217.6310] lease time 86400<br>Feb 15 10:10:17 plato-x NetworkManager[1493]: <info> [1487182217.6310] hostname 'plato-x'<br>Feb 15 10:10:17 plato-x NetworkManager[1493]: <info> [1487182217.6311] nameserver '192.168.1.1'<br>Feb 15 10:10:17 plato-x NetworkManager[1493]: <info> [1487182217.6311] dhcp4 (enp6s0): state changed bound -> bound<br>Feb 15 10:10:17 plato-x dhclient[30629]: bound to 192.168.1.172 -- renewal in 29017 seconds.<br>Feb 15 10:10:17 plato-x systemd[1]: Starting Network Manager Script Dispatcher Service...<br>Feb 15 10:10:17 plato-x dbus[1459]: [system] Successfully activated service 'org.freedesktop.nm_<wbr>dispatcher'<br>Feb 15 10:10:17 plato-x nm-dispatcher: req:1 'dhcp4-change' [enp6s0]: new request (1 scripts)<br>Feb 15 10:10:17 plato-x nm-dispatcher: req:1 'dhcp4-change' [enp6s0]: start running ordered scripts...<br>Feb 15 10:10:17 plato-x systemd[1]: Started Network Manager Script Dispatcher Service.<br>Feb 15 10:17:01 plato-x CRON[24916]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)<br>Feb 15 11:17:01 plato-x CRON[27608]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)<br>Feb 15 11:54:34 plato-x systemd[1]: Starting Daily apt activities...<br>Feb 15 11:56:56 plato-x systemd[1]: snapd.refresh.timer: Adding 5h 21min 51.717429s random time.<br>Feb 15 11:57:31 plato-x systemd[1]: Started Daily apt activities.<br>Feb 15 11:57:31 plato-x systemd[1]: apt-daily.timer: Adding 6h 3min 22.557945s random time.<br>Feb 15 11:57:31 plato-x systemd[1]: apt-daily.timer: Adding 9h 28min 33.356115s random time.<br>Feb 15 12:17:01 plato-x CRON[30798]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)<br>Feb 15 13:17:01 plato-x CRON[1056]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)<br>Feb 15 13:30:44 plato-x systemd-timesyncd[1338]: Timed out waiting for reply from <a href="http://91.189.89.199:123" target="_blank">91.189.89.199:123</a> (<a href="http://ntp.ubuntu.com" target="_blank">ntp.ubuntu.com</a>).<br>Feb 15 13:30:44 plato-x systemd-timesyncd[1338]: Synchronized to time server <a href="http://91.189.91.157:123" target="_blank">91.189.91.157:123</a> (<a href="http://ntp.ubuntu.com" target="_blank">ntp.ubuntu.com</a>).<br>Feb 15 14:17:01 plato-x CRON[4111]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)<br>Feb 15 15:17:01 plato-x CRON[6860]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)<br>Feb 15 16:17:01 plato-x CRON[9648]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)<br>Feb 15 16:48:51 plato-x kernel: [1377271.765054] NMI watchdog: Watchdog detected hard LOCKUP on cpu 1<br></div>So could these com.canonical.Unity.Webapps.<wbr>Service lines be related to a failure of the user interface, which then went quiet until I got home and around 4 PM noticed the machine was frozen, and all the CPU lockups are attempts to respond to my REISUB?<br><br></div>If so is there anything I can do about this?<br><div><div><div><br><div><br><br><div><div><div><div><br></div></div></div></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Thu, Feb 16, 2017 at 12:39 AM, Colin Law <span dir="ltr"><<a href="mailto:clanlaw@gmail.com" target="_blank">clanlaw@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On 16 February 2017 at 02:48, Kevin O'Gorman <span dir="ltr"><<a href="mailto:kogorman@gmail.com" target="_blank">kogorman@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>I've got a big machine running Ubuntu that's I'd prefer kept running all the time. I don't always get what I want, of course...<br><br></div>Today, for the second time in about three weeks, it completely froze. Fortunately, I kept a printed copy of <a href="http://blog.kember.net/articles/reisub-the-gentle-linux-restart/" target="_blank">http://blog.kember.net/article<wbr>s/reisub-the-gentle-linux-rest<wbr>art/</a> in a folder with assorted HOWTO-type articles, and was able to restart it that way. Which, by the way, proves it was still alive, at least sort of.<br><br></div>Simply put, I held down Alt and SysRq and slowly typed R E I S U B. Several times, because there's no feedback and a machine with 256GB or RAM can take a while to dump buffers, but I could not actually tell if it was doing anything at all until the boot sequence started.<br><br></div>My question is this: is there any way to investigate what caused this? I'll try looking in logs, but I am unfamiliar with them and don't really know what to look for.<span class="m_1323899562785991417m_7271438951082997684HOEnZb"><font color="#888888"><br clear="all"></font></span></div></blockquote><div><br></div></span><div>The first thing is to look in /var/log/syslog and see what is there at the point of failure.<span class="m_1323899562785991417HOEnZb"><font color="#888888"><br><br></font></span></div><span class="m_1323899562785991417HOEnZb"><font color="#888888"><div>Colin<br></div><br></font></span></div></div></div>
<br></div></div><span class="HOEnZb"><font color="#888888">--<br>
ubuntu-users mailing list<br>
<a href="mailto:ubuntu-users@lists.ubuntu.com" target="_blank">ubuntu-users@lists.ubuntu.com</a><br>
Modify settings or unsubscribe at: <a href="https://lists.ubuntu.com/mailman/listinfo/ubuntu-users" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailm<wbr>an/listinfo/ubuntu-users</a><br>
<br></font></span></blockquote></div><span class=""><br><br clear="all"><br>-- <br><div class="m_1323899562785991417gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Kevin O'Gorman<br></div>#define QUESTION ((bb) || (!bb)) /* Shakespeare */<br><br><div><span style="line-height:normal;font-variant:normal;font-size:10pt;font-style:normal;font-weight:normal"><span style="line-height:normal;font-variant:normal;font-size:10pt;font-style:normal;font-weight:normal"></span></span><table width="448" cellspacing="0" cellpadding="0" border="0"><tbody><tr><td width="25"><img src="cid:XVHDKDFDBURW.IMAGE_60.gif" width="25" height="21"></td>
<td width="423"><span style="FONT-FAMILY:Verdana,Geneva,sans-serif;COLOR:rgb(0,153,0);MARGIN-LEFT:5px;FONT-SIZE:10px">Please consider the environment before printing this email.</span></td></tr></tbody></table><br></div></div></div>
</span></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Kevin O'Gorman<br></div>#define QUESTION ((bb) || (!bb)) /* Shakespeare */<br><br><div><span style="line-height:normal;font-variant:normal;font-size:10pt;font-style:normal;font-weight:normal"><span style="line-height:normal;font-variant:normal;font-size:10pt;font-style:normal;font-weight:normal"></span></span><table border="0" cellpadding="0" cellspacing="0" width="448"><tbody><tr><td width="25"><img src="cid:XVHDKDFDBURW.IMAGE_60.gif" height="21" width="25"></td>
<td width="423"><span style="FONT-FAMILY:Verdana,Geneva,sans-serif;COLOR:rgb(0,153,0);MARGIN-LEFT:5px;FONT-SIZE:10px">Please consider the environment before printing this email.</span></td></tr></tbody></table><br></div></div></div>
</div>