<div dir="ltr"><div>This situation is now resolved with thanks to Roger, Gustavo and others in real time. There is no way we could have resolved it ourselves since there was corruption of the juju database caused by running out of disk space, which was unfortunate. We as a team were not aware that it is necessary to keep a backup of the juju database.<br>
<br>Unfortunately things did not go very smoothly even with the help of core developers, upgrading our agents from 1.13 was difficult and we had a cascade of failures from there, partly due to changes we had made outside of juju.<br>
<br></div><div>Anyway, we over at scraperwiki seriously appreciate the help you guys gave, so thanks again all.<br><br>Regards,<br><br>- Peter<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 30 November 2013 01:37, Peter Waller <span dir="ltr"><<a href="mailto:peter@scraperwiki.com" target="_blank">peter@scraperwiki.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>It appears that over the last few hours the logs grew a bit but the /var/lib/{juju,mongodb} directories grew and are now occupying a lot of disk space (5.5GB), which is now causing problems. In particular /var/lib/mongodb/journal/* is taking 3.1GB in three prealloc.N files and /var/lib/juju/db is taking up a lot of spaces across ~8 juju.N files.<br>
<br></div>Can I move these else where and/or delete them? What are the consequences of this? Has logging information which has found its way into the database?<br></div><div><div><div class="gmail_extra">
<br><br><div class="gmail_quote">
On 29 November 2013 17:38, Peter Waller <span dir="ltr"><<a href="mailto:peter@scraperwiki.com" target="_blank">peter@scraperwiki.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>I'm still experiencing rapid log growth so any advice on how to prevent it would be appreciated. I guess as a quick short term solution over the weekend I could stop the juju daemons but this doesn't seem ideal.</div>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On 29 November 2013 16:13, Peter Waller <span dir="ltr"><<a href="mailto:peter@scraperwiki.com" target="_blank">peter@scraperwiki.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">For the record I sent a more complete but redacted log to Roger.<br></div><div class="gmail_extra"><br>
<br>
<div class="gmail_quote"><div>On 29 November 2013 16:02, Peter Waller <span dir="ltr"><<a href="mailto:peter@scraperwiki.com" target="_blank">peter@scraperwiki.com</a>></span> wrote:<br>
</div><div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I've not supplied the full output because it contains private keys, but here is the bit that I think is relevant. Please advise if you need more.<br>
<br>2013-11-29 16:01:20 DEBUG juju.environs.simplestreams simplestreams.go:577 candidate matches for products ["com.ubuntu.juju:12.04:amd64" "com.ubuntu.juju:12.04:i386" "com.ubuntu.juju:12.04:arm" "com.ubuntu.juju:12.10:amd64" "com.ubuntu.juju:12.10:i386" "com.ubuntu.juju:12.10:arm" "com.ubuntu.juju:13.04:amd64" "com.ubuntu.juju:13.04:i386" "com.ubuntu.juju:13.04:arm" "com.ubuntu.juju:13.10:amd64" "com.ubuntu.juju:13.10:i386" "com.ubuntu.juju:13.10:arm" "com.ubuntu.juju:14.04:amd64" "com.ubuntu.juju:14.04:i386" "com.ubuntu.juju:14.04:arm"] are [{Thu, 07 Nov 2013 10:36:03 -0500 products:1.0 content-download [] streams/v1/com.ubuntu.juju:released:tools.json [com.ubuntu.juju:12.04:amd64 com.ubuntu.juju:12.04:armhf com.ubuntu.juju:12.04:i386 com.ubuntu.juju:12.10:amd64 com.ubuntu.juju:12.10:i386 com.ubuntu.juju:13.04:amd64 com.ubuntu.juju:13.04:i386 com.ubuntu.juju:13.10:amd64 com.ubuntu.juju:13.10:armhf com.ubuntu.juju:13.10:i386 com.ubuntu.juju:14.04:amd64 com.ubuntu.juju:14.04:armhf com.ubuntu.juju:14.04:i386]}]<br>
2013-11-29 16:01:20 DEBUG juju.environs.simplestreams simplestreams.go:889 finding products at path "streams/v1/com.ubuntu.juju:released:tools.json"<br>2013-11-29 16:01:21 ERROR juju.tools list.go:113 cannot match tools.Filter{Released:false, Number:version.Number{Major:1, Minor:14, Patch:1, Build:0}, Series:"", Arch:""}<br>
2013-11-29 16:01:21 ERROR juju supercommand.go:282 no matching tools available<br><br></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On 29 November 2013 16:00, roger peppe <span dir="ltr"><<a href="mailto:roger.peppe@canonical.com" target="_blank">roger.peppe@canonical.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Oh, that's odd; 1.14.1 was definitely found by your tools earlier.<br>
<br>
What does:<br>
<br>
juju upgrade-juju --version 1.14.1 --debug<br>
<br>
print?<br>
<div><div><br>
On 29 November 2013 15:57, Peter Waller <<a href="mailto:peter@scraperwiki.com" target="_blank">peter@scraperwiki.com</a>> wrote:<br>
> On 29 November 2013 15:51, roger peppe <<a href="mailto:rogpeppe@gmail.com" target="_blank">rogpeppe@gmail.com</a>> wrote:<br>
>><br>
>> > What's the best way to proceed now to upgrade them?<br>
>><br>
>> Let's try this:<br>
>><br>
>> juju upgrade-juju --version 1.14.1<br>
><br>
><br>
> $ juju upgrade-juju --version 1.14.1<br>
> ERROR cannot match tools.Filter{Released:false,<br>
> Number:version.Number{Major:1, Minor:14, Patch:1, Build:0}, Series:"",<br>
> Arch:""}<br>
> ERROR no matching tools available<br>
><br>
> $ juju version<br>
> 1.16.3-precise-amd64<br>
><br>
> Didn't seem to have any effect.<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>