logrotate configuration seems wrong

[John Meinel]

Looking again, I wonder if this is a 1024 vs 1000 bug. Specifically
rsyslog's configurations says:
$outchannel
logRotation,{{logDir}}/all-machines.log,512000000,{{logrotateHelperPath}}

and the logrotate configuration says:
/var/log/juju/all-machines.log {
    size 512M
...
}

I'm wondering if logrotate is running and saying 512,000,006 <
512*1024*1024 (536,870,912), and not rotating anything, and rsyslog is
refusing to write since the file is too big.

Regardless, I'm surprised to see it just-not-working in practice, as I
would have thought someone would have actually tested this before landing
it.

John
=:->

On Mon, Sep 15, 2014 at 9:38 AM, John Meinel <john at arbash-meinel.com> wrote:

> Going further on this, as I'm discovering new oddities:
>
> 1) ll /var/log/juju
> -rw------- 1 syslog adm    512000006 Sep 15 05:16 all-machines.log
>
> Seems surprising that it is 5120000* bytes long.
>
> 2) tail -f /var/log/juju/all-machines.log
> nothing happening
>
> 3) I see more data coming into /var/log/juju/machine-0.log, why isn't it
> ending up in all-machines.log
>
> 4) Given that we are using 2 different rotation mechanisms, did we test
> that messages that were just written to machine-0.log still get copied to
> all-machines.log when machine-0.log ends up getting rotated?
>
> 5) I'm pretty sure we didn't, because clearly we didn't test that
> all-machines.log actually gets any more data once it 'fills' up.
>
> 6) Note that I do still see rsyslog running and it is using a fair amount
> of CPU, so it is still getting messages from other units, etc. machine-0
> agent and mongod are both quite active as well.
>
> 7) "copytruncate" seems the wrong setting for interactive with rsyslog. I
> believe rsyslog is already aware that the file needs to be rotated, and
> thus it shouldn't be trying to write to the same file handle (and thus we
> don't need to truncate in place). I'm not 100% sure on the interactions
> here, but "copytruncate" seems to have an inherent likelyhood of dropping
> data (while you are copying, if any data gets written then you'll miss
> those last few bytes when you go to truncate, right?)
>
> I'm happy to see logrotation being added, but it seems quite half-baked in
> our current trunk.
>
> Am I just doing something wrong? Did someone actually test all of this out
> and it was working for them?
>
> John
> =:->
>
> On Mon, Sep 15, 2014 at 9:13 AM, John Meinel <john at arbash-meinel.com>
> wrote:
>
>> So I was testing scaling today which generally generates huge volumes of
>> logging (I actually wanted to keep it because I used it for seeing how
>> everything went, but I understand why we are rotating the logs.)
>>
>> However, I found this as it was running:
>> # ls -sh /var/log/juju
>> total 909M
>> 323M all-machines.log
>> 4.0K ca-cert.pem
>> 4.0K logrotate.conf
>> 4.0K logrotate.run
>> 301M machine-0-2014-09-15T05-02-27.486.log
>> 301M machine-0-2014-09-15T05-06-53.779.log
>>  80M machine-0.log
>> 4.0K rsyslog-cert.pem
>> 4.0K rsyslog-key.pem
>>
>> Notice that there is only 1 all-machines.log that is 300MB in size, while
>> there are 2 machine-0 logs.
>>
>> And when I track down the various configuration files, I find
>> # cat logrotate.conf
>>
>> /var/log/juju/all-machines.log {
>>     size 512M
>>     # don't move, but copy-and-truncate so the application won't have to
>> be
>>     # told that the file has moved.
>>     copytruncate
>>     # maximum of one old file
>>     rotate 1
>>     # counting old files starts at 1 rather than 0
>>     start 1
>>     # use compression
>>     compress
>> }
>>
>>
>> I have the feeling that someone didn't realize "rotate 1" means only keep
>> the original log file. As in, there are *no* backup files.
>>
>> Did the person who implemented this actually test it?
>>
>> Did we ever fix things so that "juju debug-log" doesn't become
>> immediately useless once you reach the rotate threshold (that it can look
>> in the backup log files)?
>>
>> I can understand not fixing debug-log, but I'm a bit surprised that our
>> idea of "all-machines.log needs to be rotated" became "all-machines.log
>> needs to be truncated".
>>
>> John
>> =:->
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20140915/3a51c686/attachment.html>