logrotate configuration seems wrong

[Wayne Witzel]

When this was being implemented and during the review process the actual
size of the rotated files got adjusted. I tested very thoroughly the actual
rotation script and also tested the triggering of the rotation script.
Enough that I was happy landing the changes.

When the sizes were adjusted the same amount of diligence was not applied
in testing. This is what introduced the bug.

The size mismatch is problem. rsyslogd will stop logging to the file and
attempted to execute the script in the outchannel every time it gets a new
log message, which runs logrotate, which does nothing because the file
isn't big enough.

Here is the rsyslog docs for implementation of sized based rotation.
    http://www.rsyslog.com/doc/log_rotation_fix_size.html

I first attempted use the mv command, but with our rsyslogd configuration,
when I performed a mv, rsyslogd would stop logging to the file until I
actually did a reload on the process. Also this didn't easily support
things like compression or adding more archives if we decided to keep more
than one rotation around. So the choice to use logrotate is was made.

As for the actual act of reloading rsyslog. You can use copytruncate to
avoid it all together or you can use postrotate to reload rsyslogd. In both
cases there is a small window of possible data loss. Though with
copyandtruncate it is more likley to happen when the system is under load.
So changing to a postrotate that reloads rsyslogd is probably a good idea.

The rotate setting is just the number of files to keep around. I asked some
people after the size update if 1 rotation, a total of 1GB was enough to
keep around. Most people thought it was fine, since the original purpose of
the sized based long rotation was to address a ticket that was complaining
about the juju log folder taking up 3GB+ of space.

I will get a ticket created to address the size mismatch issue and switch
the copyandtruncate approach to a postrotate of rsyslogd.

Wayne

On Mon, Sep 15, 2014 at 5:30 AM, John Meinel <john at arbash-meinel.com> wrote:

> So we are using rsyslog.conf to have it figure out when rotation needs to
> be done with:
> # Maximum size for the log on this outchannel is 512MB
> # The command to execute when an outchannel as reached its size limit
> cannot accept any arguments
> # that is why we have created the helper script for executing logrotate.
> $outchannel
> logRotation,{{logDir}}/all-machines.log,512000000,{{logrotateHelperPath}}
>
> I would think that would not require SIGHUP along with having it run the
> script at our request.
>
> John
> =:->
>
> On Mon, Sep 15, 2014 at 12:13 PM, Stuart Bishop <
> stuart.bishop at canonical.com> wrote:
>
>> On 15 September 2014 12:38, John Meinel <john at arbash-meinel.com> wrote:
>>
>> > 7) "copytruncate" seems the wrong setting for interactive with rsyslog.
>> I
>> > believe rsyslog is already aware that the file needs to be rotated, and
>> thus
>>
>> It is only aware if you sent it a HUP signal.
>>
>> > it shouldn't be trying to write to the same file handle (and thus we
>> don't
>> > need to truncate in place). I'm not 100% sure on the interactions here,
>> but
>> > "copytruncate" seems to have an inherent likelyhood of dropping data
>> (while
>> > you are copying, if any data gets written then you'll miss those last
>> few
>> > bytes when you go to truncate, right?)
>>
>> --
>> Stuart Bishop <stuart.bishop at canonical.com>
>>
>
>
> --
> Juju-dev mailing list
> Juju-dev at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>


-- 
Wayne Witzel III
wayne.witzel at canonical.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20140915/ac7f03d0/attachment.html>