getting rid of all-machines.log

Fri Aug 15 14:31:24 UTC 2014

On 15.08.2014 16:35, Kapil Thangavelu wrote:

On Fri, Aug 15, 2014 at 7:36 AM, Gabriel Samfira <gsamfira at cloudbasesolutions.com<mailto:gsamfira at cloudbasesolutions.com>> wrote:
I think this thread has become a bit lengthy, and we have started to loose perspective on what we are actually trying to accomplish.

agreed. afaics that's how do we support logging on windows.

Gustavo's idea to save the logs to is awesome and it works across platforms, allows immense flexibility and would give people a powerful tool. We should deffinately aspire to get that done sooner rather then later. However, at this point in time its only an idea, without a clear blueprint.

agreed.

What Nate is proposing *already exists*, its tangible, proposed as a PR and improves the way juju handles logs.

you'll have to be more specific, there's been a shotgun of statements in this thread, touching on logstash, aggregation removal, rsyslog removal, log rotation, deferring to stderr/stdout, 12factor apps, working with ha state servers, etc.

I was referring to Nate's lumberjack package (PR seams to be gone) and the syslog streaming change. Lumberjack works on windows as well. While the rest have been mentioned, they are just ideas that may or may not be implemented in the future. I don't think Nate started this thread with the intent of prioritising this *now*. From what I can tell it was just a thought that was meant to start a discussion to implement something better in the *future*.

So far Gustavo's idea seams like the best choice with the most benefits in the long term. I would take that from this whole thread, and mark the case of log aggregation *closed* (of course pending a blueprint).

The only thing I see missing, that might ease people's minds is a --log-file option (that works with --debug) to actually enforce the usage of a log file. If we omit that option, then juju should just log to stdout/stderr. So we get to keep what we have, but also solve a huge PITA on Windows or any other platform that have limitations in this respect, with a minimal change...

afaics your referencing your branch (https://github.com/gabriel-samfira/syslog) which will directly send logs to a remote aggregating syslog server on windows nodes. with regard to the default service behavior on windows, where does stdout/stderr go for a service? isn't the expected behavior for a windows service to use the event log facility? ie. to be inline with expected windows behavior and extant juju semantics, shouldn't we have multiple handlers on the log facility, one to rsyslog, and one to event log on windows.

While the initial scope of this thread was to see if we really do need to save an all-machines.log via syslog (and apparently the CI needs it) instead of using a better tool for this particular job (like logstash...not not necessarily that), it has gone off into a bunch of tangents, one of which is using lumberjack to write the logs and rotate them instead of having upstart write them via redirect, and then logrotate rotate and archive them(2 external and platform specific dependencies as opposed to one cross platform implementation).

Windows services have no way of capturing stdout/stderr in the same way upstart does (solved by using Nate's lumberjack). It is simply lost, or the service will fail because it cannot write to stdout/stderr (there is no console present).

The syslog change you mentioned is just half of the solution, and will allow the windows version to log remotely and be in-line with the ubuntu version without needing to install 3rd party applications. That was my initial goal. The fact that it removes rsyslog as a dependency from non-bootstrap nodes on Ubuntu as well, is in my opinion a bonus :). You get to remove an external dependency which you no longer have to configure separately, and we finally do ssl certificate checking of the rsyslog server on the state machine (right now the way the forwarders are configured, will ignore the SSL errors).

As for logging to event log, that is certainly a good option, and I will look into it.  If it solves the local logging issue, I am happy with it and will soon propose a PR. If there is a package to do this that implements an io.Writer, it should be trivial.

So to summarize:

* I will be looking into event log for local windows logging. This will probably require writing a package.
* the syslog change will solve in the sort term, the aggregation issue from Windows nodes (when something better comes along, I will personally send a case of beer or ice-cream...or both, to whomever removes syslog as a dependency)
* lumberjack works *now* for local logging on both Windows and Ubuntu. It simply removes 2 dependencies (for logging) with just a few lines of code...

As I have mentioned, I am willing to adapt to any decision you guys make (I am just an outsider), as long as that decision does not imply 2 more months and 3 extra packages just to get a perfect logging system. I believe its better to increment then to reinvent.

Right now, there are 2 small changes proposed that *will* work and they will make our job easier. Removing dependencies is always good...

Please keep in mind that its better to move forward no matter how small the steps, then to just stand still while we figure out the perfect logging system. I would much rather have windows support today, then 2 months from now when someone actually gets around to implement a *new* logging system.

This should not be a discussion about which logging system is _best_. This should be a discussion about which logging system is _better_ and available *now*. Otherwise we risk of getting caught up in details and loose sight of our actual goal.

Just my 2 cents.

Regards,
Gabriel

Thanks,

Kapil

Regards,
Gabriel

On 14.08.2014 23:47, Kapil Thangavelu wrote:

On Thu, Aug 14, 2014 at 2:14 PM, Nate Finch <nate.finch at canonical.com<mailto:nate.finch at canonical.com>> wrote:
I didn't bring up 12 factor, it's irrelevant to my argument.

I'm trying to make our product simpler and easier to maintain.  That is all.  If there's another cross-platform solution that we can use, I'd be happy to consider it.  We have to change the code to support Windows.  I'd rather the diff be +50 -150  than +75 -0.  I don't know how to state it any simpler than that.

the abrogation of responsibility which is what ic you adocating for in this thread,  also makes our product quite a lot less usable imo... Our product is a distributed system with emergent behavior. Having a debug log is one of the most useful things you can have to observe the system and back in py days was one of the most used features and it was just a simple dump to the db with querying. Its unfortunate that ability to use it usefully didn't land to core till recently and did so in broken fashion (still requiring internal tag names for filtering).. or lots more people would be using it. Gustavo's suggestion of storing the structured log data  in mongo sounds really good to me. Yes, features are work and require code but that sort of implementation is also cross platform portable. The current implementation and proposed alternatives I find somewhat ridicolous in that we basically dump structured data into an unstructured format only to reparse it every time we look at it (or ingest into logstash) given that we already have the structured data. Asking people to setup one of those distributed log aggregation systems systems and configure them is a huge task, and anyone suggesting punting that to an end user or charm developer has never setup one up themselves i suspect. ie. an analogy imo http://xahlee.info/comp/i/fault-tolerance_NoSQL.png As for the operations follks who do have them.. we can continue sending messages send to local syslog and let them collect per their preference.

-k

On Thu, Aug 14, 2014 at 1:35 PM, Gustavo Niemeyer <gustavo.niemeyer at canonical.com<mailto:gustavo.niemeyer at canonical.com>> wrote:
On Thu, Aug 14, 2014 at 1:35 PM, Nate Finch <nate.finch at canonical.com<mailto:nate.finch at canonical.com>> wrote:
> On Thu, Aug 14, 2014 at 12:24 PM, Gustavo Niemeyer
> <gustavo.niemeyer at canonical.com<mailto:gustavo.niemeyer at canonical.com>> wrote:
>>
>> > Why support two things when you can support just one?
>>
>> Just to be clear, you really mean "why support two existing and well
>> known things when I can implement a third thing", right?
>
> Yes, that is exactly what I mean.

Okay, on that basis and without any better rationale than "12factor
says I can do anything" I'd be tempted to request further analysis on
the problem if the decision was on my hands. There are more
interesting problems to solve than redoing what already exists.

gustavo @ http://niemeyer.net

--
Juju-dev mailing list
Juju-dev at lists.ubuntu.com<mailto:Juju-dev at lists.ubuntu.com>
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev

--
Juju-dev mailing list
Juju-dev at lists.ubuntu.com<mailto:Juju-dev at lists.ubuntu.com>
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20140815/68942bfa/attachment-0001.html>