<font size="2"><span style="background-color:rgba(255,255,255,0)"><br><br><br></span></font><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><font size="2"><span style="background-color:rgba(255,255,255,0)">I think you are attributing too much importance to some data that can hardly be considered information let me try to mention some points that I think are valid here.</span></font></div><div class="gmail_quote"><font size="2"><span style="background-color:rgba(255,255,255,0)">1) Not every message is valuable, you state that every message we throw away makes it harder to debug, but certainly a message like "downloading N%" is useless, you can record the start of the download and failure/end but the steps intermediate are quite useless. We can argue later which messages satisfy this criteria, but I am completely sure that some do.</span></font></div><div class="gmail_quote"><font size="2"><span style="background-color:rgba(255,255,255,0)">2) Not filling the history buffer with superfluos messages will help here, although I do agree we should find a more elegant deletion criteria (time sounds right) while not loosing sight of size (at this point no one has a record of the actual cost of storing these things in terms of space therefore we cannot make decisions based on the scale we want to support.</span></font></div><div class="gmail_quote"><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div class="gmail_quote"><font size="2"><span style="background-color:rgba(255,255,255,0)">Regarding "making the charm decide" I agree its something we might not want to do, I would actually not export this to the charm, I would just use it internally, since we are opinionated we can very well decide what goes there.</span></font></div><div class="gmail_quote"><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div class="gmail_quote"><font size="2"><span style="background-color:rgba(255,255,255,0)">Adding more status information will help adding observability, not having a flag for internal ephemeral statuses strikes me a bit as deleting with the left hand what you just wrote with the right one (saying might not translate well from Spanish)</span></font></div><div class="gmail_quote"><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div class="gmail_quote"><font size="2"><span style="background-color:rgba(255,255,255,0)">Finally, can this be a potential shoot in our own foot tool? yes, but so can almost any other part of our code base, this is something juju will use to report information to the user therefore we control it and, if we are not careful we will shot ourselves but, then again, if we are not careful with almost any part of the code we will do so too.</span></font></div><div class="gmail_quote"><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div class="gmail_quote"><font size="2"><span style="background-color:rgba(255,255,255,0)">Cheers</span></font></div><div class="gmail_quote"><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div class="gmail_quote"><font size="2"><span style="background-color:rgba(255,255,255,0)">On Thu, Mar 17, 2016 at 6:51 AM, William Reade <span dir="ltr"><<a target="_blank">william.reade@canonical.com</a>></span> wrote:</span></font></div><div class="gmail_quote"><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><font size="2"><span style="background-color:rgba(255,255,255,0)">I see this as a combination of two problems:<br><br></span></font></div><font size="2"><span style="background-color:rgba(255,255,255,0)">1) We're spamming the end user with "whatever's in the status-history collection" rather than presenting a digest tuned for their needs. </span></font></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div></div><font size="2"><span style="background-color:rgba(255,255,255,0)">2) Important messages get thrown away way too early, because we don't know which messages are important.<br></span></font><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">I think the pocket/transient/expiry solutions boil down to "let's make the charmer decide what's important", and I don't think that will help. The charmer is only sending those messages *because she believes they're important*; even if we had "perfect" trimming heuristics for the end user, we do the *charmer* a disservice by leaving them no record of what their charm actually did.<br></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">And, more generally: *every* message we throw away makes it hard to correctly analyse any older message. This applies within a given entity's domain, but also across entities: if you're trying to understand the interactions between 2 units, but one of those units is generating many more messages, you'll have 200 messages to inspect; but the 100 for the faster unit will only cover (say) the last 30 for the slower one, leaving 70 slow-unit messages that can't be correlated with the other unit's actions. At best, those messages are redundant; at worst, they're actively misleading.<br></span></font><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><font size="2"><span style="background-color:rgba(255,255,255,0)">So: I do not believe that any approach that can be summed up as "let's throw away *more* messages" is going to help either. We need to fix (2) so that we have raw status data that extends reasonably far back in time; and then we need to fix (1) so that we usefully precis that data for the user (...and! leave a path that makes the raw data observable, for the cases where our heuristics are unhelpful).<br><br></span></font><div><font size="2"><span style="background-color:rgba(255,255,255,0)">Cheers<br></span></font></div><font size="2"><span style="background-color:rgba(255,255,255,0)">William</span></font></div><div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">PS re: UX of asking for N entries... I can see end-user stories for timespans, and for "the last N *significant* changes". What's the scenario where a user wants to see exactly 50 message atoms?</span></font></div><div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br>On Thu, Mar 17, 2016 at 6:30 AM, John Meinel <span dir="ltr"><<a target="_blank">john@arbash-meinel.com</a>></span> wrote:<br></span></font></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div><div dir="ltr"><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font><div class="gmail_extra"><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font><div class="gmail_quote"><div><font size="2"><span style="background-color:rgba(255,255,255,0)">On Thu, Mar 17, 2016 at 8:41 AM, Ian Booth <span dir="ltr"><<a target="_blank">ian.booth@canonical.com</a>></span> wrote:<br></span></font><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><font size="2"><span style="background-color:rgba(255,255,255,0)"><br>Machines, services and units all now support recording status history. Two<br>issues have come up:<br><br>1. <a href="https://bugs.launchpad.net/juju-core/+bug/1530840" rel="noreferrer" target="_blank">https://bugs.launchpad.net/juju-core/+bug/1530840</a><br><br>For units, especially in steady state, status history is spammed with<br>update-status hook invocations which can obscure the hooks we really care about<br><br>2. <a href="https://bugs.launchpad.net/juju-core/+bug/1557918" rel="noreferrer" target="_blank">https://bugs.launchpad.net/juju-core/+bug/1557918</a><br><br>We now have the concept of recording a machine provisioning status. This is<br>great because it gives observability to what is happening as a node is being<br>allocated in the cloud. With LXD, this feature has been used to give visibility<br>to progress of the image downloads (finally, yay). But what happens is that the<br>machine status history gets filled with lots of "Downloading x%" type messages.<br><br>We have a pruner which caps the history to 100 entries per entity. But we need a<br>way to deal with the spam, and what is displayed when the user asks for juju<br>status-history.<br><br>Options to solve bug 1<br><br>A.<br>Filter out duplicate status entries when presenting to the user. eg say<br>"update-status (x43)". This still allows the circular buffer for that entity to<br>fill with "spam" though. We could make the circular buffer size much larger. But<br>there's still the issue of UX where a user ask for the X most recent entries.<br>What do we give them? The X most recent de-duped entries?<br><br>B.<br>If the we go to record history and the current previous entry is the same as<br>what we are about to record, just update the timestamp. For update status, my<br>view is we don't really care how many times the hook was run, but rather when<br>was the last time it ran.<br></span></font></blockquote><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">The problem is that it isn't the same as the "last" message. Going to the original paste:</span></font></div><div><pre><font face="Helvetica Neue, Helvetica, Arial, sans-serif" size="3"><span style="white-space:normal;background-color:rgba(255,255,255,0)">TIME TYPE STATUS MESSAGE 26 Dec 2015 13:51:59Z agent idle 26 Dec 2015 13:56:57Z agent executing running update-status hook 26 Dec 2015 13:56:59Z agent idle 26 Dec 2015 14:01:57Z agent executing running update-status hook 26 Dec 2015 14:01:59Z agent idle </span></font></pre></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">Which means there is an "running update-status" <b>and</b> a "idle" message. So we can't just say "is the last message == this message". It would have to look deeper in history, and how deep should we be looking? what happens if a given charm does one more "status-set" during its update-status hook to set the status of the unit to "still happy". Then we would have 3. (agent executing, unit happy, agent idle)</span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><font size="2"><span style="background-color:rgba(255,255,255,0)"><br>Options to solve bug 2<br><br>A.<br>Allow a flag when setting status to say "this status value is transient" and so<br>it is recorded in status but not logged in history.<br><br>B.<br>Do not record machine provisioning status in history. It could be argued this<br>info is more or less transient and once the machine comes up, we don't care so<br>much about it anymore. It was introduced to give observability to machine<br>allocation.<br></span></font></blockquote><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">Isn't this the same as (A)? We need a way to say that *this* message should be showed but not saved forever. Or are you saying that until a machine comes up as "running" we shouldn't save any of the messages? I don't think we want that, because when provisioning fails you want to know what steps were achieved.</span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"> </span></font></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><font size="2"><span style="background-color:rgba(255,255,255,0)"><br>Any other options?<br>Opinions on preferred solutions?<br><br>I really want to get this fixed before Juju 2.0<br></span></font></blockquote><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">We could do a "log level" rather than just "transient or not", and that would decide what would get displayed by default. (so you can ask for 'update-status' messages but they wouldn't be shown by default). The problem is that we want to keep status messages pruned at a sane level and with 2 updates for every 'update-status' call history of 100 is only 100/2*5/60 ~ 4hours of history. If something interesting happened yesterday, you're SOL.</span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">What if we added a "interesting lifetime" to status messages. So the status-set could indicate how long the message would be preserved? "update-status" and "idle" could be flagged as preserved for only 1 hour, and "dowloading %" could be flagged at say 5 minutes. Too complicated? It certainly complicates the pruner (not terribly, when we record them we just record an expire time that is indexed and the pruner just removes everything that is over its expiry time.)</span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">Alternatively we could have some sort of UUID for messages to indicate that "this message is actually similar to other messages with this UUID" and we prune them based on that. (UUIDs get flagged with a different number of messages to keep than the global 100 for otherwise untagged messages.)</span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">"Transient" is the easiest to understand, but doesn't really solve bug #1.</span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">If we think of the "UUID" version as something like a named "status pocket" maybe its actually tasteful. You'd have the "global" pocket that has our default 100 most-recent-messages, and then you can create any new pocket that has a default of say 10 messages. So you would be doing:</span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"> status-set --pocket hook-execution update-status</span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"> status-set --pocket download Downloading X% done</span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">That also lets charms do nice things at hook execution time when they're downloading large resources, without spamming the status-history log.</span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">It does complicate the model....</span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">John</span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)">=:-></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div></div></div></div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div><font size="2"><span style="background-color:rgba(255,255,255,0)">--<br>Juju-dev mailing list<br><a target="_blank">Juju-dev@lists.ubuntu.com</a><br>Modify settings or unsubscribe at: <a href="https://lists.ubuntu.com/mailman/listinfo/juju-dev" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailman/listinfo/juju-dev</a><br><br></span></font></blockquote></div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br></span></font></div></div></div></div><font size="2"><span style="background-color:rgba(255,255,255,0)"><br>--<br>Juju-dev mailing list<br><a target="_blank">Juju-dev@lists.ubuntu.com</a><br>Modify settings or unsubscribe at: <a href="https://lists.ubuntu.com/mailman/listinfo/juju-dev" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailman/listinfo/juju-dev</a><br><br></span></font></blockquote></div><br></div></div>