<div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Sat, Mar 19, 2016 at 8:36 PM William Reade <<a href="mailto:william.reade@canonical.com">william.reade@canonical.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">On Sat, Mar 19, 2016 at 4:39 AM, Ian Booth <span dir="ltr"><<a href="mailto:ian.booth@canonical.com" target="_blank">ian.booth@canonical.com</a>></span> wrote:<br></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br>
</div></div>I mostly agree but still believe there's a case for transient messages. The case<br>
where Juju is downloading an image and emits progress updates which go into<br>
status history is to me clearly a case where we needn't persist every single one<br>
(or any). In that case, it's not a charmer deciding but Juju. And with status<br>
updates like X% complete, as soon as a new message arrives, the old one is<br>
superseded anyway. The user is surely just interested to know the current status<br>
and when it completes they don't care anymore. And Juju agent can still decide<br>
to say make every 10% of download progress messages non-transient to they go to<br>
history for future reference.<br><div></div></blockquote></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> <br>There
are two distinct problems: collecting the data, and presenting
information gleaned from that data. Adding complexity to the first task
in the hope of simplifying the second mixes the concerns at a very deep
level, and makes the whole stack harder to understand for everybody. <br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Would this work s an initial improvement for 2.0:<br>
<br>
1. Increase limit of stored messages per entity so say 500 (from 100)<br></blockquote></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br>Messages-per-entity seems like a strange starting point, compared with either max age or max data size (or both). Storage concerns don't seem like a major risk: we're keeping a max 3 days/4
gigabytes of normal log messages in the database already, and I rather
doubt that SetStatus calls generate anything like that magnitude of
data. Shouldn't we just be following the same sort of trimming strategy there and leaving the dataset otherwise uncontaminated, and hence as useful as possible?<br><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
2. Allow messages emitted from Juju to be marked as transient<br>
eg for download progress<br></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>-1, it's just extra complexity to special-case a particular kind of status in exchange for very questionable storage gains, and muddies the dataset to boot.<br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
3. Do smarter filtering of what is displayed with status-history<br>
eg if we see the same tuple of messages over and over, consolidate<br>
<br>
TIME TYPE STATUS MESSAGE<br>
26 Dec 2015 13:51:59Z agent executing running config-changed hook<br>
<span>26 Dec 2015 13:51:59Z agent idle<br>
26 Dec 2015 13:56:57Z agent executing running update-status hook<br>
26 Dec 2015 13:56:59Z agent idle<br>
26 Dec 2015 14:01:57Z agent executing running update-status hook<br>
26 Dec 2015 14:01:59Z agent idle<br>
26 Dec 2015 14:01:57Z agent executing running update-status hook<br>
26 Dec 2015 14:01:59Z agent idle<br>
<br>
</span>becomes<br>
<br>
TIME TYPE STATUS MESSAGE<br>
26 Dec 2015 13:51:59Z agent executing running config-changed hook<br>
26 Dec 2015 13:51:59Z agent idle<br>
>> Repeated 3 times, last occurence:<br>
<div><div>26 Dec 2015 14:01:57Z agent executing running update-status hook<br>
26 Dec 2015 14:01:59Z agent idle<br></div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>+100 to this sort of thing. It won't be perfect, but where it's imperfect we'll be able to see how to improve. And if we're always calculating it from the source data, we can improve the presentation/analytics and fix those bugs in isolation; if we mangle the data at collection time we sharply limit our options in that arena. (And surely sensible filtering will render the transient-download-message problem moot *anyway*, leaving us less reason to worry about (2)?)<br></div></div></div></div></blockquote><div><br></div><div>+1 on collapsing repeats. I'd also prefer to add more data to status so that we can collapse entries, rather than dropping data so that we don't/can't see it in history.</div><div><br></div><div>What do we base "sensible filtering" on? Exact match on message isn't enough for download messages, obviously, and I'd be very hesitant to bake in any knowledge of specific message formats.</div><div><br></div><div>IIANM, our status entries can carry additional data which we don't render. If we add the concept of overarching operations to status entries (e.g. each "image download progress" entry is part of the "image download" operation), then we could collapse all adjacent entries within that operation. This could be a simple string in the status data; or we could extend the status schema. Either way.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div>Cheers<br></div><div>William<br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>
<br>
<br>
<br>
<br>
<br>
> On Thu, Mar 17, 2016 at 6:30 AM, John Meinel <<a href="mailto:john@arbash-meinel.com" target="_blank">john@arbash-meinel.com</a>> wrote:<br>
><br>
>><br>
>><br>
>> On Thu, Mar 17, 2016 at 8:41 AM, Ian Booth <<a href="mailto:ian.booth@canonical.com" target="_blank">ian.booth@canonical.com</a>><br>
>> wrote:<br>
>><br>
>>><br>
>>> Machines, services and units all now support recording status history. Two<br>
>>> issues have come up:<br>
>>><br>
>>> 1. <a href="https://bugs.launchpad.net/juju-core/+bug/1530840" rel="noreferrer" target="_blank">https://bugs.launchpad.net/juju-core/+bug/1530840</a><br>
>>><br>
>>> For units, especially in steady state, status history is spammed with<br>
>>> update-status hook invocations which can obscure the hooks we really care<br>
>>> about<br>
>>><br>
>>> 2. <a href="https://bugs.launchpad.net/juju-core/+bug/1557918" rel="noreferrer" target="_blank">https://bugs.launchpad.net/juju-core/+bug/1557918</a><br>
>>><br>
>>> We now have the concept of recording a machine provisioning status. This<br>
>>> is<br>
>>> great because it gives observability to what is happening as a node is<br>
>>> being<br>
>>> allocated in the cloud. With LXD, this feature has been used to give<br>
>>> visibility<br>
>>> to progress of the image downloads (finally, yay). But what happens is<br>
>>> that the<br>
>>> machine status history gets filled with lots of "Downloading x%" type<br>
>>> messages.<br>
>>><br>
>>> We have a pruner which caps the history to 100 entries per entity. But we<br>
>>> need a<br>
>>> way to deal with the spam, and what is displayed when the user asks for<br>
>>> juju<br>
>>> status-history.<br>
>>><br>
>>> Options to solve bug 1<br>
>>><br>
>>> A.<br>
>>> Filter out duplicate status entries when presenting to the user. eg say<br>
>>> "update-status (x43)". This still allows the circular buffer for that<br>
>>> entity to<br>
>>> fill with "spam" though. We could make the circular buffer size much<br>
>>> larger. But<br>
>>> there's still the issue of UX where a user ask for the X most recent<br>
>>> entries.<br>
>>> What do we give them? The X most recent de-duped entries?<br>
>>><br>
>>> B.<br>
>>> If the we go to record history and the current previous entry is the same<br>
>>> as<br>
>>> what we are about to record, just update the timestamp. For update<br>
>>> status, my<br>
>>> view is we don't really care how many times the hook was run, but rather<br>
>>> when<br>
>>> was the last time it ran.<br>
>>><br>
>><br>
>> The problem is that it isn't the same as the "last" message. Going to the<br>
>> original paste:<br>
>><br>
>> TIME TYPE STATUS MESSAGE<br>
>> 26 Dec 2015 13:51:59Z agent idle<br>
>> 26 Dec 2015 13:56:57Z agent executing running update-status hook<br>
>> 26 Dec 2015 13:56:59Z agent idle<br>
>> 26 Dec 2015 14:01:57Z agent executing running update-status hook<br>
>> 26 Dec 2015 14:01:59Z agent idle<br>
>><br>
</div></div>>> Which means there is an "running update-status" *and* a "idle" message.<br>
<div><div>>> So we can't just say "is the last message == this message". It would have<br>
>> to look deeper in history, and how deep should we be looking? what happens<br>
>> if a given charm does one more "status-set" during its update-status hook<br>
>> to set the status of the unit to "still happy". Then we would have 3.<br>
>> (agent executing, unit happy, agent idle)<br>
>><br>
>><br>
>>> Options to solve bug 2<br>
>>><br>
>>> A.<br>
>>> Allow a flag when setting status to say "this status value is transient"<br>
>>> and so<br>
>>> it is recorded in status but not logged in history.<br>
>>><br>
>>> B.<br>
>>> Do not record machine provisioning status in history. It could be argued<br>
>>> this<br>
>>> info is more or less transient and once the machine comes up, we don't<br>
>>> care so<br>
>>> much about it anymore. It was introduced to give observability to machine<br>
>>> allocation.<br>
>>><br>
>><br>
>> Isn't this the same as (A)? We need a way to say that *this* message<br>
>> should be showed but not saved forever. Or are you saying that until a<br>
>> machine comes up as "running" we shouldn't save any of the messages? I<br>
>> don't think we want that, because when provisioning fails you want to know<br>
>> what steps were achieved.<br>
>><br>
>><br>
>>><br>
>>> Any other options?<br>
>>> Opinions on preferred solutions?<br>
>>><br>
>>> I really want to get this fixed before Juju 2.0<br>
>>><br>
>><br>
>> We could do a "log level" rather than just "transient or not", and that<br>
>> would decide what would get displayed by default. (so you can ask for<br>
>> 'update-status' messages but they wouldn't be shown by default). The<br>
>> problem is that we want to keep status messages pruned at a sane level and<br>
>> with 2 updates for every 'update-status' call history of 100 is only<br>
>> 100/2*5/60 ~ 4hours of history. If something interesting happened<br>
>> yesterday, you're SOL.<br>
>><br>
>> What if we added a "interesting lifetime" to status messages. So the<br>
>> status-set could indicate how long the message would be preserved?<br>
>> "update-status" and "idle" could be flagged as preserved for only 1 hour,<br>
>> and "dowloading %" could be flagged at say 5 minutes. Too complicated? It<br>
>> certainly complicates the pruner (not terribly, when we record them we just<br>
>> record an expire time that is indexed and the pruner just removes<br>
>> everything that is over its expiry time.)<br>
>><br>
>> Alternatively we could have some sort of UUID for messages to indicate<br>
>> that "this message is actually similar to other messages with this UUID"<br>
>> and we prune them based on that. (UUIDs get flagged with a different number<br>
>> of messages to keep than the global 100 for otherwise untagged messages.)<br>
>><br>
>> "Transient" is the easiest to understand, but doesn't really solve bug #1.<br>
>><br>
>> If we think of the "UUID" version as something like a named "status<br>
>> pocket" maybe its actually tasteful. You'd have the "global" pocket that<br>
>> has our default 100 most-recent-messages, and then you can create any new<br>
>> pocket that has a default of say 10 messages. So you would be doing:<br>
>> status-set --pocket hook-execution update-status<br>
>> status-set --pocket download Downloading X% done<br>
>><br>
>> That also lets charms do nice things at hook execution time when they're<br>
>> downloading large resources, without spamming the status-history log.<br>
>><br>
>> It does complicate the model....<br>
>><br>
>> John<br>
>> =:-><br>
>><br>
>><br>
>><br>
>> --<br>
>> Juju-dev mailing list<br>
>> <a href="mailto:Juju-dev@lists.ubuntu.com" target="_blank">Juju-dev@lists.ubuntu.com</a><br>
>> Modify settings or unsubscribe at:<br>
>> <a href="https://lists.ubuntu.com/mailman/listinfo/juju-dev" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailman/listinfo/juju-dev</a><br>
>><br>
>><br>
><br>
</div></div></blockquote></div></div></div>
--<br>
Juju-dev mailing list<br>
<a href="mailto:Juju-dev@lists.ubuntu.com" target="_blank">Juju-dev@lists.ubuntu.com</a><br>
Modify settings or unsubscribe at: <a href="https://lists.ubuntu.com/mailman/listinfo/juju-dev" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailman/listinfo/juju-dev</a><br>
</blockquote></div></div>