Usability issues with status-history

[Ian Booth]

On 17/03/16 19:51, William Reade wrote:
> I see this as a combination of two problems:
> 
> 1) We're spamming the end user with "whatever's in the status-history
> collection" rather than presenting a digest tuned for their needs.
> 2) Important messages get thrown away way too early, because we don't know
> which messages are important.
> 
> I think the pocket/transient/expiry solutions boil down to "let's make the
> charmer decide what's important", and I don't think that will help. The
> charmer is only sending those messages *because she believes they're
> important*; even if we had "perfect" trimming heuristics for the end user,
> we do the *charmer* a disservice by leaving them no record of what their
> charm actually did.
> 
> And, more generally: *every* message we throw away makes it hard to
> correctly analyse any older message. This applies within a given entity's
> domain, but also across entities: if you're trying to understand the
> interactions between 2 units, but one of those units is generating many
> more messages, you'll have 200 messages to inspect; but the 100 for the
> faster unit will only cover (say) the last 30 for the slower one, leaving
> 70 slow-unit messages that can't be correlated with the other unit's
> actions. At best, those messages are redundant; at worst, they're actively
> misleading.
> 
> So: I do not believe that any approach that can be summed up as "let's
> throw away *more* messages" is going to help either. We need to fix (2) so
> that we have raw status data that extends reasonably far back in time; and
> then we need to fix (1) so that we usefully precis that data for the user
> (...and! leave a path that makes the raw data observable, for the cases
> where our heuristics are unhelpful).
> 

I mostly agree but still believe there's a case for transient messages. The case
where Juju is downloading an image and emits progress updates which go into
status history is to me clearly a case where we needn't persist every single one
(or any). In that case, it's not a charmer deciding but Juju. And with status
updates like X% complete, as soon as a new message arrives, the old one is
superseded anyway. The user is surely just interested to know the current status
and when it completes they don't care anymore. And Juju agent can still decide
to say make every 10% of download progress messages non-transient to they go to
history for future reference.

> Cheers
> William
> 
> PS re: UX of asking for N entries... I can see end-user stories for
> timespans, and for "the last N *significant* changes". What's the scenario
> where a user wants to see exactly 50 message atoms?
> 

No one would say they want to see exactly 50 - it's an estimate. It's like when
you git log and you ask for the last 20 commits. If that's not enough to see
what you want, you just run again with an increased number.

I do think allowing for a timespan to be specified may be useful.

John's suggestion for adding a lifetime does sounds more complicated than we
want right now.

Would this work s an initial improvement for 2.0:

1. Increase limit of stored messages per entity so say 500 (from 100)
2. Allow messages emitted from Juju to be marked as transient
eg for download progress
3. Do smarter filtering of what is displayed with status-history
eg if we see the same tuple of messages over and over, consolidate

TIME                    TYPE    STATUS          MESSAGE
26 Dec 2015 13:51:59Z   agent   executing       running config-changed hook
26 Dec 2015 13:51:59Z   agent   idle
26 Dec 2015 13:56:57Z   agent   executing       running update-status hook
26 Dec 2015 13:56:59Z   agent   idle
26 Dec 2015 14:01:57Z   agent   executing       running update-status hook
26 Dec 2015 14:01:59Z   agent   idle
26 Dec 2015 14:01:57Z   agent   executing       running update-status hook
26 Dec 2015 14:01:59Z   agent   idle

becomes

TIME TYPE STATUS MESSAGE
26 Dec 2015 13:51:59Z agent executing running config-changed hook
26 Dec 2015 13:51:59Z agent idle
>> Repeated 3 times, last occurence:
26 Dec 2015 14:01:57Z agent executing running update-status hook
26 Dec 2015 14:01:59Z agent idle





> On Thu, Mar 17, 2016 at 6:30 AM, John Meinel <john at arbash-meinel.com> wrote:
> 
>>
>>
>> On Thu, Mar 17, 2016 at 8:41 AM, Ian Booth <ian.booth at canonical.com>
>> wrote:
>>
>>>
>>> Machines, services and units all now support recording status history. Two
>>> issues have come up:
>>>
>>> 1. https://bugs.launchpad.net/juju-core/+bug/1530840
>>>
>>> For units, especially in steady state, status history is spammed with
>>> update-status hook invocations which can obscure the hooks we really care
>>> about
>>>
>>> 2. https://bugs.launchpad.net/juju-core/+bug/1557918
>>>
>>> We now have the concept of recording a machine provisioning status. This
>>> is
>>> great because it gives observability to what is happening as a node is
>>> being
>>> allocated in the cloud. With LXD, this feature has been used to give
>>> visibility
>>> to progress of the image downloads (finally, yay). But what happens is
>>> that the
>>> machine status history gets filled with lots of "Downloading x%" type
>>> messages.
>>>
>>> We have a pruner which caps the history to 100 entries per entity. But we
>>> need a
>>> way to deal with the spam, and what is displayed when the user asks for
>>> juju
>>> status-history.
>>>
>>> Options to solve bug 1
>>>
>>> A.
>>> Filter out duplicate status entries when presenting to the user. eg say
>>> "update-status (x43)". This still allows the circular buffer for that
>>> entity to
>>> fill with "spam" though. We could make the circular buffer size much
>>> larger. But
>>> there's still the issue of UX where a user ask for the X most recent
>>> entries.
>>> What do we give them? The X most recent de-duped entries?
>>>
>>> B.
>>> If the we go to record history and the current previous entry is the same
>>> as
>>> what we are about to record, just update the timestamp. For update
>>> status, my
>>> view is we don't really care how many times the hook was run, but rather
>>> when
>>> was the last time it ran.
>>>
>>
>> The problem is that it isn't the same as the "last" message. Going to the
>> original paste:
>>
>> TIME                    TYPE    STATUS          MESSAGE
>> 26 Dec 2015 13:51:59Z   agent   idle
>> 26 Dec 2015 13:56:57Z   agent   executing       running update-status hook
>> 26 Dec 2015 13:56:59Z   agent   idle
>> 26 Dec 2015 14:01:57Z   agent   executing       running update-status hook
>> 26 Dec 2015 14:01:59Z   agent   idle
>>
>> Which means there is an "running update-status" *and* a "idle" message.
>> So we can't just say "is the last message == this message". It would have
>> to look deeper in history, and how deep should we be looking? what happens
>> if a given charm does one more "status-set" during its update-status hook
>> to set the status of the unit to "still happy". Then we would have 3.
>> (agent executing, unit happy, agent idle)
>>
>>
>>> Options to solve bug 2
>>>
>>> A.
>>> Allow a flag when setting status to say "this status value is transient"
>>> and so
>>> it is recorded in status but not logged in history.
>>>
>>> B.
>>> Do not record machine provisioning status in history. It could be argued
>>> this
>>> info is more or less transient and once the machine comes up, we don't
>>> care so
>>> much about it anymore. It was introduced to give observability to machine
>>> allocation.
>>>
>>
>> Isn't this the same as (A)? We need a way to say that *this* message
>> should be showed but not saved forever. Or are you saying that until a
>> machine comes up as "running" we shouldn't save any of the messages? I
>> don't think we want that, because when provisioning fails you want to know
>> what steps were achieved.
>>
>>
>>>
>>> Any other options?
>>> Opinions on preferred solutions?
>>>
>>> I really want to get this fixed before Juju 2.0
>>>
>>
>> We could do a "log level" rather than just "transient or not", and that
>> would decide what would get displayed by default. (so you can ask for
>> 'update-status' messages but they wouldn't be shown by default). The
>> problem is that we want to keep status messages pruned at a sane level and
>> with 2 updates for every 'update-status' call history of 100 is only
>> 100/2*5/60 ~ 4hours of history. If something interesting happened
>> yesterday, you're SOL.
>>
>> What if we added a "interesting lifetime" to status messages. So the
>> status-set could indicate how long the message would be preserved?
>> "update-status" and "idle" could be flagged as preserved for only 1 hour,
>> and "dowloading %" could be flagged at say 5 minutes. Too complicated? It
>> certainly complicates the pruner (not terribly, when we record them we just
>> record an expire time that is indexed and the pruner just removes
>> everything that is over its expiry time.)
>>
>> Alternatively we could have some sort of UUID for messages to indicate
>> that "this message is actually similar to other messages with this UUID"
>> and we prune them based on that. (UUIDs get flagged with a different number
>> of messages to keep than the global 100 for otherwise untagged messages.)
>>
>> "Transient" is the easiest to understand, but doesn't really solve bug #1.
>>
>> If we think of the "UUID" version as something like a named "status
>> pocket" maybe its actually tasteful. You'd have the "global" pocket that
>> has our default 100 most-recent-messages, and then you can create any new
>> pocket that has a default of say 10 messages. So you would be doing:
>>  status-set --pocket hook-execution update-status
>>  status-set --pocket download Downloading X% done
>>
>> That also lets charms do nice things at hook execution time when they're
>> downloading large resources, without spamming the status-history log.
>>
>> It does complicate the model....
>>
>> John
>> =:->
>>
>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>>
>