We desperately need more ops in #ubuntu.

[Aaron Rainbolt]

I'd like to start by thanking you for the thoroughness of this reply, 
and for taking the time to write it. I'd also like to apologize for the 
upset I've accidentally caused here - I didn't mean to set off events 
that would make this sort of thing happen, and I certainly didn't expect 
the Community Council to be called on. I was trying to call attention to 
an issue and offer to help, and instead it looks like I've just cost 
many people their time and peace. This was entirely unintentional, and I 
will try to be far more careful in the future.

On Mon, Dec 18, 2023 at 8:40 AM Melissa Draper <melissa at meldraweb.com> 
wrote:
>
> Hi Simon.
>
> Given we are in a traditionally disruptive holiday period, it is quite premature (among other things) to accuse the IRCC of disregarding this report simply because they have kind of discussed it on IRC and not yet replied on email. I’m making some assumptions about the sources of your assumptions here, because I’m really not privy to the completeness of what you seem to be responding to. I’m also not going to inline this, so apologies to anyone irked by that.

I also do not know the full details of what he's responding to, and have 
intentionally did not ask for logs of the conversation. I don't want any 
team to feel "forced" to accept a new member.

> Of the 2/3 who I have seen respond at all anywhere, only one has offered more than one single passing IRC comment. As such, I’m going to reply to this thread again because it would appear there are expectations that deviate from reality about the amount of spare time IRCC folk have available right now to read through this and then also formulate the comprehensive response y'all seem to desire. As an indication, this email has taken me a fair bit of a day to write out and make coherent. Hopefully in doing so I can make it easier for the IRCC guys by covering some bases first. Sure, I am going to sound biased on this, but experience does burden one with bias.

For the record, I did not expect a rapid or comprehensive response. It's 
the holiday season, and I know what being insanely busy feels like. I'm 
sorry that what I said came across that way.

> Does the channel need more ops? Possibly. One of the IRCC guys even said on IRC that there is room for improvement or parity regarding timezones. The data documents a diverse set of people contributing to the maintenance of the channel. Response times are as lagged as I would expect from a channel overseen by volunteers who often don’t have push notifications and are usually multitasking, especially for one that has its level of activity and current amount of automation (ie IRC team-maintained bots). I am named in the ops call, so I know how much it gets used, and it’s less than one might expect.
>
> Aaron, I do appreciate that you have put time and effort into this, and that you are acting in what you perceive as the best interest of the community. However as indicated prior when I offered advice for improving the report, I do not feel that the investigation or the conclusion was adequate yet.
>
> Importantly, the data does _not_ convey to _me_ (and potentially also not to the IRCC) that this is as _desperate_ an issue as has been advertised. Instead of a counter-admonishment, I invite you both, and the unnamed key players, to clarify the specific urgency at hand. In the interim, I will do my best to explain what the data _does_ tell me.
>
> First, let’s spell out the facts that the data shows in raw numbers, some which Aaron provided, some which were not summarized by him:
>
> - The timespan documented is approximately 707 days, or 101 weeks (given we’re 2w short of EOY and I helped Aaron figure out how to scrape the logs one week ago.) That is almost 2 years.
> - There are 58 incidents documented.
> - On average there was a documented ops call event slightly more often than once a fortnight.
> - There are 23 “unaddressed” incidents documented.
> - This averages to _less_ than once a month where a situation was documented as ending with no discernible action causing the resolution.
> - With 9 events, October this year was abnormally busy.
> - Some months had no events.
> - 11 distinct individuals are named as having intervened in events.
> - The last event documented as “unaddressed" was 16 days ago.
> - Last new year period, early December through mid March was a ghost town (was the logging client offline?)
>
> Can you please clarify the claims of desperate need based on these facts? To be absolutely clear, "urgency demanding of immediate followup from specific volunteers in a holiday period" is the kind of thing I’d be expecting to involve a timeframe of a couple of days at most. Given the timeframes I’ve described above, I cannot in good conscience agree that this is urgent. Perhaps there is a misunderstanding, but it’s up to _you_ to fix that, with information not… that follow up.

I think my word "desperately" was misunderstood here. I did not intend 
to convey that this was an acute condition worthy of immediate follow-up 
and sudden enrollment of new operators. The reason I picked such a big 
timeframe is because I've watched severe incidents go unhandled often 
enough that, subjectively, it felt like all the ops were always absent 
any time something really bad was happening. **This was obviously an 
extremely subjective and biased opinion of mine** caused by the 
adrenaline and upset of dealing with a severe troll, but it happened 
often enough it felt like something was wrong. I know the ops team does 
a fantastic job when they are around, so my conclusion was that there 
just must not be enough ops. I voiced this concern about a year ago, and 
was reassured by some people that it wasn't that bad, while others let 
me know in private that they too had noticed the same problem and that 
it was pretty bad. I tried to see if the issue was actually an issue, 
and this is what I came up with.

The info above is flawed, yes. That's the inevitable result of dealing 
with incomplete logging and being subjective. (Also as you noticed, the 
log bot did indeed drop out for a significant period of time twice, and 
also there was one time where the channel was suddenly relatively 
peaceful.) I believed the condition is desperate because it is 
*chronic*, and is gradually wearing on our volunteers (myself included). 
It's not urgent, it's certainly not worth the level of upset I seem to 
have unintentionally set off, it's just seems to me like it needs 
considered and addressed at some point. Lots of things that are 
desperate can be put off if needed in developer work if those issues 
aren't urgent (like install failures, all web browsers segfaulting, 
etc.), so I didn't realize I was conveying a sense of urgency.

> There are also specific things not represented by the list, some of which have already been mentioned or acknowledged:
>
> - If there is a specific or consistent timezone that the data shows is inadequately supervised.
> - How (or why) events resolved without documented intervention. Did they even need warrant the ops call?
> - That some calls (notably the burst of October ones) were dealt with by network staff because they were an inter-namespace issue.
> - That in accordance with good practice (ie, https://libera.chat/guides/catalyst disclosure: I wrote that.) sometimes a participant was spoken to out of channel instead of somewhere public.
> - Some incidents in this list were looked at by a channel operator who did not agree, based on good practices, that the ops call was necessary. Shaming for a non-egregious bad call would also generally be bad practice.

I did try to filter out the unwarranted ops calls. There were many calls 
I left off the list because (for instance):

* A user immediately shaped up after the call
* The call seemed unwarranted
* There events after the call were of short enough duration that an op 
wouldn't have had time to respond

> Some concepts implied by the documentation:
>
> - That people generally thank you in the channel.
> - That all the calls were necessary and operator intervention is mandatory.
> - That resolution where the offending user left on their own after the call is a failure of the ops team.
> - That resolution by staff action is a failure of the ops team.

The latter two I disagree with strongly (i.e., I agree with what you say 
later on in this email), and tried to say as much in the initial email. 
If a user leaves on their own shortly after seeing the ops trigger, 
that's fine and I tried to filter those events out of the list. 
Resolution by Libera staff was intentionally considered equivalent to an 
operator responding, however knowing more about how Libera staff 
operate, I can see that it is much more difficult to notice when staff 
take action unless you have your own logs. I caught only one event that 
was by staff (and considered it equivalent to operator intervention), so 
I can see by that, that my study is quite a bit off.

I do think people usually say thank you when a troll is removed from the 
channel, since it helps to relieve adrenaline and stress to say thank 
you to the person who got a troll out of the channel. I see though that 
this kind of behavior isn't generally considered good, and will take 
that into account. Also, I did *try* to pick out the ops calls that were 
reasonable and leave out the ones that weren't. Obviously it's up to 
each individual operator whether they feel an event is worth handling or 
not, this is just how it looks to me as an observer.

> Again, I am biased here, but my learned opinion is that IRC regulars know there’s no good reason to celebrate operator actions in the channel. Some of that is because we have discouraged being applauded for wielding the ban hammer over the years. It’s not healthy to reward the us versus them. Some of it is also because IRC is a medium that doesn’t generally have push notifications by default, you have to set those up yourself, etc and the moment has passed by the time it is seen for the most part.
 >
> It is also not unheard of for an ops calls to be ill-advised in the first place, an act of retaliation/antagonism, or even just failing to respect that newbies often cannot — and should not be expected to — immediately conform to a detailed etiquette when they are frustrated and need acknowledgement and/or a solution. As such, an ops call should never ever be an _obligation_ to intervene if our experience in reading the room informs us otherwise. Punitive responses can and do regularly lead to further disputes, and it’s actually often healthier to have a bit of back and forth where the user leaves on their own. I know from experience that my instincts on this are not the same as Aaron’s, or those of a certain others who I’m sure might be considered key players.

I fully agree. I did try to filter out the ill-advised calls, so I share 
this line of reasoning, though I may see slightly different lines than you.

> Further, and most frustratingly, network staff intervention should _never_ be reflected onto the ops team as a failure, particularly since there are some of us with both hats, and it is quite crushing that our work at in the network context is conveying inadequacy on this team. I understand this disrespect wasn’t intended but it does end up inferred by the report's conclusion.

Agreed 100%. In real conversation, such intervention would never be seen 
poorly seen since, hey, the troll left, everyone can see they got 
K-lined or otherwise removed, everyone's happy. It only came across 
wrong in my study because of how difficult it is to pick out those 
events from the logs. (I would have used my own logs, but I lost most of 
them and was gone for six months at one point, so...)

> For the record, the most recent documented-as-unaddressed event in the list, on Dec 1, did involve conversation with a participant outside of the channel. I was the operator that followed up. I do not see the need for punitive measures in the log of that incident. If I was forced to undertake punitive measures based on that log, I’d have been removing a key player at the same time, for fairness sake.
>
> That was also _weeks_ before this thread was started. So, again, where’s the urgency? I don’t think that it is unreasonable, given that the claims made do not actually seem to add up, that one might explore potential motivations. Especially motivations that are also indicated by current applications for privileges. That doesn’t necessarily mean assuming bad faith, but rather trying to wrap our heads around what is actually being proposed because the current framing doesn’t make sense.

There is no urgency, I simply miscommunicated. That's my fault and I 
should have been more careful with my choice of wording.

> This pondering, for those reading along and confused about the out of left field tangent taken by Simon in his email, was aired in a conversation that occurred semi-privately, and was not intended for publication on a public mailing list. And while we’re talking about respecting contributors, I will be doing so by _not_ being the one publishing the non-binding opinions I gave on the matter, because not all opinions, even if they are justified and appropriate, are suitable for public search engine archival.
>
> Regards,
> el

Thank you again. I hope this clears up some misunderstandings.

I think in trying to volunteer help, I've accidentally caused as much or 
more trouble than a repeat troll would have caused in the channel. I'll 
leave my offer to assist the operator team open in case the team would 
like help, but if this has been more trouble than it's worth, please 
feel free to simply reject me as a proposed member of the #ubuntu ops 
team. I am more than content to help in the roles where I'm already 
helping, and I don't want any team to feel negatively toward me because 
they had to interact with me or those attempting to back me up. I really 
don't want to be part of a team that feels like they were somehow forced 
to accept my presence within the team.

On my own behalf, I would like to officially ask the Community Council 
(if they are reading this) to **not, I repeat, not** intervene as far as 
I'm concerned. Nothing has happened to me that is worth calling in that 
level of authority. Obviously I can't speak for everyone involved (if 
Simon or the IRC Council are invoking CC authority on their own behalf, 
that's something I have no control over), but for me, I don't need or 
want the CC to step in against any team on my behalf.

I'm sorry again for all this. I had no intention of causing this, and I 
hope we can move forward positively from here. Please don't feel any 
obligation to reply soon (or at all), and again, feel free to remove me 
from the list of proposed #ubuntu operators if you'd like. I'm thankful 
for the work the ops team and Libera staff have been doing for so long. 
I take full responsibility for what's happened here. I was trying to 
work with the IRC team, not against them. I will try very hard to not 
cause problems like this in the future.

Wishing you all happy and peaceful holidays. Thanks again.

Sincerely,
Aaron

>
> On Dec 17, 2023, at 06:15, Simon Quigley <simon at tsimonq2.net> wrote:
>
> Dear Ubuntu IRC Council,
>
> I hope this finds you in good health and high spirits. I would like to follow up on this proposal with my own comments, and propose some steps forward. As you will read, I am not happy with quite a few elements, but you should not take this personally; someone MUST be firm about this.
>
> Firstly, I am quite appalled by the reaction from the IRCC on this proposal. From my understanding, this is generally being treated as, "Aaron just wants to be an IRC operator, and is making a huge fuss about it." That is NOT how we treat contributors in Ubuntu(!), and goes directly against the Code of Conduct:
>
> (* = [*])
> "Collaboration between teams that each have their own goal and vision is essential; for the whole to be more than the sum of its parts, each part must make an effort to understand the whole. Collaboration reduces redundancy and improves the quality of our work. Internally and externally, *we celebrate good collaboration*. [...] Our community is open, and any responsibility can be carried by *any* contributor who demonstrates the required capacity and competence."
>
> Aaron took a large amount of time, as a volunteer contributor, to prepare this proposal for you all, and it is *quite* disrespectful that you seem to be discarding it. Inactivity is understandable at this time of year, but to my understanding, a quorum of the IRCC is active, so why was Melissa's response considered adequate in any way, shape, or form? #ubuntu is *literally* the frontpage of Ubuntu on IRC, this should be at the *top* of the IRCC's list.
>
> Perhaps you are misunderstanding the urgency of this request; after talking to several key players within the community, the general consensus is, even if Aaron's data may need to be slightly adjusted, *he's right*. We NEED more IRC operators in #ubuntu. Some events may be false flags, but we are all human. If a pattern needs to be corrected, address it head on, *do not* talk about it in a clandestine manner as a point against someone being an operator, without doing anything about it. If you think someone is too aggressive in calling the operators, or as an operator banning people, *you need to address it, that is your job as the Ubuntu IRC Council*.
>
> Do better. I'm not going to respond or read your angry PMs about how I'm overreacting, because *I'm not*. I will *not* let the IRC Council get away with treating someone, especially Aaron Rainbolt, like this. I am formally asking you to fix this, immediately. He deserves better than this.
>
>
> Aaron, you did a really great job with this proposal. I really appreciate all the time and effort you put into this. Comments inline.
>
> On 12/15/23 02:16 PM, Aaron Rainbolt wrote:
>
> Good morning/evening, and thanks for your time.
> I wish to call attention to the current state of affairs as far as channel moderation in the #ubuntu IRC room. Based on both recent and past activity, it is my opinion that we do not have enough active IRC operators present to handle the current rate of spam and disruption that flows into #ubuntu on a regular basis. This can be shown by viewing the IRC logs for the #ubuntu room over the last two years.
>
>
> One-hundred percent agreed, qualitatively to start. #ubuntu is the frontpage for Ubuntu on IRC, so if additional moderators put themselves forward, the IRC Council should make it a priority to review those.
>
> To obtain the needed data, I grabbed all of the #ubuntu IRC logs over 2022 and 2023, then scanned for events that needed operator intervention. I have defined an "event that needs operator intervention" as being any discrete event within the channel wherein which the !ops trigger was used due to an actual rule violation, and the offending user did not cease misbehaving after the trigger was used (or the misbehavior was very serious). Multiple !ops triggers in relation to the same user are considered a single event, and illegitimate or potentially overzealous uses of the !ops trigger are disqualified. From this list of events, I then studied the logs to find all events that needed operator intervention where no intervention was present. I have defined "operator intervention" as being any activity by an operator listed by ubottu when the !ops trigger was used, with such activity being at least somewhat in relation to the misbehaving user's activity. Any visible or suspected activity is qualified. Finally, any events which resulted in Libera staff stepping in are considered the same as being handled by the ops team statistically (though are distinguished from normal operator intervention).
>
>
> I understand some heuristics are involved here, and I honestly agree with it; there is not quite a clear definition of what exactly defines an "event that needs operator intervention" and I could not have said it better myself. If anyone has ideas on how we can define this better, without saying "most calls to !ops are overzealous" (which, to my understanding, is a common opinion), I think we'd really get somewhere.
>
> I do note that the IRC logs do not keep record of joins, leaves, bans, kicks, etc. Some of this info may be inaccurate as a result, though I suspect it is mostly reliable as usually someone says "thanks" when a user gets banned, or the operator says something.
>
>
> In my experiences, when doing these types of analyses, I tend to use my own IRC logs. Of course, there are netsplits to worry about, but that will give you a more complete view of these items. They intentionally filter some of these events out from the IRC logs.
>
> In my personal opinion, if more than 15% of events that needed operator intervention go unhandled, that likely means we need more active operators. According to my study, the percentage of missed events is over double that. I therefore conclude that we desperately need more ops in #ubuntu, and would like for the IRC operator team to consider having a vote for additional IRC operators in that channel.
>
>
> Not disagreeing, but where did you come up with 15%? Where exactly should we draw the line here, and why?
>
> Thank you for your consideration.
>
>
> For moving the needle forward, we should be thanking *you*!
>
> Here are the results of my study:
> Number of events that needed operator intervention: 58
> Number of events handled by operators: 35
> Number of events that needed operator intervention where no intervention was present: 23
> Percentage of unhandled events: 39.65%
>
>
> (Small note; the most accurate percentage given the rule of significant digits would be 40%, rounded up.)
>
> Event list:
> 2022-01-12: offending user Guest36, operator intervention from JackFrost/Unit193
> 2022-01-21: offending user elinfo, operator intervention from tomreyn
> 2022-02-23: offending user yrdsb, operator intervention from el
> 2022-03-05: offending user Kolusion, **no operator intervention**
> 2022-03-08: offending user merc (who ironically reported himself), operator intervention from krytarik
> 2022-03-19: offending user oneone, operator intervention from krytarik
> 2022-03-26: offending user thelounge8575, **no operator intervention**
> 2022-04-03: offending user 079AAAL91, Libera staff intervention from A_Dragon
> 2022-04-09: offending user Phalor, **no operator intervention**
> 2022-04-11: offending users yrdsb and x0x, **no operator intervention**
> 2022-04-15: offending user PHDQue, *minor* operator intervention from CarlFK but spam was left unchecked for over 30 minutes after the initial !ops call, thus considering this **no operator intervention**
> 2022-05-04: offending user LinuxAspy, **no operator intervention**
> 2022-05-13: offending user Guest459, operator intervention from Unit193
> 2022-05-21: offending user NateDoge, operator intervention appears to have occurred but op cannot be identified from logs
> 2022-06-03: offending user SleepyMario, operator intervention from tomreyn
> 2022-06-21: offending user funnyboy243, operator intervention from Unit193
> 2022-06-21: offending username inappropriate, operator intervention possible but not apparent
> 2022-06-22: offending user retardedme-, operator intervention present, presumably from Unit193
> 2022-07-04: offending user pitiless, **no operator intervention**, severe incident
> 2022-07-15: offending user filename, **no operator intervention**
> 2022-07-16: offending user nshire, operator intervention appears to have come from a bot of some sort
> 2022-07-25: offending user Guest1845, operator intervention very likely
> 2022-08-12: offending user blei, operator intervention from sarnold
> 2022-08-14: offending users ZuppaVideos and ronmerkle, operator intervention from krytarik and Bahhumbug
> 2022-09-05: offending user BASHitup, operator intervention very likely
> 2022-09-11: offending user scribz, **no operator intervention**
> 2022-09-14: offending user Morpheus_37, operator intervention from genii
> 2022-09-20: offending user lagunalorre, operator intervention from Eickmeyer
> 2022-10-18: offending user samouy, operator bot appears to have been triggered
> 2022-10-19: offending user supremekai, operator intervention from sarnold suspected
> 2022-12-05: offending user lagunaloire123, operator intervention from Unit193
> 2023-03-17: offending user Guest64, operator intervention possible but not apparent
> 2023-04-13: offending user invitado, **no operator intervention**
> 2023-04-23: offending user tyrell_willick6[, operator intervention from el likely
> 2023-04-24: offending user KIRIESHKA, **no operator intervention**
> 2023-04-25: offending user KIRIESHKA, **no operator intervention** again
> 2023-07-21: offending user fastwifi_, operator intervention from Unit193
> 2023-08-09: offending users zeroadrenaline and mort, operator intervention from hggdh for mort and *maybe* for zeroadrenaline
> 2023-08-17: offending user diamat, operator intervention from genii
> 2023-08-22: offending user Phalanxer, **no operator intervention**, severe incident
> 2023-08-26: offending user diamat_, operator intervention from tomreyn, no !ops trigger used but severe enough to mention nonetheless
> 2023-08-29: offending user WHATEVERYEAH (suspected to be Kolusion), **no operator intervention**
> 2023-09-06: offending user HackerII (aka `oerheks, note backtick at beginning of name), operator intervention from genii
> 2023-09-22: offending users mina34, delmina, and possibly Guest22, operator intervention from el
> 2023-09-23: offending users casualuse and casualus8, **no operator intervention**, severe incident
> 2023-10-03: offending user Reyaina, operator intervention possibly from tomreyn
> 2023-10-08: offending username inappropriate, **no operator intervention**
> 2023-10-09: offending user corey-aid (and several other nicks), **no operator intervention**, moderately severe incident
> 2023-10-10: offending user notliks, **no operator intervention**, severe incident
> 2023-10-17: offending user con, operator intervention from el
> 2023-10-25: offending user naruto69, **no operator intervention**, severe incident
> 2023-10-26: offending user ruhsayngone, **no operator intervention**
> 2023-10-27: offending user kokomop3n0r, operator intervention from krytarik
> 2023-10-27: offending user Guest42, **no operator intervention**
> 2023-11-08: offending user librehats, **no operator intervention**
> 2023-12-01: offending user elias_a, **no operator intervention**
> 2023-12-08: offending user lagunaloire123, operator intervention suspected
> 2023-12-10: offending user violetflame, operator intervention from el
> --
> Aaron Rainbolt
> Lubuntu Developer
> Matrix: @arraybolt3:matrix.org
> IRC: arraybolt3 on irc.libera.chat
> GitHub:https://github.com/ArrayBolt3
>
>
> In my own, personal capacity,
> --
> Simon Quigley
> simon at tsimonq2.net
> tsimonq2 on LiberaChat and OFTC
> @tsimonq2:linuxdelta.com on Matrix
> 5C7A BEA2 0F86 3045 9CC8
> C8B5 E27F 2CF8 458C 2FA4
>
> --
> Ubuntu-irc mailing list
> Ubuntu-irc at lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-irc
>
>
> --
> Ubuntu-irc mailing list
> Ubuntu-irc at lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-irc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0x84FC20D468BEFD4D.asc
Type: application/pgp-keys
Size: 3996 bytes
Desc: OpenPGP public key
URL: <https://lists.ubuntu.com/archives/ubuntu-irc/attachments/20231218/30d9ad5a/attachment.key>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/ubuntu-irc/attachments/20231218/30d9ad5a/attachment.sig>