Ubotu failover setup

[Paul O'Malley]

Dennis Kaarsemaker wrote:
> Hiya,
> 
> I've been pondering about a good Ubotu failover setup for a while and
> here's what I've come up with. If nobody thinks this is a terrible idea,
> I'd like to implement it soon.
> 
> - N copies of ubotu run at N hosts
> - The nicknames of all copies are linked to ubotu
> - There is a fixed order of preference between them, with ubotu itself 
>   being leading
> - Code and datafiles (except users) are synchronized automatically
> - The copies do not have users and are in read-only mode (so no factoid 
>   editing)
> - During normal operation the copies only exist in #ubuntu-bots
> - The copies all run an extra plugin, FailOver, that will detect when 
>   ubotu or any copy leaves unexpectedly or comes back
> - When a copy becomes the first preferred copy, because ubotu (and 
>   possibly other copies) are unavailable, it will join all channels 
>   ubotu needs to be in. It will not assume the nickname ubotu
> - When ubotu is back in a channel, the copy of ubotu will automatically 
>   leave it.
> 
> Thoughts, comments?
> 
> 
case:
netsplit

solution:
have all bots join the same irc server, if they "loose their primary 
server, they all have a secondary server, and they all part and ubotu is 
the only one to join all channels

case:
ubotu dies

solution:
Have bots inc +1 each other, have the second bot sit in all channels, if 
it "sees" ubotu leave, it can assume role, if it sees "ubotu" rejoin it 
shuts up

downside:
if lagged there might be two answers for the min or so of lag time (max 
lag I have seen)


case:
if bot no2. or lower dies,

solution:
bot three takes the position of number 2 and number two becomes lowest 
number.

thus last fail becomes lowest number, if it is going to have a 
connection issue it may happen more than once

case:
restoring order after some death of one or more bots
(for whatever reason)

solution:
one of the editors calls a "@reorder" command in the channel and 
whatever the predetermined listing kicks in

(copied from Dennis but with some comment perhaps implied but not said)
case:
ubotu or replacement has a database error(S)

solution:
the "primary bot" echos on db error messages into the bot channel a 
trigger command,

a different bot becomes the master and the other ones retire, and reorder

What would be brilliant would be to have a "known good db" from a day 
eariler or two days eariler become the default and see if this restores 
order,

(Dennis great idea)

cheers,
Paul