Testing Leader Election reconfiguration

Tue Mar 15 16:27:13 UTC 2016

Hey Tom,

1. You can increase the wait time until it doesn't time out:
self.d.sentry.wait(timeout=1200)
2. At what point in this sequence of commands was the status dump captured?
3. There is a bug here. You take a reference to the pdi/0 info dict on line
1. It's the same object you use to get message2 and message3 later. So,
you'll get the same message that you got on line 1. You need `message3
= self.d.sentry['pdi'][0].info['workload-status'].get('message')`
instead.

Hope this helps.

On Tue, Mar 15, 2016 at 11:41 AM, Tom Barber <tom at analytical-labs.com>
wrote:

> Okay back here again, so my nice leader election function looks like:
>
>    def test_leader_election_failover(self):
>         unit = self.d.sentry['pdi'][0].info
>         message = unit['workload-status'].get('message')
>         ip = message.split(':', 1)[-1]
>         self.d.add_unit('pdi', 2)
>         self.d.sentry.wait()
>         message2 = unit['workload-status'].get('message')
>         ip2 = message2.split(':', 1)[-1]
>         self.assertEqual(ip, ip2)
>         self.d.remove_unit('pdi/0')
>         self.d.sentry.wait()
>         message3 = unit['workload-status'].get('message')
>         ip3 = message3.split(':', 1)[-1]
>
>         self.assertNotEqual(ip3, ip2)
>
> I know there's no logic in there, but I need to make sure the stuff
> actually functions.
>
> So Tim says wait() should work, but when I tested this last night,
>
> I get a timeout error o the wait right after add_unit.
>
> https://gist.github.com/buggtb/c271dd79d782af57dea6
>
> Yet in the status dump you can see all 3 units sat there seemingly happy.
>
> Tom
>
> --------------
>
> Director Meteorite.bi - Saiku Analytics Founder
> Tel: +44(0)5603641316
>
> (Thanks to the Saiku community we reached our Kickstart
> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
> goal, but you can always help by sponsoring the project
> <http://www.meteorite.bi/products/saiku/sponsorship>)
>
> On 9 March 2016 at 18:31, Tom Barber <tom at analytical-labs.com> wrote:
>
>> Oh really?
>>
>> /me stokes his invisible beard.
>>
>>
>> Okay I'll go back and try again.
>>
>> Tom
>>
>> --------------
>>
>> Director Meteorite.bi - Saiku Analytics Founder
>> Tel: +44(0)5603641316
>>
>> (Thanks to the Saiku community we reached our Kickstart
>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>> goal, but you can always help by sponsoring the project
>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>
>> On 9 March 2016 at 16:56, Tim Van Steenburgh <
>> tim.van.steenburgh at canonical.com> wrote:
>>
>>>
>>>
>>> On Wed, Mar 9, 2016 at 6:31 AM, Tom Barber <tom at analytical-labs.com>
>>> wrote:
>>>
>>>> Thanks Stuart.
>>>>
>>>> I do put a note in my charm message indicating the leader IP address so
>>>> that users know which to connect to.
>>>>
>>>> So with juju wait, would I destroy a unit then execute juju wait? At
>>>> which point it will hang until the leader election stuff is over and all
>>>> becomes stable again?
>>>>
>>>>
>>> Since you're already using amulet, there's no need to use the juju-wait
>>> plugin
>>> since d.sentry.wait() does the same thing. So yes, you would do
>>> d.remove_unit(...)
>>> and then call d.sentry.wait().
>>>
>>>
>>>> Also, will this work if I push it upstream to the charmers and the
>>>> automated tests up there?
>>>>
>>>>
>>> Yes.
>>>
>>>
>>>> Thanks
>>>>
>>>> Tom
>>>>
>>>> --------------
>>>>
>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>> Tel: +44(0)5603641316
>>>>
>>>> (Thanks to the Saiku community we reached our Kickstart
>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>> goal, but you can always help by sponsoring the project
>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>
>>>> On 9 March 2016 at 11:00, Stuart Bishop <stuart.bishop at canonical.com>
>>>> wrote:
>>>>
>>>>> On 9 March 2016 at 20:31, Tom Barber <tom at analytical-labs.com> wrote:
>>>>> > Morning all
>>>>> >
>>>>> > I'm trying to test for charm reconfiguration if the leader goes AWOL.
>>>>>
>>>>> I put the role of the unit in its workload status, so it is easy for
>>>>> operators to see which unit is master. And this also makes it easy for
>>>>> tests to tell.
>>>>>
>>>>>
>>>>> > Adam suggested that I watch the status waiting for the next leader
>>>>> election
>>>>> > hook the wait on that and then check my service configs.
>>>>>
>>>>> You are best of waiting for all the hooks to complete and a steady
>>>>> state, not just leader elected (since things will still be in flux
>>>>> when that hook fires, such as the leader-settings-changed hooks it
>>>>> will probably trigger and the relation changes those hooks will likely
>>>>> trigger). Use the juju-wait plugin, and maybe add support to
>>>>> https://bugs.launchpad.net/juju-core/+bug/1488777 to get this into
>>>>> core.
>>>>>
>>>>> --
>>>>> Stuart Bishop <stuart.bishop at canonical.com>
>>>>>
>>>>
>>>>
>>>> --
>>>> Juju mailing list
>>>> Juju at lists.ubuntu.com
>>>> Modify settings or unsubscribe at:
>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160315/06c22f69/attachment.html>