juju is slow to do anything

Tue Sep 17 11:17:14 UTC 2013

Hi Peter,

We'll have a new development release available in a few days. [one of]
the focus of the 1.15 series is to move more logic from talking
directly to mongodb and instead use our REST API. I hope that this
will help.

Having said that, you and Kapil appear to have horrible connections to
the aws and s3 endpoints, this is something that others are having
difficulty reproducing. Is it possible for you to use another AWS
region ?

Cheers

Dave

On Tue, Sep 17, 2013 at 7:47 PM, Peter Waller <peter at scraperwiki.com> wrote:
> Is juju any closer to having this sort of functionality in the mainline?
> Especially a faster juju status and a faster juju ssh.
>
>
> On 2 September 2013 02:22, Kapil Thangavelu <kapil.thangavelu at canonical.com>
> wrote:
>>
>> Hi Dave,
>>
>> I attached some status --debug logs to the bug for us-east-1 (30s instead
>> of 35s in us-west-2).
>>
>> cheers,
>>
>> Kapil
>>
>>
>> On Sun, Sep 1, 2013 at 9:19 PM, David Cheney <david.cheney at canonical.com>
>> wrote:
>>>
>>> HI Kapil,
>>>
>>> I cannot reproduce your results, can you please post the output of
>>> juju status -v so we can see the timestamps.
>>>
>>> I 100% agree that is this a problem to be fixed, I am trying to
>>> determine if it is (yet another) ec2 region specific SNAFU.
>>>
>>> Cheers
>>>
>>> Dave
>>>
>>> On Mon, Sep 2, 2013 at 11:12 AM, Kapil Thangavelu
>>> <kapil.thangavelu at canonical.com> wrote:
>>> > Hi Folks,
>>> >
>>> > Just to followup after some more investigation. For me on an ec2
>>> > environment
>>> > with 1-2 services, juju status was taking about 35s. I instrumented the
>>> > code
>>> > base a little, and found that roughly 25s is endpoint lookup (both s3
>>> > and
>>> > ec2 api). That overhead was constant across any of the juju commands.
>>> > So
>>> > number #3 would be the biggest win, i went ahead and filed issue
>>> > 1219441 for
>>> > it. It seems like low hanging fruit that delivers a big win for the ux.
>>> >
>>> > i went ahead and put together a juju cli plugin that does a status
>>> > approximation using endpoint caching and the api and was able to drop
>>> > that
>>> > time down to 5s (from 35s on my env) and down to 2.5s on Peter's
>>> > environment. attached, inline docs and install instructions.
>>> >
>>> > also regarding the dead machines stuck in pending state, there's a bug
>>> > outstanding for that https://bugs.launchpad.net/juju-core/+bug/1217781
>>> >
>>> > cheers,
>>> >
>>> > Kapil
>>> >
>>> >
>>> >
>>> >
>>> > On Fri, Aug 30, 2013 at 2:15 PM, John Arbash Meinel
>>> > <john at arbash-meinel.com>
>>> > wrote:
>>> >>
>>> >> -----BEGIN PGP SIGNED MESSAGE-----
>>> >> Hash: SHA1
>>> >>
>>> >> On 2013-08-30 14:28, Peter Waller wrote:
>>> >> > For the record, I sent the link privately. The run took about 22s
>>> >> > but I have measured 30s to 1m.
>>> >>
>>> >> Some thoughts, nothing that I can give absolute confirmation on.
>>> >>
>>> >> 1) Next week we have a group sprinting on moving a lot of the command
>>> >> line operations from being evaluated by the client into being
>>> >> evaluated by the API server (running in the cloud) and then returned.
>>> >> The explicit benefits that I've seen in other commands are pretty
>>> >> good.
>>> >>
>>> >> 'juju status' is going to be a command that should see a rather large
>>> >> improvement. Because it does round trip queries for a lot of things
>>> >> (what machines are there, what are the details of each machine, what
>>> >> are the units running on each one, etc).
>>> >>
>>> >> I've prototyped doing those queries in parallel, or trying to do bulk
>>> >> ops, which actually helped a lot in testing (this was for hundreds of
>>> >> units/machines).
>>> >>
>>> >> Doing it on the API server means any round trips are "local" rather
>>> >> than from your machine out to Amazon.
>>> >>
>>> >> 2) From here, 'time juju status' with a single instance running on ec2
>>> >> is 10s. Which breaks down roughly 4s to lookup the IP address, 2s to
>>> >> establish the state, and 4s to "finish up". (resolution of this is 1s
>>> >> granularity)
>>> >>
>>> >> Similarly "time juju-1.13.2 get not-service" takes 8.5s to run. 4s to
>>> >> lookup the address, 2s to connect, and 3s to give the final 'not
>>> >> found' result.
>>> >>
>>> >> With trunk, "time ./juju get not-service" is 4.6s. 2s to lookup IP
>>> >> address, 2s to connect, and the not-found result is instantaneous.
>>> >>
>>> >> So I would expect the 10s of a generic "juju status" to easily drop
>>> >> down to sub 5s. Regardless of any round-trip issues.
>>> >>
>>> >> 3) We are also looking to cache the IP address of the API server, to
>>> >> shave off another ~2-4s for the common case that the address hasn't
>>> >> changed. (We'll fall back to our current discovery mechanism.)
>>> >>
>>> >> 4) There seems to be an odd timing of your screen cast. It does report
>>> >> 22s which matches the times reported in the debug output. But the
>>> >> total time of the video is 20s including typing. Is it just running
>>> >> 2:1 speed?
>>> >>
>>> >> You can see from the debug that you have 7s to lookup the address to
>>> >> connect to, and then about 1s to connect. The rest is time spent
>>> >> gathering the information.
>>> >>
>>> >> I expect it to get a whole lot faster in a couple more weeks, but I'm
>>> >> not going to guarantee that until we've finished the work.
>>> >>
>>> >> 5) If I counted correctly, you have about 23 "machines" that are being
>>> >> considered. A bunch of them down/pending/errored.
>>> >>
>>> >> I would think for the errored ones you could do some sort of "juju
>>> >> destroy-machine". It might make things better (less time spent
>>> >> checking on machines you don't care about.)
>>> >>
>>> >> What happens when you try it? (There may be other issues that make us
>>> >> think we are waiting for something to happen with a machine that we
>>> >> don't want to destroy it.)
>>> >>
>>> >>
>>> >> Anyway in summary, this should be getting better, but I won't have
>>> >> explicit numbers until the work is done.
>>> >>
>>> >> John
>>> >> =:->
>>> >> -----BEGIN PGP SIGNATURE-----
>>> >> Version: GnuPG v1.4.13 (Cygwin)
>>> >> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>> >>
>>> >> iEYEARECAAYFAlIg4UIACgkQJdeBCYSNAAOfxQCeMaRQdqvdyQ11WyRnJ/WPAccp
>>> >> IysAniDrUq6IDtM0fu9SuZg+2AQto8rP
>>> >> =JaZw
>>> >> -----END PGP SIGNATURE-----
>>> >>
>>> >> --
>>> >> Juju mailing list
>>> >> Juju at lists.ubuntu.com
>>> >> Modify settings or unsubscribe at:
>>> >> https://lists.ubuntu.com/mailman/listinfo/juju
>>> >
>>> >
>>> >
>>> > --
>>> > Juju mailing list
>>> > Juju at lists.ubuntu.com
>>> > Modify settings or unsubscribe at:
>>> > https://lists.ubuntu.com/mailman/listinfo/juju
>>> >
>>
>>
>