[ANN] charm-tools 1.9.3

[Cory Johns]

PyYAML includes a pure-Python version which would be used if building fails.

I'm not entirely averse to coming up with a way to specify an exclude list
for wheel deps that would be provided in some other way, but what if one
layer excludes a wheel that another layer explicitly includes?  And how
would we specify that list?  We'd probably want some sort of pattern
matching because there doesn't seem to be a way to tell `pip wheel` to
exclude certain packages, so we'd have to do it based off the wheel files
themselves.

On Fri, Nov 27, 2015 at 5:02 PM, Simon Davy <bloodearnest at gmail.com> wrote:

>
>
> On Friday, 27 November 2015, Marco Ceppi <marco.ceppi at canonical.com>
> wrote:
> > On Thu, Nov 26, 2015 at 3:05 AM Simon Davy <bloodearnest at gmail.com>
> wrote:
> >>
> >> On Thursday, 26 November 2015, Marco Ceppi <marco.ceppi at canonical.com>
> wrote:
> >> > On Wed, Nov 25, 2015 at 4:08 PM Simon Davy <bloodearnest at gmail.com>
> wrote:
> >> >>
> >> >> On 25 November 2015 at 16:02, Marco Ceppi <marco.ceppi at canonical.com>
> wrote:
> >> >> > ## Wheel House for layer dependencies
> >> >> >
> >> >> > Going forward we recommend all dependencies for layers and charms
> be
> >> >> > packaged in a wheelhouse.txt file. This perform the installation
> of pypi
> >> >> > packages on the unit instead of first on the local machine meaning
> Python
> >> >> > libraries that require architecture specific builds will do it on
> the units
> >> >> > architecture.
> >> >>
> >> >> If I'm understanding the above correctly, this approach is a blocker
> for us.
> >> >>
> >> >> We would not want to install direct from pypi on a production service
> >> >>
> >> >>  1) pypi packages are not signed (or when they are, pip doesn't
> verify
> >> >> the signature)
> >> >>  2) pypi is an external dependency and thus unreliable (although not
> >> >> as bad these days)
> >> >>  3) old versions can disappear from pypi at an authors whim.
> >> >>  4) installing c packages involves installing a c toolchain on your
> prod machine
> >> >>
> >> >> Additionally, our policy (Canonical's, that is), does not allow
> access
> >> >> to the internet on production machines, for very good reasons. This
> is
> >> >> the default policy in many (probably most) production environments.
> >> >>
> >> >> Any layer or charm that consumes a layer that uses this new approach
> >> >> for dependencies would thus be unusable to us :(
> >> >>
> >> >> It also harms repeatability, and I would not want to use it even if
> >> >> our access policy allowed access to pypi.
> >> >>
> >> >> For python charm dependencies, we use system python packages as much
> >> >> as possible, or if we need any wheels, we ship that wheel in the
> >> >> charm, and pip install it directly from the there. No external
> >> >> network, completely repeatable.
> >> >
> >> > So, allow me to clarify. If you review the pastebin outputs from the
> original announcement email, what this shift does is previously `charm
> build` would create and embed installed dependencies into the charm under
> lib/ much like charm-helper-sync did for instead for any arbitrary Pypi
> dependency. Issues there are for PyYAML it will build a yaml.so file which
> would be built based on the architecture of your machine and not the cloud.
> >>
> >> Right. This was the bit which confused me, I think.
> >>
> >> Can we not just use python-yaml, as its installed by default on cloud
> images anyway?
> >>
> >> We use virtualenv with --system-site-packages, and use system packages
> for python libs with c packages where possible, leaving wheels for things
> which aren't packaged or we want newer versions of.
> >>
> >
> > Again, this is for hook dependencies, not exactly for dependencies of
> the workload.
>
> Right. I understand that :)
>
> I detailed how we solve this for our python app payloads as a possible
> solution for how to solve it for python charm deps also, but of course
> those deps would be completely separate things, not even installed in the
> same virtualenv.
>
>
> > The charm could apt intall python-yaml, but using --system-site-packages
> when building is something I'd discourage since not everyone has the same
> apt pacakges installed.
>
> Except that they do specifically have python-yaml installed, I believe.
> Its installed by default in Ubuntu cloud images, due to cloud-init I think.
>
> But yes, other system python packages could be exposed. I wish once again
> there was a way to include specific list of system packages in a virtualenv
> rather than all them.
>
> And it should be easy enough to add a way to declare which system packages
> are required by a layer?
>
> >Unless that user is building on a fresh cloud-image there's a chance they
> won't catch some packages that don't get declared.
> > We'd be interested in making this a better story. The wheelhousing for
> dependencies not yet available in the archive instead of embedding them in
> the charm was a first step but certainly not the last. I'm not sure how
> this would work when we generate a wheelhouse since the wheelhouse
> generation grabs dependencies of the install. That's why PyYAML is showing
> up in the generated charm artifact. We're not explicitly saying "included
> PyYAML" we're simply saying we need charmhelpers and charms.reactive from
> PyPI as a minimum dependency for all charm hooks built with charm build to
> work. Suggestions around this are welcome.
>
> Right, the wheelhouse seems a good approach for that. I'm just wondering
> if we can do a specific solution for python-yaml that avoids the binary
> wheel (which AIUI is brittle even on same arch, due to glibc/cc verions)
>
> I would be surprised if there were many charm python deeps that had c
> extensions ?
>
> Also, iirc there is a pure python yaml lib? Given speed is not an issue
> here (as the yaml is small and infrequently parsed), maybe we could look at
> that?
>
>
> Thanks
>
> --
> Simon
>
>
>
>
> --
> Simon
>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20151130/3a6c22cb/attachment.html>