[Ecosystem-engineering] The future of Charm Helpers

Tue Aug 11 15:17:19 UTC 2015

Have we considered using Python namespace packages (
https://pythonhosted.org/setuptools/setuptools.html#namespace-packages) as
the spiritual replacement for contrib?  I have already started using the
"charms" namespace in charms.reactive, and I think it could give us a nice
balance between having specialized helpers be independently owned and
maintained, while still giving a (loose) sense of cohesion to those
packages and making it clear that they are all tools for Charming.

I'd also like to note that the upcoming "composer" addition to charm-tools
includes some very nice features that will make dependency management of
charm-helpers much easier.  Specifically, a layer can include a .pypi file
that contains a Python requirement string which, during composition of the
layers (prior to releasing to the store) will cause the package to be
inlined into the charm (sort of like a simplified charm-helpers-sync).  The
key feature here is that you can use  semantic version ranges, so that if
you specify "charmhelpers>=2.0.0,<3.0.0" then whenever you re-compose
(refresh) your charm, you will automatically get backwards compatible
updates within the 2.x version, but will never get backwards incompatible
changes (as long as charmhelpers follows SemVer).  And this all happens at
"compile time" for the charm, so run-time deployments are 100% consistent
and repeatable.

Back on track, I personally lean toward keeping unitdata in charmhelpers
simply because we have historically had a big problem with many charms
using many different means to persist data and I think we need to make it
as easy as possible to have a single, consistent, recommended way to do
that.  I think that including unitdata in charmhelpers and exposing it via
the CLI addresses that (though I'm not averse to another solution to that
core problem).

On Tue, Aug 11, 2015 at 10:15 AM, Marco Ceppi <marco.ceppi at canonical.com>
wrote:

> On Tue, Aug 11, 2015 at 10:00 AM Matt Bruzek <matthew.bruzek at canonical.com>
> wrote:
>
>> Thanks for the write up Marco.
>>
>> I am particularly interested in growing the CLI aspect of charm-helpers
>> to aid authors who still use bash charms.  Also cleaning up the docs so
>> they make sense and is easier to understand what some methods do.
>>
>> The idea of contrib was so that we could try out a few methods and if it
>> worked well and was useful we pull that into core.  Is there any other
>> contrib features that would be pulled in? How would someone propose a new
>> feature in this new model?
>>
>
> That was the idea for using contrib. Proposing new features into "core"
> will be like most other projects, either opening an issue/bug on the
> charm-helpers project to outline what you want added or submitting code as
> a merge request for review.
>
> For those who want the contrib lifestyle of developing helpers and using
> them in charms but not submitting them yet, branching on launchpad or any
> other VCS and having pip install from a VCS is a viable workflow[0] and one
> I think we should promote.
>
> Marco
>
> [0] https://pip.pypa.io/en/latest/reference/pip_install.html#vcs-support
>
>
>>
>> Thanks,
>>
>>    - Matt Bruzek <matthew.bruzek at canonical.com>
>>
>> On Tue, Aug 11, 2015 at 8:42 AM, Marco Ceppi <marco.ceppi at canonical.com>
>> wrote:
>>
>>> Hello everyone,
>>>
>>> There have been a lot of conversations had over the past few months
>>> about charm-helpers and how to proceed forward with the project. As a
>>> result of these conversations I'd like to surface the following items for
>>> discussion:
>>>
>>> # Trimming down charm-helpers
>>>
>>> The first item, and arguably the largest is a complete reorganization of
>>> the current charm-helpers code base. Originally, charm helpers was setup to
>>> allow a path for developers to place unstable code (contrib directory)
>>> where they could iterate without a guarantee of API stability. However,
>>> this quickly grew to include code that simply didn't make sense as a "core"
>>> feature of charm-helpers. This includes a large set of code for domain
>>> specific charm ecosystems (bigdata, openstack, storage) as well as quickly
>>> fixes to missing features in Juju (leader election).
>>>
>>> The amount of collaborative code in contrib is fantastic! However, it's
>>> making the release and distribution of charm-helpers more complicated. With
>>> an inability to keep up with documentation for methods contrib and the
>>> "volatile" nature of "unstable API", I'd like to propose we completely
>>> remove the contrib directory and instead urge code maintainers in contrib
>>> to create their own libraries for the feature and code they wish to
>>> maintain going forward which simply depends on charm-helpers for the code
>>> in core.
>>>
>>> This will allow groups like bigdata, openstack charmers, storage, and
>>> others to maintain their own release and review process for helpers
>>> specific to their solutions, as well as their own means of delivery (be it
>>> charm-helpers-sync, pypi, debian, or another means). This will also GREATLY
>>> slim down the charm-helpers library to instead include one of two things:
>>>
>>> - API in Python for communicating with Juju
>>> - Simple collection of tools in Python to solve shared universal problems
>>>
>>> Which roughly translates to hookenv.py, host.py, and fetch tools. Once
>>> contrib is removed and relocated I'd like to also propose flattening the
>>> library namespaces from
>>>
>>> charmhelpers.core.hookenv -> charmhelpers.hookenv
>>> charmhelpers.core.host -> charmhelpers.host
>>> charmhelpers.core.files -> charmhelpers.host
>>> charmhelpers.core.fstab -> charmhelpers.host
>>> charmhelpers.core.sysctl -> charmhelpers.host
>>> charmhelpers.core.kv -> separate project?
>>>
>>> Everything else, outside of contrib would remain the same. While I think
>>> it's important to slim down charm-helpers I'm not convinced the above
>>> outlined rename/code merges are the right or best way forward. I welcome
>>> feedback and a discussion around this.
>>>
>>> # Packaging charm-helpers
>>>
>>> Currently, charm-helpers are available via either bazaar source or PyPI.
>>> With a more stable, slim, and maintainable code base we would like to
>>> package charm-helpers still on PyPI with semantic versioning, make a
>>> dist/wheel[0] available for embedding (instead of charm-helpers-sycn), and
>>> in Debian and have them available via a PPA but also potentially in the
>>> archive proper.
>>>
>>> The hardest challenge will be keeping the archive as up to date as
>>> possible with releases of charm-helpers. Since charms and Juju versions are
>>> decoupled from packages "frozen" in the archive, just having a version of
>>> charm-helpers in the archive for the version of Juju in that archive won't
>>> necessarily last the test of time. Someone deploying a charm now from a new
>>> version of Juju against trusty would get the 1.21 equivalent of
>>> charm-helpers which may not include the new leadership or extended status
>>> methods causing the charm to fail erroneously.
>>>
>>> Having a PPA can alleviate this, but PyPI and PPAs may interfere with
>>> firewalls and policies where Juju is being deployed. Opinions on how to
>>> help with this are appreciated. Otherwise, we may simply recommend authors
>>> to use a bit of bootstrap code which attempts to install from PyPI, with a
>>> fallback to install an embedded version of charmhelpers that was included
>>> in the charm as a dist/wheel.
>>>
>>> # Charm-helpers for other languages
>>>
>>> We already have charm-helpers written for three languages! Python[0],
>>> PowerShell[1], and Rust-lang[2]. By sliming down the charm-helpers we can
>>> define a minimum requirement that each new charm-helper language should
>>> expose. This will enable us to document charm-helpers more thoroughly and
>>> include code examples for each language supported in our docs. Also, by
>>> having a narrowly defined scope of methods, it enables other communities to
>>> create helpers for their languages.
>>>
>>> The PowerShell library was written by the folks at CloubBase, I had the
>>> chance to sit down with them a few months back and review how their
>>> structure looks. It's already very far along, includes unit testing, and
>>> matches the relative structure we have had in the Python charm helpers for
>>> some time. Going forward I'd like to continue this trend and sent some
>>> guidelines around future charm-helper libraries for other languages.
>>>
>>> # More exposure via the CLI
>>>
>>> Finally, we have a CLI interface for the charm-helpers python library.
>>> Recently there have been several merges which add exposure to additional
>>> methods and features of python charm-helpers. I'd like to continue this
>>> tend and add a CLI hook for everything in charm-helpers, within reason. A
>>> recent addition, by Cory Johns, enabled the internal key-value store in
>>> charm-helpers to be accesible via CLI. While making things like the
>>> "config_get" method available by CLI is silly, other key and core methods
>>> that exist as code in Python only and aren't a 1:1 bridge to Juju commands
>>> should also be made available. This will allow authors creating first
>>> drafts of charms in bash to access these higher methods.
>>>
>>> # Conclusion
>>>
>>> I welcome discussion, dissent, and suggestions around these topics. They
>>> are by no means set in stone but are the assumed plan at the moment.
>>>
>>> TL;DR: I want to make charm-helpers very slim, versioned, packaged,
>>> documented, and easily portable to other languages.
>>>
>>> Thanks,
>>> Marco Ceppi
>>> Juju Developer Experience group
>>>
>>> [0] http://pythonwheels.com/
>>> [1] https://github.com/cloudbase/juju-powershell-modules
>>> [2] https://crates.io/crates/juju
>>>
>>> --
>>> Mailing list: https://launchpad.net/~ecosystem-engineering
>>> Post to     : ecosystem-engineering at lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~ecosystem-engineering
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
> --
> Mailing list: https://launchpad.net/~ecosystem-engineering
> Post to     : ecosystem-engineering at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~ecosystem-engineering
> More help   : https://help.launchpad.net/ListHelp
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20150811/cdee3a19/attachment.html>