The future of Charm Helpers

[Stuart Bishop]

On 11 August 2015 at 20:42, Marco Ceppi <marco.ceppi at canonical.com> wrote:


> # Trimming down charm-helpers
>
> The first item, and arguably the largest is a complete reorganization of the
> current charm-helpers code base. Originally, charm helpers was setup to
> allow a path for developers to place unstable code (contrib directory) where
> they could iterate without a guarantee of API stability. However, this
> quickly grew to include code that simply didn't make sense as a "core"
> feature of charm-helpers. This includes a large set of code for domain
> specific charm ecosystems (bigdata, openstack, storage) as well as quickly
> fixes to missing features in Juju (leader election).
>
> The amount of collaborative code in contrib is fantastic! However, it's
> making the release and distribution of charm-helpers more complicated. With
> an inability to keep up with documentation for methods contrib and the
> "volatile" nature of "unstable API", I'd like to propose we completely
> remove the contrib directory and instead urge code maintainers in contrib to
> create their own libraries for the feature and code they wish to maintain
> going forward which simply depends on charm-helpers for the code in core.
>
> This will allow groups like bigdata, openstack charmers, storage, and others
> to maintain their own release and review process for helpers specific to
> their solutions, as well as their own means of delivery (be it
> charm-helpers-sync, pypi, debian, or another means). This will also GREATLY
> slim down the charm-helpers library to instead include one of two things:
>
> - API in Python for communicating with Juju
> - Simple collection of tools in Python to solve shared universal problems
>
> Which roughly translates to hookenv.py, host.py, and fetch tools. Once
> contrib is removed and relocated I'd like to also propose flattening the
> library namespaces from
>
> charmhelpers.core.hookenv -> charmhelpers.hookenv
> charmhelpers.core.host -> charmhelpers.host
> charmhelpers.core.files -> charmhelpers.host
> charmhelpers.core.fstab -> charmhelpers.host
> charmhelpers.core.sysctl -> charmhelpers.host
> charmhelpers.core.kv -> separate project?
>
> Everything else, outside of contrib would remain the same. While I think
> it's important to slim down charm-helpers I'm not convinced the above
> outlined rename/code merges are the right or best way forward. I welcome
> feedback and a discussion around this.

I really like the big tree, including contrib.

I discover cool things in there, which I would not if the cool things
were hidden away in another branch I know nothing about. More and more
teams get created around the globe and communication between them
grows less and less. A shared repository helps in a small way.

And your test harnesses are already setup for you. It is easy to add
new cool stuff, and its tests. And those tests get run under both
Python 2 and Python 3 with the same versions of dependencies and your
cool stuff is much more likely to work with other peoples cool stuff.
I know for a fact if things get spit out they will have less rigorous
testing, and we will end up with tech debt Py2 only modules for
example, or conflicting dependencies.

But I don't see the large tree as a problem. Its not unwieldy. And you
don't have to worry about anything dumped into contrib, except from
the perspective of if it is worth migrating it to the core area and
how much work it would be (documentation, ensuring consistency,
maintainability etc.). I don't see the large tree as a problem because
you next talk about packaging.


> # Packaging charm-helpers
>
> Currently, charm-helpers are available via either bazaar source or PyPI.
> With a more stable, slim, and maintainable code base we would like to
> package charm-helpers still on PyPI with semantic versioning, make a
> dist/wheel[0] available for embedding (instead of charm-helpers-sycn), and
> in Debian and have them available via a PPA but also potentially in the
> archive proper.

Sounds cool. You don't have to package the entire tree. Just package
the core section.

If you are really enthusiastic, you could have the contrib modules
packaged separately if they provide a working setup.py. But wouldn't
bother. There will be lots of cases where we can't use the packages,
and using unstable contrib modules would just be one of them.

The main reason I will not be using your proposed packages for quite
some time is that I like to add things to charm-helpers and fix bugs
while working on my charms. I can't do that if I need to wait for a
release cycle, even if you could get it down to 24 hours. I will
remain using charm-helpers-sync.py, until I can switch to nested git
branches, but that is fine. Packages will still make lots of other
people's lives easier.


> The hardest challenge will be keeping the archive as up to date as possible
> with releases of charm-helpers. Since charms and Juju versions are decoupled
> from packages "frozen" in the archive, just having a version of
> charm-helpers in the archive for the version of Juju in that archive won't
> necessarily last the test of time. Someone deploying a charm now from a new
> version of Juju against trusty would get the 1.21 equivalent of
> charm-helpers which may not include the new leadership or extended status
> methods causing the charm to fail erroneously.
>
> Having a PPA can alleviate this, but PyPI and PPAs may interfere with
> firewalls and policies where Juju is being deployed. Opinions on how to help
> with this are appreciated. Otherwise, we may simply recommend authors to use
> a bit of bootstrap code which attempts to install from PyPI, with a fallback
> to install an embedded version of charmhelpers that was included in the
> charm as a dist/wheel.

Could we use simplestreams for this? Or whatever the mechanism is that
Juju uses to install the correct version of the juju tools tarball
when provisioning a unit?

deb packages are unsuitable, as a subordinate charm may require a
different version to the host charm. I think it has to be a wheel or
tarball, and installed in a unit specific location rather than
stuffing it into the system Python library.

Its not just bootstrapping on install either. upgrade-charm will need
to update the package too before it is imported.


> # Charm-helpers for other languages
>
> We already have charm-helpers written for three languages! Python[0],
> PowerShell[1], and Rust-lang[2]. By sliming down the charm-helpers we can
> define a minimum requirement that each new charm-helper language should
> expose. This will enable us to document charm-helpers more thoroughly and
> include code examples for each language supported in our docs. Also, by
> having a narrowly defined scope of methods, it enables other communities to
> create helpers for their languages.

Slimming down. I've bitched before about too much stuff being in
charmhelpers.core, and 'helped' by landing my own stuff I don't think
should be core in the parent package. I tend to think
charmhelpers.core should provide wrappers around the juju hook
environment, matching the documented juju hook environment behavior,
and required low level primitives you need for writing charms. It is
the foundation for the rest of charm-helpers, which can provide higher
level and friendlier interfaces. I believe that these higher level
tools should be Pythonic. So your cli wrappers can publish
charmhelpers.core for people insane or uneducated enough to write Bash
charms, and the trivial charmhelpers.core would be trivially ported to
other high level languages. But porting the Python decorators provided
by charmhelpers.coordinator or the Pythonic datastructures provided by
charmhelpers.context would not be helpful. Nor would porting
charmhelpers.core.templating, since it is currently Jinja2 specific
(and why it should be charmhelpers.templating).



> The PowerShell library was written by the folks at CloubBase, I had the
> chance to sit down with them a few months back and review how their
> structure looks. It's already very far along, includes unit testing, and
> matches the relative structure we have had in the Python charm helpers for
> some time. Going forward I'd like to continue this trend and sent some
> guidelines around future charm-helper libraries for other languages.
>
> # More exposure via the CLI
>
> Finally, we have a CLI interface for the charm-helpers python library.
> Recently there have been several merges which add exposure to additional
> methods and features of python charm-helpers. I'd like to continue this tend
> and add a CLI hook for everything in charm-helpers, within reason. A recent
> addition, by Cory Johns, enabled the internal key-value store in
> charm-helpers to be accesible via CLI. While making things like the
> "config_get" method available by CLI is silly, other key and core methods
> that exist as code in Python only and aren't a 1:1 bridge to Juju commands
> should also be made available. This will allow authors creating first drafts
> of charms in bash to access these higher methods.

Are you being helpful or are you being an enabler? At the risk of
derailing the discussion, I do wonder if we are doing people a
disservice by pretending that good, robust, maintainable charms can be
written in Bash. Sure, it is technically possible, but even simple
charms are complex, people capable or writing complex code in Bash are
rare, and the probability finding a team able to maintain it
approaches zero. We have spent a lot of time rewriting Bash charms in
Python purely to fix bugs and add features, because it was easier. Not
bitter, at all.


> # Conclusion
>
> I welcome discussion, dissent, and suggestions around these topics. They are
> by no means set in stone but are the assumed plan at the moment.

Ha ;)


> TL;DR: I want to make charm-helpers very slim, versioned, packaged,
> documented, and easily portable to other languages.

TL;DR: Cool. Save work by keeping contrib where it is and just not
packaging it. Consider charmhelpers.core low level and charmhelpers
root high level, with only charmhelpers.core exposed by cli. Move
unsuitable parts of charmhelpers.core to the top level module.

-- 
Stuart Bishop <stuart.bishop at canonical.com>