Using snap for application with plugins

[Thomas Voß]

On Tue, Aug 16, 2016 at 9:55 PM, Alexey Yakovenko <wakeroid at gmail.com> wrote:
> Hi Thomas
>
> Thanks for replying so quickly! Answering inline. :)
>

And sorry for the high latency now, your reply made it to my spam folder *sigh*

> On Tue, Aug 16, 2016 at 8:56 PM, Thomas Voß <thomas.voss at canonical.com>
> wrote:
>>
>> Hey Alexey,
>>
>> first of all: Thanks for reaching out :) And to answer your final
>> question first: Snap'ing up applications that feature
>> a runtime extension mechanism is certainly possible, complexity
>> depends on the actual application, though.
>>
>> To get you started fast: devmode (see
>> http://askubuntu.com/questions/783945/what-is-devmode-for-snaps)
>> should help to get you off the ground.
>
>
> Yes, thanks, I know about dev mode, of course running the app in dev mode
> just makes it work the same, as it would work without using snap, right?.
> I can also run my application from the mounted snap volume directly, with
> the same result. Of course this works.
>
> However, I suppose that this is not really the idea behind snap?
> Or is it one of the standard / expected use cases on the end users side, to
> install from snap, but run directly?
>
>>
>>
>> Also note that we are right now snap'ing our Unity8 scopes and dash
>> infrastructure. In a sense, our scopes are an example of a
>> really complex plugin system and we are likely facing very similar
>> problems.
>
>
> Well, yeah, perhaps, but in your case, as far as I understand, you can
> either build snaps for all of the scopes, or force the scope developers to
> do that.
> I can't have this luxury. Please, see below for more answers and more
> questions :)
>
>>
>>
>> I'll ask specific questions inline.
>>
>> On Tue, Aug 16, 2016 at 8:08 PM, Alexey Yakovenko <wakeroid at gmail.com>
>> wrote:
>> > Hi
>> >
>> > (Before I start, I'd like to apologise if this message has duplicates, I
>> > messed up with the mailing list subscription at first).
>> >
>> > Many users are requesting snap package format for my music player
>> > application.
>> >
>> > I researched this topic, and came up with a list of problems which I
>> > could
>> > not solve from documentation or google searches.
>> >
>> > So here's a short introduction to my app architecture:
>> >
>> > * The main module, which is a console application, which can load and
>> > use
>> > plugins of various kinds
>>
>> Is your application using an in- or out-of-process plugin architecture?
>
>
> In-process.
>
>>
>>
>> > * A set of standard plugins, which can link to certain libraries
>> >     * Example plugins: GTK2 UI, GTK3 UI, PulseAudio output, Various
>> > input
>> > plugins, etc
>>
>> Do you maintain those plugins in-tree? Would you be fine with
>> distributing them within your application's snap?
>> Or would you rather prefer to distribute them as separate snaps, with
>> you being the publisher?
>
>
> They are 3rd party projects, most often developed and distributed without my
> knowledge -- i.e. 100% third party.
> They can be released a few years after my application ships.
> Most of the time, they are also binary-compatible with multiple versions of
> the host application.
> There's a ton of plugins which can run on any version of the host app,
> released in the last 5 years.
> Even if I wanted to make snaps for each of them, I would not have resources
> to do that.
> So I can't really answer positively to any of the 2 questions.
> The plugins are just *.so files, sometimes with extra data files, which need
> to be dropped to ~/.local/lib/appname (that's when not using snap).
>
> Of course, the standard plugins which I ship with the host application don't
> have any problems, I think they would "just work".
>
>>
>>
>> > * External plugins, which a user may download from internet -- they can
>> > be
>> > anything, for example a plugin could implement a UI in Qt4 or Qt5, or
>> > ncurses, or play sound via Jack.
>>
>> Is the download and installation handled by your application or with
>> the help of the underlying packaging system?
>
>
> Neither. Usually a user downloads a zip file from a plugin developer's
> website, and unpacks it into ~/.local/lib/appname.
> If there are no binary builds available, a user has to compile the plugin by
> himself. Many developers are providing shell scripts to compile and install,
> or full featured build systems using Make etc.
>
> Unfortunately, neither the host application, nor the plugins, are available
> in most linux distribution package systems.
> This is what's making snap particularly interesting (if it can do what's
> needed).
>
>
>>
>> As to what degree do you "trust" those plugins and their authors?
>
>
> I don't know most of them.
> There are a few developers who I have contact with, but technically I can't
> trust any of them.
> It's completely up to the end user whether to install certain plugin, or not
> -- the same way as he would install an application downloaded from the
> internet.
>

Now that triggers a question for me: How do end users resolve
dependencies of plugins today?
Do they just assume that the downloaded plugins bring along whatever
they need? Or do they rely
on the package manager to resolve dependencies ad hoc?

In any case: It seems to me that shipping your application and its
core plugins as a snap is certainly possible.
For installing 3rd party plugins: Why not have an "app" or command on
your snap that takes care of copying files and folders
to the right place in the writable parts of the snap. I'm thinking
along the lines of:

  yourapp.install-plugin /path/to/file/or/folder

That would also help your users in that they don't need to know about
a specific directory to place plugins into.

>
>>
>> Would you be fine to publish them as well?
>
>
> I'm already publishing a bunch of the 3rd party plugins, but I can't publish
> all of them for many reasons.
> There are simply too many of these projects. I can't even keep track of
> them.
> Basically, people release new ones almost weekly.
>
>>
>>
>> >     * It is not known at the time of package creation, what these
>> > packages
>> > could be.
>> >
>> > So when I'm creating the package, I have a problem -- which UI toolkit
>> > to
>> > use, if my app supports any toolkit? It ships with GTK2 and GTK3, but
>> > there
>> > are external plugins which add more toolkits. I can't really predict
>> > which
>> > toolkit the user wants, and what libraries the plugins will use -- it
>> > can be
>> > Qt4 or 5, or ncurses, or anything really.
>> >
>>
>> I'm trying to rephrase a little: Your application supports plugins
>> which may use *any* toolkit
>> as long as the plugin bundles the respective runtime dependencies
>> correctly. So it is ultimately the
>> responsibility of the plugin to provide all the required libraries.
>
>
> Yeah, so the answer is "a plugin which is using Qt5 could work, but it would
> need to be in snap package, and has to ship its own version of Qt5" as an
> example?

Yup, but that comes back to my question: How are dependencies for the plugins
handled today?

> But still, each of the plugin developers would have to provide each of their
> plugins in snap packages?

Not necessarily, see my comment before. You could decide to support both ways.

> The problem is that I can't make them do that. I don't have communication
> with them.
> Some developers build a plugin, make a .so out of it, put this on their
> website, and forget about it forever.
> So we sometimes have only this .so file, without source code even.
>
> What is also important: a user might already have a bunch of plugins
> installed on his computer, and he would obviously want or need to keep them,
> so if he can't use them with deadbeef from Snap -- he would not be using
> deadbeef from snap at all, and stick with another installation method (one
> of the existing ones).
>

Access to common shared directories in the user's home directory is possible.
I think that should solve the problem of supporting old plugin installations.

> A more complex question that I couldn't find an answer too:
> How to make a snap, which can use both GTK2 and GTK3.
>
> To simplify the discussion, let's say I have 2 executable files, app_gtk2,
> and app_gtk3. They ship in the same snap.
> Is it possible to do this, so that either of the 2 apps can be used from
> within the same snap package?
>

Sure, you can have multiple apps/commands defined on your snap and
just bundle all of
their dependencies up. snapcraft helps you in crafting such a snap.

HTH,

  Thomas

> So what I described above is a simplified version.
>
> What I really have: there's only 1 executable file, but it can load either a
> GTK2 plugin (ui_gtk2.so), or a GTK3 plugin (ui_gtk3.so), depending on the
> configuration and command line options.
>
>
>>
>> > Another problem. Let's take the jack output plugin as an example..
>> >
>> > If the user downloads jack.so (the plugin), and puts it in the folder,
>> > the
>> > player won't be able to load it, because the jack client library is not
>> > a
>> > part of snap, or because there's no access to jack server, or something
>> > like
>> > that, correct?
>> >
>>
>> The answer depends on how your application handles plugin download &
>> installation.
>> The issues would be solved if plugins were snaps. In that case, the
>> individual plugin brings
>> along its dependencies and defines the interfaces it would like to use
>> (see [1] for an excellent introduction & overview to
>> Snappy interfaces).
>
>
> Application itself doesn't manage plugin download & installation at all.
> It can load and run the plugins which are already downloaded and present in
> one of the pre-defined folders.
>
> There's no central registry of all plugins, therefore it's not really
> possible to change this easily.
>
>>
>>
>> > So, the final question.. Is it just a terrible idea to use snap for this
>> > kind of applications? Or am I misunderstanding something, and all of
>> > this is
>> > possible?
>> >
>> > I really hope to get some useful "official" answers, which I could
>> > forward
>> > to the users which request snap packages :)
>> >
>>
>> Looking forward to your answers and to dive deeper in your specific case.
>>
>> Cheers,
>>
>>   Thomas
>>
>> P.S.: Feel free to reach out on irc(#snappy at freenode), too.
>
>
> Sure, let's see the outcome of this round of questions, and then we can see
> if it's worth to continue.
>
>>
>>
>> [1] http://www.zygoon.pl/2016/08/creating-your-first-snappy-interface.html
>>
>> > Best regards.
>> >
>> > --
>> > Snapcraft mailing list
>> > Snapcraft at lists.snapcraft.io
>> > Modify settings or unsubscribe at:
>> > https://lists.ubuntu.com/mailman/listinfo/snapcraft
>> >
>
>