Shell communication channel: simple, half-assed or fully-arsed?

[Thomas Voß]

On Thu, Dec 12, 2013 at 11:42 AM, Thomas Voß <thomas.voss at canonical.com> wrote:
> On Thu, Dec 12, 2013 at 11:11 AM, Christopher James Halse Rogers
> <raof at ubuntu.com> wrote:
>> On Thu, 2013-12-12 at 10:34 +0100, Thomas Voß wrote:
>>> On Thu, Dec 12, 2013 at 5:46 AM, Christopher James Halse Rogers
>>> <raof at ubuntu.com> wrote:
>>> > On Thu, 2013-12-12 at 10:01 +0800, Daniel van Vugt wrote:
>>> >> "Half-assed" is inaccurate I think. Under that heading you describe
>>> >> what we had already agreed to do (as much as the team can agree on
>>> >> something).
>>> >
>>> > *I* thought I was agreeing with the Simple/Simplistic method :). When
>>> > Alan brought up his stub implementation of Half-arsed it provoked a
>>> > further round of discussion, leading to here!
>>> >
>>> >> I suggest being careful trying to merge the existing areas of opacity in
>>> >> the protocol. I'm not sure they belong together.
>>> >>
>>> >> Protobuf is quite extendable already. Are we not using it in a way
>>> >> that's compatible with extension? I'm not a fan of protobuf, and am even
>>> >> open to the idea of trying the Wayland protocol, but I don't yet see a
>>> >> serious problem with protobuf that needs solving.
>>> >
>>>
>>> I don't see why we need a way to extend the protocol. We own every
>>> part of the stack, top to bottom, and I don't see why the shell would
>>> need a way to communicate to clients that is opaque to Mir.
>>
>> In part because we intend to *not* own every part of the stack - you
>> reiterated at the sprint a the goal is for Mir to be a generally-useful
>> library for writing display-server-compositor-shells, not simply a
>> unity8 subcomponent.
>>
>
> Fully agreed, but I don't think having a generic protocol extension
> mechanism helps in ensuring that we have a generally useful display
> server for a simple reason: Application developers and toolkits would
> then have to choose a shell that they specifically optimize for.
>
>> In part because we think there are things which need to be Mir-mediated
>> because they involve concepts like Surfaces, but which are Unity8
>> specific.
>>
>> In part because we want to Mir to have a long and prosperous life, and
>> X11's extension mechanism is what allowed X to survive for 30-odd years.
>>
>>
>
> Fair point, but let's not make everything an extension. While it is
> admittedly flexible to just come up with another extension (hey, it's
> just an extension), it is likely to end in a maintenance nightmare if
> it is overused. That being said, I think it's less a technical
> question here, but more a question of policy and taking decisions
> about functionality that should end up in the core.
>
> From a technical pov, just to make sure that we are on the same page,
> I have the following approach in mind (U8 as an example):
>
> Public facing API:
>   // Can fail with: Not supported by current shell and it shouldn't be
> called Mir* at all
>   u8_enable_specific_feature_for_surface(MirSurface* surface);
>
> Internally, this talks to Mir via a private API:
>
>   // Check that U8 is actually on the other side.
>   if (mir_connection_is_extension_supported(connection,
> U8_SPECIFIC_EXTENSION_ID))
>   {
>       msg = mir_connection_create_message_for_surface(connection, surface);
>       mir_message_set_destination(msg, U8_SPECIFIC_EXTENSION_RECEIVER_ID);
>       // Fill in payload
>       mir_connection_send(msg);
>   }
>

This could quite easily be implemented on top of:

  https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.dynamic_message

Cheers,

  Thomas

> Is that what you have in mind?
>
>>>  If it's an issue of unaligned development pace, we should rather fix
>>>  the process as opposed to decoupling Mir from what Unity8 needs.
>>
>> I don't think this *is* decoupling Mir from what Unity8 needs; it's
>> providing what Unity8 needs in a more flexible way.
>>
>
> As pointed out earlier, it's a very delicate balance to strike and it
> won't free us from the responsibility of deciding what is part of the
> core and what is not. I think that if we introduce the ability to
> extend the Mir API, we need to come up with a formal review process
> for extensions, too, specifically to ensure the general usefulness of
> Mir.
>
>>>
>>> > I think we are, and have been, conflating a number of separate issues
>>> > here.
>>> >
>>> > One is “what link-level IPC mechanism should we use? Sockets, shm, etc”.
>>> > We're currently using sockets+asio.
>>> >
>>> > One is “what message serialisation mechanism should we use? Protobuf,
>>> > Cap'n'Proto, Wayland, etc”. We're currently using Protobuf.
>>> >
>>> > A third is “what IPC interface definition language should we use?
>>> > Protobuf, Wayland, home-grown, etc”. We're currently not using *any*
>>> > IDL. For the bits where we are using an IDL we're using Protobuf's
>>> > descriptors, but we also have knowledge baked into the code - fd passing
>>> > because Protobuf just doesn't support that, and MirEvent, because of
>>> > reasons that I'm not aware of.
>>> >
>>>
>>> Let's fix it in one place in Mir then. MirEvent is a POD that was only
>>> meant to communicate input events, it was never meant to carry over
>>> arbitrary events to the client. This approach was triggered by our
>>> choice of input stack. Btw., protobuf offers an IDL and a DDL. I'm not
>>> sure why you think that we are not using it.
>>
>> We mostly use it, but we've gone outside protobuf in a way that means
>> mir_protobuf.proto is no longer a description of the interface.
>> Specifically, there's Event which is entirely opaque, and there's Buffer
>> which has ‘oh, by the way, there's stuff not in this interface’ field in
>> fds_on_side_channel.
>>
>> If protobuf was expressive enough to use as our IDL we'd have a
>> ‘repeated fd fds’ field and no fds_on_side_channel.
>>
>>>
>>> > I don't mind if we continue to use Protobuf for our serialisation, but
>>> > an extension mechanism needs an actual IDL. Which could be as simple as
>>> > extending Protobuf; I don't mind.
>>> >
>>>
>>> I guess it's less a question of IPC or serialization framework, but
>>> more a question of: Why do we need to support arbitrary extensions
>>> when we own the complete stack? In my understanding, this is a
>>> contradiction. With our conversation at the sprint in mind: I think we
>>> want an easy way to internally extend the protocol, that takes away
>>> most of the tedious boilerplate generation by hand. Is that correct?
>>
>> We want that, yes.
>>
>> But I think we do also want a general purpose extension mechanism,
>> designed in such a way that we can easily fold extensions into core
>> should we decide that they're useful in the general case.
>
> See before, let's not only think about technical feasibility but also
> about our responsibility towards app & toolkit developers.
>
> Cheers,
>
>   Thomas