Shell communication channel: simple, half-assed or fully-arsed?

[Thomas Voß]

On Thu, Dec 12, 2013 at 11:11 AM, Christopher James Halse Rogers
<raof at ubuntu.com> wrote:
> On Thu, 2013-12-12 at 10:34 +0100, Thomas Voß wrote:
>> On Thu, Dec 12, 2013 at 5:46 AM, Christopher James Halse Rogers
>> <raof at ubuntu.com> wrote:
>> > On Thu, 2013-12-12 at 10:01 +0800, Daniel van Vugt wrote:
>> >> "Half-assed" is inaccurate I think. Under that heading you describe
>> >> what we had already agreed to do (as much as the team can agree on
>> >> something).
>> >
>> > *I* thought I was agreeing with the Simple/Simplistic method :). When
>> > Alan brought up his stub implementation of Half-arsed it provoked a
>> > further round of discussion, leading to here!
>> >
>> >> I suggest being careful trying to merge the existing areas of opacity in
>> >> the protocol. I'm not sure they belong together.
>> >>
>> >> Protobuf is quite extendable already. Are we not using it in a way
>> >> that's compatible with extension? I'm not a fan of protobuf, and am even
>> >> open to the idea of trying the Wayland protocol, but I don't yet see a
>> >> serious problem with protobuf that needs solving.
>> >
>>
>> I don't see why we need a way to extend the protocol. We own every
>> part of the stack, top to bottom, and I don't see why the shell would
>> need a way to communicate to clients that is opaque to Mir.
>
> In part because we intend to *not* own every part of the stack - you
> reiterated at the sprint a the goal is for Mir to be a generally-useful
> library for writing display-server-compositor-shells, not simply a
> unity8 subcomponent.
>

Fully agreed, but I don't think having a generic protocol extension
mechanism helps in ensuring that we have a generally useful display
server for a simple reason: Application developers and toolkits would
then have to choose a shell that they specifically optimize for.

> In part because we think there are things which need to be Mir-mediated
> because they involve concepts like Surfaces, but which are Unity8
> specific.
>
> In part because we want to Mir to have a long and prosperous life, and
> X11's extension mechanism is what allowed X to survive for 30-odd years.
>
>

Fair point, but let's not make everything an extension. While it is
admittedly flexible to just come up with another extension (hey, it's
just an extension), it is likely to end in a maintenance nightmare if
it is overused. That being said, I think it's less a technical
question here, but more a question of policy and taking decisions
about functionality that should end up in the core.

>From a technical pov, just to make sure that we are on the same page,
I have the following approach in mind (U8 as an example):

Public facing API:
  // Can fail with: Not supported by current shell and it shouldn't be
called Mir* at all
  u8_enable_specific_feature_for_surface(MirSurface* surface);

Internally, this talks to Mir via a private API:

  // Check that U8 is actually on the other side.
  if (mir_connection_is_extension_supported(connection,
U8_SPECIFIC_EXTENSION_ID))
  {
      msg = mir_connection_create_message_for_surface(connection, surface);
      mir_message_set_destination(msg, U8_SPECIFIC_EXTENSION_RECEIVER_ID);
      // Fill in payload
      mir_connection_send(msg);
  }

Is that what you have in mind?

>>  If it's an issue of unaligned development pace, we should rather fix
>>  the process as opposed to decoupling Mir from what Unity8 needs.
>
> I don't think this *is* decoupling Mir from what Unity8 needs; it's
> providing what Unity8 needs in a more flexible way.
>

As pointed out earlier, it's a very delicate balance to strike and it
won't free us from the responsibility of deciding what is part of the
core and what is not. I think that if we introduce the ability to
extend the Mir API, we need to come up with a formal review process
for extensions, too, specifically to ensure the general usefulness of
Mir.

>>
>> > I think we are, and have been, conflating a number of separate issues
>> > here.
>> >
>> > One is “what link-level IPC mechanism should we use? Sockets, shm, etc”.
>> > We're currently using sockets+asio.
>> >
>> > One is “what message serialisation mechanism should we use? Protobuf,
>> > Cap'n'Proto, Wayland, etc”. We're currently using Protobuf.
>> >
>> > A third is “what IPC interface definition language should we use?
>> > Protobuf, Wayland, home-grown, etc”. We're currently not using *any*
>> > IDL. For the bits where we are using an IDL we're using Protobuf's
>> > descriptors, but we also have knowledge baked into the code - fd passing
>> > because Protobuf just doesn't support that, and MirEvent, because of
>> > reasons that I'm not aware of.
>> >
>>
>> Let's fix it in one place in Mir then. MirEvent is a POD that was only
>> meant to communicate input events, it was never meant to carry over
>> arbitrary events to the client. This approach was triggered by our
>> choice of input stack. Btw., protobuf offers an IDL and a DDL. I'm not
>> sure why you think that we are not using it.
>
> We mostly use it, but we've gone outside protobuf in a way that means
> mir_protobuf.proto is no longer a description of the interface.
> Specifically, there's Event which is entirely opaque, and there's Buffer
> which has ‘oh, by the way, there's stuff not in this interface’ field in
> fds_on_side_channel.
>
> If protobuf was expressive enough to use as our IDL we'd have a
> ‘repeated fd fds’ field and no fds_on_side_channel.
>
>>
>> > I don't mind if we continue to use Protobuf for our serialisation, but
>> > an extension mechanism needs an actual IDL. Which could be as simple as
>> > extending Protobuf; I don't mind.
>> >
>>
>> I guess it's less a question of IPC or serialization framework, but
>> more a question of: Why do we need to support arbitrary extensions
>> when we own the complete stack? In my understanding, this is a
>> contradiction. With our conversation at the sprint in mind: I think we
>> want an easy way to internally extend the protocol, that takes away
>> most of the tedious boilerplate generation by hand. Is that correct?
>
> We want that, yes.
>
> But I think we do also want a general purpose extension mechanism,
> designed in such a way that we can easily fold extensions into core
> should we decide that they're useful in the general case.

See before, let's not only think about technical feasibility but also
about our responsibility towards app & toolkit developers.

Cheers,

  Thomas