[ubuntu-mono] [Bug 412503] Re: OpenBve+Class 323 does not support plugin DLLs on Linux

[Paul Sladen]

** Changed in: bve-train-br-class-323 (Ubuntu)
       Status: Confirmed => Opinion

** Changed in: bve-train-br-class-323 (Ubuntu)
     Assignee: Janis Fehrmann (janis96) => (unassigned)

** Changed in: openbve (Ubuntu)
       Status: Confirmed => Opinion

-- 
OpenBve+Class 323 does not support plugin DLLs on Linux
https://bugs.launchpad.net/bugs/412503
You received this bug notification because you are a member of Ubuntu
CLI/Mono Uploaders, which is subscribed to openbve in ubuntu.

Status in “bve-train-br-class-323” package in Ubuntu: Opinion
Status in “openbve” package in Ubuntu: Opinion

Bug description:
Binary package hint: openbve

Routes and trains distributed for the BVE Trainsim (proprietary) and OpenBve (in Ubuntu) railway simulators consist primarily of text-based configuration and model files, which in turn reference to PNG/BMP/JPG images and WAV/OGG sound files.

In addition, the BVE Trainsim version 4 format allows each simulation/scenario to be programmatically extended by a compiled plugin.  The format defines that this plugin is compiled to a Win32/PE+COFF/i386 DLL file.  OpenBve (when also running on Win32) attempts to be backwards-compatible and load these Win32-based DLL plugins.  Each DLL plugin exports a series of defined callbacks (the BVE4 plugin API) that can override the internal (default) BVE4/OpenBve state engine;  for example to simulate country-specifc, or train-specific safety interlocks and cab-based signalling systems.  The naming of plugins reflects the Japanese Automatic Train Stop (ATS) signalling system, where BVE4 originated from.

Inputs to the plugin arrive every frame, giving the state of the train, and ".Beacon" updates (one-way transponder messages placed along the track) , and from keypresses submitted by the user (the keys: Insert/Delete/Home/End/PageUp/PageDown/Spacebar and 2 to 9 are exposed).  Outputs are in the form of forced changes to the state machine (for example, hitting the brakes when passing a red signal), updates to the list of sounds being played, and the exporting of up to 256 variables in the format ats0-ats255 to other parts of the simulation.  The variables can be referenced wthin a train's cab-panel description or internal/external 3D model to reflect the operational state of the train;  for example showing a raised, or lowered paragraph, or by drawing individual raindrops on the windscreen.

These plugins are generally non-free, non-distributable, and it is very unusual that the source-code is available.  In the case of the 'OS_Ats1' plugin (last release 1.3 in 2005), source code *is* available and I have had communications with the author of the plugin to investigate releasing the source code under a DFSG-free licence.  Possibly the GPL (already confirmed) or ideally placing the plugin code in the Public Domain (tentatively confirmed).  The 'bve-train-br-class-323' train already in Ubuntu would normally be shipped wth a binary plugin called 'UKMUt.dll' to control its extended functionality.  The 'UKMUt' (non-free) and 'OS_Ats1' (soon to be free) plugins share about 90% equivalent capabilities and I have written a configuration file that allows 'OS_Ats1' to be a close-enough work-alike for the missing 'UKMUt' plugin. Branches of OS_Ats1 are packaged up in Git at:

  http://github.com/sladen/bve-plugin-os-ats/tree/debian

When a train developer chooses to request the additional services provided by a plugin, a file named 'ats.cfg' is included in the appropriate 'Train/' directory (eg. '/usr/share/games/bve/Train/BR_Class_323/ats.cfg') and this file contains a single undelineated line giving the name of the plugin including its ".dll" suffix ('OS_Ats1.dll').

The code path used by OpenBve to load and link to these plugins is to first extract the (optional) plugin name from the 'ats.cfg' file, then to thunk from the portable C#/CLR code to and from the Win32 DLL via an intermediate proxy:

  'OpenBve.exe' (CLR/CLI) <> 'AtsPluginProxy.dll' (Win32/i386) <> 'Something_Ats.dll' (Win32/i386)

The upstream binary distribution includes a non-portable binary compilation of 'AtsPluginProxy.dll', this step is only present as it appears that CLR effectively resolves DllImports at *compile time*, rather than allowing *dynamic importing* (such as dlopen()).   Upstream did not really envision plugins being used on platforms other than Win32+i386 and so the codebase has included long-term checks and *hard* warning messages that active loading from being attempted:

  if(Program.CurrentPlatform != Program.Platform.Windows)
    "...plugins are not supported on operating systems other than Windows."
  if (IntPtr.Size != 4)
    "...plugins are not supported in 64-bit environments."

I can confirm that refining these two checks allows the OpenBve code base to load and use a natively compiled version of 'OS_Ats1.so', via (an also natively-compiled) 'AtsPluginProxy.so'---basically unmodified.  The patches to narrow these two checks along with the other tweaks were provisionally presented to upstream for review (see the following Git branch):

  http://github.com/sladen/openbve/tree/native-plugin-support
  http://www.paul.sladen.org/openbve/patches/1.2.0.0-native-plugin-support/

Some of the patches were accepted and integrated.  However, in the case of specifically removing/refining/fixing the Win32 check to allow loading of _natively compiled_ .so plugins on Linux, I have been informed that this would constitute a fork, requiring the renaming of .deb packages and removal of all references to OpenBve from the packages.  The renaming/debranding away from the upstream OpenBve is not desirable, even if it has been done in the cases of Iceweasel/abrowser as a last resort---additionally upstream have indicated that they would likely cut off future communication if the fork route was chosen.

Upstream have indicated they wish to see plugins continue to be distributed *only* as Win32/PE+COFFs DLLs.  The argument, as I understand it, is that the selection of the format for images, routes files and plugins was chosen by BVE Trainsim 4 (a proprietary program) and that those chosen (binary/text) formats are supported only for backwards compatibility reasons.

A fear was presented that allowing other compiled formats of plugins (eg. Libc/ELF/armle) might lead to additional incompatibility issues;  an MS Windows user might complain that a plugin compiled for some other processor+ABI type and called MyPlugin.so would not load (as a binary) on /their/ system.  A newer release of OpenBve now includes a minimal check for an 'MZ' (DOS executable) header.

Technically, the solution for "making [source available] plugins work on Linux" is very easy (removal of the overly-zealous Platform check), however, politcally it is a very different issue.  In essence, whether to achieve backwards-compatibility for plugins at the compiled binary level, or at the source code level is a difference of opinion.  Upstream have provided a list of ten (non-free) binary i386+PE+Win32 DLLs that must be proven working for any proposed solution to be deemed acceptable.

In order to satisfy OpenBve's upstream's requirements and those of Debian Policy, it will been necessary to cross-compile any plugin source-code to an i386+PE DLL (build dependency on 'mingw32').  Then to load this foreign i386+PE DLL shared library it will be necessary to parse the PE headers.  For platforms !i386 it will be necessary to use emulation.

Load leaf-node DLLs (library which do not then go on to import any further external references) is relatively easy;  the classic example is the 'win32-codecs' package and the loaders in FFMPEG.  However, upon analysis most of the reviewed plugins further import Win32 API functions;  such as to read and locate their own configuration files.  Implementations of these further imported functions must be provided and the plugin "format" provides no restriction on what can be used.

Long-term it appears possible to write (from scratch) aPublic Domain pure C# PE parser, a pure C# i386 user-space interpreter (x86 emulator) and pure C# stub implementations of any imported Win32 functions, as these are discovered.  In addition, bundling a built-in emulator+loader would on Linux/* but also Win64/amd64, Win64/ia64 and WinCE/* and Mac OSX without additional dependencies;  not of these platforms have plugin functionality either.  This avenue of development will be extremely time-intensive and although a very small amount of work has already been done in this direction, it boils down to reimplementing a subset of Qemu and Wine in a new language (C#).

A shorter-term (quicker) option would be to just *use* Wine to parse and load plugins and then to (crucially) resolve any Win32 imports---as Wine has a fairly complete set of of Win32 API function implementations already.  On !i386, Qemu can then be used in addition, to provide the dynamic binary translation from i386 to the native instruction set.  It should be possible to write a loader Win32 .exe program (somewhat equivalent to 'AtsPluginProxy'); this could be started inside Wine and to communicate and receive requests over a TCP connection opened by the parent CLR/CLI OpenBve.exe instance.  This would ensure that cross-binary and cross-instruction set data exchanges only happen using a clearly defined wire, rather than via the exchange of pointers to unmanaged in-memory arrays.

Such a solution would have to 'Depends: wine' an entail introducing a large download for those demanding plugin support (just to support a 10 kB plugin).  It would however work on Win64/amd64 systems without Wine, where the equivalent backwards-compatible Win32 API is already provided, and/or emulated.

In summary; the availability of out-of-the-box BVE plugins on Linux is not a technical problem, but a legal/political one: (a) source code of plugins is not generally available, nor DFSG-free for packaging, and (b) OpenBve actively restricts plugins to only one specific binary format and CPU instruction set.

We have a partial solution for (a) in the form of the source code and support of the OS_Ats1 author;  but for (b) more work still needs to be done to support the requirements of upstream.