<div dir="ltr"><div>Hi,</div><div>I'm reaching out to you to get some guidance how to handle a soname versioning issue I didn't face before.</div><div><br></div><div><b>TL;DR:</b></div><div>How to correctly package a source creating libraries with individual ABI's bumped separately but that depend on each other and due to that ending up in mixed versions in the executable after ld.so mapped in dependencies?</div><div>I'm reaching out to you as this is a case I have no experience with.<br></div><div>I've thought and discussed on several solutions but I'm sure on none of them yet.</div><div><ul><li>Compat packages with symlinks to the new .so version as it has ABI backward compat symbols<br></li><li>Hard inserting the major version (currently 16.11) into every library version e.g. making the new a librte_eal.so.16.11.3<br></li><li>Some magic combination of breaks/conflicts/replaces that prevents the error window (lib updated, but not consumers) to occur</li><li>Others according to your guidance</li></ul></div><div><div>See further below for details and background on these options.</div><div><ul></ul></div></div><div><br></div><div><b>## Background ##</b></div><div><br></div><div>- DPDK in the past had a single library, but no soname versioning libdpdk.so.0</div><div>- Then they dropped the option for a single lib, but OTOH ABI tracking for individual libs was improved</div><div>- This was then deb packaged as usual versioned library packages; librte_XXX<VER> in the dpdk case.</div><div>- I'm unsure if upstream ABI tracking is "complete" given the case I outline below, yet I need to adapt in packaging no matter what.</div><div>- To be clear, as soon as a consuming app got a no-change rebuild things are fine again as all versions match again.</div><div><br></div><div><br></div><div><b>## The sub-dependencies issue ##</b></div><div><br></div><div>DPDK Libraries serve many purposes, from memory allocation to package processing.</div><div>There are a few which can be considered "core" libraries like librte_eal and librte_ethdev.</div><div>That means that most sublibraries will depend on those cores as well.</div><div><br></div><div>Dependencies prior to the update (trimmed down lddtree -a on openvswitch):</div><div><font face="monospace, monospace">ovs-vswitchd-dpdk => /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk (interpreter => /lib64/ld-linux-x86-64.so.2)</font></div><div><font face="monospace, monospace"> librte_pdump.so.1 => /usr/lib/x86_64-linux-gnu/librte_pdump.so.1</font></div><div><font face="monospace, monospace"> librte_eal.so.2 => /usr/lib/x86_64-linux-gnu/librte_eal.so.2</font></div><div><font face="monospace, monospace"> librte_eal.so.2 => /usr/lib/x86_64-linux-gnu/librte_eal.so.2</font></div><div><br></div><div>Dependencies after the update:</div><div><font face="monospace, monospace">ovs-vswitchd-dpdk => /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk (interpreter => /lib64/ld-linux-x86-64.so.2)</font></div><div><font face="monospace, monospace"> librte_pdump.so.1 => /usr/lib/x86_64-linux-gnu/librte_pdump.so.1</font></div><div><font face="monospace, monospace"> librte_eal.so.3 => /usr/lib/x86_64-linux-gnu/librte_eal.so.3</font></div><div><font face="monospace, monospace"> librte_eal.so.2 => /usr/lib/x86_64-linux-gnu/librte_eal.so.2</font></div><div><br></div><div>That causes symbol collisions. When starting it runs through some new init code (only in librte_eal3): rte_timer_init -> rte_delay_us_callback_register to write to a symbol that collides "rte_delay_us".</div><div><br></div><div>gdb: info symbol rte_delay_us</div><div><font face="monospace, monospace"> rte_delay_us in section .text of /usr/lib/x86_64-linux-gnu/librte_eal.so.2</font></div><div><br></div><div>And the types differ:</div><div> eal2:</div><div><font face="monospace, monospace"> void rte_delay_us(unsigned us) </font></div><div><font face="monospace, monospace"> { </font></div><div><font face="monospace, monospace"> const uint64_t start = rte_get_timer_cycles(); </font></div><div><font face="monospace, monospace"> const uint64_t ticks = (uint64_t)us * rte_get_timer_hz() / 1E6; </font></div><div><font face="monospace, monospace"> while ((rte_get_timer_cycles() - start) < ticks) </font></div><div><font face="monospace, monospace"> rte_pause(); </font></div><div><font face="monospace, monospace"> } </font></div><div> eal3:</div><div><font face="monospace, monospace"> void (*rte_delay_us)(unsigned int) = NULL;</font></div><div><br></div><div>So the new ABIs code writes to what it thinks to be a function pointer in eal3 but instead writes to the text segment of eal2 -> segfault.</div><div><br></div><div><br></div><div><br></div><div><b>## How could this happen ##</b></div><div><br></div><div>In the DPDK world only those sub-libraries that get ABI changes get a bump.</div><div>In the last case that was librte_eal, librte_cryptodev and libethdev (the latter also getting a rename).</div><div>So the new source upload provides new versions of the same ABI as the old provided, e.g. in my example above librte_pdump.so.1.</div><div>But since the new "librte_pdump.so.1" was build with the new source it depends on the new core libraries like librte_eal3.</div><div>On the other hand the "old" Openvswitch itself still depends on the old librte_eal2 provided by the old package.</div><div><br></div><div><br></div><div><b>## Maybe A special DPDK option around that ##</b></div><div><br></div><div>There is an extra twist od DPDK to consider as an option.</div><div>DPDK provides backward ABI compatibility via symbol maps.</div><div>Not forever but at least for one version backwards (or more if maintainable).</div><div>That said a new librte_eal.so.3 should be able to serve as librte_eal.so.2 via this compatibility layer.</div><div>But I already had failed assumption based on that creating transitional packages for librte_eal2 that map to librte_eal3.</div><div>That is not how named library soname versions work, yet this DPDK compat feature lured me in there.</div><div>But if done "right" that could be a librte_eal2 containing a symlink to librte_eal.so.3 and depending on the package.</div><div>A quick verify with that worked:</div><div><font face="monospace, monospace"> $ mv /usr/lib/x86_64-linux-gnu/librte_eal.so.2 /usr/lib/x86_64-linux-gnu/backup-librte_eal.so.2</font></div><div><font face="monospace, monospace"> $ ln -s /usr/lib/x86_64-linux-gnu/librte_eal.so.3 /usr/lib/x86_64-linux-gnu/librte_eal.so.2</font></div><div><font face="monospace, monospace"> $ mv /usr/lib/x86_64-linux-gnu/libethdev.so.4 /usr/lib/x86_64-linux-gnu/backup-libethdev.so.4</font></div><div><font face="monospace, monospace"> $ ln -s /usr/lib/x86_64-linux-gnu/librte_ethdev.so.5 /usr/lib/x86_64-linux-gnu/libethdev.so.4</font></div><div>With that in place it started correctly</div><div><br></div><div>Yet DPDK releases all three months, so sometimes we skip versions in the Distribution.</div><div>That means there could be cases where even this mechanism fails - so if this solution is good for now I'd be happy. But I'd still want to clarify what we would need to do if it is not an option at some point into the future.</div><div><br></div><div><br></div><div><b>## Potential Solutions?: ##</b></div><div><b><br></b></div><div>See the TL;DR at the top, but I'm not sure what to go for at the moment, so I hope for your guidance :-/</div><div><br></div><div><br></div><div><b>## Appendix ##</b></div><div><b><br></b></div><div>(FYI Full Dependencies)</div><div>Prior upgrade: <a href="http://paste.ubuntu.com/23939692/">http://paste.ubuntu.com/23939692/</a></div><div>After DPDK Upgraded: <a href="http://paste.ubuntu.com/23939693/">http://paste.ubuntu.com/23939693/</a></div><div><br></div><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(136,136,136);font-size:12.8px">Christian Ehrhardt</span><div style="color:rgb(136,136,136);font-size:12.8px">Software Engineer, Ubuntu Server</div><div style="color:rgb(136,136,136);font-size:12.8px">Canonical Ltd</div></div></div></div></div>
</div>