[Bug 1895106] Re: output non reproducible inhibiting verification of changes

[Christian Ehrhardt ]

Interesting, the small change I had does not make it slower and fixes
more than I've expected.

Some of the noise in the actual files like "all" and "all+extra" always switched between things like:
-xserver-xorg-dev        | Rescued from xorg-server
+xserver-xorg-dev        | xorg-dev

And it was part of the non reproducibility which one was reported.
That also got resolved by my change that was targeted primarily at the console output of germinate.

This stopped to switch between the two - it stays steady at the same
value on each re-run now.

Note: like before there are still cases that have "Rescued" and others
that list the dependent package. But now in a reproducible fashion.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to germinate in Ubuntu.
https://bugs.launchpad.net/bugs/1895106

Title:
  output non reproducible inhibiting verification of changes

Status in germinate package in Ubuntu:
  New

Bug description:
  Hi,
  there are multiple classes of issues that make germinate output hard to compare for "what changed".

  1. ordering issues
  Even the output of the running program changes. I didn't see subprocess spawning or any other asynchronicity. So I'd assume that there are lists generated/gathered that we process as-is.
  Example:
    1120 Resolving supported dependencies ...                                             
    1121 * Chose dict to satisfy dictd                                                    
    1122 * Chose probert-storage to satisfy curtin                                        
    1123   Rescued python-requests-toolbelt-doc from extra to supported                                                                                                                          
    1124   Rescued libmemcached-dbg from extra to supported 
  Rerun on same content:
    1120 Resolving supported dependencies ...                                             
    1121 * Chose dict to satisfy dictd                                                    
    1122 * Chose probert-storage to satisfy curtin                                        
    1123   Rescued default-libmysqlclient-dev from extra to supported                                                                                                                            
    1124   Rescued libmpc-dev from extra to supported                                                                                                                                            
    1125   Rescued python-openstackclient-doc from extra to supported                                                                                                                            
    1126   Rescued libgnome-menu-3-dev from extra to supported 

  Why is that - could we just sort any of the gathered lists before we iterate on them.
  That could in turn make many other things in the output reproducible.

  
  2. If a package is depended on by multiple packages or seeds a random one is reported.

  Example (run with the same seeds multiple times):
  all:
  -binutils-multiarch                          | binutils                             | binutils-multiarch-dev
  +binutils-multiarch                          | binutils                             | binutils-multiarch-dbg

  It is correct that both packages and seeds depend on them but I'd think it could be much better if we'd either:
  - report an ordered full list of dependency sources (could become very long but complete)
  or
  - report the first element out of a sorted list (as short as today, but reproducible)

  We could even get "best of both worlds" if we sort the list of
  dependency anchors, then concat the fist X (an arbitrary limit we set)
  of them and if some are left append ", ..." to reflect that.

  This way we would be reproducible, in many cases even complete, and in
  corner cases don't explode the list size.

  
  There could be more sources of non reproducibility that come to mind for experts in germinate (I'm not) and I'd appreciate if there could be an overhaul to get as close as reasonable to reproducible output.
  Because then if someone modifies seeds (or checks any follow on output based on it) it would stay the same and indicate the actual impact a change has caused.

  
  Well, maybe this was already discussed but I didn't find anything. If it was and considered undoable please point me to a log of the discussion if possible.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/germinate/+bug/1895106/+subscriptions