[Launchpad Wiki] Update of "Soyuz/NativeSourceSyncing" by CelsoProvidelo

[Launchpad Wiki]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Launchpad Wiki" for change notification.

The following page has been changed by CelsoProvidelo:
https://launchpad.canonical.com/Soyuz/NativeSourceSyncing

The comment on the change is:
updating bits

------------------------------------------------------------------------------
  == Implementation ==
  
  The implementation will be staged as follows:
+ 
   1. copy between pockets (PROPOSED -> UPDATES, SECURITY -> UPDATES, no rebuild, i.e, include binaries)
   1. copy between distrorelases (feisty-RELEASE -> edgy-BACKPORTS, rebuild, do not include binaries, replace backport-source.py)
   1. copy between distribution (debian-unstable -> ubuntu.feisty-RELEASE, rebuild, do not include binaries, replace sync-source.py)
  
- Based on design requirements above we can highlight the following implementation details:
+ For the stage 1 no extra features are required, we only need to extend the ChangeOverrideAndRemove API based on DistroReleaseSourcePackageRelease for sources and DistroArchReleaseBinaryPackage for binaries. Additionally to the ''changeOverride'' & ''supersede'' methods it will also implement a ''copyTo'' method as:
  
+ {{{
+     def copyTo(distrorelease, pocket):                                            
+         """Copy this DistroReleaseSourcePackageRelease to another location.          
+                                                                                  
+         Return the publishing record in the targeted location.                    
+         """ 
+ }}}
-  * DistroReleaseQueue creation should not require a changesfiles.
-  * We will land an enhanced pre-publication override layer (required for debian syncs).  This will override component/section/priority when copying packages.
-  * Source syncs can optionally carry their binary.
-  * Synced binaries will preserve the original build information of distro/release/pocket/architecture.
-  * Reference entities will be verified for sanity during Queue.accept()
-    * Claimed sources files match found SourcePackageRelease,
-    * Claimed SourcePackageRelease matches claimed BinaryPackageRelease,
-    * All BinaryPackageRelease generated by the found Build are included
-    * ...
  
- CelsoProvidelo: As agreed with ColinWatson on IRC, we don't need the fulfill upload queue information to move publications from one pocket to another w/o rebuild (the first use case). It can happen in a similar way to how we do overrides (changes-overrides.py) - we could simply create a new publishing record to the targeted pocket, so there will be traces of this operation only in publishing-land.
+ During the stage 1 we will assert that the target_distrorelease is the same than the current published distrorelease, i.e., it will only work for copies between pockets.
+ 
+ No DistroReleaseQueue background is required for operation in this stage. Distro-team is happy with the history kept in publishing tables.
+ 
+ We will operate in COPY mode only in all stages, when necessary, the old (source) records should be explicitly removed using remove-package.py.
+ 
+ After landing PersonalPackageArchives (PPA) we will need extend this implementation to support Archive model and probably go for stage 2 & 3 (which are very similar from the Launchpad point of view)
+ 
+ The new pre-publication overrides system is only required for syncing from other distributions, specially Debian, in stage 3.
+ 
+ Debian imports will only work as expected if we setup the gina to do incremental imports of Debian into Launchpad. Long story ...
+ 
+ Also for stage 2 & 3, we will probably need be move some checks from NascentUpload pipeline to Queue pipeline, so they don't need to be duplicated but both, uploads and sync will be submitted to them. They are, for instance:
+  * Claimed sources files match found SourcePackageRelease,
+  * Claimed SourcePackageRelease matches claimed BinaryPackageRelease,
+  * All BinaryPackageRelease generated by the found Build are included
+  * ...
  
  == UI Changes  ==
  
@@ -86, +101 @@

  --to-suite        -> to-suite 
  }}}
  
- CelsoProvidelo: We can land this script with partial functionality, for instance only copying stuff from one pocket to another. This way we could quickly sort quickly only the uncovered functionality (BACKPORTS -> UPDATES w/o rebuild and SECURITY -> UPDATES propagation).
- 
- CelsoProvidelo: We will operate in COPY mode only, when necessary the old record should be explicitly removed using remove-package.py.
- 
- CelsoProvidelo: Archive support support should be included when PPA lands.
- 
- CelsoProvidelo: Overrides support can be included here, but it's not mandatory, they can continue to be done via change-override.py. See Unresolved Issues section.
- 
  === Launchpad Web UI ===
  
  It will require a page on DistroReleaseSourcePackageRelease (ubuntu/feisty/+source/foo/1.0) allowed only for restricted group.
@@ -110, +117 @@

  Should be similar to the local command-line tool.
  
  == Code Changes ==
+ 
+  DistroReleaseSourcePackageRelease and DistroArchReleaseBinaryPackage will be modified to implements the ''copyTo'' method.
+ 
+  CopyPackageHelper library/class will be created in order to allow comprehensive and testable procedure. (lib/canonical/launchpad/ftpmaster.py). Tests for this area will be in lib/canonical/launchpad/scripts/ftests/test_copypackage.py
  
  === Pre-Publication Overrides ===
  
@@ -148, +159 @@

  
  == Unresolved issues ==
  
-  * CelsoProvidelo: The new pre-publication overrides system isn't really required for having NSS inside ubuntu, where the auto-overrides will work perfectly well. However they are necessary for debian syncs, other wise we will end up trying to publish stuff to un-selected components in ubuntu. Nonetheless I'd recommend delaying this implementation until we have landed PPA code (which renames and modifies the queue tables badly).
+ CelsoProvidelo: the right place for the binary side of ChangeOverride/Remove/CopyTo API is DistroArchReleaseBinaryPackageRelease not DistroArchReleaseBinaryPackage. We may be take the advantage of touching those files and fix this problem.
  
  ----
  CategoryProposal