[Merge] lp:~tobijk/livecd-rootfs/magic-proxy into lp:livecd-rootfs

[Tobias Koch]

Tobias Koch has proposed merging lp:~tobijk/livecd-rootfs/magic-proxy into lp:livecd-rootfs.

Commit message:
Inject a proxy into the build providing a snapshot view of the package repo.
  
When the REPO_SNAPSHOT_STAMP variable is set, the auto/build script will attempt to launch a transparent HTTP proxy on port 8080, and insert an iptables rule to redirect all outgoing HTTP requests to this proxy.
  
The proxy, contained in the `magic-proxy` Python script, examines each request and silently overrides those pointing to InRelease files or files that are listed in previously-downloaded InRelease files. It will instead provide the content of the requested file as it was at REPO_SNAPSHOT_STAMP, by downloading the corresponding asset "by hash".

Requested reviews:
  Canonical Cloudware (cloudware)
  Canonical Cloudware (cloudware)
  Ubuntu Core Development Team (ubuntu-core-dev)

For more details, see:
https://code.launchpad.net/~tobijk/livecd-rootfs/magic-proxy/+merge/357643

This MP is related to

https://code.launchpad.net/~tobijk/livecd-rootfs/image-sets/+merge/356246

and aims at enabling parallel, reproducible image builds. Launchpad keeps a rolling history of InRelease files identified by their sha256 hash sum in the archive at

dists/<series>/by-hash/SHA256/<hash_of_file>

An InRelease file in turn lists the corresponding Packages files and their hashes. The Packages files can then be downloaded “by-hash”, as well. By configuring all tools that are involved in the image building to follow this trail of hashes, we can guarantee consistent package versions across all images.

While the idea of injecting a transparent proxy into the build that rewrites URLs on the fly may seem absurd, it works great and is much cleaner than

https://code.launchpad.net/~tobijk/livecd-rootfs/lp-in-release/+merge/356927

which in addition requires patches to live build

https://pastebin.canonical.com/p/HSfM8ktN4x/

and deboostrap

https://salsa.debian.org/installer-team/debootstrap/merge_requests/22

and is prone to errors, i.e. it is easy to miss a spot or break the functionality through unrelated changes.
-- 
Your team Ubuntu Core Development Team is requested to review the proposed merge of lp:~tobijk/livecd-rootfs/magic-proxy into lp:livecd-rootfs.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: review-diff.txt
Type: text/x-diff
Size: 35182 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-reviews/attachments/20181022/4039c046/attachment-0001.diff>