ARM branch maintenance proposal

Fri Aug 13 09:15:54 UTC 2010

On 08/13/2010 05:58 AM, Dechesne, Nicolas wrote:
>> On 08/12/2010 11:29 AM, Tim Gardner wrote:
>>      
>>>> On 08/12/2010 10:13 AM, Andy Whitcroft wrote:
>>>>
>>>>
>>>>          
>>>>> We have classically taken ARM branches as a stack of patches against
>>>>> the base release for Ubuntu and rebased those directly onto the tip of
>>>>> the master branch.  This has allowed us to keep those branches in sync
>>>>> with our master branch by simply rebasing them as master moves forward.
>>>>> This means that Security fixes and general Ubuntu goodness propogate to
>>>>> those branches as they rebase.
>>>>>
>>>>> More recently we have started to receive much more complex source trees.
>>>>> Typically these are based on the target release but they are not simple
>>>>> patch stacks.  They are complex merged trees pulling in changes from
>>>>> multiple sources.  The tip of them representing the required tree.
>>>>> These are not suitable for generating a patch stack from and thus present
>>>>> a maintenance issue.  We have proposed a merge based system to maintain
>>>>> these, whereby the source tree, the Ubuntu master branch, and a packaging
>>>>> branch are merged to generate the result.  This however does impose a
>>>>> new methodolgy on stable maintenance.
>>>>>
>>>>> For other subsystems (such as AppArmor and aufs2) we have yet a third
>>>>> approach.  Here we take the tip if the upstream tree and literally copy
>>>>> over the new files into our tree.  These are then committed as a single
>>>>> commit 'update aufs2 to upstream as at dd-mm-yyyy'.  This has the advantage
>>>>> that we know our tree is exactly the same as the source tree as well as
>>>>> allowing us to maintain the tree in the normal way.  The subsystem and
>>>>> any updates are mearly patches in our stack.
>>>>>
>>>>> I am proposing a new mechanism for maintaining ARM branches, specifically
>>>>> for upstream trees which are very complex and cannot be converted
>>>>> into a patch stack.  For these I propose we adopt the subsystem update
>>>>> model above.  That we convert the delta between the branch tip and the
>>>>> that trees base version into a patch, and apply that patch as normal
>>>>> to the branch.  This allows maintenance of the tree to continue in the
>>>>> normal way, via rebase against the master branch.  When updates to the
>>>>> upstream ARM tree occur we simply can diff the new tip against that
>>>>> taken in the last update and again apply the delta as an update patch,
>>>>> this does require we carefully track our sync points when committing them.
>>>>>
>>>>> Taking the current ARM ti-omap4 update as an example, we have a tree from
>>>>> the vendor which is based on v2.6.35.  We therefore can simply diff the
>>>>> that branch against v2.6.35.  The resulting patch can then be applied to
>>>>> the tip of the master branch.
>>>>>
>>>>>         $ git diff v2.6.35..L29.4>UPDATE
>>>>>         $ git checkout -b temp v2.6.35
>>>>>         $ git apply -C1 --index UPDATE
>>>>>         $ git commit -s -m 'update ARM thingy to SHA1'
>>>>>         $ git rebase origin/master
>>>>>
>>>>> Then we can simply apply the branch packaging and we are ready to upload.
>>>>> For a subsequent updates we would simply diff from the SHA1 recorded
>>>>> above to the tip of the branch and again apply the patch.
>>>>>
>>>>> By taking this approach we do lose the ability to bisect the upstream
>>>>> tree.  But where the upsteam tree needs such bisection we would have to
>>>>> hand generate the trees in order to have the full debian infrastructure
>>>>> in place.  Which is equally easy to do using the original tree, overall
>>>>> not a loss.
>>>>>
>>>>> Comments?
>>>>>
>>>>> -apw
>>>>>
>>>>>
>>>>>
>>>>>            
>>>> This seems like quite a reasonable approach. As package maintainers we
>>>> really don't care that much about the content of the ginormous patch
>>>> that gets the vendor from v2.6.35 to their working tree. We only care
>>>> that the patch exists and that we can easily rebase against _our_ master
>>>> in order to maintain ongoing Ubuntu goodness.
>>>>
>>>> rtg
>>>>
>>>>
>>>>          
>>> I like this.  I think it will make my life as the linaro kernel package
>>> maintainer easier.
>>>
>>>
>>>        
>> Andy, as you know, I am the provider of this big giant patch in the
>> ti-omap4 branch ;-) i understand what you are trying to achieve here,
>> and my fear is that the rebase on ubuntu master might be quite
>> problematic... well I just did the test now with the current branch
>> states, and the rebase went quite fine (3 merge conflicts out of this
>> huge patchset). But how do we manage updates now? Do you expect that we
>> continue to put our patches in our tree, and you will apply them on top
>> of 'UPDATE' or do you plan to generate UPDATE for each new patch set
>> from us? Especially because OMAP3 support is managed in master and OMAP4
>> in ti-omap4..
>>
>> Also this model fits well once the vendor is able to provide a BSP which
>> is aligned with the kernel version which is in master which is just
>> going to happen now for OMAP4. How do we proceed before (e.g. projecting
>> to 11.04) while master is moving at a faster pace than ti branch?
>>
>> Regarding the linaro kernel, I don't think that might help too much.
>> Linaro kernel is mostly based on mainline stuff and it does not get all
>> the "fancy" stuff that we put in ti-omap4. at the some point of course
>> most of these 1300+ patches might end up in mainline, but until there
>> ti-omap4 branch and linaro branch will be a lot different ( i think the
>> linaro does not even enable OMAP4 for now). So Linaro kernel should not
>> blindly take the big 'UPDATE' and merge it until the code is close to
>> get into mainline.  Finally our huge patch set includes patches outside
>> of omap (ARM, network, and other). So when merging other ARM vendors
>> with the TI patches you might get lots of conflicts, having individual
>> patches as commits might help git rebase do a better job, also getting
>> the granularity of the commit would ease the debug/bisect as you said.
>>
>> this is just a thought... in any case we will continue to provide you a
>> branch with all TI patches that need to be pulled in ti-omap4. i recall
>> that when we started to work together we had decided to maintain a
>> branch that would look like: mainline + TI patches + Ubuntu sauce that
>> could later become ti-omap4 branch.
>>
>>
>>      
> Andy,
>
> I totally understand your feeling about the TI L24.9 release tree, which
> contains several merging operations. That will cause the issue when we do
> rebasing on our Ubuntu master tree.
>
> And the simply updating a large patch will make our Ubuntu tree much easier. But
> I still got some ideas about this method:
>
> 1. We will lose the git log history about the big patches in our tree. Although
> the history is supposed to be kept in upstream (here is TI L24.9 branch), we
> can't make sure it won't change or lose. If we got some critical issue and we
> need bisect or checking the history information, we might cannot find any
> accurate log. Because we can't control upstream tree, but we can control ours.
>
> 2. After we applying the large patch file, we are still supposed to get some
> rebasing conflicts when we rebase on our original/master. This won't save us
> much effort, moreover we need to look up upstream tree git logs to decide how to
> solve this conflicts. I guess that will be painful.
>
> 3. For further TI releases, I believe they will be quite small compare to this
> one. It is not necessary to use this method, it will be easier to just apply
> those additional patches on our tip branch. Just for this first TI 2.6.35
> release, I think we got this merging and rebasing pain.
>
> 4. As Nico mentioned here, it might be not very difficult to solve the merging
> and rebasing issue. So we can get a 2.6.35 + TI patches + Ubuntu master
> SAUCE/packaging stuff. After a short pain, we can get a tree in good shape. I
> think we can buy it.
>
> But on the other hand, Nico. Is that possible for TI just release a pure patch
> stack style branch instead of merging from other dev trees?
>    
Bryan, thanks I think you wrote what I actually think, but in a better 
way ;-) So couple of points:
- the current L24.9 branch that you are looking at is not your upstream, 
this branch is not maintained once it's release (tagged). Our dev team 
will move to a new mainline for L24.10 branch likely based on .36-rc1. 
The key point is that our dev team focus on mainline work and rebase 
their patches regularly. your upstream for this project is Seb's branch 
on ubuntu-kernel.git (on TI git server). When we started we decided to 
create 'stable' branches on this tree where we would maintain and 
backport OMAP4 related patches (from mainline, arm tree and from L24.10 
and above TI branches).
- we initially decided that Seb's branches would be mainline + OMAP4 
patches + Ubuntu sauce, but we can easily decide to remove the Ubuntu 
sauce if this is what you propose and have this a 'stable' .35 branch 
that we maintain, and that you can pull from.
- I don't understand your last remark. we have several teams working on 
various OMAP4 sub systems or features. each team has a tree, and we need 
to merge them all. does it answer your question?

also please note that I will be out next week, and Seb will be back on Thu.
> Thanks,
>