ARM branch maintenance proposal

[Bryan Wu]

On 08/13/2010 05:15 PM, Dechesne, Nicolas wrote:
> On 08/13/2010 05:58 AM, Dechesne, Nicolas wrote:
>>> On 08/12/2010 11:29 AM, Tim Gardner wrote:
>>>     
>>>>> On 08/12/2010 10:13 AM, Andy Whitcroft wrote:
>>>>>
>>>>>
>>>>>         
>>>>>> We have classically taken ARM branches as a stack of patches against
>>>>>> the base release for Ubuntu and rebased those directly onto the
>>>>>> tip of
>>>>>> the master branch.  This has allowed us to keep those branches in
>>>>>> sync
>>>>>> with our master branch by simply rebasing them as master moves
>>>>>> forward.
>>>>>> This means that Security fixes and general Ubuntu goodness
>>>>>> propogate to
>>>>>> those branches as they rebase.
>>>>>>
>>>>>> More recently we have started to receive much more complex source
>>>>>> trees.
>>>>>> Typically these are based on the target release but they are not
>>>>>> simple
>>>>>> patch stacks.  They are complex merged trees pulling in changes from
>>>>>> multiple sources.  The tip of them representing the required tree.
>>>>>> These are not suitable for generating a patch stack from and thus
>>>>>> present
>>>>>> a maintenance issue.  We have proposed a merge based system to
>>>>>> maintain
>>>>>> these, whereby the source tree, the Ubuntu master branch, and a
>>>>>> packaging
>>>>>> branch are merged to generate the result.  This however does impose a
>>>>>> new methodolgy on stable maintenance.
>>>>>>
>>>>>> For other subsystems (such as AppArmor and aufs2) we have yet a third
>>>>>> approach.  Here we take the tip if the upstream tree and literally
>>>>>> copy
>>>>>> over the new files into our tree.  These are then committed as a
>>>>>> single
>>>>>> commit 'update aufs2 to upstream as at dd-mm-yyyy'.  This has the
>>>>>> advantage
>>>>>> that we know our tree is exactly the same as the source tree as
>>>>>> well as
>>>>>> allowing us to maintain the tree in the normal way.  The subsystem
>>>>>> and
>>>>>> any updates are mearly patches in our stack.
>>>>>>
>>>>>> I am proposing a new mechanism for maintaining ARM branches,
>>>>>> specifically
>>>>>> for upstream trees which are very complex and cannot be converted
>>>>>> into a patch stack.  For these I propose we adopt the subsystem
>>>>>> update
>>>>>> model above.  That we convert the delta between the branch tip and
>>>>>> the
>>>>>> that trees base version into a patch, and apply that patch as normal
>>>>>> to the branch.  This allows maintenance of the tree to continue in
>>>>>> the
>>>>>> normal way, via rebase against the master branch.  When updates to
>>>>>> the
>>>>>> upstream ARM tree occur we simply can diff the new tip against that
>>>>>> taken in the last update and again apply the delta as an update
>>>>>> patch,
>>>>>> this does require we carefully track our sync points when
>>>>>> committing them.
>>>>>>
>>>>>> Taking the current ARM ti-omap4 update as an example, we have a
>>>>>> tree from
>>>>>> the vendor which is based on v2.6.35.  We therefore can simply
>>>>>> diff the
>>>>>> that branch against v2.6.35.  The resulting patch can then be
>>>>>> applied to
>>>>>> the tip of the master branch.
>>>>>>
>>>>>>         $ git diff v2.6.35..L29.4>UPDATE
>>>>>>         $ git checkout -b temp v2.6.35
>>>>>>         $ git apply -C1 --index UPDATE
>>>>>>         $ git commit -s -m 'update ARM thingy to SHA1'
>>>>>>         $ git rebase origin/master
>>>>>>
>>>>>> Then we can simply apply the branch packaging and we are ready to
>>>>>> upload.
>>>>>> For a subsequent updates we would simply diff from the SHA1 recorded
>>>>>> above to the tip of the branch and again apply the patch.
>>>>>>
>>>>>> By taking this approach we do lose the ability to bisect the upstream
>>>>>> tree.  But where the upsteam tree needs such bisection we would
>>>>>> have to
>>>>>> hand generate the trees in order to have the full debian
>>>>>> infrastructure
>>>>>> in place.  Which is equally easy to do using the original tree,
>>>>>> overall
>>>>>> not a loss.
>>>>>>
>>>>>> Comments?
>>>>>>
>>>>>> -apw
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>> This seems like quite a reasonable approach. As package maintainers we
>>>>> really don't care that much about the content of the ginormous patch
>>>>> that gets the vendor from v2.6.35 to their working tree. We only care
>>>>> that the patch exists and that we can easily rebase against _our_
>>>>> master
>>>>> in order to maintain ongoing Ubuntu goodness.
>>>>>
>>>>> rtg
>>>>>
>>>>>
>>>>>          
>>>> I like this.  I think it will make my life as the linaro kernel package
>>>> maintainer easier.
>>>>
>>>>
>>>>        
>>> Andy, as you know, I am the provider of this big giant patch in the
>>> ti-omap4 branch ;-) i understand what you are trying to achieve here,
>>> and my fear is that the rebase on ubuntu master might be quite
>>> problematic... well I just did the test now with the current branch
>>> states, and the rebase went quite fine (3 merge conflicts out of this
>>> huge patchset). But how do we manage updates now? Do you expect that we
>>> continue to put our patches in our tree, and you will apply them on top
>>> of 'UPDATE' or do you plan to generate UPDATE for each new patch set
>>> from us? Especially because OMAP3 support is managed in master and OMAP4
>>> in ti-omap4..
>>>
>>> Also this model fits well once the vendor is able to provide a BSP which
>>> is aligned with the kernel version which is in master which is just
>>> going to happen now for OMAP4. How do we proceed before (e.g. projecting
>>> to 11.04) while master is moving at a faster pace than ti branch?
>>>
>>> Regarding the linaro kernel, I don't think that might help too much.
>>> Linaro kernel is mostly based on mainline stuff and it does not get all
>>> the "fancy" stuff that we put in ti-omap4. at the some point of course
>>> most of these 1300+ patches might end up in mainline, but until there
>>> ti-omap4 branch and linaro branch will be a lot different ( i think the
>>> linaro does not even enable OMAP4 for now). So Linaro kernel should not
>>> blindly take the big 'UPDATE' and merge it until the code is close to
>>> get into mainline.  Finally our huge patch set includes patches outside
>>> of omap (ARM, network, and other). So when merging other ARM vendors
>>> with the TI patches you might get lots of conflicts, having individual
>>> patches as commits might help git rebase do a better job, also getting
>>> the granularity of the commit would ease the debug/bisect as you said.
>>>
>>> this is just a thought... in any case we will continue to provide you a
>>> branch with all TI patches that need to be pulled in ti-omap4. i recall
>>> that when we started to work together we had decided to maintain a
>>> branch that would look like: mainline + TI patches + Ubuntu sauce that
>>> could later become ti-omap4 branch.
>>>
>>>
>>>      
>> Andy,
>>
>> I totally understand your feeling about the TI L24.9 release tree, which
>> contains several merging operations. That will cause the issue when we do
>> rebasing on our Ubuntu master tree.
>>
>> And the simply updating a large patch will make our Ubuntu tree much
>> easier. But
>> I still got some ideas about this method:
>>
>> 1. We will lose the git log history about the big patches in our tree.
>> Although
>> the history is supposed to be kept in upstream (here is TI L24.9
>> branch), we
>> can't make sure it won't change or lose. If we got some critical issue
>> and we
>> need bisect or checking the history information, we might cannot find any
>> accurate log. Because we can't control upstream tree, but we can
>> control ours.
>>
>> 2. After we applying the large patch file, we are still supposed to
>> get some
>> rebasing conflicts when we rebase on our original/master. This won't
>> save us
>> much effort, moreover we need to look up upstream tree git logs to
>> decide how to
>> solve this conflicts. I guess that will be painful.
>>
>> 3. For further TI releases, I believe they will be quite small compare
>> to this
>> one. It is not necessary to use this method, it will be easier to just
>> apply
>> those additional patches on our tip branch. Just for this first TI 2.6.35
>> release, I think we got this merging and rebasing pain.
>>
>> 4. As Nico mentioned here, it might be not very difficult to solve the
>> merging
>> and rebasing issue. So we can get a 2.6.35 + TI patches + Ubuntu master
>> SAUCE/packaging stuff. After a short pain, we can get a tree in good
>> shape. I
>> think we can buy it.
>>
>> But on the other hand, Nico. Is that possible for TI just release a
>> pure patch
>> stack style branch instead of merging from other dev trees?
>>    
> Bryan, thanks I think you wrote what I actually think, but in a better
> way ;-) So couple of points:
> - the current L24.9 branch that you are looking at is not your upstream,
> this branch is not maintained once it's release (tagged). Our dev team
> will move to a new mainline for L24.10 branch likely based on .36-rc1.
> The key point is that our dev team focus on mainline work and rebase
> their patches regularly. your upstream for this project is Seb's branch
> on ubuntu-kernel.git (on TI git server). When we started we decided to
> create 'stable' branches on this tree where we would maintain and
> backport OMAP4 related patches (from mainline, arm tree and from L24.10
> and above TI branches).

Yep, I got it. L24.9 is just for us as a reference. We'll wait for your ubuntu
release branch.

> - we initially decided that Seb's branches would be mainline + OMAP4
> patches + Ubuntu sauce, but we can easily decide to remove the Ubuntu
> sauce if this is what you propose and have this a 'stable' .35 branch
> that we maintain, and that you can pull from.

Right, I think firstly we just wanna mainline 2.6.35 + OMAP4 L24.9 release
patches. Then I can cowork with you guys to add our Ubuntu stuff. I've already
simply rebase our Ubuntu stuff of 10.10 2.6.35 master branch on top of L24.9, it
is just for a testing. After that we can sync up with -ti-omap4 specific Ubuntu
stuff. That will be in a good shape.

> - I don't understand your last remark. we have several teams working on
> various OMAP4 sub systems or features. each team has a tree, and we need
> to merge them all. does it answer your question?
> 

I understood now, we can get those patch stack from Seb's release.

Thanks,
-Bryan