Rev 3366: (Mark Hammond) Document outlining strategies for TortoiseBzr in file:///home/pqm/archives/thelove/bzr/%2Btrunk/
Canonical.com Patch Queue Manager
pqm at pqm.ubuntu.com
Mon Apr 14 21:56:36 BST 2008
At file:///home/pqm/archives/thelove/bzr/%2Btrunk/
------------------------------------------------------------
revno: 3366
revision-id:pqm at pqm.ubuntu.com-20080414205628-khxum18d23suak10
parent: pqm at pqm.ubuntu.com-20080413194429-a5e4pft9sffa2ycu
parent: bialix at ukr.net-20080414183811-cy4cofsy9af1lyoe
committer: Canonical.com Patch Queue Manager <pqm at pqm.ubuntu.com>
branch nick: +trunk
timestamp: Mon 2008-04-14 21:56:28 +0100
message:
(Mark Hammond) Document outlining strategies for TortoiseBzr
added:
doc/developers/tortoise-strategy.txt tortoisestrategy.txt-20080403024510-2ahdqrvnwqrb5p5t-1
modified:
NEWS NEWS-20050323055033-4e00b5db738777ff
------------------------------------------------------------
revno: 3327.1.6
revision-id:bialix at ukr.net-20080414183811-cy4cofsy9af1lyoe
parent: bialix at ukr.net-20080414182943-43sglxt1oo9cotdy
committer: Alexander Belchenko <bialix at ukr.net>
branch nick: tortoise-doc
timestamp: Mon 2008-04-14 21:38:11 +0300
message:
NEWS
modified:
NEWS NEWS-20050323055033-4e00b5db738777ff
------------------------------------------------------------
revno: 3327.1.5
revision-id:bialix at ukr.net-20080414182943-43sglxt1oo9cotdy
parent: mhammond at skippinet.com.au-20080409081938-pgbl7xc4jb1lt3fq
parent: pqm at pqm.ubuntu.com-20080413194429-a5e4pft9sffa2ycu
committer: Alexander Belchenko <bialix at ukr.net>
branch nick: tortoise-doc
timestamp: Mon 2008-04-14 21:29:43 +0300
message:
merge bzr.dev
added:
doc/developers/integration.txt integration.txt-20080404022341-2lorxocp1in07zij-1
modified:
NEWS NEWS-20050323055033-4e00b5db738777ff
bzrlib/annotate.py annotate.py-20050922133147-7c60541d2614f022
bzrlib/branch.py branch.py-20050309040759-e4baf4e0d046576e
bzrlib/builtins.py builtins.py-20050830033751-fc01482b9ca23183
bzrlib/bzrdir.py bzrdir.py-20060131065624-156dfea39c4387cb
bzrlib/commands.py bzr.py-20050309040720-d10f4714595cf8c3
bzrlib/config.py config.py-20051011043216-070c74f4e9e338e8
bzrlib/errors.py errors.py-20050309040759-20512168c4e14fbd
bzrlib/fetch.py fetch.py-20050818234941-26fea6105696365d
bzrlib/help_topics/__init__.py help_topics.py-20060920210027-rnim90q9e0bwxvy4-1
bzrlib/help_topics/en/hooks.txt hooks.txt-20070830033044-xxu2rced13f72dka-1
bzrlib/inventory.py inventory.py-20050309040759-6648b84ca2005b37
bzrlib/knit.py knit.py-20051212171256-f056ac8f0fbe1bd9
bzrlib/lockable_files.py control_files.py-20051111201905-bb88546e799d669f
bzrlib/mail_client.py mail_client.py-20070809192806-vuxt3t19srtpjpdn-1
bzrlib/multiparent.py __init__.py-20070410133617-n1jdhcc1n1mibarp-1
bzrlib/osutils.py osutils.py-20050309040759-eeaff12fbf77ac86
bzrlib/reconfigure.py reconfigure.py-20070908040425-6ykgo7escxhyrg9p-1
bzrlib/remote.py remote.py-20060720103555-yeeg2x51vn0rbtdp-1
bzrlib/repofmt/knitrepo.py knitrepo.py-20070206081537-pyy4a00xdas0j4pf-1
bzrlib/repofmt/pack_repo.py pack_repo.py-20070813041115-gjv5ma7ktfqwsjgn-1
bzrlib/repofmt/weaverepo.py presplitout.py-20070125045333-wfav3tsh73oxu3zk-1
bzrlib/repository.py rev_storage.py-20051111201905-119e9401e46257e3
bzrlib/revisiontree.py revisiontree.py-20060724012533-bg8xyryhxd0o0i0h-1
bzrlib/smart/branch.py branch.py-20061124031907-mzh3pla28r83r97f-1
bzrlib/smart/client.py client.py-20061116014825-2k6ada6xgulslami-1
bzrlib/smart/medium.py medium.py-20061103051856-rgu2huy59fkz902q-1
bzrlib/smart/protocol.py protocol.py-20061108035435-ot0lstk2590yqhzr-1
bzrlib/smart/request.py request.py-20061108095550-gunadhxmzkdjfeek-1
bzrlib/store/revision/knit.py knit.py-20060303020652-de5fa299e941a3c7
bzrlib/store/versioned/__init__.py weavestore.py-20050907094258-88262e0434babab9
bzrlib/tests/blackbox/test_reconfigure.py test_reconfigure.py-20070908173426-khfo5fi2rgzgtwj3-1
bzrlib/tests/blackbox/test_status.py teststatus.py-20050712014354-508855eb9f29f7dc
bzrlib/tests/branch_implementations/test_hooks.py test_hooks.py-20070129154855-blhpwxmvjs07waei-1
bzrlib/tests/bzrdir_implementations/__init__.py __init__.py-20060131065642-34c39b54f42dd048
bzrlib/tests/bzrdir_implementations/test_bzrdir.py test_bzrdir.py-20060131065642-0ebeca5e30e30866
bzrlib/tests/interversionedfile_implementations/__init__.py __init__.py-20060302012326-981af525594d02ed
bzrlib/tests/repository_implementations/test_check_reconcile.py test_broken.py-20070928125406-62236394w0jpbpd6-2
bzrlib/tests/repository_implementations/test_fetch.py test_fetch.py-20070814052151-5cxha9slx4c93uog-1
bzrlib/tests/repository_implementations/test_repository.py test_repository.py-20060131092128-ad07f494f5c9d26c
bzrlib/tests/revisionstore_implementations/test_all.py test_all.py-20060303020702-9b2d4c1d75407f31
bzrlib/tests/test_branch.py test_branch.py-20060116013032-97819aa07b8ab3b5
bzrlib/tests/test_bzrdir.py test_bzrdir.py-20060131065654-deba40eef51cf220
bzrlib/tests/test_config.py testconfig.py-20051011041908-742d0c15d8d8c8eb
bzrlib/tests/test_fetch.py testfetch.py-20050825090644-f73e07e7dfb1765a
bzrlib/tests/test_knit.py test_knit.py-20051212171302-95d4c00dd5f11f2b
bzrlib/tests/test_mail_client.py test_mail_client.py-20070809192806-vuxt3t19srtpjpdn-2
bzrlib/tests/test_merge.py testmerge.py-20050905070950-c1b5aa49ff911024
bzrlib/tests/test_reconfigure.py test_reconfigure.py-20070908040425-6ykgo7escxhyrg9p-2
bzrlib/tests/test_remote.py test_remote.py-20060720103555-yeeg2x51vn0rbtdp-2
bzrlib/tests/test_smart.py test_smart.py-20061122024551-ol0l0o0oofsu9b3t-2
bzrlib/tests/test_smart_transport.py test_ssh_transport.py-20060608202016-c25gvf1ob7ypbus6-2
bzrlib/tests/test_store.py teststore.py-20050826022702-f6caadb647395769
bzrlib/tests/test_subsume.py test_subsume.py-20060927040024-tsvh4pchajoayymg-1
bzrlib/tests/test_versionedfile.py test_versionedfile.py-20060222045249-db45c9ed14a1c2e5
bzrlib/tests/test_weave.py testknit.py-20050627023648-9833cc5562ffb785
bzrlib/transport/remote.py ssh.py-20060608202016-c25gvf1ob7ypbus6-1
bzrlib/transport/sftp.py sftp.py-20051019050329-ab48ce71b7e32dfe
bzrlib/transport/ssh.py ssh.py-20060824042150-0s9787kng6zv1nwq-1
bzrlib/versionedfile.py versionedfile.py-20060222045106-5039c71ee3b65490
bzrlib/weave.py knit.py-20050627021749-759c29984154256b
bzrlib/workingtree_4.py workingtree_4.py-20070208044105-5fgpc5j3ljlh5q6c-1
doc/developers/HACKING.txt HACKING-20050805200004-2a5dc975d870f78c
doc/developers/index.txt index.txt-20070508041241-qznziunkg0nffhiw-1
doc/en/user-guide/controlling_registration.txt controlling_registra-20071121073725-0corxykv5irjal00-3
------------------------------------------------------------
revno: 3327.1.4
revision-id:mhammond at skippinet.com.au-20080409081938-pgbl7xc4jb1lt3fq
parent: mhammond at skippinet.com.au-20080407053215-m0l6lsv1qwz0fy0e
committer: Mark Hammond <mhammond at skippinet.com.au>
branch nick: bzr.work2
timestamp: Wed 2008-04-09 18:19:38 +1000
message:
based on the suggestions of a few people, make the tone of the document
more authoritative, documenting decisions made rather than the options
available.
modified:
doc/developers/tortoise-strategy.txt tortoisestrategy.txt-20080403024510-2ahdqrvnwqrb5p5t-1
------------------------------------------------------------
revno: 3327.1.3
revision-id:mhammond at skippinet.com.au-20080407053215-m0l6lsv1qwz0fy0e
parent: mhammond at skippinet.com.au-20080407052213-j3oqlr0rsubk0eci
committer: Mark Hammond <mhammond at skippinet.com.au>
branch nick: bzr.work2
timestamp: Mon 2008-04-07 15:32:15 +1000
message:
Typos spotted by Alexander Belchenko.
modified:
doc/developers/tortoise-strategy.txt tortoisestrategy.txt-20080403024510-2ahdqrvnwqrb5p5t-1
------------------------------------------------------------
revno: 3327.1.2
revision-id:mhammond at skippinet.com.au-20080407052213-j3oqlr0rsubk0eci
parent: mhammond at skippinet.com.au-20080403024532-xkq0nt29natiws0h
parent: pqm at pqm.ubuntu.com-20080407044456-s1a9orh0kssphdh9
committer: Mark Hammond <mhammond at skippinet.com.au>
branch nick: bzr.work2
timestamp: Mon 2008-04-07 15:22:13 +1000
message:
merge
added:
bzrlib/tests/test_mutabletree.py test_mutabletree.py-20080405014429-2v0cdi3re320p8db-1
bzrlib/transport/nosmart.py nosmart.py-20080402095843-6ib17idympwy1zkr-1
doc/developers/plugin-api.txt pluginapi.txt-20080229110225-q2j5y4agqhlkjn0s-1
renamed:
bzrlib/tests/test_revisionnamespaces.py => bzrlib/tests/test_revisionspec.py testrevisionnamespaces.py-20050711050225-8b4af89e6b1efe84
modified:
NEWS NEWS-20050323055033-4e00b5db738777ff
bzrlib/branch.py branch.py-20050309040759-e4baf4e0d046576e
bzrlib/bugtracker.py bugtracker.py-20070410073305-vu1vu1qosjurg8kb-1
bzrlib/builtins.py builtins.py-20050830033751-fc01482b9ca23183
bzrlib/errors.py errors.py-20050309040759-20512168c4e14fbd
bzrlib/knit.py knit.py-20051212171256-f056ac8f0fbe1bd9
bzrlib/mutabletree.py mutabletree.py-20060906023413-4wlkalbdpsxi2r4y-2
bzrlib/plugin.py plugin.py-20050622060424-829b654519533d69
bzrlib/revisionspec.py revisionspec.py-20050907152633-17567659fd5c0ddb
bzrlib/smart/client.py client.py-20061116014825-2k6ada6xgulslami-1
bzrlib/status.py status.py-20050505062338-431bfa63ec9b19e6
bzrlib/tests/TestUtil.py TestUtil.py-20050824080200-5f70140a2d938694
bzrlib/tests/__init__.py selftest.py-20050531073622-8d0e3c8845c97a64
bzrlib/tests/blackbox/test_switch.py test_switch.py-20071122111948-0c5en6uz92bwl76h-1
bzrlib/tests/test_bugtracker.py test_bugtracker.py-20070410073305-vu1vu1qosjurg8kb-2
bzrlib/tests/test_errors.py test_errors.py-20060210110251-41aba2deddf936a8
bzrlib/tests/test_knit.py test_knit.py-20051212171302-95d4c00dd5f11f2b
bzrlib/tests/test_merge.py testmerge.py-20050905070950-c1b5aa49ff911024
bzrlib/tests/test_plugins.py plugins.py-20050622075746-32002b55e5e943e9
bzrlib/tests/test_remote.py test_remote.py-20060720103555-yeeg2x51vn0rbtdp-2
bzrlib/tests/test_selftest.py test_selftest.py-20051202044319-c110a115d8c0456a
bzrlib/tests/test_versionedfile.py test_versionedfile.py-20060222045249-db45c9ed14a1c2e5
bzrlib/tests/workingtree_implementations/test_commit.py test_commit.py-20060421013633-1610ec2331c8190f
bzrlib/transform.py transform.py-20060105172343-dd99e54394d91687
bzrlib/transport/__init__.py transport.py-20050711165921-4978aa7ce1285ad5
bzrlib/versionedfile.py versionedfile.py-20060222045106-5039c71ee3b65490
bzrlib/weave.py knit.py-20050627021749-759c29984154256b
doc/developers/index.txt index.txt-20070508041241-qznziunkg0nffhiw-1
doc/en/user-guide/core_concepts.txt core_concepts.txt-20071114035000-q36a9h57ps06uvnl-2
bzrlib/tests/test_revisionspec.py testrevisionnamespaces.py-20050711050225-8b4af89e6b1efe84
------------------------------------------------------------
revno: 3327.1.1
revision-id:mhammond at skippinet.com.au-20080403024532-xkq0nt29natiws0h
parent: pqm at pqm.ubuntu.com-20080402090835-uq4ef5jnwo7opd9t
committer: Mark Hammond <mhammond at skippinet.com.au>
branch nick: bzr.work2
timestamp: Thu 2008-04-03 13:45:32 +1100
message:
Add document outlining strategies for TortoiseBzr.
added:
doc/developers/tortoise-strategy.txt tortoisestrategy.txt-20080403024510-2ahdqrvnwqrb5p5t-1
=== added file 'doc/developers/tortoise-strategy.txt'
--- a/doc/developers/tortoise-strategy.txt 1970-01-01 00:00:00 +0000
+++ b/doc/developers/tortoise-strategy.txt 2008-04-09 08:19:38 +0000
@@ -0,0 +1,354 @@
+Bazaar Windows Shell Extension Options
+========================================
+
+.. contents:: :local:
+
+Introduction
+------------
+
+This document details the imlpementation strategy chosen for the
+Bazaar Windows Shell Extensions, otherwise known as TortoiseBzr, or TBZR.
+As justification for the strategy, it also describes the general architecture
+of Windows Shell Extensions, then looks at the C++ implemented TortoiseSvn
+and the Python implemented TortoiseBzr, and discusses alternative
+implementation strategies, and the reasons they were not chosen.
+
+The following points summarize the strategy.
+
+* Main shell extension code will be implemented in C++, and be as thin as
+ possible. It will not directly do any VCS work, but instead will perform
+ all operations via either external applications or an RPC server.
+
+* Most VCS operations will be performed by external applications. For
+ example, committing changes or viewing history will spawn a child
+ process that provides its own UI.
+
+* For operations where spawning a child process is not practical, an
+ external RPC server will be implemented in Python and will directly use
+ the VCS library. In the short term, there will be no attempt to create a
+ general purpose RPC mechanism, but instead will be focused on keeping the
+ C++ RPC client as thin, fast and dumb as possible.
+
+Background Information
+----------------------
+
+The facts about shell extensions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Well - the facts as I understand them :)
+
+Shell Extensions are COM objects. They are implemented as DLLs which are
+loaded by the Windows shell. There is no facility for shell extensions to
+exist in a separate process - DLLs are the only option, and they are loaded
+into other processes which take advantage of the Windows shell (although
+obviously this DLL is free to do whatever it likes)
+
+For the sake of this discussion, there are 2 categories of shell extensions:
+
+* Ones that create a new "namespace". The file-system itself is an example of
+ such a namespace, as is the "Recycle Bin". For a user-created example,
+ picture a new tree under "My Computer" which allows you to browse a remote
+ server - it creates a new, stand-alone tree that doesn't really interact
+ with the existing namespaces.
+
+* Ones that enhance existing namespaces, including the filesystem. An example
+ would be an extension which uses Icon Overlays to modify how existing files
+ on disk are displayed or add items to their context menu, for example.
+
+The latter category is the kind of shell extension relevant for TortoiseBzr,
+and it has an important implication - it will be pulled into any process
+which uses the shell to display a list of files. While this is somewhat
+obvious for Windows Explorer (which many people consider the shell), every
+other process that shows a FileOpen/FileSave dialog will have these shell
+extensions loaded into its process space. This may surprise many people - the
+simple fact of allowing the user to select a filename will result in an
+unknown number of DLLs being loaded into your process. For a concrete
+example, when notepad.exe first starts with an empty file it is using around
+3.5MB of RAM. As soon as the FileOpen dialog is loaded, TortoiseSvn loads
+well over 20 additional DLLs, including the MSVC8 runtime, into the Notepad
+process causing its memory usage to more than double - all without doing
+anything tortoise specific at all.
+
+This has wide-ranging implications. It means that such shell extensions
+should be developed using a tool which can never cause conflict with
+arbitrary processes. For this very reason, MS recommend against using .NET
+to write shell extensions[1], as there is a significant risk of being loaded
+into a process that uses a different version of the .NET runtime, and this
+will kill the process. Similarly, Python implemented shell extension may well
+conflict badly with other Python implemented applications (and will certainly
+kill them in some situations). A similar issue exists with GUI toolkits used
+- using (say) PyGTK directly in the shell extension would need to be avoided
+(which it currently is best I can tell). It should also be obvious the shell
+extension will be in many processes simultaneously, meaning use of a simple
+log-file etc is problematic.
+
+In practice, there is only 1 truly safe option - a low-level language (such
+as C/C++) which makes use of only the win32 API, and a static version of the
+C runtime library if necessary. Obviously, this sucks from our POV :)
+
+[1]: http://blogs.msdn.com/oldnewthing/archive/2006/12/18/1317290.aspx
+
+Analysis of TortoiseSVN code
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+TortoiseSVN is implemented in C++. It relies on an external process to
+perform most UI (such as diff, log, commit etc commands), but it appears to
+directly embed the SVN C libraries for the purposes of obtaining status for
+icon overlays, context menu, drag&drop, etc.
+
+The use of an external process to perform commands is fairly simplistic in
+terms of parent and modal windows - for example, when selecting "Commit", a
+new process starts and *usually* ends up as the foreground window, but it may
+occasionally be lost underneath the window which created it, and the user may
+accidently start many processes when they only need 1. Best I can tell, this
+isn't necessarily a limitation of the approach, just the implementation.
+
+Advantages of using the external process is that it keeps all the UI code
+outside Windows explorer - only the minimum needed to perform operations
+directly needed by the shell are part of the "shell extension" and the rest
+of TortoiseSvn is "just" a fairly large GUI application implementing many
+commands. The command-line to the app has even been documented for people who
+wish to automate tasks using that GUI. This GUI appears to also be
+implemented in C++ using Windows resource files.
+
+TortoiseSvn appears to cache using a separate process, aptly named
+TSVNCache.exe. It uses a named pipe to accept connections from other
+processes for various operations. At this stage, it's still unclear exactly
+what is fetched from the cache and exactly what the shell extension fetches
+directly via the subversion C libraries.
+
+There doesn't seem to be a good story for logging or debugging - which is
+what you expect from C++ based apps :( Most of the heavy lifting is done by
+the external application, which might offer better facilities.
+
+Analysis of existing TortoiseBzr code
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The existing code is actually quite cool given its history (SoC student,
+etc), so this should not be taken as criticism of the implementer nor of the
+implementation - indeed, many criticisms are also true of the TortoiseSvn
+implementation - see above. However, I have attempted to list the bad things
+rather than the good things so a clear future strategy can be agreed, with
+all limitations understood.
+
+The existing TortoiseBzr code has been ported into Python from other tortoise
+implementations (probably svn). This means it is very nice to implement and
+develop, but suffers the problems described above - it is likely to conflict
+with other Python based processes, and it means the entire CPython runtime
+and libraries are pulled into many arbitrary processes.
+
+The existing TortoiseBzr code pulls in the bzrlib library to determine the
+path of the bzr library, and also to determine the status of files, but uses
+an external process for most GUI commands - ie, very similar to TortoiseSvn
+as described above - and as such, all comments above apply equally here - but
+note that the bzr library *is* pulled into the shell, and therefore every
+application using the shell. The GUI in the external application is written
+in PyGTK, which may not offer the best Windows "look and feel", but that
+discussion is beyond the scope of this document.
+
+It has a better story for logging and debugging for the developer - but not
+for diagnosing issues in the field - although again, much of the heavy
+lifting remains done by the external application.
+
+It uses a rudimentary in-memory cache for the status of files and
+directories, the implementation of which isn't really suitable (ie, no
+theoretical upper bound on cache size), and also means that there is no
+sharing of cached information between processes, which is unfortunate (eg,
+imagine a user using Windows explorer, then switching back to their editor)
+and also error prone (it's possible the editor will check the file in,
+meaning Windows explorer will be showing stale data). This may be possible to
+address via file-system notifications, but a shared cache would be preferred
+(although clearly more difficult to implement)
+
+One tortoise port recently announced a technique for all tortoise ports to
+share the same icon overlays to help work around a limitation in Windows on
+the total number of overlays (its limited to 15, due to the number of bits
+reserved in a 32bit int for overlays). TBZR needs to take advantage of that
+(but to be fair, this overlay sharing technique was probably done after the
+TBZR implementation)
+
+The current code appears to recursively walk a tree to check if *any* file in
+the tree has changed, so it can reflect this in the parent directory status.
+This is almost certainly an evil thing to do (Shell Extensions are optimized
+so that a folder doesn't even need to look in its direct children for another
+folder, let alone recurse for any reason at all. It may be a network mounted
+drive that doesn't perform at all)
+
+Although somewhat dependent on bzr itself, we need a strategy for binary
+releases (ie, it assumes python.exe, etc) and integration into an existing
+"blessed" installer.
+
+Trivially, the code is not PEP8 compliant and was written by someone fairly
+inexperienced with the language.
+
+Detailed Implementation Strategy
+---------------------------------
+
+We will create a hybrid Python and C++ implementation. In this model, we
+would still use something like TSVNCache.exe (this external
+process doesn't have the same restrictions as the shell extension itself) but
+go one step further - use this remote process for *all* interactions with
+bzr, including status and other "must be fast" operations. This would allow
+the shell extension itself to be implemented in C++, but still take advantage
+of Python for much of the logic.
+
+A pragmatic implementation strategy will be used to work towards the above
+infrastructure - we will keep the shell extension implemented in Python - but
+without using bzrlib. This would allow us to focus on this
+shared-cache/remote-process infrastructure without immediately
+re-implementing a shell extension in C++. Longer term, once the
+infrastructure is in place and as optimized as possible, we can move to C++
+code in the shell calling our remote Python process. This port should try and
+share as much code as possible from TortoiseSvn, including overlay handlers.
+
+External Command Processor
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The external command application (ie, the app invoked by the shell extension
+to perform commands) can remain as-is, and remain a "shell" for other
+external commands. The implementation of this application is not particularly
+relevant to the shell extension, just the interface to the application (ie,
+its command-line) is. In the short term this will remain PyGTK and will only
+change if there is compelling reason - cross-platform GUI tools are a better
+for bazaar than Windows specific ones, although native look-and-feel is
+important. Either way, this can change independently from the shell
+extension.
+
+Performance considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As discussed above, the model used by Tortoise is that most "interesting"
+things are done by external applications. Most Tortoise implementations
+show read-only columns in the "detail" view, and shows a few read only
+properties in the "Properties" dialog - but most of these properties are
+"state" related (eg, revision number), or editing of others is done by
+launching an external application. This means that the shell extension itself
+really has 2 basic requirements WRT RPC: 1) get the local state of a file and
+2) get some named state-related "properties" for a file. Everything else can
+be built on that.
+
+There are 2 aspects of the shell integration which are performance critical -
+the "icon overlays" and "column providers"
+
+The short-story with Icon Overlays is that we need to register 12 global
+"overlay providers" - one for each state we show. Each provider is called for
+every icon ever shown in Windows explorer or in any application's FileOpen
+dialog. While most versions of Windows update icons in the background, we
+still need to perform well. On the positive side, this just needs the simple
+"local state" of a file - information that can probably be carried in a
+single byte. On the negative side, it is the shell which makes a synchronous
+call to us with a single filename as an arg, which makes it difficult to
+"batch" multiple status requests into a single RPC call.
+
+The story with columns is messier - these have changed significantly for
+Vista and the new system may not work with the VCS model (see below).
+However, if we implement this, it will be fairly critical to have
+high-performance name/value pairs implemented, as described above.
+
+Note that the nature of the shell implementation means we will have a large
+number of "unrelated" handlers, each called somewhat independently by the
+shell, often for information about the same file (eg, imagine each of our
+overlay providers all called in turn with the same filename, followed by our
+column providers called in turn with the same filename. However, that isn't
+exactly what happens!). This means we will need a kind of cache, geared
+towards reducing the number of status or property requests we make to the RPC
+server.
+
+We will also allow all of the above to be disabled via user preferences.
+Thus, Icon Overlays could be disabled if it did cause a problem for some
+people, for example.
+
+RPC options
+~~~~~~~~~~~~
+
+Due to the high number of calls for icon overlays, the RPC overhead must be
+kept as low as possible. Due to the client side being implemented in C++,
+reducing complexity is also a goal. Our requirements are quite simple and no
+existing RPC options exist we can leverage. It does not seen prudent to build
+an XMLRPC solution for tbzr - which is not to preclude the use of such a
+server in the future, but tbzr need not become the "pilot" project for an
+XMLRPC server given these constraints.
+
+I propose that a custom RPC mechanism, built initially using windows-specific
+named-pipes, be used. A binary format, designed with an eye towards
+implementation speed and C++ simplicity, will be used. If we succeed here, we
+can build on that infrastructure, and even replace it should other more
+general frameworks materialize.
+
+FWIW, with a Python process at each end, my P4 2.4G machine can achieve
+around 25000 "calls" per-second across an open named pipe. C++ at one end
+should increase this a little, but obviously any real work done by the Python
+side of the process will be the bottle-neck. However, this throughput would
+appear sufficient to implement a prototype.
+
+Vista versus XP
+~~~~~~~~~~~~~~~~
+
+Let's try and avoid an OS advocacy debate :) But it is probably true that
+TBZR will, over its life, be used by more Vista computers than XP ones. In
+short, Vista has changed a number of shell related interfaces, and while TSVN
+is slowly catching up (http://tortoisesvn.net/vistaproblems) they are a pain.
+
+XP has IColumnProvider (as implemented by Tortoise), but Vista changes this
+model. The new model is based around "file types" (eg, .jpg files) and it
+appears each file type can only have 1 provider! TSVN also seems to think the
+Vista model isn't going to work (see previous link). It's not clear how much
+effort we should expend on a column system that has already been abandoned by
+MS. I would argue we spend effort on other parts of the system (ie, the
+external GUI apps themselves, etc) and see if a path forward does emerge for
+Vista. We can re-evaluate this based on user feedback and more information
+about features of the Vista property system.
+
+Implementation plan:
+--------------------
+
+The following is a high-level set of milestones for the implementation:
+
+* Design the RPC mechanism used for icon overlays (ie, binary format used for
+ communication)
+
+* Create Python prototype of the C++ "shim": modify the existing TBZR Python
+ code so that all references to "bzrlib" are removed. Implement the client
+ side of the RPC mechanism and implement icon overlays using this RPC
+ mechanism.
+
+* Create initial implementation of RPC server in Python. This will use
+ bzrlib, but will also maintain a local cache to achieve the required
+ performance. The initial implementation may even be single-threaded, just
+ to keep synchronization issues to a minimum.
+
+* Analyze performance of prototype. Verify that technique is feasible and
+ will offer reasonable performance and user experience.
+
+* Implement C++ shim: replace the Python prototype with a light-weight C++
+ version. We would work from the current TSVN sources, including its new
+ support for sharing icon overlays. Advice on if we should "fork" TSVN, or
+ try and manage our own svn based branch in bazaar are invited.
+
+* Implement property pages and context menus in C++. Expand RPC server as
+ necessary.
+
+* Create binary for alpha releases, then go round-and-round until its baked.
+
+Alternative Implementation Strategies
+-------------------------------------
+
+Only one credible alternative strategy was identified, as discussed below. No
+languages other than Python and C++ were considered; Python as the bzr
+library and existing extensions are written in Python and otherwise only C++
+for reasons outlined in the background on shell extensions above.
+
+Implement Completely in Python
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This would keep the basic structure of the existing TBZR code, with the
+shell extension continuing to pull in Python and all libraries used by Bzr
+into various processes.
+
+Although implementation simplicity is a key benefit to this option, it was
+not chosen for various reasons; The use of Python means that there is a
+larger chance of conflicting with existing applications, or even existing
+Python implemented shell extensions. It will also increase the memory usage
+of all applications which use the shell. While this may create problems for a
+small number of users, it may create a wider perception of instability or
+resource hogging.
=== modified file 'NEWS'
--- a/NEWS 2008-04-13 17:41:14 +0000
+++ b/NEWS 2008-04-14 18:38:11 +0000
@@ -25,6 +25,8 @@
DOCUMENTATION:
+ * Document outlining strategies for TortoiseBzr. (Mark Hammond)
+
TESTING:
INTERNALS:
More information about the bazaar-commits
mailing list