Rev 3815: Merge in the debug hacks in http://bzr.arbash-meinel.com/branches/bzr/brisbane/hack

John Arbash Meinel john at arbash-meinel.com
Tue Dec 23 21:18:44 GMT 2008


At http://bzr.arbash-meinel.com/branches/bzr/brisbane/hack

------------------------------------------------------------
revno: 3815
revision-id: john at arbash-meinel.com-20081223211826-lkia2y1fjvs6vsrx
parent: john at arbash-meinel.com-20081223170214-sbuus38ifyfa9hin
parent: john at arbash-meinel.com-20081223171750-pker2zueia9u5gs2
committer: John Arbash Meinel <john at arbash-meinel.com>
branch nick: hack
timestamp: Tue 2008-12-23 15:18:26 -0600
message:
  Merge in the debug hacks
modified:
  bzrlib/repofmt/pack_repo.py    pack_repo.py-20070813041115-gjv5ma7ktfqwsjgn-1
  bzrlib/trace.py                trace.py-20050309040759-c8ed824bdcd4748a
  bzrlib/xml8.py                 xml5.py-20050907032657-aac8f960815b66b1
    ------------------------------------------------------------
    revno: 3800.2.6
    revision-id: john at arbash-meinel.com-20081223171750-pker2zueia9u5gs2
    parent: john at arbash-meinel.com-20081223170918-5y3wgrpl1qp9j86y
    committer: John Arbash Meinel <john at arbash-meinel.com>
    branch nick: debug_hacks
    timestamp: Tue 2008-12-23 11:17:50 -0600
    message:
      Add a flush call to mutter()
    modified:
      bzrlib/trace.py                trace.py-20050309040759-c8ed824bdcd4748a
    ------------------------------------------------------------
    revno: 3800.2.5
    revision-id: john at arbash-meinel.com-20081223170918-5y3wgrpl1qp9j86y
    parent: john at arbash-meinel.com-20081223170721-8ezsrdfl23ugptzm
    committer: John Arbash Meinel <john at arbash-meinel.com>
    branch nick: debug_hacks
    timestamp: Tue 2008-12-23 11:09:18 -0600
    message:
      Bring back the xml non-copy caching.
    modified:
      bzrlib/xml8.py                 xml5.py-20050907032657-aac8f960815b66b1
    ------------------------------------------------------------
    revno: 3800.2.4
    revision-id: john at arbash-meinel.com-20081223170721-8ezsrdfl23ugptzm
    parent: john at arbash-meinel.com-20081208183149-0m60m72x4pw378ib
    parent: john at arbash-meinel.com-20081219230732-ri1i1tujtrh2d3sl
    committer: John Arbash Meinel <john at arbash-meinel.com>
    branch nick: debug_hacks
    timestamp: Tue 2008-12-23 11:07:21 -0600
    message:
      bring in the latest brisbane-core, resolve differences
    added:
      bzrlib/_chunks_to_lines_py.py  _chunks_to_lines_py.-20081211024848-6uc3mtuje8j14l60-1
      bzrlib/_chunks_to_lines_pyx.pyx _chunks_to_lines_pyx-20081211021736-op7n8vrxgrd8snfi-1
      bzrlib/fifo_cache.py           fifo_cache.py-20081209212307-31ffjwvteyvmydnf-1
      bzrlib/tests/blackbox/test_shelve.py test_ls_shelf.py-20081202053526-thlo8yt0pi1cgor1-1
      bzrlib/tests/test__chunks_to_lines.py test__chunks_to_line-20081211024848-6uc3mtuje8j14l60-2
      bzrlib/tests/test_fifo_cache.py test_fifo_cache.py-20081209212307-31ffjwvteyvmydnf-2
    modified:
      .bzrignore                     bzrignore-20050311232317-81f7b71efa2db11a
      Makefile                       Makefile-20050805140406-d96e3498bb61c5bb
      NEWS                           NEWS-20050323055033-4e00b5db738777ff
      bzrlib/_dirstate_helpers_c.h   _dirstate_helpers_c.-20070802205935-hqo9yzuzjix271dd-1
      bzrlib/atomicfile.py           atomicfile.py-20050509044450-dbd24e6c564f7c66
      bzrlib/branch.py               branch.py-20050309040759-e4baf4e0d046576e
      bzrlib/builtins.py             builtins.py-20050830033751-fc01482b9ca23183
      bzrlib/bzrdir.py               bzrdir.py-20060131065624-156dfea39c4387cb
      bzrlib/chk_map.py              chk_map.py-20081001014447-ue6kkuhofvdecvxa-1
      bzrlib/delta.py                delta.py-20050729221636-54cf14ef94783d0a
      bzrlib/errors.py               errors.py-20050309040759-20512168c4e14fbd
      bzrlib/graph.py                graph_walker.py-20070525030359-y852guab65d4wtn0-1
      bzrlib/help_topics/__init__.py help_topics.py-20060920210027-rnim90q9e0bwxvy4-1
      bzrlib/inventory.py            inventory.py-20050309040759-6648b84ca2005b37
      bzrlib/knit.py                 knit.py-20051212171256-f056ac8f0fbe1bd9
      bzrlib/log.py                  log.py-20050505065812-c40ce11702fe5fb1
      bzrlib/lru_cache.py            lru_cache.py-20070119165515-tlw203kuwh0id5gv-1
      bzrlib/merge.py                merge.py-20050513021216-953b65a438527106
      bzrlib/msgeditor.py            msgeditor.py-20050901111708-ef6d8de98f5d8f2f
      bzrlib/osutils.py              osutils.py-20050309040759-eeaff12fbf77ac86
      bzrlib/push.py                 push.py-20080606021927-5fe39050e8xne9un-1
      bzrlib/registry.py             lazy_factory.py-20060809213415-2gfvqadtvdn0phtg-1
      bzrlib/remote.py               remote.py-20060720103555-yeeg2x51vn0rbtdp-1
      bzrlib/repofmt/pack_repo.py    pack_repo.py-20070813041115-gjv5ma7ktfqwsjgn-1
      bzrlib/repository.py           rev_storage.py-20051111201905-119e9401e46257e3
      bzrlib/shelf.py                prepare_shelf.py-20081005181341-n74qe6gu1e65ad4v-1
      bzrlib/shelf_ui.py             shelver.py-20081005210102-33worgzwrtdw0yrm-1
      bzrlib/tests/__init__.py       selftest.py-20050531073622-8d0e3c8845c97a64
      bzrlib/tests/blackbox/__init__.py __init__.py-20051128053524-eba30d8255e08dc3
      bzrlib/tests/blackbox/test_log.py test_log.py-20060112090212-78f6ea560c868e24
      bzrlib/tests/blackbox/test_ls.py test_ls.py-20060712232047-0jraqpecwngee12y-1
      bzrlib/tests/blackbox/test_pull.py test_pull.py-20051201144907-64959364f629947f
      bzrlib/tests/blackbox/test_push.py test_push.py-20060329002750-929af230d5d22663
      bzrlib/tests/blackbox/test_revision_info.py test_revision_info.py-20050917162600-21dab3877aa348d7
      bzrlib/tests/blackbox/test_tags.py test_tags.py-20070116132048-5h4qak2cm22jlb9e-1
      bzrlib/tests/branch_implementations/__init__.py __init__.py-20060123013057-b12a52c3f361daf4
      bzrlib/tests/branch_implementations/test_push.py test_push.py-20070130153159-fhfap8uoifevg30j-1
      bzrlib/tests/per_repository/test_add_fallback_repository.py test_add_fallback_re-20080215040003-8w9n4ck9uqdxj18m-1
      bzrlib/tests/per_repository/test_add_inventory_by_delta.py test_add_inventory_d-20081013002626-rut81igtlqb4590z-1
      bzrlib/tests/per_repository/test_repository.py test_repository.py-20060131092128-ad07f494f5c9d26c
      bzrlib/tests/test_bzrdir.py    test_bzrdir.py-20060131065654-deba40eef51cf220
      bzrlib/tests/test_graph.py     test_graph_walker.py-20070525030405-enq4r60hhi9xrujc-1
      bzrlib/tests/test_http.py      testhttp.py-20051018020158-b2eef6e867c514d9
      bzrlib/tests/test_knit.py      test_knit.py-20051212171302-95d4c00dd5f11f2b
      bzrlib/tests/test_msgeditor.py test_msgeditor.py-20051202041359-920315ec6011ee51
      bzrlib/tests/test_osutils.py   test_osutils.py-20051201224856-e48ee24c12182989
      bzrlib/tests/test_remote.py    test_remote.py-20060720103555-yeeg2x51vn0rbtdp-2
      bzrlib/tests/test_repository.py test_repository.py-20060131075918-65c555b881612f4d
      bzrlib/tests/test_revisionspec.py testrevisionnamespaces.py-20050711050225-8b4af89e6b1efe84
      bzrlib/tests/test_shelf.py     test_prepare_shelf.p-20081005181341-n74qe6gu1e65ad4v-2
      bzrlib/tests/test_smart_transport.py test_ssh_transport.py-20060608202016-c25gvf1ob7ypbus6-2
      bzrlib/tests/test_source.py    test_source.py-20051207061333-a58dea6abecc030d
      bzrlib/tests/test_versionedfile.py test_versionedfile.py-20060222045249-db45c9ed14a1c2e5
      bzrlib/transform.py            transform.py-20060105172343-dd99e54394d91687
      bzrlib/transport/__init__.py   transport.py-20050711165921-4978aa7ce1285ad5
      bzrlib/transport/decorator.py  decorator.py-20060402223305-e913a0f25319ab42
      bzrlib/transport/http/__init__.py http_transport.py-20050711212304-506c5fd1059ace96
      bzrlib/transport/http/_pycurl.py pycurlhttp.py-20060110060940-4e2a705911af77a6
      bzrlib/transport/http/_urllib.py _urlgrabber.py-20060113083826-0bbf7d992fbf090c
      bzrlib/transport/remote.py     ssh.py-20060608202016-c25gvf1ob7ypbus6-1
      bzrlib/versionedfile.py        versionedfile.py-20060222045106-5039c71ee3b65490
      bzrlib/weave.py                knit.py-20050627021749-759c29984154256b
      bzrlib/xml4.py                 xml4.py-20050916091259-db5ab55e7e6ca324
      bzrlib/xml5.py                 xml5.py-20080328030717-t9guwinq8hom0ar3-1
      bzrlib/xml7.py                 xml7.py-20061029182747-d5tiiny21bvrd2jj-1
      bzrlib/xml8.py                 xml5.py-20050907032657-aac8f960815b66b1
      bzrlib/xml_serializer.py       xml.py-20050309040759-57d51586fdec365d
      doc/developers/HACKING.txt     HACKING-20050805200004-2a5dc975d870f78c
      doc/en/user-guide/svn_plugin.txt svn_plugin.txt-20080509065016-cjc90f46407vi9a0-2
      setup.py                       setup.py-20050314065409-02f8a0a6e3f9bc70
      tools/win32/build_release.py   build_release.py-20081105204355-2ghh5cv01v1x4rzz-1
    ------------------------------------------------------------
    revno: 3800.2.3
    revision-id: john at arbash-meinel.com-20081208183149-0m60m72x4pw378ib
    parent: john at arbash-meinel.com-20081207184233-fu65des8q4jhr8ig
    parent: john at arbash-meinel.com-20081208183041-9r88rfoms0y8cr5b
    committer: John Arbash Meinel <john at arbash-meinel.com>
    branch nick: debug_hacks
    timestamp: Mon 2008-12-08 12:31:49 -0600
    message:
      Merge the XML entry cache.
    modified:
      NEWS                           NEWS-20050323055033-4e00b5db738777ff
      bzrlib/lru_cache.py            lru_cache.py-20070119165515-tlw203kuwh0id5gv-1
      bzrlib/tests/test_lru_cache.py test_lru_cache.py-20070119165535-hph6rk4h9rzy4180-1
      bzrlib/xml8.py                 xml5.py-20050907032657-aac8f960815b66b1
    ------------------------------------------------------------
    revno: 3800.2.2
    revision-id: john at arbash-meinel.com-20081207184233-fu65des8q4jhr8ig
    parent: john at arbash-meinel.com-20081207174338-vx3sm4yzedmk338j
    committer: John Arbash Meinel <john at arbash-meinel.com>
    branch nick: debug_hacks
    timestamp: Sun 2008-12-07 12:42:33 -0600
    message:
      Don't execute pack ops twice.
    modified:
      bzrlib/repofmt/pack_repo.py    pack_repo.py-20070813041115-gjv5ma7ktfqwsjgn-1
    ------------------------------------------------------------
    revno: 3800.2.1
    revision-id: john at arbash-meinel.com-20081207174338-vx3sm4yzedmk338j
    parent: john at arbash-meinel.com-20081207174043-xjxckelplsqdey73
    parent: john at arbash-meinel.com-20081203043230-f0riipwg6wkjr6ae
    committer: John Arbash Meinel <john at arbash-meinel.com>
    branch nick: merge_dev
    timestamp: Sun 2008-12-07 11:43:38 -0600
    message:
      Merge in the debug_hacks.
    modified:
      bzrlib/repofmt/pack_repo.py    pack_repo.py-20070813041115-gjv5ma7ktfqwsjgn-1
    ------------------------------------------------------------
    revno: 3791.1.16
    revision-id: john at arbash-meinel.com-20081203043230-f0riipwg6wkjr6ae
    parent: john at arbash-meinel.com-20081203041138-ecssp5m0mqxjgzhu
    committer: John Arbash Meinel <john at arbash-meinel.com>
    branch nick: chk_map
    timestamp: Tue 2008-12-02 22:32:30 -0600
    message:
      Hack in some other code, so we can determine how much compression we get.
      
      This just tracks the 'old size' of all the packs that are getting combined versus the
      'new size' of the newly created pack file.
    modified:
      bzrlib/repofmt/pack_repo.py    pack_repo.py-20070813041115-gjv5ma7ktfqwsjgn-1
    ------------------------------------------------------------
    revno: 3791.1.15
    revision-id: john at arbash-meinel.com-20081203041138-ecssp5m0mqxjgzhu
    parent: john at arbash-meinel.com-20081203035643-0x4npc4s8mh8nqlh
    committer: John Arbash Meinel <john at arbash-meinel.com>
    branch nick: chk_map
    timestamp: Tue 2008-12-02 22:11:38 -0600
    message:
      Add size information to the mutter when -Dpack is used.
      
      Also fix a bug in -Dpack when the repository doesn't support chk_bytes.
    modified:
      bzrlib/repofmt/pack_repo.py    pack_repo.py-20070813041115-gjv5ma7ktfqwsjgn-1
-------------- next part --------------
=== modified file 'bzrlib/repofmt/pack_repo.py'
--- a/bzrlib/repofmt/pack_repo.py	2008-12-20 21:16:24 +0000
+++ b/bzrlib/repofmt/pack_repo.py	2008-12-23 21:18:26 +0000
@@ -420,9 +420,15 @@
                 '../packs/' + self.name + '.pack')
         self._state = 'finished'
         if 'pack' in debug.debug_flags:
+            try:
+                size = self.pack_transport.stat(self.name + '.pack').st_size
+                size /= 1024.*1024
+            except errors.TransportNotPossible:
+                size = -1
             # XXX: size might be interesting?
-            mutter('%s: create_pack: pack renamed into place: %s%s->%s%s t+%6.3fs',
-                time.ctime(), self.upload_transport.base, self.random_name,
+            mutter('%s: create_pack: pack renamed into place (%.3fMB): %s%s->%s%s'
+                   ' t+%6.3fs',
+                time.ctime(), size, self.upload_transport.base, self.random_name,
                 self.pack_transport, self.name,
                 time.time() - self.start_time)
 
@@ -823,10 +829,17 @@
                 rev_count = len(self.revision_ids)
             else:
                 rev_count = 'all'
-            mutter('%s: create_pack: creating pack from source packs: '
+            size = 0
+            for a_pack in self.packs:
+                try:
+                    size += a_pack.pack_transport.stat(a_pack.name + '.pack').st_size
+                except errors.TransportNotPossible:
+                    pass
+            size /= 1024.*1024
+            mutter('%s: create_pack: creating pack from source packs (%.3fMB): '
                 '%s%s %s revisions wanted %s t=0',
-                time.ctime(), self._pack_collection._upload_transport.base, new_pack.random_name,
-                plain_pack_list, rev_count)
+                time.ctime(), size, self._pack_collection._upload_transport.base,
+                new_pack.random_name, plain_pack_list, rev_count)
         self._copy_revision_texts()
         self._copy_inventory_texts()
         self._copy_text_texts()
@@ -1403,9 +1416,18 @@
             'containing %d revisions. Packing %d files into %d affecting %d'
             ' revisions', self, total_packs, total_revisions, num_old_packs,
             num_new_packs, num_revs_affected)
-        self._execute_pack_operations(pack_operations,
+        old_size, new_size = self._execute_pack_operations(pack_operations,
                                       reload_func=self._restart_autopack)
-        mutter('Auto-packing repository %s completed', self)
+        if old_size is None:
+            old_size = -1
+        else:
+            old_size /= (1024.0*1024)
+        if new_size is None:
+            new_size = -1
+        else:
+            new_size /= (1024.0*1024)
+        mutter('Auto-packing repository %s completed %.3fMB => %.3fMB',
+               self.transport.base, old_size, new_size)
         return True
 
     def _execute_pack_operations(self, pack_operations, _packer_class=Packer,
@@ -1416,6 +1438,7 @@
         :param _packer_class: The class of packer to use (default: Packer).
         :return: None.
         """
+        new_size = 0
         for revision_count, packs in pack_operations:
             # we may have no-ops from the setup logic
             if len(packs) == 0:
@@ -1423,7 +1446,7 @@
             packer = _packer_class(self, packs, '.autopack',
                                    reload_func=reload_func)
             try:
-                packer.pack()
+                new_pack = packer.pack()
             except errors.RetryWithNewPacks:
                 # An exception is propagating out of this context, make sure
                 # this packer has cleaned up. Packer() doesn't set its new_pack
@@ -1432,14 +1455,26 @@
                 if packer.new_pack is not None:
                     packer.new_pack.abort()
                 raise
+            try:
+                new_size += new_pack.pack_transport.stat(new_pack.name + '.pack').st_size
+            except errors.TransportNotPossible:
+                new_size = None
             for pack in packs:
                 self._remove_pack_from_memory(pack)
         # record the newly available packs and stop advertising the old
         # packs
+        if new_size is None:
+            old_size = None
+        else:
+            old_size = 0
+            for revision_count, packs in pack_operations:
+                for a_pack in packs:
+                    old_size += a_pack.pack_transport.stat(a_pack.name + '.pack').st_size
         self._save_pack_names(clear_obsolete_packs=True)
         # Move the old packs out of the way now they are no longer referenced.
         for revision_count, packs in pack_operations:
             self._obsolete_packs(packs)
+        return old_size, new_size
 
     def lock_names(self):
         """Acquire the mutex around the pack-names index.

=== modified file 'bzrlib/trace.py'
--- a/bzrlib/trace.py	2008-11-01 14:44:47 +0000
+++ b/bzrlib/trace.py	2008-12-23 17:17:50 +0000
@@ -129,9 +129,11 @@
 info = note
 log_error = _bzr_logger.error
 error =     _bzr_logger.error
+_last_mutter_flush_time = None
 
 
 def mutter(fmt, *args):
+    global _last_mutter_flush_time
     if _trace_file is None:
         return
     if (getattr(_trace_file, 'closed', None) is not None) and _trace_file.closed:
@@ -152,11 +154,16 @@
         out = fmt % tuple(real_args)
     else:
         out = fmt
-    timestamp = '%0.3f  ' % (time.time() - _bzr_log_start_time,)
+    now = time.time()
+    timestamp = '%0.3f  ' % (now - _bzr_log_start_time,)
     out = timestamp + out + '\n'
     _trace_file.write(out)
     # no need to flush here, the trace file is now linebuffered when it's
     # opened.
+    if (_last_mutter_flush_time is None
+        or (now - _last_mutter_flush_time) > 2.0):
+        _trace_file.flush()
+        _last_mutter_flush_time = now
 
 
 def mutter_callsite(stacklevel, fmt, *args):

=== modified file 'bzrlib/xml8.py'
--- a/bzrlib/xml8.py	2008-12-13 03:19:40 +0000
+++ b/bzrlib/xml8.py	2008-12-23 17:09:18 +0000
@@ -428,10 +428,10 @@
                 pass
             else:
                 # Only copying directory entries drops us 2.85s => 2.35s
-                # if cached_ie.kind == 'directory':
-                #     return cached_ie.copy()
-                # return cached_ie
-                return cached_ie.copy()
+                if cached_ie.kind == 'directory':
+                    return cached_ie.copy()
+                return cached_ie
+                # return cached_ie.copy()
 
         kind = elt.tag
         if not InventoryEntry.versionable_kind(kind):



More information about the bazaar-commits mailing list