[MERGE] Improvements to is_ignored, factored out glob stuff.
Jan Hudec
bulb at ucw.cz
Tue Jan 17 21:29:30 GMT 2006
Hello All,
Commited http://drak.ucw.cz/~bulb/bzr/bzr.ignore revision 1527, which factors out
the glob stuff into a separate file.
On Mon, Jan 16, 2006 at 20:50:40 +0100, Jan Hudec wrote:
> On Mon, Jan 16, 2006 at 09:10:28 -0600, John A Meinel wrote:
> > Jan Hudec wrote:
> > > + def test_leading_dotslash(self):
> > > + self.assertMatch(u'./foo', [u'foo'])
> > > + self.assertNotMatch(u'./foo', [u'\u8336/foo', u'barfoo', u'x/y/foo'])
> >
> > I'm a little curious about './foo' matching 'foo'. But it does seem to
> > do the rest correctly (it doesn't match foo in another directory)
> >
> > But what about:
> > self.assertMatch(u'./foo', [u'foo', u'./foo'])
> >
> > If './foo' doesn't match './foo' I would definitely be surprised by the
> > system.
>
> No, ./foo really does NOT match ./foo and in this respect the behaviour did
> not change. The semantics of patterns in .bzrignore always was:
> - pattern not containing / is matched against filename only
> - pattern containing / is matched agains full path from root
> - to request matching only entries in the tree root, start the pattern with
> ./
>
> Since the filenames are passed to is_ignored without leading ./, we need to
> strip it from the pattern as well.
With factoring out the glob stuff, I made it independent from the ignore
logic (with helper function for it), but this feature remained. The point is,
that *//*, */./* and */* match the same files in shell. So they are
canonicalized, because otherwise the former two would usually match nothing
in bzr. But to avoid overcomplicating the patterns, they must be matched
against canonical paths -- which never starts with './' (I have described
this in help for bzrlib.glob.translate function).
I attach a patch for review again (cumulative for the whole change -- use the
branch to see individual steps). Please merge from above url
(http://drak.ucw.cz/~bulb/bzr/bzr.ignore)
=== added file 'bzrlib/glob.py'
--- /dev/null
+++ bzrlib/glob.py
@@ -0,0 +1,171 @@
+# Copyright (C) 2006 Jan Hudec
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+"""Tools for converting globs to regular expressions.
+
+This module provides functions for converting shell-like globs to regular
+expressions. See translate function for implemented glob semantics.
+"""
+
+
+import re
+
+
+class replacer(object):
+ """Do a multiple-pattern substitution.
+
+ The patterns and substitutions are combined into one, so the result of
+ one replacement is never substituted again. Add the patterns and
+ replacements via the add method and then call the object. The patterns
+ must not contain capturing groups.
+ """
+ _expand = re.compile(ur'\\&')
+ def __init__(self):
+ self._pat = None
+ self._pats = []
+ self._funs = []
+
+ def add(self, pat, fun):
+ r"""Add a pattern and replacement.
+
+ The pattern must not contain capturing groups.
+ The replacement might be either a string template in which \& will be
+ replaced with the match, or a function that will get the matching text as
+ argument. It does not get match object, because capturing is forbidden
+ anyway.
+ """
+ self._pat = None
+ self._pats.append(pat)
+ self._funs.append(fun)
+
+ def __call__(self, text):
+ if self._pat is None:
+ self._pat = re.compile(
+ u'|'.join([u'(%s)' % p for p in self._pats]),
+ re.UNICODE)
+ return self._pat.sub(self._do_sub, text)
+
+ def _do_sub(self, m):
+ fun = self._funs[m.lastindex - 1]
+ if hasattr(fun, '__call__'):
+ return fun(m.group(0))
+ else:
+ return self._expand.sub(m.group(0), fun)
+
+
+_sub_named = replacer()
+_sub_named.add(u'\[:digit:\]', ur'\d')
+_sub_named.add(u'\[:space:\]', ur'\s')
+_sub_named.add(u'\[:alnum:\]', ur'\w')
+_sub_named.add(u'\[:ascii:\]', ur'\0-\x7f')
+_sub_named.add(u'\[:blank:\]', ur' \t')
+_sub_named.add(u'\[:cntrl:\]', ur'\0-\x1f\x7f-\x9f')
+# I would gladly support many others like [:alpha:], [:graph:] and [:print:]
+# but python regular expression engine does not provide their equivalents.
+
+def _sub_group(m):
+ if m[1] == u'!':
+ m[1] = u'^'
+ return u'[' + _sub_named(m[1:-1]) + u']'
+
+def _sub_re(m):
+ return m[3:]
+
+_sub_glob = replacer()
+_sub_glob.add(ur'^RE:.*', _sub_re) # RE:<anything> is a regex.
+_sub_glob.add(ur'(?:(?<=/)|^)(?:\.?/)+', u'') # Canonicalize
+_sub_glob.add(ur'\\.', ur'\&') # keep anything backslashed...
+_sub_glob.add(ur'[(){}|^$+.]', ur'\\&') # escape specials...
+_sub_glob.add(ur'(?:(?<=/)|^)\*\*\*/', ur'(?:[^/]+/)*') # ***, includes .*
+_sub_glob.add(ur'(?:(?<=/)|^)\*\*/', ur'(?:[^./][^/]*/)*') # **, zsh-style
+_sub_glob.add(ur'(?:(?<=/)|^)\*\.', ur'(?:[^./][^/]*)\.') # *. after / or at start
+_sub_glob.add(ur'(?:(?<=/)|^)\*', ur'(?:[^./][^/]*)?') # * after / or at start
+_sub_glob.add(ur'\*', ur'[^/]*') # * elsewhere
+_sub_glob.add(ur'(?:(?<=/)|^)\?', ur'[^./]') # ? after / or at start
+_sub_glob.add(ur'\?', ur'[^/]') # ? elsewhere
+_sub_glob.add(ur'\[\^?\]?(?:[^][]|\[:[^]]+:\])+\]', _sub_group) # character group
+
+
+def translate(pat):
+ r"""Convert a unix glob to regular expression.
+
+ Globs implement *, ?, [] character groups (both ! and ^ negate), named
+ character classes [:digit:], [:space:], [:alnum:], [:ascii:], [:blank:],
+ [:cntrl:] (use /inside/ []), zsh-style **/, ***/ which includes hidden
+ directories and escapes regular expression special characters.
+
+ If a pattern starts with RE:, the rest is considered to be regular
+ expression.
+
+ During conversion the regexp is canonicalized and must be matched against
+ canonical path. The path must NOT start with '/' and must not contain
+ '.' components nor multiple '/'es.
+
+ Pattern is returned as string.
+ """
+ return _sub_glob(pat) + u'$'
+
+
+def translate_list(pats, wrap=u'(?:%s)'):
+ """Convert a list of unix globs to a regular expression.
+
+ The pattern is returned as string. The wrap is % format applied to each
+ individual glob pattern. It has to apply group.
+
+ See translate for glob semantics.
+ """
+ return u'|'.join([wrap % translate(x) for x in pats])
+
+ """
+ Patterns containing '/' need to match whole path; others match
+ against only the last component - as per requirement of
+ WorkingTree.is_ignored().
+ """
+
+def compile(pat):
+ """Convert a unix glob to regular expression and compile it.
+
+ This converts a glob to regex via translate and compiles the regex. See
+ translate for glob semantics.
+ """
+ return re.compile(translate(pat), re.UNICODE)
+
+
+def compile_list(pats, wrap=u'(?:%s)'):
+ """Convert a list of unix globs to a regular expression and compile it.
+
+ The pattern is returned as compiled regex object. The wrap is % format
+ applied to each individual glob pattern. It has to apply group.
+ """
+ return re.compile(translate_list(pats, wrap), re.UNICODE)
+
+
+def anchor_glob(pat):
+ if '/' in pat:
+ return pat
+ else:
+ return u'***/' + pat
+
+
+def anchor_globs(pats):
+ """Convert file-globs to path globs as used in ignore patterns.
+
+ If a pattern contains '/' or starts with './', it should match whole path
+ from root (optional './' is stripped), otherwise it should match the
+ filename only. Thus such patterns are prefixed with '***/'.
+
+ """
+ return [anchor_glob(pat) for pat in pats]
=== added file 'bzrlib/tests/test_ignore.py'
--- /dev/null
+++ bzrlib/tests/test_ignore.py
@@ -0,0 +1,191 @@
+# Copyright (C) 2006 by Jan Hudec
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import re
+
+import logging
+from cStringIO import StringIO
+import bzrlib.trace
+
+from bzrlib.branch import Branch
+from bzrlib.tests import TestCase, TestCaseInTempDir
+
+from bzrlib.glob import anchor_glob, compile
+
+class TestGlobs(TestCase):
+
+ def assertMatch(self, glob, names):
+ rx = compile(anchor_glob(glob))
+ for name in names:
+ if not rx.match(name):
+ raise AssertionError(repr(
+ u'name "%s" does not match glob "%s" (rx="%s")' %
+ (name, glob, rx.pattern)))
+
+ def assertNotMatch(self, glob, names):
+ rx = compile(anchor_glob(glob))
+ for name in names:
+ if rx.match(name):
+ raise AssertionError(repr(
+ u'name "%s" does match glob "%s" (rx="%s")' %
+ (name, glob, rx.pattern)))
+
+ def test_char_groups(self):
+ # The definition of digit this uses includes arabic digits from
+ # non-latin scripts (arabic, indic, etc.) and subscript/superscript
+ # digits, but neither roman numerals nor vulgar fractions.
+ self.assertMatch(u'[[:digit:]]', [u'0', u'5', u'\u0663', u'\u06f9',
+ u'\u0f21', u'\xb9'])
+ self.assertNotMatch(u'[[:digit:]]', [u'T', u'q', u' ', u'\u8336'])
+
+ self.assertMatch(u'[[:space:]]', [u' ', u'\t', u'\n', u'\xa0',
+ u'\u2000', u'\u2002'])
+ self.assertMatch(u'[^[:space:]]', [u'a', u'-', u'\u8336'])
+ self.assertNotMatch(u'[[:space:]]', [u'a', u'-', u'\u8336'])
+
+ self.assertMatch(u'[[:alnum:]]', [u'a', u'Z', u'\u017e', u'\u8336'])
+ self.assertMatch(u'[^[:alnum:]]', [u':', u'-', u'\u25cf'])
+ self.assertNotMatch(u'[[:alnum:]]', [u':', u'-', u'\u25cf'])
+
+ self.assertMatch(u'[[:ascii:]]', [u'a', u'Q', u'^'])
+ self.assertNotMatch(u'[^[:ascii:]]', [u'a', u'Q', u'^'])
+ self.assertNotMatch(u'[[:ascii:]]', [u'\xcc', u'\u8336'])
+
+ self.assertMatch(u'[[:blank:]]', [u'\t'])
+ self.assertNotMatch(u'[^[:blank:]]', [u'\t'])
+ self.assertNotMatch(u'[[:blank:]]', [u'x', u'y', u'z'])
+
+ self.assertMatch(u'[[:cntrl:]]', [u'\b', u'\t', '\x7f'])
+ self.assertNotMatch(u'[[:cntrl:]]', [u'a', u'Q', u'\u8336'])
+
+ self.assertMatch(u'[a-z]', [u'a', u'q', u'f'])
+ self.assertNotMatch(u'[a-z]', [u'A', u'Q', u'F'])
+ self.assertMatch(u'[^a-z]', [u'A', u'Q', u'F'])
+ self.assertNotMatch(u'[^a-z]', [u'a', u'q', u'f'])
+
+ self.assertMatch(ur'[\x20-\x30\u8336]', [u'\040', u'\044', u'\u8336'])
+ self.assertNotMatch(ur'[^\x20-\x30\u8336]', [u'\040', u'\044', u'\u8336'])
+
+ def test_question_mark(self):
+ self.assertMatch(u'?foo', [u'xfoo', u'bar/xfoo', u'bar/\u8336foo'])
+ self.assertNotMatch(u'?foo', [u'.foo', u'bar/.foo', u'bar/foo',
+ u'foo'])
+
+ self.assertMatch(u'foo?bar', [u'fooxbar', u'foo.bar',
+ u'foo\u8336bar', u'qyzzy/foo.bar'])
+ self.assertNotMatch(u'foo?bar', [u'foo/bar'])
+
+ self.assertMatch(u'foo/?bar', [u'foo/xbar'])
+ self.assertNotMatch(u'foo/?bar', [u'foo/.bar', u'foo/bar',
+ u'bar/foo/xbar'])
+
+ def test_asterisk(self):
+ self.assertMatch(u'*.x', [u'foo/bar/baz.x', u'\u8336/Q.x'])
+ self.assertNotMatch(u'*.x', [u'.foo.x', u'bar/.foo.x', u'.x'])
+
+ self.assertMatch(u'x*x', [u'xx', u'x.x', u'x\u8336..x',
+ u'\u8336/x.x'])
+ self.assertNotMatch(u'x*x', [u'x/x', u'bar/x/bar/x', u'bax/abaxab'])
+
+ self.assertMatch(u'*/*x', [u'\u8336/x', u'foo/bax', u'x/a.x'])
+ self.assertNotMatch(u'*/*x', [u'.foo/x', u'\u8336/.x', u'foo/.q.x',
+ u'foo/bar/bax'])
+
+ def test_double_asterisk(self):
+ self.assertMatch(u'**/\u8336', [u'\u8336', u'foo/\u8336',
+ u'q/y/z/z/y/\u8336'])
+ self.assertNotMatch(u'**/\u8336', [u'that\u8336',
+ u'q/y/z/.z/y/\u8336', u'a/b\u8336'])
+
+ self.assertMatch(u'x**x', [u'xaaargx', u'boo/x-x'])
+ self.assertNotMatch(u'x**x', [u'x/y/z/bax', u'boo/x/x'])
+ # ... because it's not **, but rather **/ and it is only recognized
+ # after / or at the begining.
+
+ self.assertMatch(u'x**/x', [u'xaarg/x', u'x/x'])
+ self.assertNotMatch(u'x**/x', [u'xa/b/x', u'foo/xfoo/x'])
+
+ def test_leading_dotslash(self):
+ self.assertMatch(u'./foo', [u'foo'])
+ self.assertNotMatch(u'./foo', [u'\u8336/foo', u'barfoo', u'x/y/foo'])
+
+ def test_end_anchor(self):
+ self.assertMatch(u'*.333', [u'foo.333'])
+ self.assertNotMatch(u'*.3', [u'foo.333'])
+
+class TestBzrignore(TestCaseInTempDir):
+
+ shape = None
+
+ def setUp(self):
+ super(TestBzrignore, self).setUp()
+ self.branch = Branch.initialize(u'.')
+ self.wt = self.branch.working_tree()
+
+ def putIgnores(self, ignores):
+ bzrignore = file(u'.bzrignore', 'wb')
+ bzrignore.write(ignores)
+
+ def assertIgnored(self, name):
+ if not self.wt.is_ignored(name):
+ raise AssertionError(repr(u'name "%s" is not ignored' % name))
+
+ def assertNotIgnored(self, name):
+ if self.wt.is_ignored(name):
+ raise AssertionError(repr(u'name "%s" is ignored' % name))
+
+ def assertIgnoredBy(self, name, pattern):
+ by = self.wt.is_ignored_by(name)
+ if by != pattern:
+ raise AssertionError(repr(
+ u'name "%s" ignored by "%s" instead of "%s"' %
+ (name, by, pattern)))
+
+
+ def test_unicode(self):
+ self.putIgnores(u'*.\u8336\n'.encode('utf-8'))
+ self.assertIgnored(u'foo.\u8336')
+ self.assertNotIgnored(u'.boo.\u8336')
+ self.assertNotIgnored(u'this')
+
+ def test_misencoded(self):
+ stderr = StringIO()
+ handler = logging.StreamHandler(stderr)
+ handler.setFormatter(bzrlib.trace.QuietFormatter())
+ handler.setLevel(logging.INFO)
+ logger = logging.getLogger('')
+ logger.addHandler(handler)
+ self.putIgnores(u'*.[1\xc1]\n*.2\n'.encode('iso8859-1'))
+
+ self.assertNotIgnored(u'whatever.1')
+ self.assertNotIgnored(u'whatever.2')
+ self.assertContainsRe(stderr.getvalue(), 'WARN.*not utf-8')
+
+ def test_long(self):
+ self.putIgnores(u''.join([u'*.%i\n' % i
+ for i in range(1, 999)]).encode('utf-8'))
+ self.assertIgnoredBy(u'foo.333', u'*.333')
+ self.assertIgnoredBy(u'qyzzy.666', u'*.666')
+ self.assertIgnoredBy(u'\u8336.42', u'*.42')
+ self.assertNotIgnored(u'\u8336')
+ self.assertIgnoredBy(u'42', None)
+
+ def test_escapes(self):
+ self.putIgnores(u'*.\\*\n\\[*\\]\n'.encode('utf-8'))
+ # XXX: \uNNNN sequence does not work for me here, though it does
+ # work for me in TestGlob.test_char_groups above.
+ self.assertIgnored(u'foo.*')
+ self.assertIgnored(u'[foo]')
=== modified file 'bzrlib/add.py'
--- bzrlib/add.py
+++ bzrlib/add.py
@@ -155,7 +155,7 @@
if subf == bzrlib.BZRDIR:
mutter("skip control directory %r", subp)
else:
- ignore_glob = tree.is_ignored(subp)
+ ignore_glob = tree.is_ignored_by(subp)
if ignore_glob is not None:
mutter("skip ignored sub-file %r", subp)
if ignore_glob not in ignored:
=== modified file 'bzrlib/builtins.py'
--- bzrlib/builtins.py
+++ bzrlib/builtins.py
@@ -1135,7 +1135,7 @@
if file_class != 'I':
continue
## XXX: Slightly inefficient since this was already calculated
- pat = tree.is_ignored(path)
+ pat = tree.is_ignored_by(path)
print '%-50s %s' % (path, pat)
=== modified file 'bzrlib/tests/__init__.py'
--- bzrlib/tests/__init__.py
+++ bzrlib/tests/__init__.py
@@ -692,6 +692,7 @@
'bzrlib.tests.test_hashcache',
'bzrlib.tests.test_http',
'bzrlib.tests.test_identitymap',
+ 'bzrlib.tests.test_ignore',
'bzrlib.tests.test_inv',
'bzrlib.tests.test_log',
'bzrlib.tests.test_merge',
=== modified file 'bzrlib/workingtree.py'
--- bzrlib/workingtree.py
+++ bzrlib/workingtree.py
@@ -44,14 +44,15 @@
from copy import deepcopy
import os
+import re
import stat
-import fnmatch
from bzrlib.branch import (Branch,
is_control_file,
needs_read_lock,
needs_write_lock,
quotefn)
+from bzrlib.glob import anchor_globs, compile_list
from bzrlib.errors import (BzrCheckError,
BzrError,
DivergedBranches,
@@ -75,7 +76,7 @@
rename)
from bzrlib.textui import show_status
import bzrlib.tree
-from bzrlib.trace import mutter
+from bzrlib.trace import mutter, warning
import bzrlib.xml5
@@ -85,7 +86,6 @@
This should probably generate proper UUIDs, but for the moment we
cope with just randomness because running uuidgen every time is
slow."""
- import re
from binascii import hexlify
from time import time
@@ -712,11 +712,10 @@
def ignored_files(self):
- """Yield list of PATH, IGNORE_PATTERN"""
+ """Yield list of paths"""
for subp in self.extras():
- pat = self.is_ignored(subp)
- if pat != None:
- yield subp, pat
+ if self.is_ignored(subp):
+ yield subp
def get_ignore_list(self):
@@ -730,10 +729,43 @@
l = bzrlib.DEFAULT_IGNORE[:]
if self.has_filename(bzrlib.IGNORE_FILENAME):
f = self.get_file_byname(bzrlib.IGNORE_FILENAME)
- l.extend([line.rstrip("\n\r") for line in f.readlines()])
+ try:
+ l.extend([line.decode('utf-8').rstrip("\n\r")
+ for line in f.readlines()])
+ except UnicodeDecodeError:
+ warning("'%s' is not utf-8 encoded, not reading ignore patterns"
+ % bzrlib.IGNORE_FILENAME)
self._ignorelist = l
return l
+ def _get_ignore_regex(self):
+ """Return a regular expression composed of ignore patterns.
+
+ Cached in the Tree object after the first call.
+ """
+ if not hasattr(self, '_ignoreregex'):
+ self._ignoreregex = compile_list(
+ anchor_globs(self.get_ignore_list()))
+ return self._ignoreregex
+
+ def _get_ignore_by_regex_list(self):
+ """Return regex list for is_ignored_by method.
+
+ Cached in the Tree object after the first call.
+
+ The return is a list of lists, each having pattern as the first
+ element, followed by list of globs it is composed from.
+ """
+ if not hasattr(self, '_ignore_by_regex_list'):
+ pats = self.get_ignore_list() # So we can shift...
+ self._ignore_by_regex_list = []
+ while pats:
+ self._ignore_by_regex_list.append(
+ [compile_list(anchor_globs(pats[0:50]),
+ wrap=u'(%s)')]
+ + pats[0:50])
+ pats = pats[50:]
+ return self._ignore_by_regex_list
def is_ignored(self, filename):
r"""Check whether the filename matches an ignore pattern.
@@ -741,37 +773,27 @@
Patterns containing '/' or '\' need to match the whole path;
others match against only the last component.
- If the file is ignored, returns the pattern which caused it to
- be ignored, otherwise None. So this can simply be used as a
- boolean if desired."""
-
- # TODO: Use '**' to match directories, and other extended
- # globbing stuff from cvs/rsync.
-
- # XXX: fnmatch is actually not quite what we want: it's only
- # approximately the same as real Unix fnmatch, and doesn't
- # treat dotfiles correctly and allows * to match /.
- # Eventually it should be replaced with something more
- # accurate.
-
- for pat in self.get_ignore_list():
- if '/' in pat or '\\' in pat:
-
- # as a special case, you can put ./ at the start of a
- # pattern; this is good to match in the top-level
- # only;
-
- if (pat[:2] == './') or (pat[:2] == '.\\'):
- newpat = pat[2:]
- else:
- newpat = pat
- if fnmatch.fnmatchcase(filename, newpat):
- return pat
- else:
- if fnmatch.fnmatchcase(splitpath(filename)[-1], pat):
- return pat
- else:
- return None
+ If the file is ignored, returns a match object, otherwise None. So
+ this can simply be used as a boolean if desired. The match object is
+ really not very useful, because the individual patterns are not
+ captured.
+ """
+ pat = self._get_ignore_regex()
+ return pat.match(filename)
+
+ def is_ignored_by(self, filename):
+ r"""Check whether the filename matches and return the pattern it matches.
+
+ This method is similar to is_ignored, but makes the extra effort to
+ return the pattern that matched.
+ """
+
+ pats = self._get_ignore_by_regex_list()
+ for pat in pats:
+ m = pat[0].match(filename)
+ if m:
+ return pat[m.lastindex]
+ return None
def kind(self, file_id):
return file_kind(self.id2abspath(file_id))
--
Jan 'Bulb' Hudec <bulb at ucw.cz>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060117/a5418750/attachment.pgp
More information about the bazaar
mailing list