[rfc] Proposal: fake symlinks support on windows

[Martin Pool]

On 18 May 2006, Alexander Belchenko <bialix at ukr.net> wrote:
> Short summary: make a full cross-platform support and access to symlinks 
> in working tree from non-native filesystem is painful terrible work. 
> Windows and Cygwin share the same filesystem therefore they have good 
> compatibility. But anyway it's better don't mix them.
> 
> So I restrict my proposal for fake symlinks support to deal with 
> symlinks only inside bounds of native filesystem, without mixing access 
> to working tree from outside.

That sounds fine for now.

> I wrote plugin that run-time-patches some modules:
> * os module (adds symlink(), readlink() methods that absent on native 
> windows),
> * os.path module (os.path.islink() function),
> * bzrlib.osutils module (function file_kind()).
> 
> In addition I slightly change selftest and run tests with and without 
> patch. Fortunately all symlinks-specific tests passed, and I don't find 
> any regression in overall test suite.
> 
> If you insist to make all work with symlinks via WorkingTree methods 
> then there is need to grep and change all existing code and change 
> explicit os.symlink/os.readlink. That will be *very* big patch.

That may be good, but also doesn't need to be done now.

> So at this moment I'd like to hear comments from developers that 
> interested in this symlink simulation. What my simple os-patching plugin 
> is missed? What additional tests should be added, improved?



> In attachment you'll find patch for selftest subsystem of bzr. This 
> changes is not depend on windows nor cygwin. I've add test_symlinks.py 
> module with tests somewhat specific for my plugin. This tests looks 
> trivial for unix where  I want to keep it separately and independetly in 
> main bzrlib code with reason to split namespaces.
> 
> I think this patch is worth to include in main bzr.dev.

I think this is fine - essentially it makes the tests be skipped if
there are no symlinks, rather than just silently passing.  That's a
useful improvement.

We do have a specification which uses some finer distinctions than just
TestSkipped; this is a step in that direction and when we do add more
subclasses we can grep through and adjust them.

> # Written by Alexander Belchenko for Bazzar-NG project
> # win32 fake symlinks support
> # based on cygwin symlinks simulation
> 
> """Fake symlinks support on win32"""
> 
> import sys
> 
> if sys.platform == "win32":
>     import errno
>     import os
>     from stat import (S_ISREG, S_ISDIR, S_ISLNK, ST_MODE, ST_SIZE,
>                       S_ISCHR, S_ISBLK, S_ISFIFO, S_ISSOCK)
>     
>     import bzrlib.osutils
>     
>     from win32_symlinks import *
> 
>     # patching os module
>     os.symlink = symlink
>     os.readlink = readlink

To bring this into bzr proper I'd rather not patch the os module but
rather update all callers to either go through bzrlib.osutils or the
workingtree.

>     # patching bzrlib.osutils module
>     def file_kind(f):
>         """Detecting file kind on win32 with fake symlink support"""
>         mode = os.lstat(f)[ST_MODE]
>         if S_ISREG(mode):
>             try:
>                 readlink(f)
>             except OSError, errno.EINVAL:
>                 return 'file'
>             else:
>                 return 'symlink'
>         elif S_ISDIR(mode):
>             return 'directory'
>         elif S_ISLNK(mode):
>             return 'symlink'
>         elif S_ISCHR(mode):
>             return 'chardev'
>         elif S_ISBLK(mode):
>             return 'block'
>         elif S_ISFIFO(mode):
>             return 'fifo'
>         elif S_ISSOCK(mode):
>             return 'socket'
>         else:
>             return 'unknown'
>     
>     bzrlib.osutils.file_kind = file_kind
> 
>     # patching os.path.islink function
>     def islink(path):
>         """Return True if path refers to a directory entry that is a symbolic link.
>         Re-implemented for fake symlink support on win32
>         """
>         return 'symlink' == file_kind(path)
> 
>     os.path.islink = islink


> #!/usr/bin/python
> # This code written by Alexander Belchenko for the Bazaar-NG project
> #
> # Cygwin's symlink file format:
> # '!<symlink>path/to/target\0'
> # file should have SYSTEM attribute in windows
> 
> 
> __all__ = ['symlink', 'readlink']
> 
> 
> import errno
> import os
> 
> # from pywin32 package (http://pywin32.sf.net)
> from win32file import (GetFileAttributes, SetFileAttributes,
>                        FILE_ATTRIBUTE_SYSTEM, FILE_ATTRIBUTE_READONLY)
> from win32com.shell import shell
> from pythoncom import (CoCreateInstance, CLSCTX_INPROC_SERVER,
>                        IID_IPersistFile)
> import pywintypes
> 
> 
> def symlink(src, dst):
>     """Create a symbolic link (in cygwin format) pointing to src named dst.
>     """
>     if os.path.isfile(dst):
>         raise OSError, errno.EEXIST     # 'File exists' error
> 
>     f = file(dst, 'wb')
>     f.write('!<symlink>%s\0' % src)
>     f.close()

The close should be a try/finally just as generally good practice.

> 
>     # change attributes of file to System
>     SetFileAttributes(dst, FILE_ATTRIBUTE_SYSTEM)
> 
> 
> def readlink(path):
>     """Return a string representing the path to which the symbolic link points.
>     The result may be either an absolute or relative pathname;
>     if it is relative, it may be converted to an absolute pathname
>     using os.path.join(os.path.dirname(path), result).
>     """
>     if os.path.isfile(path):            # cygwin symlink to file or directory
>         # check if this file is cygwin symlink
>         f = file(path, 'rb')
>         data = f.read()
>         f.close()
> 
>         if GetFileAttributes(path) & FILE_ATTRIBUTE_SYSTEM \
>             and data.startswith('!<symlink>') \
>             and data.endswith('\0'):
>                 return data[10:-1]
>         else:
>             raise OSError, errno.EINVAL     # 'Invalid argument' error
> 
>     elif os.path.isfile(path+'.lnk'):   # cygwin symlink to directory
>         path = path + '.lnk'
> 
>         # Cygwin create symlink to directory as windows shell shortcut
>         # with READONLY file attribute.
>         if GetFileAttributes(path) & FILE_ATTRIBUTE_READONLY:
>             # this code to work with shortcut is based on pywin32 manual
>             # see 'win32com.shell and Windows Shell Links'
>             shortcut = CoCreateInstance(shell.CLSID_ShellLink,
>                                         None,
>                                         CLSCTX_INPROC_SERVER,
>                                         shell.IID_IShellLink
>                                        )
>             try:
>                 shortcut.QueryInterface(IID_IPersistFile).Load(path)
>                 target = shortcut.GetDescription()
>     
>                 # [bialix 20060517]
>                 # we probably could ensure that this shortcut
>                 # is really cygwin symlink to directory
>                 # in some complex way
>                 # but I'm not sure that it will works for all cygwin versions
>                 if 0:
>                     name, find_data = shortcut.GetPath(shell.SLGP_RAWPATH)
>                     if 0 != find_data[0] \
>                         or [-109205.] * 3 != map(float, find_data[1:4]):
>                             raise pywintypes.com_error
>     
>                 return target
>     
>             except pywintypes.com_error:
>                 # if we get this error then path is not shortcut
>                 # or this file is not cygwin symlink
>                 pass
> 
>         # if we are here then path is not [cygwin] symlink
>         raise OSError, errno.EINVAL     # 'Invalid argument' error

These branches are large enough that they should be split into separate
functions, e.g. _read_cygwin_symlink, etc.

>     else:
>         raise OSError, errno.ENOENT     # 'No such file or directory' error

Can you really conclude that this is ENOENT, as opposed to say EACCESS?
I suppose they'll come from os.path.isfile?

> if __name__ == "__main__":
>     # manual testing
>     if os.path.isfile('test_symlink'):
>         os.remove('test_symlink')
> 
>     symlink('../', 'test_symlink')
> 
>     print readlink('test_symlink')
> 
>     print readlink('dir_symlink')

OK, so to test this better, we would want to make a subclass of
TestCaseInTempDir, and split each of these into separate tests.  You
would also want to test pointing to a nonexistent file (which should
make no difference atm) and failures such as the symlink not existing,
or the directory containing it not existing.  Some of these will be
windows specific but more can be cross-platform, which will help use
define what behaviour is common.

I'm cautiously positive on merging this if those things are fixed, but
would like an opinion from someone who uses windows or cygwin more
often.

-- 
Martin