[ubuntu-mono] [Bug 816945] [NEW] mono: GC_install_header() in libgc/headers.c will segfault if mmap() fails

[Peter Maydell]

Public bug reported:

If mmap() fails mono can crash with a segfault rather than handling the
memory allocation failure cleanly. This cropped up in bug 816791 (where
mono is running under qemu; qemu is prone to returning failure from
mmap()).

This is the function GC_install_header from libgc/headers.c:

/* Install a header for block h. */
/* The header is uninitialized. */
/* Returns the header or 0 on failure. */
struct hblkhdr * GC_install_header(h)
register struct hblk * h;
{
    hdr * result;
    
    if (!get_index((word) h)) return(0);
    result = alloc_hdr();
    SET_HDR(h, result);
# ifdef USE_MUNMAP
result -> hb_last_reclaimed = GC_gc_no;
# endif
    return(result);
}

The comment correctly claims we can return 0 on failure, because
alloc_hdr() might return 0. However if USE_MUNMAP is defined then before
we return 0 we will try to dereference it, and crash instead.

Found with mono 2.6.7-5ubuntu3, but this bug appears to still be present in upstream:
https://github.com/mono/mono/blob/master/libgc/headers.c#L201

** Affects: mono (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
CLI/Mono Uploaders, which is subscribed to mono in Ubuntu.
https://bugs.launchpad.net/bugs/816945

Title:
  mono: GC_install_header() in libgc/headers.c will segfault if mmap()
  fails

Status in “mono” package in Ubuntu:
  New

Bug description:
  If mmap() fails mono can crash with a segfault rather than handling
  the memory allocation failure cleanly. This cropped up in bug 816791
  (where mono is running under qemu; qemu is prone to returning failure
  from mmap()).

  This is the function GC_install_header from libgc/headers.c:

  /* Install a header for block h. */
  /* The header is uninitialized. */
  /* Returns the header or 0 on failure. */
  struct hblkhdr * GC_install_header(h)
  register struct hblk * h;
  {
      hdr * result;
      
      if (!get_index((word) h)) return(0);
      result = alloc_hdr();
      SET_HDR(h, result);
  # ifdef USE_MUNMAP
  result -> hb_last_reclaimed = GC_gc_no;
  # endif
      return(result);
  }

  The comment correctly claims we can return 0 on failure, because
  alloc_hdr() might return 0. However if USE_MUNMAP is defined then
  before we return 0 we will try to dereference it, and crash instead.

  Found with mono 2.6.7-5ubuntu3, but this bug appears to still be present in upstream:
  https://github.com/mono/mono/blob/master/libgc/headers.c#L201

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mono/+bug/816945/+subscriptions