[Bug 1668093] Re: ssh-keygen -H corrupts already hashed entries

[ChristianEhrhardt]

Testing and shell into SRU for Xenial (others have the same, so just one is ok)
Former Version:
$ autopkgtest --apt-upgrade --shell --no-built-binaries openssh_7.2p2-4ubuntu2.1.dsc -- lxd ubuntu-daily:xenial/amd64
New Version:
$ autopkgtest --apt-upgrade --shell --setup-commands="add-apt-repository ppa:ci-train-ppa-service/2585; apt update; apt -y dist-upgrade" --no-built-binaries openssh_7.2p2-4ubuntu2.2.dsc -- lxd ubuntu-daily:xenial/amd64

Rerun full regression check
 $

Isolated execution of the Regression Test with:
 $ cd ${AUTOPKGTEST_TMP}/tree/regress
 $ env TEST_SHELL="sh" $(pwd)/test-exec.sh $(pwd) $(pwd)/keygen-knownhosts.sh

There are actually two tests that report "find simple in hashed".
The second one is the one failing:

rm -f $OBJ/kh.expect                                                             
expect_key host-h host-h host-f                                                  
check_hashed_find host-h "find simple in hashed" $OBJ/kh.hosts 

That creates an expected hosts file and compares vs the created one.
In the bad case the result has 3x the same line, expected is just one (the content is good, just x3).

Expected is "constructed" by the script and always the same.
The input here is kh.hosts which shall be filtered for host-h.
The "special" thing on host-h is that it has multiple entries host-f, host-g, host-h

good case:
cat kh.hosts
# host-f
|1|qOQ8g9WMxPbnPpDWt+uKrbIDg+I=|CRVR2cmi1FQ2yOQOJI92OsewK0Y= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHu6oWC3VvOmfMSUXHCnCalWeV+BwsdbFohYI4pbt/bk host-f
|1|iR+WMF7scwVxrq6gMEvU7FDT3rw=|nKJaAF8aL7QBcrdhr5ktbvNpphc= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHu6oWC3VvOmfMSUXHCnCalWeV+BwsdbFohYI4pbt/bk host-f
|1|I7Fz/FqixsITHRgQMQ1qh3Z0MOs=|35k5Pgsdi26LnX8qEa/x5AA1Ra0= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHu6oWC3VvOmfMSUXHCnCalWeV+BwsdbFohYI4pbt/bk host-f
$ ssh-keygen -f kh.hosts -H -F host-h
|1|I7Fz/FqixsITHRgQMQ1qh3Z0MOs=|35k5Pgsdi26LnX8qEa/x5AA1Ra0= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHu6oWC3VvOmfMSUXHCnCalWeV+BwsdbFohYI4pbt/bk host-f

bad case:
cat kh.hosts
# host-f
host-f,host-g,host-h ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJUaapZHmhvIshkUwydfXMBUucV8bDnJQzRr5w3KLvhC host-f
ssh-keygen -f kh.hosts -H -F host-h
|1|2F+CF0DKXdtLe0Hnb/4/uk82u7g=|oY3P3goocNH/nGO8I+851qQUBMQ= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJUaapZHmhvIshkUwydfXMBUucV8bDnJQzRr5w3KLvhC host-f
|1|ggIXIbTzzzTUU1boKmqEyIkzZEo=|7RZjQXl9O0GdwqwKJI7IVqnZq58= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJUaapZHmhvIshkUwydfXMBUucV8bDnJQzRr5w3KLvhC host-f
|1|mzIFYbXFU5rWYEAQVK1u+T4OaAM=|kXqon1yIIgUGSPQ5iJx4Qy97rHM= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJUaapZHmhvIshkUwydfXMBUucV8bDnJQzRr5w3KLvhC host-f

So in the old case it was already hashed when it got here in the test (that might be the failure actually).
Now since it is not hashed (yet) when it gets here the code generates 3x (one for each hostname) hashed entries.

Both agree on "kh.hosts.orig" content other than the actual generated
pubs.

Next step: need to debug if the error is either:
1. that kh.hosts is different when it gets to the test
or
2. the handling by ssh-keygen on it then.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/1668093

Title:
  ssh-keygen -H corrupts already hashed entries

Status in openssh package in Ubuntu:
  Fix Committed
Status in openssh source package in Xenial:
  Triaged
Status in openssh source package in Yakkety:
  Triaged
Status in openssh package in Debian:
  Fix Released

Bug description:
  [Impact]

   * re-execution of ssh-keygen -H can clobber known-hosts
   * Due to that users might get spurious re-warnings of known systems. For Automation it might be worse as it might stop to work when re-executed.

   * This is a regression from Trusty (working) to Xenial (fail) upgrade
  due to an upstream bug in the versions we merged.

   * This is a backport of the upstream fix

  [Test Case]

   * Pick a Host IP to scan keys from that you can reach and replies with SSH, then run the following trivial loop:
    $ ssh-keyscan ${IP} > ~/.ssh/known_hosts; for i in $(seq 1 20); do ssh-keygen -H; diff -Naur ~/.ssh/known_hosts.old ~/.ssh/known_hosts; done

   * Expected: no diff reported, since already hashed entries should be left as-is
   * Without fix: - diff in the hashes

  [Regression Potential]

   * The fix is upstream and soon in Debian as well, so we are not
  custom diverting here.

   * The risk should be minimal as this only changes ssh-keygen so
  despite openssh being really critical this doesn't affect ssh itself
  at all.

  [Other Info]
   
   * n/a

  
  ---

  xenial @ 1:7.2p2-4ubuntu2.1 on amd64 has this bug. trusty @
  1:6.6p1-2ubuntu2.8 on amd64 does not have this bug. I have not tested
  any other ssh versions.

  The following should reproduce the issue:

  #ssh-keyscan XXXX > ~/.ssh/known_hosts
  # ssh root at XXXXX
  Permission denied (publickey).
  # ssh-keygen -H
  /root/.ssh/known_hosts updated.
  Original contents retained as /root/.ssh/known_hosts.old
  WARNING: /root/.ssh/known_hosts.old contains unhashed entries
  Delete this file to ensure privacy of hostnames
  # ssh root at XXXXXX
  Permission denied (publickey).
  # ssh-keygen -H
  /root/.ssh/known_hosts updated.
  Original contents retained as /root/.ssh/known_hosts.old
  WARNING: /root/.ssh/known_hosts.old contains unhashed entries
  Delete this file to ensure privacy of hostnames
  # ssh root at XXXXX
  The authenticity of host 'XXXXXX' can't be established.
  RSA key fingerprint is XXXXXX.
  Are you sure you want to continue connecting (yes/no)?

  # diff known_hosts.old known_hosts
  1c1
  < |1|BoAbRpUE3F5AzyprJcbjdepeDh8=|x/1AcaLxh45FlShmVQnlgx2qjxY= XXXXX
  ---
  > |1|nTPsoLxCugQyZi3pqOa2pc/cX64=|bUH5qwZlZPp8msMGHdLtslf3Huk= XXXXX

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1668093/+subscriptions