All ports blocked, ping works, firewall and apparmor off

[Tony Baechler - BATS]

I have tried your suggestions as far as I'm able and it would seem to be a
boot problem.  I think you're right that it's somehow stuck in single user
mode, but I don't know how to undo it.  Obviously networking isn't coming
up.  I'll answer your specific questions below.

On 2015-04-10 04:31 PM, Karl Auer wrote:
> On Fri, 2015-04-10 at 05:51 -0700, Tony Baechler - BATS wrote:
>> Everything was working fine, but I couldn't connect after the reboot.
>> [...]
>> I completely purged ufw and iptables but no luck.
> 
> If you cannot connect to it, how are you modifying the configuration?
> Just to double-check, this is a real system, not a virtual? Do you still
> have physical access to the system?

Yes, it's a real dedicated server.  I never had physical access.  Everything
is through the rescue system.  I'm in the US and it's in Germany, so it
would be impossible for me to physically access it.  If I could, I could
figure out if there is an error on the screen and that would probably clear
up the mystery.

> 
> You mentioned calling someone's support - whose?

No, I wrote to them.  They are Hetzner. <http://www.hetzner.com/> and they
told me that they give me full root access, so I'm on my own.

> 
>> What's really strange is that it boots fine with kvm from the rescue system.
>> I can get to the login prompt and everything seems to be fine.  It acts
>> like a boot problem, but I don't see why ping would work if it isn't
>> booting.
> 
> Add a crontab entry that runs every minute as root, collects some
> information (the output from runlevel, ps, dmesg, ifconfig, iptables,
> mount - whatever you can think of), and writes it into a known location
> (but NOT /tmp). Reboot, wait at least ten minutes, then go in with the
> rescue system and look at what's been written. If nothing's been written
> then yes, you have a boot problem.  I suggest you write a very simple
> one first and see if it works at all. That way you haven't wasted a lot
> of time if it doesn't. If it does work, go wild with version 2 :-)

I set a cron job to write the output of dmesg to a file every minute and the
file wasn't created.  I also added a similar line to /etc/rc.local and that
wasn't created either.  It would seem to be a boot problem.  Now that I
think about it, I did at one point try to go to single user mode to make a
full backup.  I wasn't able to connect, but rebooting via the web-based
robot seemed to fix the problem at the time.  Now that I'm again totally
locked out, it would seem that it went back to single user mode, thus my
question of how to get it to boot normally.  Again, KVm booted to a normal
login prompt with nothing about single user mode.


> 
> Perhaps the system is booting into single-user mode for some reason. You
> could try adding a job in /etc/init.d/rc1.d that collects info if level
> 1 is entered.

I tried rcS.d but nothing happened.  I'll look at rc1.d.

> 
> Also, check the default run level in /etc/init/rc-sysinit. It should be
> 2. 

Yes, it's 2 and I haven't changed anything in that directory.

> 
> Check the kernel command line.

I checked /etc/default/grub several times and /etc/grub.d and
/boot/grub/grub.cfg and they look fine.  The job to write /proc/cmdline to a
file doesn't seem to be working.
> 
> Check the BIOS boot order - this is a very long shot.

I can't since I have no physical access to the machine, but I don't think
that's it.  Then again, it almost seems like a BIOS issue, but why would
ping work?

> 
> Also, check /etc/resolv.conf. Make sure the nameservers are correctly
> entered and reachable from that system, otherwise all sorts of weird
> delays can happen, especially if things like Apache try to check their
> own address, or things like ssh try to check yours.

Yes, it checks out fine.  It's using the Hetzner nameservers.

> 
> Check the IP address you have configured. Make sure it is legal - not a
> broadcast or network address. Check the mask, check the gateway. Easy to
> get wrong. It seems unlikely if you can ping the address, but still -
> check it.

Yes, that's why I restored /etc/network/interfaces from a known good backup.
 It looks fine.

> 
> Check that the IP address of your server is not a duplicate. If some
> other system has your server's IP address, your server may not be able
> to bring up networking, but the other system with that address may well
> respond only to ping, either because it doesn't have services
> configured, or is firewalling you.

I don't think that's the case.  The rescue system on the same IP address
works fine and I doubt if Hetzner would assign duplicate addresses.
Apparently the rescue system is loaded from the Hetzner tftp server when I
activate it from the robot.

> 
> How are you trying to (for example) connect with ssh? Via the known IP
> address or via the name of the system? If via the name, try via the IP
> address.

I've tried both.  That's why I tried nmap from an outside system and it said
all 1,000 ports are closed.

> 
> If you connect by name, and the system has DNS entries for IPv4 and
> IPv6, and YOUR system has IPv6 enabled, the connection will be attempted
> via IPv6. Specifying the IP address rather than the name bypasses that
> mechanism. If connection via the address works and connection via the
> name doesn't suspect  DNS issue at your end, or an IPv6/IPv4 issue such
> as misconfigured IPv6.

Yes, I think IPV6 isn't configured correctly but would that block IPV4
connections?

> 
> Try connecting from a completely other machine in a completely other
> well-maintained network. Just to make sure it's not a problem at your
> end.

Yes, I tried from a different Hetzner server in a completely different
datacenter and a different IP address block.  I can connect to other servers
fine from my local box.  I have a UK server for example which connects fine.

> 
> How long have you waited for the system to come up? Some networking
> issues cause a delay of up to a minute or more.

Several hours.

> 
> Maybe try re-installing just the new kernel?

What do you mean?  I removed the old "generic" kernels completely and
installed "lowlatency" to eliminate a kernel problem.  The machine has 32 GB
of RAM.  I could switch back to the 3.13.0-48-lowlatency kernel as it's
currently booting 3.16.

> 
> Regards, K.
> 
> 

--