[Bug 293705] Re: expired kerberos credentials cause significant syslog spam
Bug Watch Updater
293705 at bugs.launchpad.net
Tue Jul 12 07:39:14 UTC 2011
Launchpad has imported 7 comments from the remote bug at
https://bugzilla.novell.com/show_bug.cgi?id=620066.
If you reply to an imported comment from within Launchpad, your comment
will be sent to the remote bug automatically. Read more about
Launchpad's inter-bugtracker facilities at
https://help.launchpad.net/InterBugTracking.
------------------------------------------------------------------------
On 2010-07-06T12:05:34+00:00 Mika Fischer wrote:
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.6)
Gecko/20100628 Ubuntu/10.04 (lucid) Firefox/3.6.6
SSH by default deletes Kerberos credentials when a user logs out.
If the user left a program running (for instance via screen), and if
Kerberos credentials are needed to access the home directories
(kerberized NFS), rpc.gssd will fail to obtain Kerberos credentials.
The problem is that it generates excessive amounts of warnings in the
syslog to this effect (about 1100 wrnings per second), which then
quickly fill up the hard drive.
Reproducible: Always
Steps to Reproduce:
1. Log in (via SSH) to host that mounts home directory via kerberized NFS
2. Start screen with some process accessing the home dir inside
3. Detach screen
4. Close SSH session
5. Wait for rpc.gssd credentials cache to expire
Actual Results:
When the process still running on the target host tries to access the home directory, rpc.gssd will try and fail to obtain kerberos credentials for the user. It will then spam the syslog with the following warning
----
<date> <hostname> rpc.gssd[<pid>]: WARNING: Failed to create krb5 context for user with uid <uid> for server <other hostname>
----
This is repeated ad infinitum until the offending process is killed manually. The logfile otherwise quickly fills up the partition.
Expected Results:
Maybe one warning or no warning at all should be emitted (the latter is the case for *expired* credentials). See also https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/293705 for the case of expired credentials.
A fix fo this should probably also be coordinated with upstream.
Reply at: https://bugs.launchpad.net/ubuntu/+source/nfs-
utils/+bug/293705/comments/25
------------------------------------------------------------------------
On 2010-07-29T06:45:48+00:00 Sjayaraman wrote:
Thanks for the bug report. This issue has been already fixed in
upstream.
I have built an updated package including the fix and made it available here:
http://www.suse.com/~sjayaraman/test-pkgs/nfs-utils/
(syncing could take a few hours)
You would need to update your nfs-kernel-server package from the above
location. Please report back if this fixes the issue for you.
Reply at: https://bugs.launchpad.net/ubuntu/+source/nfs-
utils/+bug/293705/comments/26
------------------------------------------------------------------------
On 2010-07-30T06:02:42+00:00 Sjayaraman wrote:
Created an attachment (id=379436)
Upstream patch
Attaching the upstream patch that fixes the problem for completeness.
Reply at: https://bugs.launchpad.net/ubuntu/+source/nfs-
utils/+bug/293705/comments/27
------------------------------------------------------------------------
On 2010-08-03T15:45:37+00:00 Mika Fischer wrote:
After testing the packages I can confirm that they fix the problem for
us.
Do you recommend that we deploy them on all our 11.2 hosts or should we
wait for an official update?
Also, this probably should be fixed in 11.3. However there we have a
similar but slightly different behaviour. The error message does not
come from rpc.gssd but from the kernel itself. It is however caused by
the same circumstances and also spams the log so quickly that there's a
good chance of filling up the /var partition.
The error message in this case is (on the NFS client):
kernel: [1301515.320931] Error: state manager failed on NFSv4 server <NFS server hostname> with error 13
Error 13 probably means NFSERR_ACCES. Which probably means that the
process does not have permissions to access the file because the
Credentials Cache was removed when the user logged out.
Do you want me to open a separate bug report for this?
Reply at: https://bugs.launchpad.net/ubuntu/+source/nfs-
utils/+bug/293705/comments/28
------------------------------------------------------------------------
On 2010-08-03T15:57:28+00:00 Sjayaraman wrote:
(In reply to comment #4)
> After testing the packages I can confirm that they fix the problem for us.
Thanks for confirming.
> Do you recommend that we deploy them on all our 11.2 hosts or should we wait
> for an official update?
You should wait for an official update.
> Also, this probably should be fixed in 11.3. However there we have a similar
> but slightly different behaviour. The error message does not come from rpc.gssd
> but from the kernel itself. It is however caused by the same circumstances and
> also spams the log so quickly that there's a good chance of filling up the /var
> partition.
>
>
> Do you want me to open a separate bug report for this?
Yes, it sounds different from this one. Please open a separate bugzilla
for that issue.
Reply at: https://bugs.launchpad.net/ubuntu/+source/nfs-
utils/+bug/293705/comments/29
------------------------------------------------------------------------
On 2010-08-11T06:22:43+00:00 Nfbrown wrote:
I have submitted an update for 11.2 containing this patch, but I'm not
confident that an update will be released in any great hurry.
This bug is fixed in 11.3, It might be appropriate to upgrade to 11.3, or
just get the nfs-utils package from there.
The update request id is 45345
Reply at: https://bugs.launchpad.net/ubuntu/+source/nfs-
utils/+bug/293705/comments/30
------------------------------------------------------------------------
On 2011-07-04T07:51:14+00:00 Joschibrauchle wrote:
Hello everyone,
as Mika Fischer described, this bug exists also in OpenSUSE 11.3/11.4, except that the error message is:
--------
kernel: [<timestamp>] Error: state manager failed on NFSv4 server <IP_of_nfs_server> with error 13
--------
Otherwise, the description is still exactly valid:
- SSH login
- start user job, which accesses kerberized nfs user home
- SSH logout
- Kerberos cache expires
- /var/log/messages is spammed with ~1000 errors PER SECOND!
- /var partition out of space!
So, I could not find the corresponding bug report for 11.3/11.4. Is
there one yet?
Reply at: https://bugs.launchpad.net/ubuntu/+source/nfs-
utils/+bug/293705/comments/31
** Changed in: nfs-utils (openSUSE)
Status: Unknown => Fix Released
** Changed in: nfs-utils (openSUSE)
Importance: Unknown => Medium
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/293705
Title:
expired kerberos credentials cause significant syslog spam
Status in “nfs-utils” package in Ubuntu:
Fix Released
Status in “nfs-utils” source package in Hardy:
Fix Released
Status in “nfs-utils” source package in Intrepid:
Won't Fix
Status in “nfs-utils” package in openSUSE:
Fix Released
Bug description:
This bug can cause an installation's filesystem to fill up due to
excessive logging by rpc.gssd, when the user's Kerberos credentials
have expired and they have /home mounted via Kerberised NFS.
An explanation of how the bug has been addressed in the development
branch, including the relevant version numbers of packages modified in
order to implement the fix.
This bug has been fixed in the development branch, by way of a patch
(fixed in 1:1.1.2-4ubuntu2)
The patch is attached to this bug report.
TEST CASE: have a system that mounts /home via Kerberised NFS.
kdestroy. Wait ~15 minutes for rpc.gssd's cached credentials to
expire. Perform activities that attempt to access files in $HOME. Much
logging like what is below will occur.
This is a very simple patch. The potential for regression is slim, as
all this patch does is decrease the logging verbosity of a few
messages below that of normal operation.
Oct 8 17:05:18 swan rpc.gssd[4747]: WARNING: Failed to create krb5 context for user with uid 85153 for server *REDACTED*
Oct 8 17:05:18 swan rpc.gssd[4747]: ERROR: GSS-API: error in gss_acquire_cred(): Unspecified GSS failure. Minor code may provide more information - No credentials cache found
Oct 8 17:05:18 swan rpc.gssd[4747]: WARNING: Failed to create krb5 context for user with uid 85153 for server *REDACTED*
Oct 8 17:05:18 swan rpc.gssd[4747]: ERROR: GSS-API: error in gss_acquire_cred(): Unspecified GSS failure. Minor code may provide more information - No credentials cache found
Oct 8 17:05:18 swan rpc.gssd[4747]: WARNING: Failed to create krb5 context for user with uid 85153 for server *REDACTED*
Oct 8 17:05:18 swan rpc.gssd[4747]: ERROR: GSS-API: error in gss_acquire_cred(): Unspecified GSS failure. Minor code may provide more information - No credentials cache found
Oct 8 17:05:18 swan rpc.gssd[4747]: WARNING: Failed to create krb5 context for user with uid 85153 for server *REDACTED*
Oct 8 17:05:18 swan rpc.gssd[4747]: ERROR: GSS-API: error in gss_acquire_cred(): Unspecified GSS failure. Minor code may provide more information - No credentials cache found
Oct 8 17:05:18 swan rpc.gssd[4747]: WARNING: Failed to create krb5 context for user with uid 85153 for server *REDACTED*
Oct 8 17:05:18 swan rpc.gssd[4747]: ERROR: GSS-API: error in gss_acquire_cred(): Unspecified GSS failure. Minor code may provide more information - No credentials cache found
Oct 8 17:05:18 swan rpc.gssd[4747]: WARNING: Failed to create krb5 context for user with uid 85153 for server *REDACTED*
Oct 8 17:05:18 swan rpc.gssd[4747]: ERROR: GSS-API: error in gss_acquire_cred(): Unspecified GSS failure. Minor code may provide more information - No credentials cache found
Oct 8 17:05:18 swan rpc.gssd[4747]: WARNING: Failed to create krb5 context for user with uid 85153 for server *REDACTED*
Oct 8 17:05:18 swan rpc.gssd[4747]: ERROR: GSS-API: error in gss_acquire_cred(): Unspecified GSS failure. Minor code may provide more information - No credentials cache found
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/293705/+subscriptions
More information about the foundations-bugs
mailing list