[Bug 1421377] Re: "MAAS failed to start" recurs if postgresql is stopped
Mike Pontillo
mike.pontillo at canonical.com
Thu Feb 12 20:40:24 UTC 2015
** Description changed:
This is using the MAAS 1.8 experimental branch.
A race condition is possible in which a persistent exception is thrown
when hitting http://<server>/MAAS, if maas-regiond starts before
postgresql. (that is, even if you start postgresql, the MAAS web
interface never recovers.)
==> Exception seen on the web page <==
MAAS failed to start
Traceback (most recent call last):
- File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap
- self.__bootstrap_inner()
- File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
- self.run()
- File "/usr/lib/python2.7/threading.py", line 763, in run
- self.__target(*self.__args, **self.__kwargs)
+ File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap
+ self.__bootstrap_inner()
+ File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
+ self.run()
+ File "/usr/lib/python2.7/threading.py", line 763, in run
+ self.__target(*self.__args, **self.__kwargs)
--- <exception caught here> ---
- File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
- result = context.call(ctx, function, *args, **kwargs)
- File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
- return self.currentContext().callWithContext(ctx, func, *args, **kw)
- File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
- return func(*args,**kw)
- File "/usr/lib/python2.7/dist-packages/maasserver/webapp.py", line 83, in prepareApplication
- start_up.start_up()
- File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 48, in start_up
- security.get_shared_secret()
- File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 113, in wrapper
- return func_in_reactor(*args, **kwargs).wait(timeout)
- File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 219, in wait
- result.raiseException()
- File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
- result = context.call(ctx, function, *args, **kwargs)
- File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
- return self.currentContext().callWithContext(ctx, func, *args, **kw)
- File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
- return func(*args,**kw)
- File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 148, in wrapper
- return func(*args, **kwargs)
- File "/usr/lib/python2.7/dist-packages/maasserver/utils/async.py", line 164, in call_within_transaction
- return func_outside_txn(*args, **kwargs)
- File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 216, in retrier
- return func(*args, **kwargs)
- File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 338, in inner
- with self:
- File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 237, in __enter__
- if not connection.get_autocommit():
- File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 324, in get_autocommit
- self.ensure_connection()
- File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 124, in ensure_connection
- self.connect()
- File "/usr/lib/python2.7/dist-packages/django/db/utils.py", line 99, in __exit__
- six.reraise(dj_exc_type, dj_exc_value, traceback)
- File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 124, in ensure_connection
- self.connect()
- File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 112, in connect
- self.connection = self.get_new_connection(conn_params)
- File "/usr/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 116, in get_new_connection
- return Database.connect(**conn_params)
- File "/usr/lib/python2.7/dist-packages/psycopg2/__init__.py", line 179, in connect
- connection_factory=connection_factory, async=async)
+ File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
+ result = context.call(ctx, function, *args, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
+ return self.currentContext().callWithContext(ctx, func, *args, **kw)
+ File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
+ return func(*args,**kw)
+ File "/usr/lib/python2.7/dist-packages/maasserver/webapp.py", line 83, in prepareApplication
+ start_up.start_up()
+ File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 48, in start_up
+ security.get_shared_secret()
+ File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 113, in wrapper
+ return func_in_reactor(*args, **kwargs).wait(timeout)
+ File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 219, in wait
+ result.raiseException()
+ File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
+ result = context.call(ctx, function, *args, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
+ return self.currentContext().callWithContext(ctx, func, *args, **kw)
+ File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
+ return func(*args,**kw)
+ File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 148, in wrapper
+ return func(*args, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/maasserver/utils/async.py", line 164, in call_within_transaction
+ return func_outside_txn(*args, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 216, in retrier
+ return func(*args, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 338, in inner
+ with self:
+ File "/usr/lib/python2.7/dist-packages/django/db/transaction.py", line 237, in __enter__
+ if not connection.get_autocommit():
+ File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 324, in get_autocommit
+ self.ensure_connection()
+ File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 124, in ensure_connection
+ self.connect()
+ File "/usr/lib/python2.7/dist-packages/django/db/utils.py", line 99, in __exit__
+ six.reraise(dj_exc_type, dj_exc_value, traceback)
+ File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 124, in ensure_connection
+ self.connect()
+ File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 112, in connect
+ self.connection = self.get_new_connection(conn_params)
+ File "/usr/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 116, in get_new_connection
+ return Database.connect(**conn_params)
+ File "/usr/lib/python2.7/dist-packages/psycopg2/__init__.py", line 179, in connect
+ connection_factory=connection_factory, async=async)
django.db.utils.OperationalError: could not connect to server: Connection refused
- Is the server running on host "localhost" (127.0.0.1) and accepting
- TCP/IP connections on port 5432?
-
+ Is the server running on host "localhost" (127.0.0.1) and accepting
+ TCP/IP connections on port 5432?
==> Steps to Reproduce <==
$ sudo service postgresql stop
- * Stopping PostgreSQL 9.3 database server [ OK ]
+ * Stopping PostgreSQL 9.3 database server [ OK ]
$ sudo service maas-clusterd stop
maas-clusterd stop/waiting
$ sudo service maas-regiond stop
maas-regiond stop/waiting
$ sudo service maas-regiond start
maas-regiond start/running, process 10964
(then hit the MAAS web page)
(Note: this was seen upon system boot.)
==> Workaround <==
- $ sudo service maas-clusterd restart
+ $ sudo service maas-clusterd stop
+ $ sudo service maas-regiond restart
+ $ sudo service maas-clusterd start
+
+ (note: the workaround implies a second race condition which is a
+ separate issue; if maas-clusterd starts before maas-regiond, it will
+ fail with "Service Unavailable" when you hit the web interface)
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to maas in Ubuntu.
https://bugs.launchpad.net/bugs/1421377
Title:
"MAAS failed to start" recurs if postgresql is stopped
To manage notifications about this bug go to:
https://bugs.launchpad.net/maas/+bug/1421377/+subscriptions
More information about the Ubuntu-server-bugs
mailing list