[Bug 1236433] [NEW] maas-cluster-celery did not survive upgrade
Scott Moser
smoser at ubuntu.com
Mon Oct 7 15:50:44 UTC 2013
Public bug reported:
This morning I upgraded maas.
Then I went to deploy a node, and the node did not power on.
I found that 'maas-cluster-celery' was 'stop/waiting'.
I am able to 'sudo start maas-cluster-celery' now and it is functioning, but I do not expect for apt-upgrade to break the system.
Further investigation shows the following in /var/log/maas/celery.log
[2013-10-07 13:35:49,020: INFO/Beat] Scheduler: Sending due task provisioningserver.tasks.upload_dhcp_leases
[2013-10-07 13:35:49,022: INFO/MainProcess] Got task from broker: provisioningserver.tasks.upload_dhcp_leases[76f1caad-3ba9-4cbe-8894-9ca96534a903]
[2013-10-07 13:35:49,169: INFO/MainProcess] Task provisioningserver.tasks.upload_dhcp_leases[76f1caad-3ba9-4cbe-8894-9ca96534a903] succeeded in 0.141157150269s: None
[2013-10-07 13:36:06,647: INFO/Beat] Scheduler: Sending due task provisioningserver.tasks.report_boot_images
[2013-10-07 13:36:06,649: INFO/MainProcess] Got task from broker: provisioningserver.tasks.report_boot_images[fa862682-3ffb-44bf-9733-268701d73e9a]
[2013-10-07 13:36:06,826: INFO/MainProcess] Task provisioningserver.tasks.report_boot_images[fa862682-3ffb-44bf-9733-268701d73e9a] succeeded in 0.128345966339s: None
[2013-10-07 13:36:10,644: WARNING/MainProcess] celeryd: Warm shutdown (MainProcess)
[2013-10-07 13:36:10,644: INFO/MainProcess] Celerybeat: Shutting down...
[2013-10-07 13:36:10,759: INFO/MainProcess] process shutting down
The time stamp there correlates to /var/log/apt/history.log
Start-Date: 2013-10-07 13:36:09
Commandline: apt-get dist-upgrade
Install: libsasl2-modules-db:amd64 (2.1.25.dfsg1-17, automatic), python-markdown:amd64 (2.3.1-1, automatic)
Upgrade: python-maas-provisioningserver:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), libboost-thread1.53.0:amd64 (1.53.0-6+exp3ubuntu6, 1.53.0-6+exp3ubuntu7), juju-core:amd64 (1.15.0-0~1903~ubuntu13.10.1, 1.15.1-0~1954~ubuntu13.10.1), maas-common:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), maas-region-controller:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), maas-dns:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), libboost-system1.53.0:amd64 (1.53.0-6+exp3ubuntu6, 1.53.0-6+exp3ubuntu7), charm-tools:amd64 (1.0.0-0ubuntu1, 1.0.0-0ubuntu2), maas-cli:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), libgmp10:amd64 (5.1.2+dfsg-2, 5.1.2+dfsg-2ubuntu1), upstart:amd64 (1.10-0ubuntu3, 1.10-0ubuntu6), maas-dhcp:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), maas-cluster-controller:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), libboost-iostreams1.53.0:amd64 (1.53.0-6+exp3ubuntu6, 1.53.0-6+exp3ubuntu7), python-django-maas:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), libsasl2-modules:amd64 (2.1.25.dfsg1-14, 2.1.25.dfsg1-17), libsasl2-2:amd64 (2.1.25.dfsg1-14, 2.1.25.dfsg1-17), maas:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), python-maas-client:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2)
End-Date: 2013-10-07 13:36:48
So I'm assuming this was shut down during upgrade and did not get restarted.
/var/log/upstart/maas-cluster-celery.log shows:
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/lib/python2.7/dist-packages/provisioningserver/__main__.py", line 39, in <module>
main()
File "/usr/lib/python2.7/dist-packages/provisioningserver/utils.py", line 431, in __call__
self.execute(argv)
File "/usr/lib/python2.7/dist-packages/provisioningserver/utils.py", line 426, in execute
args.handler.run(args)
File "/usr/lib/python2.7/dist-packages/provisioningserver/start_cluster_controller.py", line 178, in run
connection_details = register(args.server_url)
File "/usr/lib/python2.7/dist-packages/provisioningserver/start_cluster_controller.py", line 90, in register
interfaces=interfaces, uuid=cluster_uuid)
File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 216, in post
url, method="POST", headers=headers, data=body)
File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 99, in dispatch_query
res = urllib2.urlopen(req)
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 404, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 422, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1187, in do_open
r = h.getresponse(buffering=True)
File "/usr/lib/python2.7/httplib.py", line 1045, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 409, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 373, in _read_status
raise BadStatusLine(line)
httplib.BadStatusLine: ''
[2013-10-07 13:36:29,586: INFO/MainProcess] process shutting down
ProblemType: Bug
DistroRelease: Ubuntu 13.10
Package: maas-region-controller 1.4+bzr1656+dfsg-0ubuntu2
ProcVersionSignature: Ubuntu 3.11.0-11.17-generic 3.11.3
Uname: Linux 3.11.0-11-generic x86_64
ApportVersion: 2.12.5-0ubuntu1
Architecture: amd64
Date: Mon Oct 7 16:39:36 2013
MAASLog:
MAASPservLog:
2013-10-07 13:36:15+0100 [-] Received SIGTERM, shutting down.
2013-10-07 13:36:15+0100 [-] Received SIGTERM, shutting down.
MarkForUpload: True
PackageArchitecture: all
RelatedPackageVersions:
python-django-maas 1.4+bzr1656+dfsg-0ubuntu2
apparmor 2.8.0-0ubuntu29
SourcePackage: maas
UpgradeStatus: Upgraded to saucy on 2013-05-22 (138 days ago)
modified.conffile..etc.maas.maas.local.celeryconfig.py: [inaccessible: [Errno 13] Permission denied: '/etc/maas/maas_local_celeryconfig.py']
modified.conffile..etc.maas.maas.local.settings.py: [inaccessible: [Errno 13] Permission denied: '/etc/maas/maas_local_settings.py']
mtime.conffile..etc.maas.preseeds.preseed.xinstall: 2013-06-01T06:28:12.745041
mtime.conffile..etc.maas.txlongpoll.yaml: 2013-10-07T13:36:38.915836
** Affects: maas (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug saucy
** Description changed:
This morning I upgraded maas.
Then I went to deploy a node, and the node did not power on.
I found that 'maas-cluster-celery' was 'stop/waiting'.
+ I am able to 'sudo start maas-cluster-celery' now and it is functioning, but I do not expect for apt-upgrade to break the system.
Further investigation shows the following in /var/log/maas/celery.log
[2013-10-07 13:35:49,020: INFO/Beat] Scheduler: Sending due task provisioningserver.tasks.upload_dhcp_leases
[2013-10-07 13:35:49,022: INFO/MainProcess] Got task from broker: provisioningserver.tasks.upload_dhcp_leases[76f1caad-3ba9-4cbe-8894-9ca96534a903]
[2013-10-07 13:35:49,169: INFO/MainProcess] Task provisioningserver.tasks.upload_dhcp_leases[76f1caad-3ba9-4cbe-8894-9ca96534a903] succeeded in 0.141157150269s: None
[2013-10-07 13:36:06,647: INFO/Beat] Scheduler: Sending due task provisioningserver.tasks.report_boot_images
[2013-10-07 13:36:06,649: INFO/MainProcess] Got task from broker: provisioningserver.tasks.report_boot_images[fa862682-3ffb-44bf-9733-268701d73e9a]
[2013-10-07 13:36:06,826: INFO/MainProcess] Task provisioningserver.tasks.report_boot_images[fa862682-3ffb-44bf-9733-268701d73e9a] succeeded in 0.128345966339s: None
[2013-10-07 13:36:10,644: WARNING/MainProcess] celeryd: Warm shutdown (MainProcess)
[2013-10-07 13:36:10,644: INFO/MainProcess] Celerybeat: Shutting down...
[2013-10-07 13:36:10,759: INFO/MainProcess] process shutting down
The time stamp there correlates to /var/log/apt/history.log
Start-Date: 2013-10-07 13:36:09
Commandline: apt-get dist-upgrade
Install: libsasl2-modules-db:amd64 (2.1.25.dfsg1-17, automatic), python-markdown:amd64 (2.3.1-1, automatic)
Upgrade: python-maas-provisioningserver:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), libboost-thread1.53.0:amd64 (1.53.0-6+exp3ubuntu6, 1.53.0-6+exp3ubuntu7), juju-core:amd64 (1.15.0-0~1903~ubuntu13.10.1, 1.15.1-0~1954~ubuntu13.10.1), maas-common:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), maas-region-controller:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), maas-dns:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), libboost-system1.53.0:amd64 (1.53.0-6+exp3ubuntu6, 1.53.0-6+exp3ubuntu7), charm-tools:amd64 (1.0.0-0ubuntu1, 1.0.0-0ubuntu2), maas-cli:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), libgmp10:amd64 (5.1.2+dfsg-2, 5.1.2+dfsg-2ubuntu1), upstart:amd64 (1.10-0ubuntu3, 1.10-0ubuntu6), maas-dhcp:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), maas-cluster-controller:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), libboost-iostreams1.53.0:amd64 (1.53.0-6+exp3ubuntu6, 1.53.0-6+exp3ubuntu7), python-django-maas:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), libsasl2-modules:amd64 (2.1.25.dfsg1-14, 2.1.25.dfsg1-17), libsasl2-2:amd64 (2.1.25.dfsg1-14, 2.1.25.dfsg1-17), maas:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2), python-maas-client:amd64 (1.4+bzr1656+dfsg-0ubuntu1, 1.4+bzr1656+dfsg-0ubuntu2)
End-Date: 2013-10-07 13:36:48
-
So I'm assuming this was shut down during upgrade and did not get restarted.
/var/log/upstart/maas-cluster-celery.log shows:
Traceback (most recent call last):
- File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
- "__main__", fname, loader, pkg_name)
- File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
- exec code in run_globals
- File "/usr/lib/python2.7/dist-packages/provisioningserver/__main__.py", line 39, in <module>
- main()
- File "/usr/lib/python2.7/dist-packages/provisioningserver/utils.py", line 431, in __call__
- self.execute(argv)
- File "/usr/lib/python2.7/dist-packages/provisioningserver/utils.py", line 426, in execute
- args.handler.run(args)
- File "/usr/lib/python2.7/dist-packages/provisioningserver/start_cluster_controller.py", line 178, in run
- connection_details = register(args.server_url)
- File "/usr/lib/python2.7/dist-packages/provisioningserver/start_cluster_controller.py", line 90, in register
- interfaces=interfaces, uuid=cluster_uuid)
- File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 216, in post
- url, method="POST", headers=headers, data=body)
- File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 99, in dispatch_query
- res = urllib2.urlopen(req)
- File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
- return _opener.open(url, data, timeout)
- File "/usr/lib/python2.7/urllib2.py", line 404, in open
- response = self._open(req, data)
- File "/usr/lib/python2.7/urllib2.py", line 422, in _open
- '_open', req)
- File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
- result = func(*args)
- File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
- return self.do_open(httplib.HTTPConnection, req)
- File "/usr/lib/python2.7/urllib2.py", line 1187, in do_open
- r = h.getresponse(buffering=True)
- File "/usr/lib/python2.7/httplib.py", line 1045, in getresponse
- response.begin()
- File "/usr/lib/python2.7/httplib.py", line 409, in begin
- version, status, reason = self._read_status()
- File "/usr/lib/python2.7/httplib.py", line 373, in _read_status
- raise BadStatusLine(line)
+ File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
+ "__main__", fname, loader, pkg_name)
+ File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
+ exec code in run_globals
+ File "/usr/lib/python2.7/dist-packages/provisioningserver/__main__.py", line 39, in <module>
+ main()
+ File "/usr/lib/python2.7/dist-packages/provisioningserver/utils.py", line 431, in __call__
+ self.execute(argv)
+ File "/usr/lib/python2.7/dist-packages/provisioningserver/utils.py", line 426, in execute
+ args.handler.run(args)
+ File "/usr/lib/python2.7/dist-packages/provisioningserver/start_cluster_controller.py", line 178, in run
+ connection_details = register(args.server_url)
+ File "/usr/lib/python2.7/dist-packages/provisioningserver/start_cluster_controller.py", line 90, in register
+ interfaces=interfaces, uuid=cluster_uuid)
+ File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 216, in post
+ url, method="POST", headers=headers, data=body)
+ File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 99, in dispatch_query
+ res = urllib2.urlopen(req)
+ File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
+ return _opener.open(url, data, timeout)
+ File "/usr/lib/python2.7/urllib2.py", line 404, in open
+ response = self._open(req, data)
+ File "/usr/lib/python2.7/urllib2.py", line 422, in _open
+ '_open', req)
+ File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
+ result = func(*args)
+ File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
+ return self.do_open(httplib.HTTPConnection, req)
+ File "/usr/lib/python2.7/urllib2.py", line 1187, in do_open
+ r = h.getresponse(buffering=True)
+ File "/usr/lib/python2.7/httplib.py", line 1045, in getresponse
+ response.begin()
+ File "/usr/lib/python2.7/httplib.py", line 409, in begin
+ version, status, reason = self._read_status()
+ File "/usr/lib/python2.7/httplib.py", line 373, in _read_status
+ raise BadStatusLine(line)
httplib.BadStatusLine: ''
[2013-10-07 13:36:29,586: INFO/MainProcess] process shutting down
ProblemType: Bug
DistroRelease: Ubuntu 13.10
Package: maas-region-controller 1.4+bzr1656+dfsg-0ubuntu2
ProcVersionSignature: Ubuntu 3.11.0-11.17-generic 3.11.3
Uname: Linux 3.11.0-11-generic x86_64
ApportVersion: 2.12.5-0ubuntu1
Architecture: amd64
Date: Mon Oct 7 16:39:36 2013
MAASLog:
-
+
MAASPservLog:
- 2013-10-07 13:36:15+0100 [-] Received SIGTERM, shutting down.
- 2013-10-07 13:36:15+0100 [-] Received SIGTERM, shutting down.
+ 2013-10-07 13:36:15+0100 [-] Received SIGTERM, shutting down.
+ 2013-10-07 13:36:15+0100 [-] Received SIGTERM, shutting down.
MarkForUpload: True
PackageArchitecture: all
RelatedPackageVersions:
- python-django-maas 1.4+bzr1656+dfsg-0ubuntu2
- apparmor 2.8.0-0ubuntu29
+ python-django-maas 1.4+bzr1656+dfsg-0ubuntu2
+ apparmor 2.8.0-0ubuntu29
SourcePackage: maas
UpgradeStatus: Upgraded to saucy on 2013-05-22 (138 days ago)
modified.conffile..etc.maas.maas.local.celeryconfig.py: [inaccessible: [Errno 13] Permission denied: '/etc/maas/maas_local_celeryconfig.py']
modified.conffile..etc.maas.maas.local.settings.py: [inaccessible: [Errno 13] Permission denied: '/etc/maas/maas_local_settings.py']
mtime.conffile..etc.maas.preseeds.preseed.xinstall: 2013-06-01T06:28:12.745041
mtime.conffile..etc.maas.txlongpoll.yaml: 2013-10-07T13:36:38.915836
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to maas in Ubuntu.
https://bugs.launchpad.net/bugs/1236433
Title:
maas-cluster-celery did not survive upgrade
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1236433/+subscriptions
More information about the Ubuntu-server-bugs
mailing list