[Bug 1347322] Re: juju ssh results in a panic: runtime error
Dave Cheney
1347322 at bugs.launchpad.net
Sat Jul 26 03:15:11 UTC 2014
btw. This has nothing to do with ssh, it is to do with the length of
time the juju cli process is running for. Normally this is a very
short period, but juju ssh forks ssh as a child then waits for it to
finish. The crash is caused by a bug in the previous runtime (compiled
statically into the binary) when the garbage collector had been
running for long enough to call madvise(DONT_NEED, ...) with the wrong
page size.
On Sat, Jul 26, 2014 at 8:22 AM, Matt Bruzek
<matthew.bruzek at canonical.com> wrote:
> With sinzui's help I have compiled the latest version of Juju 1.20.2
> with the gccgo-4.9_4.9.1-1ubuntu3_ppc64el.deb.
>
> NOT able to reproduce the “juju ssh” panic/problem having several ssh
> sessions open simultaneously and having output streaming for over 2
> hours.
>
> For clarity and review here are the details of how we build the deb
> packages for Juju with the gccgo 4.9.1 compiler on the CI machine
> stilson-09 (a P8 64k page size machine).
>
> ubuntu at stilson-09:~⟫ uname -a
> Linux stilson-09 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:29:18 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux
> ubuntu at stilson-09:~⟫ getconf PAGE_SIZE
> 65536
> ubuntu at stilson-09:~⟫ more /proc/cpuinfo
> processor : 0
> cpu : POWER8E (raw), altivec supported
> clock : 4116.000000MHz
> revision : 2.0 (pvr 004b 0200)
>
> processor : 1
> cpu : POWER8E (raw), altivec supported
> clock : 4116.000000MHz
> revision : 2.0 (pvr 004b 0200)
>
> timebase : 512000000
> platform : pSeries
> model : IBM pSeries (emulated by qemu)
> machine : CHRP IBM pSeries (emulated by qemu)
>
> Downloaded the following packages from:
> https://launchpad.net/ubuntu/utopic/ppc64el/gccgo-4.9/4.9.1-1ubuntu3
>
> binutils_2.24.51.20140709-1ubuntu1_ppc64el.deb
> cpp-4.9_4.9.1-1ubuntu3_ppc64el.deb
> gcc-4.9_4.9.1-1ubuntu3_ppc64el.deb
> gcc-4.9-base_4.9.1-1ubuntu3_ppc64el.deb
> gccgo-4.9_4.9.1-1ubuntu3_ppc64el.deb
> libatomic1_4.9.1-1ubuntu3_ppc64el.deb
> libgcc1_4.9.1-1ubuntu3_ppc64el.deb
> libgcc-4.9-dev_4.9.1-1ubuntu3_ppc64el.deb
> libgo5_4.9.1-1ubuntu3_ppc64el.deb
> libgomp1_4.9.1-1ubuntu3_ppc64el.deb
> libitm1_4.9.1-1ubuntu3_ppc64el.deb
>
> Installed the packages on stilson-09:
>
> ubuntu at stilson-09:~/gccgo⟫ sudo dpkg -i *.deb
>
> ubuntu at stilson-09:~/gccgo⟫ dpkg -l | grep gcc
> ii gcc 4:4.8.2-1ubuntu6 ppc64el GNU C compiler
> ii gcc-4.8 4.8.3-3ubuntu0.2 ppc64el GNU C compiler
> ii gcc-4.8-base:ppc64el 4.8.3-3ubuntu0.2 ppc64el GCC, the GNU Compiler Collection (base package)
> ii gcc-4.9 4.9.1-1ubuntu3 ppc64el GNU C compiler
> ii gcc-4.9-base:ppc64el 4.9.1-1ubuntu3 ppc64el GCC, the GNU Compiler Collection (base package)
> ii gccgo 4:4.9-1ubuntu6 ppc64el Go compiler, based on the GCC backend
> ii gccgo-4.9 4.9.1-1ubuntu3 ppc64el GNU Go compiler
> ii gccgo-go 1.2.1-0ubuntu1 ppc64el Go tool for use with gccgo
> ii libgcc-4.8-dev:ppc64el 4.8.3-3ubuntu0.2 ppc64el GCC support library (development files)
> ii libgcc-4.9-dev:ppc64el 4.9.1-1ubuntu3 ppc64el GCC support library (development files)
> ii libgcc1:ppc64el 1:4.9.1-1ubuntu3 ppc64el GCC support library
>
> Built Juju from source on stilson-09:
>
> <sinzui> ~/Work/juju-release-tools/make-source-packages.bash stable juju-core_1.20.2.tar.gz 'Curtis Hovey <curtis.hovey at canonical.com>' 1340893 1345014 1344940 1345832 1307434 1347371 1337091 1342747 1343301 1341589
> <sinzui> ~/Work/juju-release-tools/build-package-with-dsc.bash ubuntu at stilson-09 '' /home/curtis/Downloads/tmp.WUZeXOfFz1/juju-core_1.20.2-0ubuntu1~14.04.1~juju1.dsc
>
> Resulting in:
>
> ubuntu at stilson-09:~⟫ ls -l juju-build
> total 17332
> -rw-r--r-- 1 ubuntu ubuntu 6944 Jul 25 15:05 juju_1.20.2-0ubuntu1~14.04.1~juju1_all.deb
> drwxrwxr-x 7 ubuntu ubuntu 4096 Jul 25 15:04 juju-core-1.20.2
> -rw------- 1 ubuntu ubuntu 8356 Jul 25 15:04 juju-core_1.20.2-0ubuntu1~14.04.1~juju1.debian.tar.gz
> -rw------- 1 ubuntu ubuntu 1183 Jul 25 15:04 juju-core_1.20.2-0ubuntu1~14.04.1~juju1.dsc
> -rw-rw-r-- 1 ubuntu ubuntu 3133 Jul 25 15:05 juju-core_1.20.2-0ubuntu1~14.04.1~juju1_ppc64el.changes
> -rw-r--r-- 1 ubuntu ubuntu 14341816 Jul 25 15:05 juju-core_1.20.2-0ubuntu1~14.04.1~juju1_ppc64el.deb
> -rw-rw-r-- 1 ubuntu ubuntu 3354392 Jul 25 15:03 juju-core_1.20.2.orig.tar.gz
> -rw-r--r-- 1 ubuntu ubuntu 7020 Jul 25 15:05 juju-local_1.20.2-0ubuntu1~14.04.1~juju1_all.deb
> -rw-r--r-- 1 ubuntu ubuntu 7022 Jul 25 15:05 juju-local-kvm_1.20.2-0ubuntu1~14.04.1~juju1_all.deb
>
> I copied these deb packages to stilson-01:
> ubuntu at stilson-01:~$ ls -l juju-build
> total 14032
> -rw-r--r-- 1 ubuntu ubuntu 6944 Jul 25 15:21 juju_1.20.2-0ubuntu1~14.04.1~juju1_all.deb
> -rw-r--r-- 1 ubuntu ubuntu 14341816 Jul 25 15:22 juju-core_1.20.2-0ubuntu1~14.04.1~juju1_ppc64el.deb
> -rw-r--r-- 1 ubuntu ubuntu 7020 Jul 25 15:22 juju-local_1.20.2-0ubuntu1~14.04.1~juju1_all.deb
> -rw-r--r-- 1 ubuntu ubuntu 7022 Jul 25 15:22 juju-local-kvm_1.20.2-0ubuntu1~14.04.1~juju1_all.deb
>
> Installed the deb packages on stilson-01:
>
> sudo dpkg -i *.deb
>
> ubuntu at stilson-01:~$ dpkg -l | grep juju
> ii juju 1.20.2-0ubuntu1~14.04.1~juju1 all next generation service orchestration system
> ii juju-core 1.20.2-0ubuntu1~14.04.1~juju1 ppc64el Juju is devops distilled - client
> ii juju-deployer 0.3.6-0ubuntu2 all Deploy complex stacks of services using Juju
> ii juju-jitsu 0.20-1 all external tools to enhance juju
> ii juju-local 1.20.2-0ubuntu1~14.04.1~juju1 all dependency package for the Juju local provider
> ii juju-local-kvm 1.20.2-0ubuntu1~14.04.1~juju1 all dependency package for the Juju local provider
> ii juju-mongodb 2.4.9-0ubuntu3 ppc64el MongoDB object/document-oriented database for Juju
> ii juju-quickstart 1.4.1+bzr88+ppa25~ubuntu14.04.1 all Easy configuration of Juju environments
> ii python-jujuclient 0.17.5-0ubuntu2 all Python API client for juju
>
> Examined the strings in juju and jujud:
>
> ubuntu at stilson-01:~$ strings $(which juju) | grep gcc
> __gccgo_personality_v0
> libgcc_s.so.1
> -gccgoflags=-static-libgo
> gccgo
> ubuntu at stilson-01:~$ strings $(which jujud) | grep gcc
> libgcc_s.so.1
> gccgo
> -gccgoflags=-static-libgo
> gccgo
> xgcc (Ubuntu 4.9.1-1ubuntu3) 4.9.1
>
>
> Doing some testing on power that involves running the “juju test” command which bootstraps an environment deploys waits for success/fail and then destroys the environment. The test run will attempt to run 141 times and is on 9 iterations without a juju failure. I am confident this gccgo fix is good.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1347322
>
> Title:
> juju ssh results in a panic: runtime error
>
> Status in juju-core:
> Triaged
> Status in “juju-core” package in Ubuntu:
> Confirmed
>
> Bug description:
> I am using Juju on Power 8 hardware and I get a panic when I use juju
> to ssh to a system. The reproduction steps are
>
> juju init
> juju bootstrap -e local
> juju switch local
> juju deploy local:trusty/ubuntu
> juju ssh ubuntu/0
>
> (use the session to the local machine and after some time the panic
> happens and the screen is unreadable.)
>
> ubuntu at ubuntu-local-machine-1:~$ panic: runtime error: invalid memory address or nil pointer dereference
> [signal 0xb code=0x1 addr=0x8]
>
> goroutine 11 [running]:
> code.google.com_p_go.net_websocket.Send.N40_code.
> google.com_p_go.net_websocket.Codec
> /build/buildd/juju-core-1.18.1/src/code.google.com/p/go.net/webso
> cket/websocket.go:293
> launchpad.net_juju_core_rpc_jsoncodec.Send.N48_launchpad.net_juju_core_rpc_jsonc
> odec.wsJSONConn
> /build/buildd/juju-core-1.18.1/src/launchpad.net/juju-core/rpc/jsoncodec/conn.go:
> 21
> launchpad.net_juju_core_rpc_jsoncodec.WriteMessage.pN43_launchpad.net_juju_core_rpc_jsoncodec.Codec
> /build/buildd/juju-core-1.18.1/src/launchpad.net/juju-core/rpc/jsoncodec/codec.go:178
> launchpad.ne
> t_juju_core_rpc.send.pN32_launchpad.net_juju_core_rpc.Conn
> /build/buildd/juju-core-1.18.1/src/launch
> pad.net/juju-core/rpc/client.go:72
> launchpad.net_juju_core_rpc.Go.pN32_launchpad.net_juju_core_rpc.Con
> n
> /build/buildd/juju-core-1.18.1/src/launchpad.net/juju-core/rpc/client.go:174
> launchpad.net_juju_co
> re_rpc.Call.pN32_launchpad.net_juju_core_rpc.Conn
> /build/buildd/juju-core-1.18.1/src/launchpad.net/
> juju-core/rpc/client.go:148
> launchpad.net_juju_core_state_api.Call.pN39_launchpad.net_juju_core_state_
> api.State
> /build/buildd/juju-core-1.18.1/src/launchpad.net/juju-core/state/api/apiclient.go:168
> launchpad.net_juju_core_state_api.Ping.pN39_launchpad.net_juju_core_state_api.State
> /build/buildd/juj
> u-core-1.18.1/src/launchpad.net/juju-core/state/api/apiclient.go:158
> launchpad.net_juju_core_state_api
> .heartbeatMonitor.pN39_launchpad.net_juju_core_state_api.State
> /build/buildd/juju-core-1.18.1/sr
> c/launchpad.net/juju-core/state/api/apiclient.go:149
> created by launchpad.net_juju_core_state_api.Open
> /build/buildd/juju-core-1.18.1/src/launchpad.net/juju-core/state/api/apiclient.go:143
>
> goroutine 1
> [syscall]:
> goroutine in C code; stack unavailable
>
> goroutine 3 [syscall]:
> goroutine in C code; stac
> k unavailable
>
> goroutine 10 [IO wait]:
> code.google.com_p_go.net_websocket.ReadByte.N57_code.google.com_p
> _go.net_websocket.hybiFrameReaderFactory
> /build/buildd/juju-core-1.18.1/src/code.google.com/p/go.n
> et/websocket/hybi.go:113
> code.google.com_p_go.net_websocket.NewFrameReader.N57_code.google.com_p_go.ne
> t_websocket.hybiFrameReaderFactory
> /build/buildd/juju-core-1.18.1/src/code.google.com/p/go.net/webso
> cket/hybi.go:126
> code.google.com_p_go.net_websocket.Receive.N40_code.google.com_p_go.net_websocket.Cod
> ec
> /build/buildd/juju-core-1.18.1/src/code.google.com/p/go.net/websocket/websocket.go:314
> launchpad.n
> et_juju_core_rpc_jsoncodec.Receive.N48_launchpad.net_juju_core_rpc_jsoncodec.wsJSONConn
> /build/bu
> ildd/juju-core-1.18.1/src/launchpad.net/juju-core/rpc/jsoncodec/conn.go:25
> launchpad.net_juju_core_rpc
> _jsoncodec.ReadHeader.pN43_launchpad.net_juju_core_rpc_jsoncodec.Codec
> /build/buildd/juju-core-1
> .18.1/src/launchpad.net/juju-core/rpc/jsoncodec/codec.go:113
> launchpad.net_juju_core_rpc.loop.pN32_lau
> nchpad.net_juju_core_rpc.Conn
> /build/buildd/juju-core-1.18.1/src/launchpad.net/juju-core/rpc/se
> rver.go:344
> launchpad.net_juju_core_rpc.input.pN32_launchpad.net_juju_core_rpc.Conn
> /build/buildd/juj
> u-core-1.18.1/src/launchpad.net/juju-core/rpc/server.go:317
> created by launchpad.net_juju_core_rpc.Sta
> rt.pN32_launchpad.net_juju_core_rpc.Conn
> /build/buildd/juju-core-1.18.1/src/launchpad.net/juju-cor
> e/rpc/server.go:200
>
> Here are the specifics on the host system.
>
> ubuntu at stilson-01:~$ uname -a
> Linux stilson-01 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:50:31 UTC 2014 ppc64le ppc64le ppc64le GN
> U/Linux
> ubuntu at stilson-01:~$ dpkg -l | grep juju
> ii juju 1.18.1-0ubuntu1.1 all next generation
> service orchestration system
> ii juju-core 1.18.1-0ubuntu1.1 ppc64el Juju is devops
> distilled - client
> ii juju-deployer 0.3.6-0ubuntu2 all Deploy complex
> stacks of services using Juju
> ii juju-jitsu 0.20-1 all external tools
> to enhance juju
> ii juju-local 1.18.1-0ubuntu1.1 all dependency pack
> age for the Juju local provider
> ii juju-mongodb 2.4.9-0ubuntu3 ppc64el MongoDB object/
> document-oriented database for Juju
> ii juju-quickstart 1.4.1+bzr88+ppa25~ubuntu14.04.1 all Easy configurat
> ion of Juju environments
> ii python-jujuclient 0.17.5-0ubuntu2 all Python API clie
> nt for juju
> ubuntu at stilson-01:~$ getconf PAGE_SIZE
> 65536
>
> I got this error when testing a fix for another bug:
> https://bugs.launchpad.net/ubuntu/+source/gccgo-4.9/+bug/1304754
>
> Since the dmesg looks different we believe this is a new problem.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1347322/+subscriptions
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju-core in Ubuntu.
https://bugs.launchpad.net/bugs/1347322
Title:
juju ssh results in a panic: runtime error
To manage notifications about this bug go to:
https://bugs.launchpad.net/juju-core/+bug/1347322/+subscriptions
More information about the Ubuntu-server-bugs
mailing list