<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Jan 30, 2017 at 5:13 PM, Reed O'Brien <span dir="ltr"><<a href="mailto:reed.obrien@canonical.com" target="_blank">reed.obrien@canonical.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span id="gmail-m_-4366630366306112198gmail-docs-internal-guid-85ba55ef-f208-cb13-8bce-c3c9709065b5">KVM as a juju deploy target now works on AMD64 and ARM64. There is an issue on ARM64 using trusty, so it only works with xenial on the hardware I could test on atm. There are a number of issues that prevent running juju on trusty on ARM64. Following is a support matrix, an outline of known issues, and scenarios where we might run into the issue of trusty not working as expected. Finally there are a couple thoughts about handling the issue. Any input on options/solutions for this issue are appreciated.</span><div><span><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:ubuntu;background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">ARM64 QEMU/KVM UEFI Support Matrix</span></p></span></div></div></blockquote><div><br></div><div>I've put the support matrix here: <a href="http://pastebin.ubuntu.com/23907437/">http://pastebin.ubuntu.com/23907437/</a> since the html table was stripped from the list archive. </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><span><ul style="margin-top:0pt;margin-bottom:0pt">- EFI guests require qemu-efi, which is only available in xenial.</ul><ul style="margin-top:0pt;margin-bottom:0pt">- Trusty’s 3.13 kernel can’t boot on GICv3 systems, nor as a guest on GICv3 systems.</ul><ul style="margin-top:0pt;margin-bottom:0pt">- Trusty’s virt stack does not support GICv3 guests, but that support is available in the cloud archive.</ul><div><br></div></span></div><div><span>The first issue is that there is a very limited set of hardware that was certified for trusty on ARM64. So that severely limits the number of systems where we would expect juju to be able to deploy to trusty. As noted in the next issue is possible using the hwe-x kernel, but that is not something juju can control.<br><br>The second issue is that those systems include GICv2 (General Interrupt Controller v2), whereas newer systems include GICv3. This would require trusty to boot with hwe-x kernel. AFAIU there are no official cloud images with that kernel pre-installed. Additionally, it currently seems it isn't possible to set hwe-x as the minimum kernel version in MAAS: <a href="https://bugs.launchpad.net/maas/+bug/1659694" target="_blank">https://bugs.launchpad.net/<wbr>maas/+bug/1659694</a><br><br>Newer versions of qemu/libvirt support setting the GIC version to "host" which allows the VM to use whatever the host supports as long as the guest supports. Trusty guests will not boot on a GICv3 host. Also "host" as a GIC element’s version attribute isn't supported by the virt stack that ships with trusty.</span></div><div><span><br>One final issue is that there is no qemu-efi package built for trusty. It is feasible that someone could build their own ppa for use. Other options include:</span></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>- adding qemu-efi to trusty-backports which is not easy to use,</div><div>- push a stable release update back to trusty, which is a regression risk for x86 by updating the source that also builds ovmf,</div><div>- introduce qemu-efi in trusty which would require an exception from the security team. </div></blockquote><div><span><br></span></div><div><span>There are a few scenarios which involve deploying trusty.<br></span></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><span>- juju bootstrap a-cloud arm64-controller --default-series trusty</span></div><div><span> This would a require a trusty cloud image with hwe-x kernel pre-installed.</span></div><div><span>- juju add-machine --series trusty</span></div><div><span> This would require the underlying provider to boot with hwe-x kernel</span></div><div><span> minimum. This also seems like it is going to fail to deploy with the</span></div><div><span> wily kernel.</span></div><div>- juju deploy wordpress --to kvm:0</div><div><span> This would require the machine-0 to be able to install qemu-efi and it</span></div><div><span> would also require a trusty cloud image with hwe-x kernel pre-installed.</span></div></blockquote><div><span><br><br>So given all the possible knobs one would need to turn to make it work, should we:<br><br></span></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><span>- block it from working in juju and return an error noting that trusty on ARM64 isn’t supported?</span></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><span>This in conjunction with the deploy scenario in the last section seems to be a real issue. One could decide to deploy only xenial hosts and guests on ARM64, but it would hobble the usefulness of quite a few charms that require trusty or bundles that require such charms. This seems draconian when selecting a particular minimum kernel in MAAS could make it work -- that is if qemu-efi was available and there weren’t hardware version issues.</span></div></blockquote><div><span>- warn that it won't work without a custom setup and document it as a known issue pointing in to a wiki page in the warning/error that contains the information above?</span></div><div><span>- something else/other ideas?</span></div></blockquote><div><div><br></div><div>Cheers,</div><div>Reed</div><span class="gmail-HOEnZb"><font color="#888888"><div><br></div></font></span></div></div></blockquote></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr">Reed O'Brien <div><div><span style="font-size:12.8px">✉ </span><a href="mailto:reed.obrien@canonical.com" target="_blank">reed.obrien@canonical.com</a></div><div style="font-size:12.8px">✆ <span title="Call with Google Voice">415-562-6797</span></div></div><div style="font-size:12.8px"><span title="Call with Google Voice">💻 redir</span></div></div></div></div></div></div> </div></div>