Kernel Vulnerability Mitigation Issue
Dimitri John Ledkov
dimitri.ledkov at canonical.com
Mon May 17 17:22:35 UTC 2021
Hi,
Canonical official Ubuntu images ship with the latest microcode available
which we do attempt to update on all instances, for all instance types, in
all regions, in all current Ubuntu releases.
Many of the recent named vulnerabilities require microcode updates on the
host, hypervisor patches, host and guest kernel patches.
Ubuntu can only update the microcode only on some instance types - mostly
those that are .metal, and only when the microcode revisions that we ship
are newer than the ones installed on the host already.
You will observe different microcode levels across different regions, and
instance types. But also different ones from account to account, as regions
do not look the same to everyone.
Many of the kernel versions that you list below, are quite out of date. Are
you sure you always lookup latest published Ubuntu AMIs when launching new
instances? Also you mention generic flavour a lot, when normally we
recommend to only use aws kernel flavours (aws, aws-fips, aws-edge) as
these flavours have full hardware support for all AWS instance types,
hardware and features which is not guaranteed to be available on the
generic flavour.
For example if I
launch ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210511 -
ami-0f76aec439103b9fd on an m5.metal in us-east-2 region I can see that
uptodate kernel 5.4.0-1048-aws is in use, that latest microcode is applied,
and all the vulnerabilities are either not applicable or mitigated in the
lscpu output.
Note that in my case I've tested various .metal instances. Host OS, host
microcode, host hypervisor mitigations that are required to protect against
some of the vulnerabilities must be acted upon by Amazon themselves. They
should be updating microcode on the host nodes that your virtualized guests
run on. However, they may also do other things in the hypervisor to
mitigate vulnerabilities in other ways without microcode update (for
example by pinning cpu sockets or nodes to a single tenant). It is outside
of the guest OS control. It is not possible for the guest to know how
hypervisor mitigated things, apart from things looking kind of, potentially
not mitigated.
I see that you mention FIPS kernels, it must mean that you are a Canonical
Ubuntu Advantage customer. It is not quite appropriate for Amazon support
to redirect your query to a public mailing list of Ubuntu kernel team
working on development of next kernel versions. Instead you should have
been directed to the Canonical Customer support portal if you have access
for that. I will reach out to our support staff to see if we can reach out
via customer support channels, instead of this community support forum.
If you have access to Canonical Customer salesforce support portal please
raise a ticket with information of which AMIs you are running, in which
regions, and what your concerns are such that your account manager can
correctly route your query.
On Fri, May 14, 2021 at 7:45 AM Parker, James [USA] <Parker_James2 at bah.com>
wrote:
> Good afternoon Ubuntu Kernel Team!
>
>
>
> I’m James, a System Administrator with Booz Allen Hamilton, and we’ve been
> having some kernel vulnerability issues we think you will hopefully be able
> to assist us with getting to a resolution of some sort.
>
> After a lengthy back and forth discussion with the AWS Support Center,
> they pointed us to the Ubuntu Kernel Team, so I’m reaching out about some
> vulnerabilities in several of the Ubuntu OS kernels that we’re having
> trouble with getting the mitigations to take effect. We can discuss the
> majority of the details in follow-on conversations, but for starters here
> is the basic gist of what we’ve done so far in a chart that shows most of
> our tests with the results.
>
> The following AWS EC2 instance types, Ubuntu OS, and kernel versions have
> been tested in a virtual environment. These have all been scanned (as a
> privileged user) with Greenbone’s *OpenVAS* vulnerability scanner with
> the latest feeds, which resulted in the below CVE vulnerabilities missing
> mitigations. These same vulnerabilities also showed up in the ‘*lscpu*’
> command, (see *lscpu* output below).
>
>
>
> Despite our various attempts to add the recommended mitigations to the
> */etc/default/grub* file, running the *update-grub* command, and
> power-cycling the AWS EC2 Instances, we continue to see these kernel
> vulnerabilities showing up in the results of ‘*lscpu*’ and *OpenVAS*
> scans as *missing mitigations* and are looking for a way to apply the
> mitigations or update to a different kernel that is not vulnerable for
> these EC2 Instance typens (Primarily Intel Xeon CPUs):
>
>
>
>
>
> *OS Version*
>
> *Kernel Version*
>
> *AWS EC2 Instance Type*
>
>
> *Kernel Vulnerabilities (missing mitigations)*
>
> Ubuntu 18.04.5 LTS
>
> 4.15.0-2000-aws-fips
>
> t3.medium
>
> iTLB, MDS, SSB, TAA
>
> Ubuntu 20.04.2 LTS
>
> 4.15.0-2000-aws-fips
>
> t3.medium
>
> iTLB, MDS, SSB, TAA
>
> Ubuntu 20.04.2 LTS
>
> 5.12.0-051200-generic
>
> m5.xlarge
>
> Not Scanned
>
> Ubuntu 20.04.2 LTS
>
> 5.4.0-1022-aws
>
> t3.large
>
> SSB, iTLB, MDS
>
> Ubuntu 20.04.2 LTS
>
> 5.4.0-1045-aws
>
> t3.large
>
> SSB, iTLB, MDS
>
> Ubuntu 20.04.2 LTS
>
> 5.4.0-1047-aws
>
> t3.large
>
> SSB, iTLB, MDS
>
> Ubuntu 20.04.2 LTS
>
> 5.4.0-1047-aws
>
> t3.small
>
> SSB, iTLB, MDS
>
> Ubuntu 20.04.2 LTS
>
> 5.4.0-72-generic
>
> t3.large
>
> SSB, iTLB, MDS, TAA
>
> Ubuntu 20.04.2 LTS
>
> 5.8.0-50-generic
>
> m5.xlarge
>
> SSB, MDS
>
> Ubuntu 20.04.2 LTS
>
> 5.8.0-50-generic
>
> t3.large
>
> SSB, MDS, TAA
>
> Ubuntu 20.04.2 LTS
>
> 5.8.0-50-generic
>
> c5.xlarge
>
> SSB, MDS
>
> Ubuntu 20.04.2 LTS
>
> 5.8.0-50-generic
>
> inf1.xlarge
>
> SSB, MDS
>
> Ubuntu 20.04.2 LTS
>
> 5.8.0-50-generic
>
> t3.large
>
> SSB, MDS
>
> Ubuntu 20.04.2 LTS
>
> 5.8.0-50-generic
>
> t3.xlarge
>
> SSB, MDS, TAA
>
> Ubuntu 20.04.2 LTS
>
> 5.8.0-50-generic
>
> r5.xlarge
>
> SSB, MDS, TAA
>
> Ubuntu 20.04.2 LTS
>
> 5.8.0-50-generic
>
> c5n.4xlarge
>
> SSB, MDS, TAA
>
> Ubuntu 20.04.2 LTS
>
> 5.8.0-50-generic
>
> t3.large
>
> MDS, SSB
>
> Ubuntu 20.04.2 LTS
>
> 5.8.0-50-generic
>
> t3.small
>
> MDS, SSB, TAA
>
>
>
The above is missing AMI & region information. Without this it is hard to
identify why things look the way they do, or how the instances were
customized.
> *See the section on “Choice of Processors” at this link for more details
> about the various AWS EC2 Instance Types and which CPU it’s using: Amazon
> EC2
> <https://aws.amazon.com/ec2/?hp=tile&so-exp=below&ec2-whats-new.sort-by=item.additionalFields.postDateTime&ec2-whats-new.sort-order=desc>
>
>
>
> *LSCPU Output:*
> root at ubuntu:~# lscpu
>
> Architecture: x86_64
>
> CPU op-mode(s): 32-bit, 64-bit
>
> Byte Order: Little Endian
>
> Address sizes: 46 bits physical, 48 bits virtual
>
> CPU(s): 16
>
> On-line CPU(s) list: 0-15
>
> Thread(s) per core: 2
>
> Core(s) per socket: 8
>
> Socket(s): 1
>
> NUMA node(s): 1
>
> Vendor ID: GenuineIntel
>
> CPU family: 6
>
> Model: 85
>
> Model name: Intel(R) Xeon(R) Platinum 8259CL CPU @
> 2.50GHz
>
> Stepping: 7
>
> CPU MHz: 3100.427
>
> BogoMIPS: 4999.97
>
> Hypervisor vendor: KVM
>
> Virtualization type: full
>
> L1d cache: 256 KiB
>
> L1i cache: 256 KiB
>
> L2 cache: 8 MiB
>
> L3 cache: 35.8 MiB
>
> NUMA node0 CPU(s): 0-15
>
> Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported
>
> Vulnerability L1tf: Mitigation; PTE Inversion
>
> *Vulnerability Mds: Vulnerable: Clear CPU buffers attempted,
> no microcode; SMT Host state unknown*
>
> Vulnerability Meltdown: Mitigation; PTI
>
> *Vulnerability Spec store bypass: Vulnerable*
>
> Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and
> __user pointer sanitization
>
> Vulnerability Spectre v2: Mitigation; Full generic retpoline, STIBP
> disabled, RSB filling
>
> Vulnerability Srbds: Not affected
>
> Vulnerability Tsx async abort: Not affected
>
> Flags: fpu vme de pse tsc msr pae mce cx8 apic
> sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss
>
> ht syscall nx pdpe1gb rdtscp lm
> constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc
>
> _known_freq pni pclmulqdq ssse3 fma cx16
> pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer ae
>
> s xsave avx f16c rdrand hypervisor
> lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust b
>
> mi1 avx2 smep bmi2 erms invpcid mpx
> avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512b
>
> w avx512vl xsaveopt xsavec xgetbv1 xsaves
> ida arat pku ospke
>
>
> *OpenVAS Results:*
>
>
>
>
>
> Is there a way to mitigate these vulnerabilities in the AWS EC2 instances?
> Or do you have any recommendations as far as configuration settings or a
> specific kernel we should be using, etc.?
>
>
>
> Please let us know if you have any questions regarding this issue.
>
>
>
> Thank you.
>
>
>
> v/r
>
>
>
> James Parker
>
> Lead Technologist
>
>
> *Booz | Allen | Hamilton **---------------------------------------------*
>
> Mobile: 865.607.5117
>
> Desk: 240.547.2981
>
>
> --
> kernel-team mailing list
> kernel-team at lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/kernel-team
>
--
Regards,
Dimitri.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20210517/d7b2cb38/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 80810 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20210517/d7b2cb38/attachment-0001.jpg>
More information about the kernel-team
mailing list