[FWTS-Live] fwts MTRR tests fail on AMD platform if MMIO is above 4G
Alex Hung
alex.hung at canonical.com
Sat Oct 24 00:26:14 UTC 2020
On Fri, Oct 23, 2020 at 2:09 AM Tung, Larry <Larry.Tung at amd.com> wrote:
> [AMD Official Use Only - Internal Distribution Only]
>
>
>
> Hi Alex,
>
>
>
> I’ve submitted a bug below,
>
> https://bugs.launchpad.net/fwts/+bug/1901146
>
Hi Larry,
Thanks for sharing the information. Let's move our discussion to the
launchpad so it is easier to manage.
>
>
> =====================
>
> The effective memory type can be override by PAT mechanism and MTRRs.
>
>
>
> The PAT(MSR0000_0277) register contains 8 page-attribute fields from PA0
> to PA7. PA fields are selected by using three bits from the page-table
> entries {PAT, PCD, PWT}, and each fields have corresponding memory type in
> the PAT register.
>
>
>
> If the memory range is not specified using the fixed-range and
> variable-range MTRRs, the default memory type in MTRRdefType(MSR0000_02FF)
> would be used.
>
>
>
> The combination of MTRR and PAT memtype are described in AMD64
> Architecture Programmer's Manual Volume 2 (#24593)
>
> https://www.amd.com/system/files/TechDocs/24593.pdf
>
>
>
>
>
> Best Regards,
>
> Larry
>
>
>
> *From:* Alex Hung <alex.hung at canonical.com>
> *Sent:* Thursday, October 22, 2020 3:52 PM
> *To:* Tung, Larry <Larry.Tung at amd.com>
> *Cc:* Lien, Eliot <Eliot.Lien at amd.com>
> *Subject:* Re: [FWTS-Live] fwts MTRR tests fail on AMD platform if MMIO
> is above 4G
>
>
>
> [CAUTION: External Email]
>
>
>
>
>
> On Thu, Oct 22, 2020 at 1:33 AM Tung, Larry <Larry.Tung at amd.com> wrote:
>
> [AMD Official Use Only - Internal Distribution Only]
>
>
>
> Hi Alex,
>
>
>
> Thanks for the information.
>
>
>
> After clarifying, the TOM2-WB (Sys_Cfg[22]=1) would be override with PAT
> (MSR0000_0277) memtype. The page-attribute used to access the PCIe video
> controller MMIO should be WC.
>
>
>
> So, I wonder if it’s possible to add a “page-attribute” checking into the
> test tool ?
>
>
>
> Certainly. We really appreciate your feedback and technical details. I was
> able to find a public AMD datasheet
> "54945_PPR_Family_17h_Models_00h-0Fh.pdf", but I don't quite understand how
> it overrides TOM2-WB.
>
>
>
> Will you be able to share some details?
>
>
>
> FYI, you can also send emails to our mailing list "
> fwts-devel at lists.ubuntu.com" where more people can discuss if all
> information is publicly available.
>
>
>
>
>
>
>
>
>
> Thanks,
>
>
>
> Best Regards,
>
> Larry
>
>
>
> *From:* Alex Hung <alex.hung at canonical.com>
> *Sent:* Thursday, October 22, 2020 3:06 AM
> *To:* Tung, Larry <Larry.Tung at amd.com>
> *Cc:* Lien, Eliot <Eliot.Lien at amd.com>
> *Subject:* Re: [FWTS-Live] fwts MTRR tests fail on AMD platform if MMIO
> is above 4G
>
>
>
> [CAUTION: External Email]
>
> Hi,
>
>
>
> That MSR is used to check memory types such as below:
>
>
>
> /* Get the default memory type of memory between 4GB and second top of
> * memory (TOM2) - i.e. is it write back (WB)
> */
> if (strstr(fwts_cpuinfo->vendor_id, "AMD") ||
> strstr(fwts_cpuinfo->vendor_id, "Hygon")) {
> if (fwts_cpu_readmsr(fw, 0, AMD_SYS_CFG_MSR, &amd_sys_conf) == FWTS_OK)
> if (amd_sys_conf & 0x200000)
> amd_Tom2ForceMemTypeWB = true;
> }
>
>
>
> ....
>
>
>
>
>
> /* On AMD platforms, Tom2ForceMemTypeWB overwrites other memory types */
> if (amd_Tom2ForceMemTypeWB && start >= 0x100000000) {
> type = WRITE_BACK;
> return type;
> }
>
>
>
>
>
> However, memory-mapped IO (like this one) should not be write-back:
>
>
>
> "Write back is a storage method in which data is written into the cache
> every time a change occurs, but is written into the corresponding location
> in main memory only at specified intervals or under certain conditions.
>
>
>
> PCI memory-mapped I/O registers aren't RAM that stays the same until the
> CPU changes them - and MMIO can change by itself to reflect hardware
> changes unlike memory. As a result, it is unsuitable for the CPU to write
> to MMIO "only at specific intervals or under certain conditions".
>
>
>
> The scenario should be: you write a command to PCI MMIO, expect the PCI
> hardware to react as soon as writes are issued, and want to read an ack
> from it; however the PCI hardware will not do anything until many
> milliseconds later, long after your reads....
>
>
>
> There are similar discussion @
> https://community.intel.com/t5/Intel-ISA-Extensions/TSX-and-PCI-consistent-memory/td-p/1107558
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.intel.com%2Ft5%2FIntel-ISA-Extensions%2FTSX-and-PCI-consistent-memory%2Ftd-p%2F1107558&data=04%7C01%7CLarry.Tung%40amd.com%7C337e705aab604c47282708d8765f7261%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637389499601755643%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MYGzTQbZ9cAwTJ40oIoVDe90oZ1oRIZU73tyz7BiANk%3D&reserved=0>,
> and I quote
>
>
>
> =======================
>
> FYI: The WB type will not work with memory-mapped IO. You can program
> the bits to set up the mapping as WB, but the system will crash as soon as
> it gets a transaction that it does not know how to handle. It is
> theoretically possible to use WP or WT to get cached reads from MMIO, but
> coherence has to be handled in software.
>
> =======================
>
>
>
>
>
> On Wed, Oct 21, 2020 at 3:04 AM Tung, Larry <Larry.Tung at amd.com> wrote:
>
> [AMD Official Use Only - Internal Distribution Only]
>
>
>
> Hi Alex,
>
>
>
> On AMD system, there is a MSR to force the 4GB ~ TOM2 as WB memory type.
> However, we encounter MTRR test failed on PCIe GFx resource with incorrect
> WB memory type, if BAR is assigned to above 4G memory range. I looked into
> /mtrr.c form git hub, code logic seems would check if PCIe BAR is
> prefetchable, then it will return the “mustnot” mask as WB which may cause
> this failure.
>
>
>
> May I have your comment on this please? Thanks
>
>
>
> 1.
>
> /* if it's PCI mmio -> uncached typically except for video */
>
> if (strstr(string, "0000:")) {
>
> bool pref;
>
>
>
> if (check_prefetchable(fw, string, address,
> &pref) != FWTS_OK)
>
> return FWTS_ERROR;
>
>
>
> if (pref) {
>
> *must = 0;
>
> *mustnot = WRITE_BACK;
>
> } else {
>
> *must = UNCACHED;
>
> *mustnot = (~UNCACHED) &
> (~DEFAULT);
>
> }
>
> }
>
> 1.
>
> if ((type & type_mustnot) != 0) {
>
> failed++;
>
> fwts_failed(fw,
> LOG_LEVEL_CRITICAL,
>
>
> "MTRRIncorrectAttr",
>
> "Memory range
> 0x%" PRIx64 " to 0x%" PRIx64 " (%s) "
>
> "has incorrect
> attribute%s.",
>
> start, end,
>
> c2,
> cache_to_string(type & type_mustnot));
>
> if (type_must == UNCACHED)
>
> skiperror = true;
>
> }
>
>
>
> Error log with FWTS Version V20.07.00
>
> mtrr: MTRR tests.
>
>
> --------------------------------------------------------------------------------
>
> MTRR overview
>
> -------------
>
> Reg 0: 0x0000000000000000 - 0x000000007fffffff ( 2048 MB) Write-Back
>
> Reg 1: 0x0000000080000000 - 0x00000000bfffffff ( 1024 MB) Write-Back
>
>
>
> Test 1 of 3: Validate the kernel MTRR IOMEM setup.
>
> FAILED [CRITICAL] MTRRIncorrectAttr: Test 1, Memory range 0xfce0000000 to
>
> 0xfcefffffff (0000:08:00.0) has incorrect attribute Write-Back.
>
> FAILED [CRITICAL] MTRRIncorrectAttr: Test 1, Memory range 0xfcf0000000 to
>
> 0xfcf01fffff (0000:08:00.0) has incorrect attribute Write-Back.
>
>
>
> Test 2 of 3: Validate the MTRR setup across all processors.
>
> PASSED: Test 2, All processors have the a consistent MTRR setup.
>
>
>
> Test 3 of 3: Test for AMD MtrrFixDramModEn being cleared by the BIOS.
>
> PASSED: Test 3, No MtrrFixDramModEn error detected.
>
>
>
>
>
>
>
> Best Regards,
>
> Larry
>
>
>
>
>
> --
>
> Cheers,
> Alex Hung
>
>
>
> --
>
> Cheers,
> Alex Hung
>
--
Cheers,
Alex Hung
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/fwts-devel/attachments/20201023/3a250d67/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 88184 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/fwts-devel/attachments/20201023/3a250d67/attachment-0003.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 12314 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/fwts-devel/attachments/20201023/3a250d67/attachment-0004.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.jpg
Type: image/jpeg
Size: 38936 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/fwts-devel/attachments/20201023/3a250d67/attachment-0005.jpg>
More information about the fwts-devel
mailing list