<div dir="ltr">Hi Dimitri,<div> </div><div>Thanks for checking this!</div><div>And yes, that would be very helpful to enable it in focal lowlatency-hwe and jammy lowlatency-hwe kernels,</div><div>at least users/customers could have a way to use this feature.</div><div> </div><div>Another question I'd like to ask is roughly when will 6.5 lowlatency-hwe kernel be landed in Jammy? Should I also submit this to 6.2 lowlatency kernel?</div><div> </div><div>Thanks,</div><div>Gerald</div><div> </div></div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 31, 2023 at 8:36 PM Dimitri John Ledkov <<a href="mailto:dimitri.ledkov@canonical.com">dimitri.ledkov@canonical.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi, On Tue, 31 Oct 2023 at 10:54, Gerald Yang <<a href="mailto:gerald.yang@canonical.com" target="_blank">gerald.yang@canonical.com</a>> wrote: > > Hi Andrea, > > Thanks for the feedback! please let me explain this in more detail > > I sent out this SRU because some of our customers need to use NO_HZ_FULL. > In their scenario, the CPU time is pretty sensitive, and must avoid as much noise as possible to keep > most CPUs doing only one single task. > I found some CPU steal time happen on their VM because RCU callback softirqs are still triggered on CPUs > with isolcpus, nohz_full and rcu_nocbs configured, and lead me to find NO_HZ_FULL is not built into generic kernel. > > I think the use case for NO_HZ_FULL should be different than generic kernel, e.g. performance vs low latency > > On generic kernel, we also did some tests with NO_HZ_FULL built-in here: > <a href="https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1919154" rel="noreferrer" target="_blank">https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1919154</a> > > The results show performance degradation on AMD EPYC series machines only > when NO_HZ_FULL is built-in but not enabled, but this doesn't happen on Intel and arm64 machines. > > Because one of the customer is still on Focal, if it's possible to merge this into 5.15 kernel, it would be more convenient > for them to use HWE lowlatency kernel. > And if there are users need to use NO_HZ_FULL on Focal, lowlatency kernel could be a proper choice for them, > because so far there is no kernel with NO_HZ_FULL built-in on Focal. I am ok having this enabled in focal:lowlatency-hwe-5.15 and jammy:lowlatency-hwe-6.5 but not change anything in focal:lowlatency (5.4 / ga) jammy:lowlatency (5.15 / ga). Would that help you? As an exception to our policy. I really feel uncomfortable changing default GA flavours, but risk levels of making hwe kernels more like the future sounds safer. > > Thanks, > Gerald > > On Tue, Oct 31, 2023 at 2:28 PM Andrea Righi <<a href="mailto:andrea.righi@canonical.com" target="_blank">andrea.righi@canonical.com</a>> wrote: >> >> On Tue, Oct 31, 2023 at 12:57:39AM +0800, Gerald Yang wrote: >> > BugLink: <a href="https://bugs.launchpad.net/bugs/2023007" rel="noreferrer" target="_blank">https://bugs.launchpad.net/bugs/2023007</a> >> > >> > [Impact] >> > >> > The lowlatency kernel in Ubuntu is specifically designed to prioritize >> > high responsiveness, making it ideal for multimedia environments like >> > DAWs and audio processing platforms, as well as soft real-time >> > environments. >> > >> > With the introduction of a real-time kernel, it might be worth >> > reconsidering the role of the lowlatency kernel and potentially >> > including it as the default kernel in desktop images, focusing on its >> > suitability for desktop-oriented usage. >> > >> > To achieve this, we can enable additional configuration settings and >> > make it more focused for a low-latency and highly responsive desktop >> > environment. >> > >> > Optionally (for the future) provide also an additional user-space >> > package that would enable specific run-time kernel settings focused at >> > certain preset workload profiles (e.g, web navigation, gaming, audio >> > processing, etc.). >> > >> > [Test case] >> > >> > Use linux-lowlatency in a desktop environment and measure responsiveness >> > of interactive applications. >> > >> > [Fix] >> > >> > Enable the following additional .config settings to make this kernel >> > more suitable for a low-latency desktop kernel: >> > >> > - CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown >> > clock tick when possible across all the enabled CPUs if they are >> > either idle or running 1 task - reduce kernel jitter of running tasks >> > due to the periodic clock tick, must be enabled at boot time passing >> > `nohz_full=<cpu_list>`) >> > >> > - CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y: move RCU >> > callbacks from softirq context to kthread context (reduce time spent >> > in softirqs with preemption disabled to improve the overall system >> > responsiveness, at the cost of introducing a potential performance >> > penalty, because RCU callbacks are not processed by kernel threads) >> > >> > [Regression potential] >> > >> > Enabling all these settings can introduce a potential performance >> > regression, but the kernel should result more responsive and make >> > it more suitable for a desktop/multimedia/gaming/audio processing >> > context. >> > >> > Signed-off-by: Gerald Yang <<a href="mailto:gerald.yang@canonical.com" target="_blank">gerald.yang@canonical.com</a>> >> >> It's worth mentioning that we already applied these configs to the >> lowlatency kernels in mantic and in general I don't see any downside if >> we apply this. >> >> However, I'm not sure if a change like this is proper SRU material, we >> are basically saying "now lowlatency is a desktop kernel everywhere". >> >> Certain workloads may experience regressions (especially with the >> RCU_NOCB_CPU), while NO_HZ_FULL is a lot safer (since it can be >> enabled/disabled via boot options). >> >> We also need to keep in mind that Jammy will get these changes with the >> lowlatency-hwe-6.5 kernel, so I'm not still 100% convinced to backport >> this and apply it everywhere... opinions? >> >> -Andrea >> >> > --- >> > debian.lowlatency/config/annotations | 13 +++++++++++++ >> > 1 file changed, 13 insertions(+) >> > >> > diff --git a/debian.lowlatency/config/annotations b/debian.lowlatency/config/annotations >> > index 694ae26b6745..3e1556d60809 100644 >> > --- a/debian.lowlatency/config/annotations >> > +++ b/debian.lowlatency/config/annotations >> > @@ -15,12 +15,20 @@ CONFIG_HZ_250 note<'Override default HZ used i >> > CONFIG_LATENCYTOP policy<{'amd64': 'y', 'arm64': 'y'}> >> > CONFIG_LATENCYTOP note<'<a href="https://lists.ubuntu.com/archives/kernel-team/2014-July/045006.html" rel="noreferrer" target="_blank">https://lists.ubuntu.com/archives/kernel-team/2014-July/045006.html</a>, LP#1655986'> >> > >> > +CONFIG_NO_HZ_FULL policy<{'amd64': 'y', 'arm64': 'y'}> >> > +CONFIG_NO_HZ_FULL note<'Enable access to "Full tickless mode" (LP: #2023007)'> >> > + >> > +CONFIG_NO_HZ_IDLE policy<{'amd64': 'n', 'arm64': 'n'}> >> > +CONFIG_NO_HZ_IDLE note<'Disabled in favor of CONFIG_NO_HZ_FULL (LP: #2023007)'> >> > + >> > CONFIG_PREEMPT policy<{'amd64': 'y', 'arm64': 'y'}> >> > CONFIG_PREEMPT note<'Enable fully preemptible kernel'> >> > >> > CONFIG_PREEMPT_VOLUNTARY policy<{'amd64': 'n', 'arm64': 'n'}> >> > CONFIG_PREEMPT_VOLUNTARY note<'Disable voluntary preemption model'> >> > >> > +CONFIG_RCU_NOCB_CPU policy<{'amd64': 'y', 'arm64': 'y'}> >> > +CONFIG_RCU_NOCB_CPU note<'Move RCU callbacks from softirq context to kthread context (LP: #2023007)'> >> > >> > # ---- Annotations without notes ---- >> > >> > @@ -55,6 +63,8 @@ CONFIG_CEC_PIN policy<{'amd64': 'y', 'arm64': ' >> > CONFIG_CEC_PIN_ERROR_INJ policy<{'amd64': 'n', 'arm64': 'n'}> >> > CONFIG_COMEDI_TESTS_EXAMPLE policy<{'amd64': 'n', 'arm64': 'm'}> >> > CONFIG_COMEDI_TESTS_NI_ROUTES policy<{'amd64': 'n', 'arm64': 'm'}> >> > +CONFIG_CONTEXT_TRACKING policy<{'amd64': 'y', 'arm64': 'y'}> >> > +CONFIG_CONTEXT_TRACKING_FORCE policy<{'amd64': 'n', 'arm64': 'n'}> >> > CONFIG_DEBUG_PREEMPT policy<{'amd64': 'n', 'arm64': 'n'}> >> > CONFIG_HZ policy<{'amd64': '1000', 'arm64': '1000'}> >> > CONFIG_INLINE_READ_LOCK policy<{'arm64': '-'}> >> > @@ -89,4 +99,7 @@ CONFIG_PREEMPT_RCU policy<{'amd64': 'y', 'arm64': ' >> > CONFIG_PREEMPT_TRACER policy<{'amd64': 'n', 'arm64': 'n'}> >> > CONFIG_TASKS_RCU policy<{'amd64': 'y', 'arm64': 'y'}> >> > CONFIG_TEST_DIV64 policy<{'amd64': 'm', 'arm64': 'm'}> >> > +CONFIG_TICK_CPU_ACCOUNTING policy<{'amd64': '-', 'arm64': '-'}> >> > CONFIG_UNINLINE_SPIN_UNLOCK policy<{'amd64': 'y', 'arm64': 'y'}> >> > +CONFIG_VIRT_CPU_ACCOUNTING policy<{'amd64': 'y', 'arm64': 'y'}> >> > +CONFIG_VIRT_CPU_ACCOUNTING_GEN policy<{'amd64': 'y', 'arm64': 'y'}> >> > -- >> > 2.25.1 >> > >> > >> > -- >> > kernel-team mailing list >> > <a href="mailto:kernel-team@lists.ubuntu.com" target="_blank">kernel-team@lists.ubuntu.com</a> >> > <a href="https://lists.ubuntu.com/mailman/listinfo/kernel-team" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailman/listinfo/kernel-team</a> > > -- > kernel-team mailing list > <a href="mailto:kernel-team@lists.ubuntu.com" target="_blank">kernel-team@lists.ubuntu.com</a> > <a href="https://lists.ubuntu.com/mailman/listinfo/kernel-team" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailman/listinfo/kernel-team</a> -- okurrr, Dimitri </blockquote></div>