<div dir="ltr">Hi Andrea,<div><br></div><div>Thanks for the feedback! please let me explain this in more detail</div><div><br></div><div>I sent out this SRU because some of our customers need to use NO_HZ_FULL.</div><div>In their scenario, the CPU time is pretty sensitive, and must avoid as much noise as possible to keep </div><div>most CPUs doing only one single task.</div><div>I found some CPU steal time happen on their VM because RCU callback softirqs are still triggered on CPUs</div><div>with isolcpus, nohz_full and rcu_nocbs configured, and lead me to find NO_HZ_FULL is not built into generic kernel.</div><div><div><br></div><div>I think the use case for NO_HZ_FULL should be different than generic kernel, e.g. performance vs low latency<br></div><div><br></div><div>On generic kernel, we also did some tests with NO_HZ_FULL built-in here:</div><div><a href="https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1919154">https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1919154</a><br></div><div><br></div><div>The results show performance degradation on AMD EPYC series machines only </div><div>when NO_HZ_FULL is built-in but not enabled, but this doesn't happen on Intel and arm64 machines.</div></div><div><br></div><div>Because one of the customer is still on Focal, if it's possible to merge this into 5.15 kernel, it would be more convenient</div><div>for them to use HWE lowlatency kernel.</div><div>And if there are users need to use NO_HZ_FULL on Focal, lowlatency kernel could be a proper choice for them,</div><div>because so far there is no kernel with NO_HZ_FULL built-in on Focal.</div><div><br></div><div>Thanks,</div><div>Gerald</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 31, 2023 at 2:28 PM Andrea Righi <<a href="mailto:andrea.righi@canonical.com">andrea.righi@canonical.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Tue, Oct 31, 2023 at 12:57:39AM +0800, Gerald Yang wrote:<br>
> BugLink: <a href="https://bugs.launchpad.net/bugs/2023007" rel="noreferrer" target="_blank">https://bugs.launchpad.net/bugs/2023007</a><br>
> <br>
> [Impact]<br>
> <br>
> The lowlatency kernel in Ubuntu is specifically designed to prioritize<br>
> high responsiveness, making it ideal for multimedia environments like<br>
> DAWs and audio processing platforms, as well as soft real-time<br>
> environments.<br>
> <br>
> With the introduction of a real-time kernel, it might be worth<br>
> reconsidering the role of the lowlatency kernel and potentially<br>
> including it as the default kernel in desktop images, focusing on its<br>
> suitability for desktop-oriented usage.<br>
> <br>
> To achieve this, we can enable additional configuration settings and<br>
> make it more focused for a low-latency and highly responsive desktop<br>
> environment.<br>
> <br>
> Optionally (for the future) provide also an additional user-space<br>
> package that would enable specific run-time kernel settings focused at<br>
> certain preset workload profiles (e.g, web navigation, gaming, audio<br>
> processing, etc.).<br>
> <br>
> [Test case]<br>
> <br>
> Use linux-lowlatency in a desktop environment and measure responsiveness<br>
> of interactive applications.<br>
> <br>
> [Fix]<br>
> <br>
> Enable the following additional .config settings to make this kernel<br>
> more suitable for a low-latency desktop kernel:<br>
> <br>
> - CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown<br>
> clock tick when possible across all the enabled CPUs if they are<br>
> either idle or running 1 task - reduce kernel jitter of running tasks<br>
> due to the periodic clock tick, must be enabled at boot time passing<br>
> `nohz_full=<cpu_list>`)<br>
> <br>
> - CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y: move RCU<br>
> callbacks from softirq context to kthread context (reduce time spent<br>
> in softirqs with preemption disabled to improve the overall system<br>
> responsiveness, at the cost of introducing a potential performance<br>
> penalty, because RCU callbacks are not processed by kernel threads)<br>
> <br>
> [Regression potential]<br>
> <br>
> Enabling all these settings can introduce a potential performance<br>
> regression, but the kernel should result more responsive and make<br>
> it more suitable for a desktop/multimedia/gaming/audio processing<br>
> context.<br>
> <br>
> Signed-off-by: Gerald Yang <<a href="mailto:gerald.yang@canonical.com" target="_blank">gerald.yang@canonical.com</a>><br>
<br>
It's worth mentioning that we already applied these configs to the<br>
lowlatency kernels in mantic and in general I don't see any downside if<br>
we apply this.<br>
<br>
However, I'm not sure if a change like this is proper SRU material, we<br>
are basically saying "now lowlatency is a desktop kernel everywhere".<br>
<br>
Certain workloads may experience regressions (especially with the<br>
RCU_NOCB_CPU), while NO_HZ_FULL is a lot safer (since it can be<br>
enabled/disabled via boot options).<br>
<br>
We also need to keep in mind that Jammy will get these changes with the<br>
lowlatency-hwe-6.5 kernel, so I'm not still 100% convinced to backport<br>
this and apply it everywhere... opinions?<br>
<br>
-Andrea<br>
<br>
> ---<br>
> debian.lowlatency/config/annotations | 13 +++++++++++++<br>
> 1 file changed, 13 insertions(+)<br>
> <br>
> diff --git a/debian.lowlatency/config/annotations b/debian.lowlatency/config/annotations<br>
> index 694ae26b6745..3e1556d60809 100644<br>
> --- a/debian.lowlatency/config/annotations<br>
> +++ b/debian.lowlatency/config/annotations<br>
> @@ -15,12 +15,20 @@ CONFIG_HZ_250 note<'Override default HZ used i<br>
> CONFIG_LATENCYTOP policy<{'amd64': 'y', 'arm64': 'y'}><br>
> CONFIG_LATENCYTOP note<'<a href="https://lists.ubuntu.com/archives/kernel-team/2014-July/045006.html" rel="noreferrer" target="_blank">https://lists.ubuntu.com/archives/kernel-team/2014-July/045006.html</a>, LP#1655986'><br>
> <br>
> +CONFIG_NO_HZ_FULL policy<{'amd64': 'y', 'arm64': 'y'}><br>
> +CONFIG_NO_HZ_FULL note<'Enable access to "Full tickless mode" (LP: #2023007)'><br>
> +<br>
> +CONFIG_NO_HZ_IDLE policy<{'amd64': 'n', 'arm64': 'n'}><br>
> +CONFIG_NO_HZ_IDLE note<'Disabled in favor of CONFIG_NO_HZ_FULL (LP: #2023007)'><br>
> +<br>
> CONFIG_PREEMPT policy<{'amd64': 'y', 'arm64': 'y'}><br>
> CONFIG_PREEMPT note<'Enable fully preemptible kernel'><br>
> <br>
> CONFIG_PREEMPT_VOLUNTARY policy<{'amd64': 'n', 'arm64': 'n'}><br>
> CONFIG_PREEMPT_VOLUNTARY note<'Disable voluntary preemption model'><br>
> <br>
> +CONFIG_RCU_NOCB_CPU policy<{'amd64': 'y', 'arm64': 'y'}><br>
> +CONFIG_RCU_NOCB_CPU note<'Move RCU callbacks from softirq context to kthread context (LP: #2023007)'><br>
> <br>
> # ---- Annotations without notes ----<br>
> <br>
> @@ -55,6 +63,8 @@ CONFIG_CEC_PIN policy<{'amd64': 'y', 'arm64': '<br>
> CONFIG_CEC_PIN_ERROR_INJ policy<{'amd64': 'n', 'arm64': 'n'}><br>
> CONFIG_COMEDI_TESTS_EXAMPLE policy<{'amd64': 'n', 'arm64': 'm'}><br>
> CONFIG_COMEDI_TESTS_NI_ROUTES policy<{'amd64': 'n', 'arm64': 'm'}><br>
> +CONFIG_CONTEXT_TRACKING policy<{'amd64': 'y', 'arm64': 'y'}><br>
> +CONFIG_CONTEXT_TRACKING_FORCE policy<{'amd64': 'n', 'arm64': 'n'}><br>
> CONFIG_DEBUG_PREEMPT policy<{'amd64': 'n', 'arm64': 'n'}><br>
> CONFIG_HZ policy<{'amd64': '1000', 'arm64': '1000'}><br>
> CONFIG_INLINE_READ_LOCK policy<{'arm64': '-'}><br>
> @@ -89,4 +99,7 @@ CONFIG_PREEMPT_RCU policy<{'amd64': 'y', 'arm64': '<br>
> CONFIG_PREEMPT_TRACER policy<{'amd64': 'n', 'arm64': 'n'}><br>
> CONFIG_TASKS_RCU policy<{'amd64': 'y', 'arm64': 'y'}><br>
> CONFIG_TEST_DIV64 policy<{'amd64': 'm', 'arm64': 'm'}><br>
> +CONFIG_TICK_CPU_ACCOUNTING policy<{'amd64': '-', 'arm64': '-'}><br>
> CONFIG_UNINLINE_SPIN_UNLOCK policy<{'amd64': 'y', 'arm64': 'y'}><br>
> +CONFIG_VIRT_CPU_ACCOUNTING policy<{'amd64': 'y', 'arm64': 'y'}><br>
> +CONFIG_VIRT_CPU_ACCOUNTING_GEN policy<{'amd64': 'y', 'arm64': 'y'}><br>
> -- <br>
> 2.25.1<br>
> <br>
> <br>
> -- <br>
> kernel-team mailing list<br>
> <a href="mailto:kernel-team@lists.ubuntu.com" target="_blank">kernel-team@lists.ubuntu.com</a><br>
> <a href="https://lists.ubuntu.com/mailman/listinfo/kernel-team" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailman/listinfo/kernel-team</a><br>
</blockquote></div>