[apparmor] [PATCH] [parser] Fix jobs not scaling up to meet available resources when cpus are brought online during compilation

[Seth Arnold]

On Tue, Apr 05, 2016 at 12:37:07PM -0700, John Johansen wrote:
> Enable dynamically scaling max jobs if new resources are brought online
> 
> BugLink: http://bugs.launchpad.net/bugs/1566490
> 
> This patch enables to parser to scale the max jobs if new resources are
> being brought online by the scheduler.
> 
> It only enables the scaling check if there is a difference between the
> maximum number of cpus (CONF) and the number of online (ONLN) cpus.
> 
> Instead of checking for more resources regardless, of whether the online
> cpu count is increasing it limits its checking to a maximum of
> MAX CPUS + 1 - ONLN cpus times. With each check coming after fork spawns a
> new work unit, giving the scheduler a chance to bring new cpus online
> before the next check.  The +1 ensures the checks will be done at least
> once after the scheduling task sleeps waiting for its children giving
> the scheduler an extra chance to bring cpus online.
> 
> Signed-off-by: John Johansen <john.johansen at canonical.com>

This feels more complicated than it could be but I must admit I can't
suggest any modifications to the algorithm to simplify it.

Acked-by: Seth Arnold <seth.arnold at canonical.com>

Thanks

> 
> === modified file 'parser/parser_main.c'
> --- a/parser/parser_main.c	2016-01-15 01:26:26 +0000
> +++ b/parser/parser_main.c	2016-04-05 07:27:06 +0000
> @@ -89,6 +89,12 @@
>  long jobs_max = -8;			/* 8 * cpus */
>  long jobs = JOBS_AUTO;			/* default: number of processor cores */
>  long njobs = 0;
> +long jobs_scale = 0;			/* number of chance to resample online
> +					 * cpus. This allows jobs spawning to
> +					 * scale when scheduling policy is
> +					 * taking cpus off line, and brings
> +					 * them back with load
> +					 */
>  bool debug_jobs = false;
>  
>  struct timespec cache_tstamp, mru_policy_tstamp;
> @@ -898,6 +904,16 @@
>  		RESULT(WORK);						\
>  		break;							\
>  	}*/								\
> +	if (jobs_scale) {						\
> +		long n = sysconf(_SC_NPROCESSORS_ONLN);			\
> +		if (n > jobs) {						\
> +			/* reset sample chances - potentially reduce to 0 */ \
> +			jobs_scale = jobs_max + 1 - n;			\
> +			jobs = n;					\
> +		} else							\
> +			/* reduce scaling chance by 1 */		\
> +			jobs_scale--;					\
> +	}								\
>  	if (njobs == jobs) {						\
>  		/* wait for a child */					\
>  		if (debug_jobs)						\
> @@ -959,11 +975,18 @@
>  {
>  	/* jobs and paralell_max set by default, config or args */
>  	long n = sysconf(_SC_NPROCESSORS_ONLN);
> +	long maxn = sysconf(_SC_NPROCESSORS_CONF);
>  	if (n == -1)
>  		/* unable to determine number of processors, default to 1 */
>  		n = 1;
> +	if (maxn == -1)
> +		/* unable to determine number of processors, default to 1 */
> +		maxn = 1;
> +	if (n < maxn)
> +		/* the bigger the difference the more sample chances given */
> +		jobs_scale = maxn + 1 - n;
>  	jobs = compute_jobs(n, jobs);
> -	jobs_max = compute_jobs(n, jobs_max);
> +	jobs_max = compute_jobs(maxn, jobs_max);
>  
>  	if (jobs > jobs_max) {
>  		pwarn("%s: Warning capping number of jobs to %ld * # of cpus == '%ld'",
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/apparmor/attachments/20160405/85e72512/attachment.pgp>