[Bug 459628] [NEW] erratic thread scheduling, kernel 2.6.31
Launchpad Bug Tracker
459628 at bugs.launchpad.net
Fri Aug 27 20:45:52 UTC 2010
You have been subscribed to a public bug:
There is apparently a problem in the latest kernel used in Ubuntu 9.10.
The scheduling of contending threads has become erratic in comparison
with Ubuntu 9.04.
Summary of Tests
test program: loop-bench, 4 independent looping threads
kernel CPU time variance throughput variance
2.6.28 1% 1%
2.6.31 26% 30%
test program: mutex-bench, 4 looping threads contending for one mutex
kernel CPU time variance throughput variance
2.6.28 9% 30%
2.6.31 66% 70%
I wrote two simple benchmark programs to explore what is happening.
The program loop-bench has 4 independent looping threads. Each thread
measures its own throughput (loop iterations). The main thread measures
the total CPU time for the entire process.
The program mutex-bench has 4 looping threads which contend for a single
mutex resource. Each thread measures its own throughput (loop
iterations). The loop consists of: lock the mutex, release the mutex,
increment the loop counter. The main thread measures the total CPU time
for the entire process.
The binaries of these programs were compiled on Ubuntu 9.04. The same
binaries were used for the Ubuntu 9.10 tests. This was done to rule out
compiler differences. Recompiling on 9.10 actually makes no difference.
Below is the output from each test. The programs run forever until
stopped.
After the test output please find both source programs.
===========================================================================
Ubuntu 9.04 kernel 2.6.28-15
./loop-bench
thread counts: 56513 56512 56520 56470 total: 226015 cpu: 4.00
thread counts: 57295 56940 56839 57274 total: 228348 cpu: 4.00
thread counts: 57297 57295 57245 57277 total: 229114 cpu: 4.00
thread counts: 57296 57293 57277 57246 total: 229112 cpu: 4.00
thread counts: 57295 56881 56837 57279 total: 228292 cpu: 4.00
thread counts: 57295 56818 56911 57290 total: 228314 cpu: 3.99
thread counts: 57275 57295 56435 56714 total: 227719 cpu: 4.00
thread counts: 57251 57275 56780 56674 total: 227980 cpu: 4.00
thread counts: 57006 56959 57294 57277 total: 228536 cpu: 4.00
thread counts: 57036 57027 57291 57244 total: 228598 cpu: 4.00
thread counts: 57274 57296 56814 56694 total: 228078 cpu: 4.00
thread counts: 56582 56728 57296 57278 total: 227884 cpu: 4.00
thread counts: 57292 57244 57297 57278 total: 229111 cpu: 4.00
thread counts: 57292 56847 56748 57297 total: 228184 cpu: 4.00
thread counts: 57273 56873 56757 57297 total: 228200 cpu: 4.00
thread counts: 57274 56723 57296 56701 total: 227994 cpu: 4.01
thread counts: 57272 56925 57296 57012 total: 228505 cpu: 4.00
thread counts: 57273 56865 56823 57189 total: 228150 cpu: 3.99
thread counts: 57272 57294 56775 56824 total: 228165 cpu: 3.99
thread counts: 57269 57295 56824 56837 total: 228225 cpu: 4.00
thread counts: 57001 56846 57296 57252 total: 228395 cpu: 4.00
thread counts: 57146 56851 57250 56869 total: 228116 cpu: 4.00
thread counts: 57295 57248 57143 57251 total: 228937 cpu: 4.00
thread counts: 57294 57267 56760 56692 total: 228013 cpu: 4.00
thread counts: 57163 57296 57297 57244 total: 229000 cpu: 4.01
thread counts: 56645 56595 57296 57297 total: 227833 cpu: 4.00
thread counts: 57276 57098 57277 57296 total: 228947 cpu: 4.00
thread counts: 57278 57238 57296 57297 total: 229109 cpu: 4.00
./mutex-bench
thread counts: 1386616 1351903 1372874 1350698 total: 5462091 cpu: 3.53
thread counts: 1386219 1395610 1396089 1422855 total: 5600773 cpu: 3.45
thread counts: 2010729 1532559 2143082 1335095 total: 7021465 cpu: 3.30 **
thread counts: 1375342 1342083 1326799 1365190 total: 5409414 cpu: 3.52 **
thread counts: 1487265 1438809 1438392 1471736 total: 5836202 cpu: 3.53
thread counts: 1378093 1391842 1347232 1313966 total: 5431133 cpu: 3.50
thread counts: 1357851 1369293 1401537 1336267 total: 5464948 cpu: 3.58
thread counts: 1437049 1439325 1427853 1377762 total: 5681989 cpu: 3.56
thread counts: 1467866 1496237 1442375 1443847 total: 5850325 cpu: 3.57
thread counts: 1433046 1480726 1456441 1453021 total: 5823234 cpu: 3.44
thread counts: 1367688 1371358 1406184 1342367 total: 5487597 cpu: 3.60
thread counts: 1450414 1475509 1438713 1407877 total: 5772513 cpu: 3.55
thread counts: 1462166 1476551 1439815 1436066 total: 5814598 cpu: 3.48
thread counts: 1360315 1378476 1338497 1339656 total: 5416944 cpu: 3.48
thread counts: 1335259 1362162 1352755 1370795 total: 5420971 cpu: 3.53
thread counts: 1372602 1390689 1354660 1365219 total: 5483170 cpu: 3.42
thread counts: 1387776 1357859 1341409 1376913 total: 5463957 cpu: 3.51
thread counts: 1411252 1357439 1409672 1359153 total: 5537516 cpu: 3.50
thread counts: 1362966 1379526 1351444 1326267 total: 5420203 cpu: 3.44
thread counts: 1398544 1386933 1333636 1337658 total: 5456771 cpu: 3.50
thread counts: 1415511 1389943 1397077 1449271 total: 5651802 cpu: 3.54
thread counts: 1357685 1362500 1419309 1327220 total: 5466714 cpu: 3.44
thread counts: 1376264 1348499 1378064 1416585 total: 5519412 cpu: 3.56
thread counts: 1396253 1396613 1347499 1385890 total: 5526255 cpu: 3.50
thread counts: 1484030 1419015 1459998 1422398 total: 5785441 cpu: 3.52
thread counts: 1407998 1433006 1472601 1284169 total: 5597774 cpu: 3.54
thread counts: 1420319 1461719 1462053 1449576 total: 5793667 cpu: 3.57
thread counts: 1400848 1409122 1340463 1370147 total: 5520580 cpu: 3.50
thread counts: 1404472 1407983 1409094 1351917 total: 5573466 cpu: 3.39
thread counts: 1359954 1355746 1378467 1398860 total: 5493027 cpu: 3.53
thread counts: 1438503 1473504 1443501 1465028 total: 5820536 cpu: 3.56
thread counts: 1393750 1444156 1424676 1449060 total: 5711642 cpu: 3.51
thread counts: 1416971 1475893 1422139 1449518 total: 5764521 cpu: 3.57
thread counts: 1430823 1470149 1418519 1448440 total: 5767931 cpu: 3.47
thread counts: 1432612 1473566 1464045 1475357 total: 5845580 cpu: 3.51
thread counts: 1324637 1372631 1354401 1402721 total: 5454390 cpu: 3.47
thread counts: 1472826 1354701 1384159 1347553 total: 5559239 cpu: 3.48
thread counts: 1454837 1475362 1433196 1439087 total: 5802482 cpu: 3.51
thread counts: 1355788 1372150 1396890 1332695 total: 5457523 cpu: 3.51
thread counts: 1405096 1367768 1427919 1375766 total: 5576549 cpu: 3.52
thread counts: 1337253 1382325 1398567 1338998 total: 5457143 cpu: 3.47
thread counts: 1336811 1431545 1378790 1361479 total: 5508625 cpu: 3.53
thread counts: 1418853 1470286 1376820 1397196 total: 5663155 cpu: 3.50
thread counts: 1402107 1409531 1329849 1348216 total: 5489703 cpu: 3.49
thread counts: 1352628 1380044 1354502 1382921 total: 5470095 cpu: 3.49
thread counts: 1353565 1380356 1347274 1353020 total: 5434215 cpu: 3.46
thread counts: 1472132 1447214 1455167 1473006 total: 5847519 cpu: 3.51
thread counts: 1323590 1330872 1402937 1385639 total: 5443038 cpu: 3.49
thread counts: 1347971 1380832 1345910 1399361 total: 5474074 cpu: 3.45
thread counts: 1317167 1348667 1370679 1398638 total: 5435151 cpu: 3.51
thread counts: 1469092 1339734 1351313 1354283 total: 5514422 cpu: 3.60
thread counts: 1494367 1358360 1388689 1329650 total: 5571066 cpu: 3.55
thread counts: 1423782 1279174 1287753 1265609 total: 5256318 cpu: 3.41
thread counts: 1423841 1473347 1477843 1415771 total: 5790802 cpu: 3.54
thread counts: 1374764 1386644 1363009 1374196 total: 5498613 cpu: 3.45
thread counts: 1306398 1365510 1387019 1366253 total: 5425180 cpu: 3.51
thread counts: 1506709 1366217 1389855 1357988 total: 5620769 cpu: 3.48
thread counts: 1490922 1491552 1445878 1418908 total: 5847260 cpu: 3.61
===========================================================================
Ubuntu 9.10 kernel 2.6.31-14
./loop-bench
thread counts: 56482 56482 56486 56458 total: 225908 cpu: 3.96
thread counts: 43890 43879 57333 57337 total: 202439 cpu: 3.94
thread counts: 57329 57334 43890 43880 total: 202433 cpu: 3.97
thread counts: 57328 57310 43891 43891 total: 202420 cpu: 3.96
thread counts: 57329 57335 53086 53080 total: 220830 cpu: 3.96
thread counts: 43897 43864 57336 57341 total: 202438 cpu: 3.95
thread counts: 57154 43761 43751 57337 total: 202003 cpu: 3.95
thread counts: 56676 56665 43385 43373 total: 200099 cpu: 3.93
thread counts: 57305 43919 43782 57340 total: 202346 cpu: 3.93
thread counts: 57329 43891 43881 57341 total: 202442 cpu: 3.94
thread counts: 56534 43281 43266 57339 total: 200420 cpu: 3.93
thread counts: 57067 57064 43689 43688 total: 201508 cpu: 3.94
thread counts: 43749 43633 57328 57343 total: 202053 cpu: 3.95
thread counts: 56438 43211 43189 57327 total: 200165 cpu: 3.95
thread counts: 56431 56423 43197 43184 total: 199235 cpu: 3.95
thread counts: 43894 57305 43865 57337 total: 202401 cpu: 3.94
thread counts: 43893 43878 57328 57343 total: 202442 cpu: 3.95
thread counts: 56437 43223 43141 57330 total: 200131 cpu: 3.95
thread counts: 56378 56401 43169 43164 total: 199112 cpu: 3.95
thread counts: 43892 43902 43853 43886 total: 175533 cpu: 3.94
thread counts: 43896 43868 57341 57339 total: 202444 cpu: 3.95
thread counts: 57338 43916 43774 57333 total: 202361 cpu: 3.94
thread counts: 56422 56432 43199 43169 total: 199222 cpu: 3.97
thread counts: 43891 57308 43891 57337 total: 202427 cpu: 3.94
thread counts: 57318 43892 43877 57343 total: 202430 cpu: 3.95
thread counts: 56654 56675 43376 43377 total: 200082 cpu: 4.96 **
thread counts: 43897 43893 43876 43885 total: 175551 cpu: 3.95
thread counts: 56628 43419 43075 57325 total: 200447 cpu: 3.94
thread counts: 56817 56816 43380 43506 total: 200519 cpu: 3.93
thread counts: 43855 43891 43892 43900 total: 175538 cpu: 3.94
thread counts: 57160 43447 43827 57335 total: 201769 cpu: 3.94
thread counts: 56430 56430 42856 43278 total: 198994 cpu: 3.94
thread counts: 43890 57291 43890 57341 total: 202412 cpu: 3.94
thread counts: 57335 43970 43577 57335 total: 202217 cpu: 3.93
thread counts: 57332 57341 57339 57291 total: 229303 cpu: 3.96
thread counts: 43982 43534 57339 57342 total: 202197 cpu: 3.93
thread counts: 56693 43482 43060 57333 total: 200568 cpu: 3.94
thread counts: 56676 56675 43460 43080 total: 199891 cpu: 3.94
thread counts: 43892 57292 43892 57341 total: 202417 cpu: 3.94
thread counts: 57332 52292 52004 57343 total: 218971 cpu: 3.95
thread counts: 56676 56677 43470 43055 total: 199878 cpu: 3.95
thread counts: 55905 57293 55809 57342 total: 226349 cpu: 3.94
thread counts: 43890 57290 43890 57339 total: 202409 cpu: 3.95
thread counts: 57339 57336 43970 43581 total: 202226 cpu: 3.95
thread counts: 43905 57293 43837 57341 total: 202376 cpu: 3.96
thread counts: 57335 43967 43583 57335 total: 202220 cpu: 3.97
thread counts: 56863 56863 43165 43610 total: 200501 cpu: 3.95
thread counts: 43892 57295 43892 57337 total: 202416 cpu: 3.95
thread counts: 57340 43971 43582 57332 total: 202225 cpu: 3.95
thread counts: 56648 56636 43374 43294 total: 199952 cpu: 3.96
thread counts: 57331 57309 43892 43892 total: 202424 cpu: 3.95
thread counts: 57330 57325 43892 43875 total: 202422 cpu: 4.96 **
thread counts: 43892 57309 43892 57332 total: 202425 cpu: 3.96
thread counts: 57324 43892 43875 57343 total: 202434 cpu: 3.95
thread counts: 56664 56663 43385 43369 total: 200081 cpu: 3.95
thread counts: 43893 43891 57332 57311 total: 202427 cpu: 3.94
thread counts: 57329 57327 43893 43880 total: 202429 cpu: 3.96
thread counts: 43892 43893 43891 43879 total: 175555 cpu: 3.97
thread counts: 43892 57329 57299 43887 total: 202407 cpu: 3.96
thread counts: 57336 43886 43883 57343 total: 202448 cpu: 3.96
thread counts: 56437 56432 43202 43200 total: 199271 cpu: 3.94
thread counts: 57330 57304 43891 43890 total: 202415 cpu: 3.94
thread counts: 57324 43886 43888 57333 total: 202431 cpu: 3.95
thread counts: 56948 43599 43583 57340 total: 201470 cpu: 3.94
thread counts: 56632 56634 43362 43346 total: 199974 cpu: 3.94
thread counts: 43893 57309 43888 57332 total: 202422 cpu: 3.94
./mutex-bench
thread counts: 1368376 1371722 1368929 1429373 total: 5538400 cpu: 3.53
thread counts: 1915284 1928430 3530136 2190232 total: 9564082 cpu: 2.51
thread counts: 3110392 3086282 2483567 4882237 total: 13562478 cpu: 2.10
thread counts: 2525415 4250569 2975315 2279842 total: 12031141 cpu: 2.27
thread counts: 2772195 2810243 3286730 3560976 total: 12430144 cpu: 2.14
thread counts: 1594847 1523772 1524091 1604299 total: 6247009 cpu: 2.86
thread counts: 4470288 3205561 2504720 2865932 total: 13046501 cpu: 2.00
thread counts: 2484798 2577016 2144983 3784977 total: 10991774 cpu: 2.30
thread counts: 3100625 3288606 5271904 2223420 total: 13884555 cpu: 1.99
thread counts: 1553419 1625698 1545399 1496500 total: 6221016 cpu: 2.87
thread counts: 2774464 3516284 3186190 2737983 total: 12214921 cpu: 2.02
thread counts: 2935428 3528820 2310856 2938276 total: 11713380 cpu: 2.42
thread counts: 2396074 2611892 4417722 2694076 total: 12119764 cpu: 2.18
thread counts: 2781538 2870678 3504373 2946348 total: 12102937 cpu: 2.26
thread counts: 1803045 1723898 1703833 1734871 total: 6965647 cpu: 2.80
thread counts: 3086962 4042160 3933631 3664278 total: 14727031 cpu: 1.19 **
thread counts: 1577623 1497828 1548924 1576739 total: 6201114 cpu: 2.87
thread counts: 2684337 3263193 5203930 3000744 total: 14152204 cpu: 1.58
thread counts: 3655164 3682537 3670353 2940867 total: 13948921 cpu: 2.07
thread counts: 1502591 1612421 1579027 1549353 total: 6243392 cpu: 2.86
thread counts: 3352068 2940188 4461258 2468785 total: 13222299 cpu: 2.03
thread counts: 4509419 3868584 3532675 2939812 total: 14850490 cpu: 1.87
thread counts: 3110005 3680400 3757980 3444667 total: 13993052 cpu: 1.52
thread counts: 1478133 1565752 1620530 1528830 total: 6193245 cpu: 2.87
thread counts: 1517548 1581164 1563860 1530313 total: 6192885 cpu: 2.86
thread counts: 3983783 4778611 5141953 3912456 total: 17816803 cpu: 2.11
thread counts: 2670202 3552656 3407125 3364797 total: 12994780 cpu: 2.09
thread counts: 3305922 3603738 3076325 2781303 total: 12767288 cpu: 2.14
thread counts: 1512421 1508050 1634607 1528920 total: 6183998 cpu: 2.88
thread counts: 2860995 3594953 3562578 2707053 total: 12725579 cpu: 2.16
thread counts: 1429166 1722046 1598985 1725740 total: 6475937 cpu: 2.85
thread counts: 3067732 2686763 3708798 2509704 total: 11972997 cpu: 2.19
thread counts: 1591057 1464516 1605583 1513675 total: 6174831 cpu: 2.89
thread counts: 2409497 2177906 2057195 2429560 total: 9074158 cpu: 2.54
thread counts: 3306836 3103516 3159204 3394419 total: 12963975 cpu: 2.06
thread counts: 4595398 3226518 3570597 3120376 total: 14512889 cpu: 1.99
thread counts: 1584855 1545397 1475950 1560216 total: 6166418 cpu: 2.87
thread counts: 2916919 4661226 3418193 2502552 total: 13498890 cpu: 1.97
thread counts: 1503413 1631554 1583769 1455205 total: 6173941 cpu: 2.87
thread counts: 2755663 4845651 4012224 3164140 total: 14777678 cpu: 1.51
thread counts: 2862638 3022413 2416316 3069710 total: 11371077 cpu: 2.22
thread counts: 4261418 2498152 2741344 3177890 total: 12678804 cpu: 2.06
thread counts: 1553793 1539409 1608343 1502662 total: 6204207 cpu: 2.89
thread counts: 3529285 3748373 3445962 3359241 total: 14082861 cpu: 2.09
thread counts: 2059544 1746512 1867893 1829735 total: 7503684 cpu: 2.69
thread counts: 2770944 3306708 4300948 2735480 total: 13114080 cpu: 2.09
thread counts: 1509390 1532763 1590248 1579340 total: 6211741 cpu: 2.87
thread counts: 3313247 3151852 3852792 2490807 total: 12808698 cpu: 2.09
thread counts: 1565078 1551175 1580439 1481357 total: 6178049 cpu: 2.88
thread counts: 2949780 2329638 3103444 2531422 total: 10914284 cpu: 2.37
thread counts: 3957242 4462240 4596818 4971173 total: 17987473 cpu: 1.86
thread counts: 3919540 3523495 3227946 3419620 total: 14090601 cpu: 2.06
thread counts: 4058515 3340403 2407451 4750843 total: 14557212 cpu: 1.98
thread counts: 1500656 1607491 1547774 1549650 total: 6205571 cpu: 2.87
thread counts: 2342774 2190523 3537443 5906503 total: 13977243 cpu: 1.98
thread counts: 1391138 1394731 1426851 1380975 total: 5593695 cpu: 3.54 **
thread counts: 3080208 3677328 3929946 3664969 total: 14352451 cpu: 2.02
thread counts: 3352103 4963351 4177006 3444167 total: 15936627 cpu: 1.96
thread counts: 3089638 2363115 3428596 5099717 total: 13981066 cpu: 1.99
thread counts: 3597469 2438833 2799484 2557122 total: 11392908 cpu: 2.18
=================== loop-bench program ===================================
// measure throughput and CPU time for contending threads
// build: g++ -lpthread -o loop-bench loop-bench.cpp
#include <unistd.h>
#include <stdio.h>
#include <sys/time.h>
#include <pthread.h>
#include <math.h>
void start_detached_thread(void * threadfunc(void *), void * arg);
double get_seconds();
void * thread(void *);
int main(int argc, char *argv[])
{
int ii, count[4], total;
double time0, time1, cputime;
while (1)
{
count[0] = count[1] = count[2] = count[3] = 0;
time0 = 1.0 * clock() / CLOCKS_PER_SEC;
for (ii = 0; ii < 4; ii++) // launch threads
start_detached_thread(thread,&count[ii]);
for (ii = 0; ii < 4; ii++) // wait for all done
while (count[ii] == 0) usleep(1000);
time1 = 1.0 * clock() / CLOCKS_PER_SEC; // report
cputime = time1 - time0;
total = count[0] + count[1] + count[2] + count[3];
printf("thread counts: %6d %6d %6d %6d total: %7d cpu: %4.2f \n",
count[0], count[1], count[2], count[3], total, cputime);
}
}
void * thread(void *varg)
{
int *count = (int *) varg;
int loops = 0;
double time0 = get_seconds();
while (get_seconds() - time0 < 1.0) // loop 1.0 seconds
{
for (int ii = 0; ii < 1000; ii++) sqrt(ii);
loops++;
}
*count = loops;
pthread_exit(0);
}
void start_detached_thread(void * threadfunc(void *), void * arg)
{
pthread_t ptid;
pthread_attr_t ptattr;
int pterr;
pthread_attr_init(&ptattr);
pthread_attr_setdetachstate(&ptattr,PTHREAD_CREATE_DETACHED);
pterr = pthread_create(&ptid,&ptattr,threadfunc,arg);
if (pterr) printf("start_detached_thread() failure");
return;
}
double get_seconds()
{
timeval time1;
gettimeofday(&time1,0);
return time1.tv_sec + 0.000001 * time1.tv_usec;
}
=================== mutex-bench program ===================================
// measure mutex ownership changes and CPU time
// build: g++ -lpthread -o mutex-bench mutex-bench.cpp
#include <unistd.h>
#include <stdio.h>
#include <sys/time.h>
#include <pthread.h>
#define mutex pthread_mutex_t
#define mutex_lock pthread_mutex_lock
#define mutex_unlock pthread_mutex_unlock
void start_detached_thread(void * threadfunc(void *), void * arg);
double get_seconds();
void * thread(void *);
mutex mlock = PTHREAD_MUTEX_INITIALIZER;
int main(int argc, char *argv[])
{
int ii, count[4], total;
double time0, time1, cputime;
while (1)
{
count[0] = count[1] = count[2] = count[3] = 0;
time0 = 1.0 * clock() / CLOCKS_PER_SEC;
for (ii = 0; ii < 4; ii++) // launch threads
start_detached_thread(thread,&count[ii]);
for (ii = 0; ii < 4; ii++) // wait for all done
while (count[ii] == 0) usleep(1000);
time1 = 1.0 * clock() / CLOCKS_PER_SEC; // report
cputime = time1 - time0;
total = count[0] + count[1] + count[2] + count[3];
printf("thread counts: %8d %8d %8d %8d total: %8d cpu: %4.2f \n",
count[0], count[1], count[2], count[3], total, cputime);
}
}
void * thread(void *varg)
{
int *count = (int *) varg;
int loops = 0;
double time0 = get_seconds();
while (get_seconds() - time0 < 1.0) // loop 1.0 seconds
{
mutex_lock(&mlock);
mutex_unlock(&mlock);
loops++;
}
*count = loops;
pthread_exit(0);
}
void start_detached_thread(void * threadfunc(void *), void * arg)
{
pthread_t ptid;
pthread_attr_t ptattr;
int pterr;
pthread_attr_init(&ptattr);
pthread_attr_setdetachstate(&ptattr,PTHREAD_CREATE_DETACHED);
pterr = pthread_create(&ptid,&ptattr,threadfunc,arg);
if (pterr) printf("start_detached_thread() failure");
return;
}
double get_seconds()
{
timeval time1;
gettimeofday(&time1,0);
return time1.tv_sec + 0.000001 * time1.tv_usec;
}
** Affects: linux (Ubuntu)
Importance: Undecided
Status: Invalid
** Tags: karmic regression-release
--
erratic thread scheduling, kernel 2.6.31
https://bugs.launchpad.net/bugs/459628
You received this bug notification because you are a member of Kernel Bugs, which is subscribed to linux in ubuntu.
More information about the kernel-bugs
mailing list