[Bug 1979715] Re: out of memory, unantteded killed by kernel. apt-daily systemd fails.
sascha arthur
1979715 at bugs.launchpad.net
Wed Jun 29 13:35:23 UTC 2022
and we hit this again.
# unattended-upgrade -v -d --apt-debug
Killed
# dmesg
[51111.470168] oom_reaper: reaped process 407772 (unattended-upgr), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[51345.730386] sshd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[51345.730393] CPU: 0 PID: 409066 Comm: sshd Kdump: loaded Not tainted 5.15.0-40-lowlatency #43-Ubuntu
[51345.730395] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
[51345.730397] Call Trace:
[51345.730398] <TASK>
[51345.730401] show_stack+0x52/0x58
[51345.730405] dump_stack_lvl+0x4a/0x5f
[51345.730408] dump_stack+0x10/0x12
[51345.730410] dump_header+0x53/0x22e
[51345.730412] oom_kill_process.cold+0xb/0x10
[51345.730413] out_of_memory+0x106/0x2e0
[51345.730417] __alloc_pages_slowpath.constprop.0+0x993/0xa50
[51345.730420] __alloc_pages+0x326/0x340
[51345.730421] alloc_pages+0x9e/0x1d0
[51345.730453] __page_cache_alloc+0x7a/0x90
[51345.730454] pagecache_get_page+0x1dd/0x550
[51345.730455] ? page_cache_ra_unbounded+0x166/0x210
[51345.730457] filemap_fault+0x484/0xab0
[51345.730459] ? filemap_map_pages+0x30c/0x400
[51345.730460] __do_fault+0x3c/0x120
[51345.730462] do_read_fault+0xeb/0x160
[51345.730464] do_fault+0xa0/0x2e0
[51345.730465] handle_pte_fault+0x1c5/0x230
[51345.730467] __handle_mm_fault+0x3c7/0x700
[51345.730469] handle_mm_fault+0xdd/0x2c0
[51345.730471] do_user_addr_fault+0x1e2/0x680
[51345.730473] exc_page_fault+0x77/0x180
[51345.730476] ? asm_exc_page_fault+0x8/0x30
[51345.730479] asm_exc_page_fault+0x1e/0x30
[51345.730481] RIP: 0033:0x7fec21ea6660
[51345.730485] Code: 17 0f be 0e 49 89 f9 49 89 f0 84 d2 74 32 8d 42 d0 8d 71 d0 3c 09 76 32 40 80 fe 09 0f 86 9b 00 00 00 38 ca 0f 85 99 00 00 00 <41> 0f b6 51 01 41 0f be 48 01 49 83 c1 01 49 83 c0 01 84 d2 75 ce
[51345.730487] RSP: 002b:00007ffcfd7731f8 EFLAGS: 00010246
[51345.730489] RAX: 0000000000000039 RBX: 00000000000000ae RCX: 0000000000000069
[51345.730490] RDX: 0000000000000069 RSI: 0000000000000039 RDI: 00007fec21e7b61c
[51345.730491] RBP: 00007ffcfd7732f0 R08: 00007fec21e8afa6 R09: 00007fec21e7b61d
[51345.730492] R10: 00007fec21eb8140 R11: 0000000000001050 R12: 00007fec21e86030
[51345.730493] R13: 00007fec21e7b61c R14: 000000000000009f R15: 00007fec21e86000
[51345.730496] </TASK>
[51345.730497] Mem-Info:
[51345.730499] active_anon:307 inactive_anon:1895160 isolated_anon:0
active_file:26 inactive_file:224 isolated_file:0
unevictable:8844 dirty:0 writeback:0
slab_reclaimable:6026 slab_unreclaimable:47254
mapped:4273 shmem:340 pagetables:4757 bounce:0
kernel_misc_reclaimable:0
free:27378 free_pcp:435 free_cma:0
[51345.730503] Node 0 active_anon:1228kB inactive_anon:7580640kB active_file:104kB inactive_file:896kB unevictable:35376kB isolated(anon):0kB isolated(file):0kB mapped:17092kB dirty:0kB writeback:0kB shmem:1360kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:3344kB pagetables:19028kB all_unreclaimable? no
[51345.730507] Node 0 DMA free:14336kB min:128kB low:160kB high:192kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[51345.730510] lowmem_reserve[]: 0 1837 7806 7806 7806
[51345.730512] Node 0 DMA32 free:38892kB min:15216kB low:19020kB high:22824kB reserved_highatomic:0KB active_anon:64kB inactive_anon:1835764kB active_file:0kB inactive_file:200kB unevictable:0kB writepending:0kB present:2079032kB managed:1881728kB mlocked:0kB bounce:0kB free_pcp:372kB local_pcp:140kB free_cma:0kB
[51345.730515] lowmem_reserve[]: 0 0 5968 5968 5968
[51345.730517] Node 0 Normal free:56284kB min:54280kB low:67336kB high:80392kB reserved_highatomic:4096KB active_anon:1164kB inactive_anon:5744876kB active_file:524kB inactive_file:1052kB unevictable:35376kB writepending:0kB present:6291456kB managed:6111812kB mlocked:35376kB bounce:0kB free_pcp:1368kB local_pcp:1004kB free_cma:0kB
[51345.730520] lowmem_reserve[]: 0 0 0 0 0
[51345.730522] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (M) 3*4096kB (M) = 14336kB
[51345.730529] Node 0 DMA32: 409*4kB (UME) 197*8kB (UME) 182*16kB (UME) 130*32kB (UME) 95*64kB (UE) 46*128kB (UE) 23*256kB (UE) 7*512kB (UME) 5*1024kB (UME) 1*2048kB (U) 0*4096kB = 38892kB
[51345.730537] Node 0 Normal: 617*4kB (UME) 515*8kB (UME) 261*16kB (UME) 194*32kB (UME) 132*64kB (UME) 67*128kB (UME) 43*256kB (UME) 23*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 56780kB
[51345.730545] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[51345.730546] 4413 total pagecache pages
[51345.730547] 0 pages in swap cache
[51345.730547] Swap cache stats: add 0, delete 0, find 0/0
[51345.730548] Free swap = 0kB
[51345.730549] Total swap = 0kB
[51345.730549] 2096621 pages RAM
[51345.730550] 0 pages HighMem/MovableOnly
[51345.730550] 94396 pages reserved
[51345.730550] 0 pages hwpoisoned
[51345.730551] Tasks state (memory values in pages):
[51345.730561] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[51345.730565] [ 414] 0 414 72329 6776 110592 0 -1000 multipathd
[51345.730567] [ 418] 0 418 6576 1306 77824 0 -1000 systemd-udevd
[51345.730569] [ 790] 103 790 2319 772 57344 0 -900 dbus-daemon
[51345.730571] [ 795] 0 795 20699 608 69632 0 0 irqbalance
[51345.730572] [ 802] 0 802 7537 2974 106496 0 0 networkd-dispat
[51345.730574] [ 807] 0 807 1076 557 49152 0 0 cron
[51345.730575] [ 809] 0 809 58621 889 86016 0 0 polkitd
[51345.730577] [ 813] 0 813 3834 1215 69632 0 0 systemd-logind
[51345.730578] [ 815] 0 815 98114 911 131072 0 0 udisksd
[51345.730579] [ 893] 0 893 26776 2855 106496 0 0 unattended-upgr
[51345.730581] [ 896] 0 896 60818 697 110592 0 0 ModemManager
[51345.730582] [ 900] 0 900 1543 206 45056 0 0 agetty
[51345.730583] [ 7171] 0 7171 21119 14384 204800 0 0 puppet
[51345.730585] [ 8461] 101 8461 100199 97078 847872 0 0 systemd-network
[51345.730587] [ 8466] 102 8466 6348 1763 94208 0 0 systemd-resolve
[51345.730588] [ 8468] 0 8468 35600 1352 270336 0 -250 systemd-journal
[51345.730589] [ 9904] 0 9904 3228 176 61440 0 0 iscsid
[51345.730591] [ 9905] 0 9905 3354 3249 65536 0 -17 iscsid
[51345.730592] [ 9921] 0 9921 75668 3033 167936 0 0 packagekitd
[51345.730593] [ 9985] 0 9985 3863 1693 77824 0 -1000 sshd
[51345.730595] [ 10750] 107 10750 55601 895 81920 0 0 rsyslogd
[51345.730596] [ 12822] 104 12822 22338 984 73728 0 0 systemd-timesyn
[51345.730598] [ 16614] 0 16614 3753 1185 69632 0 0 systemd-machine
[51345.730599] [ 16617] 1001 16617 4547 1403 73728 0 0 systemd
[51345.730600] [ 16618] 1001 16618 26278 1244 90112 0 0 (sd-pam)
[51345.730602] [ 16816] 0 16816 2171 653 57344 0 0 check_mk_agent
[51345.730603] [ 312917] 0 312917 4253 2017 81920 0 0 sshd
[51345.730605] [ 312920] 1000 312920 4264 1407 73728 0 0 systemd
[51345.730606] [ 312922] 1000 312922 26233 1271 90112 0 0 (sd-pam)
[51345.730608] [ 313044] 1000 313044 4438 1520 81920 0 0 sshd
[51345.730609] [ 313045] 1000 313045 2209 944 49152 0 0 bash
[51345.730610] [ 316175] 0 316175 4316 2047 81920 0 0 sshd
[51345.730612] [ 316343] 1000 316343 4351 1599 81920 0 0 sshd
[51345.730613] [ 316344] 1000 316344 2209 921 49152 0 0 bash
[51345.730614] [ 316358] 1000 316358 4209 1513 69632 0 0 sudo
[51345.730616] [ 316359] 1000 316359 4209 792 65536 0 0 sudo
[51345.730617] [ 316360] 0 316360 2553 503 53248 0 0 su
[51345.730618] [ 316361] 0 316361 2229 979 53248 0 0 bash
[51345.730620] [ 324229] 0 324229 2017 628 53248 0 0 watchfrr
[51345.730621] [ 324246] 113 324246 574178 477901 3964928 0 0 zebra
[51345.730623] [ 324251] 113 324251 457450 405667 3403776 0 0 bgpd
[51345.730632] [ 324256] 113 324256 2342 503 61440 0 0 staticd
[51345.730634] [ 395869] 109 395869 4116 1200 73728 0 0 tcpdump
[51345.730635] [ 405621] 0 405621 4262 2041 69632 0 0 sshd
[51345.730636] [ 405763] 1000 405763 4478 1605 73728 0 0 sshd
[51345.730644] [ 405764] 1000 405764 2209 911 53248 0 0 bash
[51345.730645] [ 405778] 1000 405778 4208 1666 73728 0 0 sudo
[51345.730646] [ 405779] 1000 405779 4208 734 65536 0 0 sudo
[51345.730648] [ 405780] 0 405780 2553 554 61440 0 0 su
[51345.730649] [ 405781] 0 405781 2209 915 57344 0 0 bash
[51345.730651] [ 408564] 1000 408564 4006 1036 69632 0 0 sudo
[51345.730652] [ 408565] 1000 408565 4006 317 65536 0 0 sudo
[51345.730653] [ 408566] 0 408566 2553 521 57344 0 0 su
[51345.730654] [ 408567] 0 408567 2198 919 57344 0 0 bash
[51345.730656] [ 408583] 0 408583 898069 876636 7143424 0 0 unattended-upgr
[51345.730657] [ 409058] 0 409058 1795 415 57344 0 0 cron
[51345.730658] [ 409059] 0 409059 1787 399 57344 0 0 cron
[51345.730659] [ 409060] 0 409060 1787 529 57344 0 0 cron
[51345.730660] [ 409061] 0 409061 3124 1142 61440 0 0 htop
[51345.730662] [ 409064] 0 409064 2714 167 57344 0 0 systemctl
[51345.730663] [ 409065] 0 409065 2609 416 65536 0 0 systemctl
[51345.730664] [ 409066] 0 409066 2678 140 53248 0 0 sshd
[51345.730665] [ 409067] 0 409067 1622 372 53248 0 0 cron
[51345.730667] [ 409068] 0 409068 1622 475 53248 0 0 cron
[51345.730668] [ 409069] 0 409069 1622 389 53248 0 0 cron
[51345.730669] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-1964.scope,task=unattended-upgr,pid=408583,uid=0
[51345.730680] Out of memory: Killed process 408583 (unattended-upgr) total-vm:3592276kB, anon-rss:3501496kB, file-rss:5048kB, shmem-rss:0kB, UID:0 pgtables:6976kB oom_score_adj:0
[51345.848147] oom_reaper: reaped process 408583 (unattended-upgr), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
painfully freezing the whole server which is the worst ever can happen to unattended upgrades..
This whole thing took now 3,5 GB memory (which should be far more then enough) before it got wiped by the kernel. In this current state its kind of useless for us.
We consider disabling this tool now on hunderts of our ubuntu servers.
The downside leaving (them open for security risks), is far less painful
then having processed killed and swapping triggered up to an
unresponsive server.
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1979715
Title:
out of memory, unantteded killed by kernel. apt-daily systemd fails.
Status in unattended-upgrades package in Ubuntu:
New
Bug description:
had a case where
/usr/bin/python3 /usr/bin/unattended-upgrade reproducable just using
100% cpu and loading stuff into memory, until OOM killer of kernel
killed it.
noticed it because apt-daily systemd failed.
Memory of my machine was 8 GB, nothing else was running.
machine was running 22.04 LTS (yammy, lately installed)
Reinstalled my machine now. If someone hits this, feel free to attach
more information.
Also ubuntu devs should consider to introduce an memory limit for
unattended-upgrade. I mean its unattended, and not nice to flood
"unattended" the memory..
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1979715/+subscriptions
More information about the foundations-bugs
mailing list