Периодически при запуске virt-df в логах сыпет page allocation failure: order:4, mode:0xc0d0
Самой памяти на сервере предостаточно.
free -h
total used free shared buff/cache available
Mem: 251G 127G 37G 4,0G 86G 119G
Swap: 0B 0B 0B
Как я понял order:4 это 2^4 страницы = 64 Kb
в состояние памяти:
Apr 21 06:00:05 h5 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 0*64kB 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15780kB
Apr 21 06:00:05 h5 kernel: Node 0 DMA32: 3294*4kB (UEM) 1418*8kB (UEM) 2604*16kB (UEM) 2198*32kB (UEM) 13*64kB (UEM) 18*128kB (UEM) 33*256kB (UM) 146*512kB (UM) 27*1024kB (UM) 0*2048kB 0*4096kB = 250504kB
Apr 21 06:00:05 h5 kernel: Node 0 Normal: 655474*4kB (UE) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2621896kB
Apr 21 06:00:05 h5 kernel: Node 1 Normal: 2636415*4kB (UEM) 367372*8kB (UEM) 8*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 13484764kB
Apr 21 06:00:05 h5 kernel: Node 2 Normal: 893338*4kB (UEM) 1081153*8kB (UEM) 376889*16kB (UEM) 676*32kB (UM) 29*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 18276416kB
Apr 21 06:00:05 h5 kernel: Node 3 Normal: 458731*4kB (UEM) 311281*8kB (UEM) 152481*16kB (UEM) 1179*32kB (UM) 7*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6803044kB
Полный лог:
Apr 21 06:00:01 h5 systemd-machined: New machine qemu-36204-guestfs-ivr52gsm3jqr02gj.
Apr 21 06:00:01 h5 systemd: Started Virtual Machine qemu-36204-guestfs-ivr52gsm3jqr02gj.
Apr 21 06:00:01 h5 kvm: 13 guests now active
Apr 21 06:00:03 h5 kvm: 12 guests now active
Apr 21 06:00:03 h5 systemd-machined: Machine qemu-36204-guestfs-ivr52gsm3jqr02gj terminated.
Apr 21 06:00:05 h5 libvirtd: 2020-04-21 03:00:05.118+0000: 3719: error : virHostCPUGetTscInfo:1395 : Unable to create KVM VM for TSC probing: Cannot allocate memory
Apr 21 06:00:05 h5 kernel: libvirtd: page allocation failure: order:4, mode:0xc0d0
Apr 21 06:00:05 h5 kernel: CPU: 10 PID: 3719 Comm: libvirtd Kdump: loaded Tainted: P OE ------------ 3.10.0-1062.9.1.el7.x86_64 #1
Apr 21 06:00:05 h5 kernel: Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 07/21/2019
Apr 21 06:00:05 h5 kernel: Call Trace:
Apr 21 06:00:05 h5 kernel: [<ffffffffab57ac23>] dump_stack+0x19/0x1b
Apr 21 06:00:05 h5 kernel: [<ffffffffaafc3d70>] warn_alloc_failed+0x110/0x180
Apr 21 06:00:05 h5 kernel: [<ffffffffaafc897f>] __alloc_pages_nodemask+0x9df/0xbe0
Apr 21 06:00:05 h5 kernel: [<ffffffffab016b28>] alloc_pages_current+0x98/0x110
Apr 21 06:00:05 h5 kernel: [<ffffffffaafe3b28>] kmalloc_order+0x18/0x40
Apr 21 06:00:05 h5 kernel: [<ffffffffab022056>] kmalloc_order_trace+0x26/0xa0
Apr 21 06:00:05 h5 kernel: [<ffffffffc0c396a4>] kvm_dev_ioctl+0xc4/0x6a0 [kvm]
Apr 21 06:00:05 h5 kernel: [<ffffffffab05fb40>] do_vfs_ioctl+0x3a0/0x5a0
Apr 21 06:00:05 h5 kernel: [<ffffffffab05c3ed>] ? putname+0x3d/0x60
Apr 21 06:00:05 h5 kernel: [<ffffffffab05fde1>] SyS_ioctl+0xa1/0xc0
Apr 21 06:00:05 h5 kernel: [<ffffffffab58dede>] system_call_fastpath+0x25/0x2a
Apr 21 06:00:05 h5 kernel: Mem-Info:
Apr 21 06:00:05 h5 kernel: active_anon:27111341 inactive_anon:421738 isolated_anon:0#012 active_file:11492436 inactive_file:11412231 isolated_file:0#012 unevictable:26289 dirty:525 writeback:0 unstable:0#012 slab_reclaimable:552429 slab_unreclaimable:238869#012 mapped:21532 shmem:1058031 pagetables:59029 bounce:0#012 free:10361900 free_pcp:2134 free_cma:0
Apr 21 06:00:05 h5 kernel: Node 0 DMA free:15780kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 21 06:00:05 h5 kernel: lowmem_reserve[]: 0 1533 63995 63995
Apr 21 06:00:05 h5 kernel: Node 0 DMA32 free:250504kB min:804kB low:1004kB high:1204kB active_anon:186248kB inactive_anon:2884kB active_file:8kB inactive_file:4kB unevictable:280kB isolated(anon):0kB isolated(file):0kB present:1817084kB managed:1569928kB mlocked:280kB dirty:4kB writeback:0kB mapped:408kB shmem:4692kB slab_reclaimable:115748kB slab_unreclaimable:56744kB kernel_stack:496kB pagetables:848kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 21 06:00:05 h5 kernel: lowmem_reserve[]: 0 0 62462 62462
Apr 21 06:00:05 h5 kernel: Node 0 Normal free:2621236kB min:32784kB low:40980kB high:49176kB active_anon:30963012kB inactive_anon:682504kB active_file:14033880kB inactive_file:13571704kB unevictable:6140kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63961120kB mlocked:6140kB dirty:164kB writeback:0kB mapped:31516kB shmem:1814604kB slab_reclaimable:557304kB slab_unreclaimable:117908kB kernel_stack:2864kB pagetables:70048kB unstable:0kB bounce:0kB free_pcp:2168kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 21 06:00:05 h5 kernel: lowmem_reserve[]: 0 0 0 0
Apr 21 06:00:05 h5 kernel: Node 1 Normal free:13483532kB min:33848kB low:42308kB high:50772kB active_anon:22545500kB inactive_anon:223496kB active_file:17418308kB inactive_file:10923700kB unevictable:1088kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66042524kB mlocked:1088kB dirty:548kB writeback:52kB mapped:27808kB shmem:541424kB slab_reclaimable:420740kB slab_unreclaimable:163264kB kernel_stack:4000kB pagetables:52736kB unstable:0kB bounce:0kB free_pcp:2852kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 21 06:00:05 h5 kernel: lowmem_reserve[]: 0 0 0 0
Apr 21 06:00:05 h5 kernel: Node 2 Normal free:18275136kB min:33860kB low:42324kB high:50788kB active_anon:20074712kB inactive_anon:427620kB active_file:9429012kB inactive_file:13488620kB unevictable:82000kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66058476kB mlocked:82000kB dirty:488kB writeback:204kB mapped:12244kB shmem:974216kB slab_reclaimable:665568kB slab_unreclaimable:272732kB kernel_stack:8128kB pagetables:53268kB unstable:0kB bounce:0kB free_pcp:1296kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 21 06:00:05 h5 kernel: lowmem_reserve[]: 0 0 0 0
Apr 21 06:00:05 h5 kernel: Node 3 Normal free:6801412kB min:33856kB low:42320kB high:50784kB active_anon:34675892kB inactive_anon:350448kB active_file:5088536kB inactive_file:7664896kB unevictable:15648kB isolated(anon):0kB isolated(file):0kB present:67108864kB managed:66058032kB mlocked:15648kB dirty:896kB writeback:0kB mapped:14152kB shmem:897188kB slab_reclaimable:450356kB slab_unreclaimable:344828kB kernel_stack:7808kB pagetables:59216kB unstable:0kB bounce:0kB free_pcp:2220kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 21 06:00:05 h5 kernel: lowmem_reserve[]: 0 0 0 0
Apr 21 06:00:05 h5 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 0*64kB 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15780kB
Apr 21 06:00:05 h5 kernel: Node 0 DMA32: 3294*4kB (UEM) 1418*8kB (UEM) 2604*16kB (UEM) 2198*32kB (UEM) 13*64kB (UEM) 18*128kB (UEM) 33*256kB (UM) 146*512kB (UM) 27*1024kB (UM) 0*2048kB 0*4096kB = 250504kB
Apr 21 06:00:05 h5 kernel: Node 0 Normal: 655474*4kB (UE) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2621896kB
Apr 21 06:00:05 h5 kernel: Node 1 Normal: 2636415*4kB (UEM) 367372*8kB (UEM) 8*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 13484764kB
Apr 21 06:00:05 h5 kernel: Node 2 Normal: 893338*4kB (UEM) 1081153*8kB (UEM) 376889*16kB (UEM) 676*32kB (UM) 29*64kB (UM) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 18276416kB
Apr 21 06:00:05 h5 kernel: Node 3 Normal: 458731*4kB (UEM) 311281*8kB (UEM) 152481*16kB (UEM) 1179*32kB (UM) 7*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6803044kB
Apr 21 06:00:05 h5 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 21 06:00:05 h5 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 21 06:00:05 h5 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 21 06:00:05 h5 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 21 06:00:05 h5 kernel: Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 21 06:00:05 h5 kernel: Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 21 06:00:05 h5 kernel: Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 21 06:00:05 h5 kernel: Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 21 06:00:05 h5 kernel: 23964568 total pagecache pages
Apr 21 06:00:05 h5 kernel: 0 pages in swap cache
Apr 21 06:00:05 h5 kernel: Swap cache stats: add 0, delete 0, find 0/0
Apr 21 06:00:05 h5 kernel: Free swap = 0kB
Apr 21 06:00:05 h5 kernel: Total swap = 0kB
Apr 21 06:00:05 h5 kernel: 67042845 pages RAM
Apr 21 06:00:05 h5 kernel: 0 pages HighMem/MovableOnly
Apr 21 06:00:05 h5 kernel: 1116349 pages reserved
