LINUX.ORG.RU
ФорумAdmin

page scan

 , ,


0

2

Добрый день.

На sda только OS. Своп отключён. На сервере запущены ВМ с nvme0n1, nvme1n1

Периодически возникает жуткий page scan, локальный диск sda грузится по чтению. OS зависает по D+, так минут 5, потом отвисает само.

Помогите разобраться. Свободная память вроде есть, ~9Гб. В page cache ~200-300мб.

Даже если возникает page scan, своп у меня отключен. Почему грузится локальный диск?

В 12:58

                                                   12:58:02                                                    --------------                                                    1m0s elapsed
PRC | sys   51.27s  | user   2m57s  |               |               | #proc  15621  | #trun	 5  | #tslpi   705 |  #tslpu     1 |  #zombie    0 |  clones 16042 |               |               |  #exit  15276 |
CPU | sys      82%  | user    300%  | irq       1%  |               |               | idle    755%  | wait     63% |               |  steal     0% |  guest   252% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       8%  | user     36%  | irq       0%  |               |               | idle     50%  | cpu005 w  5% |               |  steal     0% |  guest    34% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       7%  | user     32%  | irq       0%  |               |               | idle     55%  | cpu001 w  5% |               |  steal     0% |  guest    28% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       9%  | user     30%  | irq	0%  |               |               | idle     57%  | cpu004 w  4% |               |  steal     0% |  guest    28% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       8%  | user     30%  | irq       0%  |               |               | idle     55%  | cpu003 w  6% |               |  steal     0% |  guest    27% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       8%  | user     28%  | irq       0%  |               |               | idle     59%  | cpu002 w  5% |               |  steal     0% |  guest    24% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       9%  | user     25%  | irq	0%  |               |               | idle     61%  | cpu000 w  5% |               |  steal     0% |  guest    22% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       5%  | user     26%  | irq	0%  |               |               | idle     63%  | cpu008 w  6% |               |  steal     0% |  guest    17% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       4%  | user     24%  | irq       0%  |               |               | idle     66%  | cpu006 w  6% |               |  steal     0% |  guest    21% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       5%  | user     21%  | irq	0%  |               |               | idle     70%  | cpu009 w  5% |               |  steal     0% |  guest    17% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       6%  | user     19%  | irq	0%  |               |               | idle     71%  | cpu007 w  5% |               |  steal     0% |  guest    15% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       5%  | user     19%  | irq       0%  |               |               | idle     72%  | cpu010 w  4% |               |  steal     0% |  guest    13% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       7%  | user      9%  | irq       0%  |               |               | idle     76%  | cpu011 w  6% |               |  steal     0% |  guest     5% |               |  curf 4.30GHz |  curscal  82% |
CPL | avg1    6.13  |               | avg5    6.06  | avg15   5.73  |               |               | csw  1718695 |               |  intr 1178894 |               |               |  numcpu    12 |               |
MEM | tot    62.6G  | free    9.6G  | cache 285.6M  | dirty   1.0M  | buff    0.0M  | slab    1.1G  | slrec  50.3M |  shmem 211.1M |  shrss   0.8M |  shswp   0.0M |  vmbal   0.0M |  hptot   0.0M |  hpuse   0.0M |
SWP | tot     0.0M  | free    0.0M  |               |               |               |               |              |               |               |               |  vmcom  65.8G |               |  vmlim  31.3G |
PAG | scan   13054  | steal  11869  |               | stall    391  |               |               |              |               |               |               |  swin	 0 |               |  swout	 0 |
DSK |          sda  | busy     45%  |               | read  102463  | write    558  | KiB/r      8  | KiB/w      9 |               |  MBr/s   14.3 |  MBw/s    0.1 |  avq     1.17 |               |  avio 0.26 ms |
DSK |      nvme0n1  | busy     28%  |               | read    1429  | write  16738  | KiB/r	39  | KiB/w	39 |               |  MBr/s    0.9 |  MBw/s   10.8 |  avq     2.52 |               |  avio 0.91 ms |
DSK |      nvme1n1  | busy     28%  |               | read    1400  | write  16857  | KiB/r	39  | KiB/w	39 |               |  MBr/s    0.9 |  MBw/s   10.8 |  avq     2.45 |               |  avio 0.91 ms |
DSK |          sdb  | busy	0%  |               | read	37  | write     95  | KiB/r	 7  | KiB/w	 6 |               |  MBr/s    0.0 |  MBw/s    0.0 |  avq     1.04 |               |  avio 1.23 ms |
  PID                             TID                             RDDSK                             WRDSK                           WCANCL                             DSK                            CMD      1/908
22149                               -                            790.6M                                0K                               0K                             34%                            logrotate
 9230                               -                              320K                            130.4M                               0K                              6%                            qemu-kvm
23863                               -                             8264K                            84132K                               0K                              4%                            qemu-kvm
  749                               -                            84255K                                0K                               0K                              4%                            z_wr_int_4
  748                               -                            81314K                                0K                               0K                              3%                            z_wr_int_3
  745                               -                            80129K                                0K                               0K                              3%                            z_wr_int_0
  746                               -                            80111K                                0K                               0K                              3%                            z_wr_int_1
  747                               -                            79480K                                0K                               0K                              3%                            z_wr_int_2
  750                               -                            79309K                                0K                               0K                              3%                            z_wr_int_5
  752                               -                            79283K                                0K                               0K                              3%                            z_wr_int_7
  751                               -                            78957K                                0K                               0K                              3%                            z_wr_int_6
  737                               -                            58453K                                0K                               0K                              2%                            z_wr_iss
  743                               -                            58065K                                0K                               0K                              2%                            z_wr_iss
  739                               -                            57923K                                0K                               0K                              2%                            z_wr_iss
  738                               -                            55525K                                0K                               0K                              2%                            z_wr_iss
  735                               -                            55319K                                0K                               0K                              2%                            z_wr_iss
  741                               -                            54982K                                0K                               0K                              2%                            z_wr_iss

Через минуту начинается PAG scan:

12:59:02                                                    --------------                                                    1m0s elapsed
PRC | sys   44.04s  | user   2m26s  |               |               | #proc  13924  | #trun	 3  | #tslpi   678 |  #tslpu     3 |  #zombie    0 |  clones 13649 |               |               |  #exit  13580 |
CPU | sys      69%  | user    237%  | irq       1%  |               |               | idle    811%  | wait     83% |               |  steal     0% |  guest   232% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       8%  | user     29%  | irq       0%  |               |               | idle     54%  | cpu005 w  9% |               |  steal     0% |  guest    28% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       7%  | user     28%  | irq       0%  |               |               | idle     58%  | cpu000 w  6% |               |  steal     0% |  guest    28% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       7%  | user     25%  | irq	0%  |               |               | idle     61%  | cpu004 w  7% |               |  steal     0% |  guest    24% |               |  curf 4.38GHz |  curscal  84% |
cpu | sys       6%  | user     25%  | irq       0%  |               |               | idle     61%  | cpu002 w  7% |               |  steal     0% |  guest    25% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       7%  | user     25%  | irq       0%  |               |               | idle     60%  | cpu003 w  8% |               |  steal     0% |  guest    24% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       8%  | user     23%  | irq	0%  |               |               | idle     61%  | cpu001 w  8% |               |  steal     0% |  guest    22% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       3%  | user     17%  | irq	0%  |               |               | idle     74%  | cpu009 w  6% |               |  steal     0% |  guest    16% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       3%  | user     16%  | irq       0%  |               |               | idle     74%  | cpu007 w  7% |               |  steal     0% |  guest    16% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       2%  | user     17%  | irq	0%  |               |               | idle     75%  | cpu008 w  6% |               |  steal     0% |  guest    17% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       3%  | user     16%  | irq	0%  |               |               | idle     76%  | cpu006 w  5% |               |  steal     0% |  guest    16% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       5%  | user     13%  | irq       0%  |               |               | idle     73%  | cpu010 w  9% |               |  steal     0% |  guest    13% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys       7%  | user      3%  | irq       1%  |               |               | idle     84%  | cpu011 w  5% |               |  steal     0% |  guest     3% |               |  curf 4.30GHz |  curscal  82% |
CPL | avg1    5.11  |               | avg5    5.88  | avg15   5.69  |               |               | csw  2123017 |               |  intr 1221074 |               |               |  numcpu    12 |               |
MEM | tot    62.6G  | free    9.6G  | cache 277.9M  | dirty   1.1M  | buff    0.0M  | slab    1.1G  | slrec  50.6M |  shmem 211.1M |  shrss   0.8M |  shswp   0.0M |  vmbal   0.0M |  hptot   0.0M |  hpuse   0.0M |
SWP | tot     0.0M  | free    0.0M  |               |               |               |               |              |               |               |               |  vmcom  65.6G |               |  vmlim  31.3G |
PAG | scan 22067e3  | steal 103970  |               | stall   6716  |               |               |              |               |               |               |  swin	 0 |               |  swout	 0 |
DSK |          sda  | busy     88%  |               | read  194668  | write    420  | KiB/r      9  | KiB/w      8 |               |  MBr/s   31.4 |  MBw/s    0.1 |  avq     3.70 |               |  avio 0.27 ms |
DSK |      nvme1n1  | busy     27%  |               | read     911  | write  14127  | KiB/r	41  | KiB/w	40 |               |  MBr/s    0.6 |  MBw/s    9.4 |  avq     2.13 |               |  avio 1.07 ms |
DSK |      nvme0n1  | busy     20%  |               | read    1175  | write  14615  | KiB/r	37  | KiB/w	39 |               |  MBr/s    0.7 |  MBw/s    9.4 |  avq     1.68 |               |  avio 0.75 ms |
DSK |          sdb  | busy	0%  |               | read	 3  | write     50  | KiB/r	 4  | KiB/w	 5 |               |  MBr/s    0.0 |  MBw/s    0.0 |  avq     1.07 |               |  avio 3.04 ms |
  PID                             TID                             RDDSK                             WRDSK                           WCANCL                             DSK                            CMD      1/808
22149                               -                              1.5G                                8K                               0K                             49%                            logrotate
 9230                               -                             7264K                            114.5M                               0K                              4%                            qemu-kvm
23863                               -                            14880K                            81816K                               0K                              3%                            qemu-kvm
  746                               -                            75510K                                0K                               0K                              2%                            z_wr_int_1
  751                               -                            74760K                                0K                               0K                              2%                            z_wr_int_6
  745                               -                            74597K                                0K                               0K                              2%                            z_wr_int_0
  749                               -                            73683K                                0K                               0K                              2%                            z_wr_int_4
  750                               -                            73506K                                0K                               0K                              2%                            z_wr_int_5
  748                               -                            70174K                                0K                               0K                              2%                            z_wr_int_3
  752                               -                            69027K                                0K                               0K                              2%                            z_wr_int_7
  747                               -                            68774K                                0K                               0K                              2%                            z_wr_int_2
 8858                               -                            47880K                             2496K                               0K                              2%                            qemu-kvm
  737                               -                            48999K                                0K                               0K                              2%                            z_wr_iss
  736                               -                            48744K                                0K                               0K                              2%                            z_wr_iss
  738                               -                            46608K                                0K                               0K                              1%                            z_wr_iss
 8662                               -                            16640K                            28676K                               0K                              1%                            qemu-kvm
  743                               -                            45208K                                0K                               0K                              1%                            z_wr_iss

В 13:01:

PRC | sys    4m19s  | user   2m33s  |               |               | #proc   4205  | #trun	22  | #tslpi   497 |  #tslpu    29 |  #zombie    0 |  clones  4040 |               |               |  #exit   3815 |
CPU | sys     416%  | user    242%  | irq       1%  |               |               | idle    135%  | wait    407% |               |  steal     0% |  guest   241% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys      41%  | user     22%  | irq       0%  |               |               | idle     12%  | cpu004 w 26% |               |  steal     0% |  guest    22% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys      40%  | user     22%  | irq       0%  |               |               | idle      9%  | cpu002 w 29% |               |  steal     0% |  guest    22% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys      38%  | user     21%  | irq	0%  |               |               | idle     11%  | cpu003 w 30% |               |  steal     0% |  guest    21% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys      33%  | user     24%  | irq       0%  |               |               | idle     10%  | cpu000 w 32% |               |  steal     0% |  guest    24% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys      31%  | user     26%  | irq       0%  |               |               | idle     12%  | cpu006 w 31% |               |  steal     0% |  guest    26% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys      39%  | user     17%  | irq	0%  |               |               | idle      3%  | cpu005 w 40% |               |  steal     0% |  guest    17% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys      34%  | user     20%  | irq	0%  |               |               | idle     13%  | cpu001 w 33% |               |  steal     0% |  guest    20% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys      30%  | user     21%  | irq       0%  |               |               | idle     13%  | cpu008 w 35% |               |  steal     0% |  guest    21% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys      37%  | user     14%  | irq	0%  |               |               | idle     21%  | cpu010 w 28% |               |  steal     0% |  guest    14% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys      33%  | user     16%  | irq	1%  |               |               | idle      1%  | cpu011 w 49% |               |  steal     0% |  guest    16% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys      25%  | user     25%  | irq       0%  |               |               | idle     16%  | cpu009 w 34% |               |  steal     0% |  guest    25% |               |  curf 4.30GHz |  curscal  82% |
cpu | sys      34%  | user     13%  | irq       0%  |               |               | idle     14%  | cpu007 w 39% |               |  steal     0% |  guest    13% |               |  curf 4.30GHz |  curscal  82% |
CPL | avg1   26.51  |               | avg5   11.44  | avg15   7.60  |               |               | csw  2045975 |               |  intr 1640155 |               |               |  numcpu    12 |               |
MEM | tot    62.6G  | free    9.7G  | cache 229.4M  | dirty   0.0M  | buff    0.0M  | slab    1.1G  | slrec  49.7M |  shmem 211.2M |  shrss   0.8M |  shswp   0.0M |  vmbal   0.0M |  hptot   0.0M |  hpuse   0.0M |
SWP | tot     0.0M  | free    0.0M  |               |               |               |               |              |               |               |               |  vmcom  64.4G |               |  vmlim  31.3G |
PAG | scan 18111e4  | steal 1582e3  |               | stall  84968  |               |               |              |               |               |               |  swin	 0 |               |  swout	 0 |
DSK |          sda  | busy    102%  |               | read   38361  | write     85  | KiB/r    164  | KiB/w     10 |               |  MBr/s  102.6 |  MBw/s    0.0 |  avq    50.81 |               |  avio 1.56 ms |
DSK |      nvme1n1  | busy      7%  |               | read     829  | write   5582  | KiB/r	37  | KiB/w	32 |               |  MBr/s    0.5 |  MBw/s    2.9 |  avq     4.20 |               |  avio 0.67 ms |
DSK |      nvme0n1  | busy      7%  |               | read     980  | write   5521  | KiB/r	36  | KiB/w	32 |               |  MBr/s    0.6 |  MBw/s    2.9 |  avq     3.75 |               |  avio 0.66 ms |
DSK |          sdb  | busy	0%  |               | read       1  | write     21  | KiB/r	 4  | KiB/w	 5 |               |  MBr/s    0.0 |  MBw/s    0.0 |  avq     1.80 |               |  avio 1.14 ms |
NFC | rpc        1  | read       0  |               | write      0  | retxmit    0  | autref     1  |              |               |               |               |               |               |               |
NET | transport     | tcpi     275  | tcpo     342  | udpi       1  | udpo       1  | tcpao      1  | tcppo     33 |               |  tcprs      6 |  tcpie      0 |  tcpor      2 |  udpnp      1 |  udpie      0 |
NET | network       | ipi      298  |               | ipo      293  | ipfrw      0  | deliv    287  |              |               |               |               |  icmpi      3 |               |  icmpo      0 |
NET | kvm2032  17%  | pcki    4070  | pcko    8167  | sp   10 Mbps  | si 1772 Kbps  | so  210 Kbps  |              |  coll	 0 |  mlti	 0 |  erri	 0 |  erro	 0 |  drpi	 0 |  drpo	 0 |
NET | kvm2068   3%  | pcki    2866  | pcko    6357  | sp   10 Mbps  | si  357 Kbps  | so  158 Kbps  |              |  coll	 0 |  mlti	 0 |  erri	 0 |  erro	 0 |  drpi	 0 |  drpo       0 |
NET | kvm2061   1%  | pcki     335  | pcko    3886  | sp   10 Mbps  | si  179 Kbps  | so   34 Kbps  |              |  coll	 0 |  mlti	 0 |  erri	 0 |  erro	 0 |  drpi	 0 |  drpo       0 |
NET | kvm2229   0%  | pcki      37  | pcko    3402  | sp   10 Mbps  | si    0 Kbps  | so   27 Kbps  |              |  coll	 0 |  mlti	 0 |  erri	 0 |  erro	 0 |  drpi	 0 |  drpo       0 |
NET | kvm2011   0%  | pcki      22  | pcko    3382  | sp   10 Mbps  | si    0 Kbps  | so   27 Kbps  |              |  coll	 0 |  mlti	 0 |  erri	 0 |  erro	 0 |  drpi	 0 |  drpo       0 |
NET | kvm2067   0%  | pcki      15  | pcko    3379  | sp   10 Mbps  | si    1 Kbps  | so   27 Kbps  |              |  coll	 0 |  mlti	 0 |  erri	 0 |  erro	 0 |  drpi	 0 |  drpo       0 |
NET | kvm2030	0%  | pcki      17  | pcko    3374  | sp   10 Mbps  | si    0 Kbps  | so   27 Kbps  |              |  coll	 0 |  mlti	 0 |  erri	 0 |  erro	 0 |  drpi	 0 |  drpo	 0 |
NET | kvm2549   0%  | pcki	 9  | pcko    3377  | sp   10 Mbps  | si    0 Kbps  | so   27 Kbps  |              |  coll	 0 |  mlti	 0 |  erri	 0 |  erro	 0 |  drpi	 0 |  drpo       0 |
NET | kvm2021   0%  | pcki	 9  | pcko    3372  | sp   10 Mbps  | si    0 Kbps  | so   27 Kbps  |              |  coll	 0 |  mlti	 0 |  erri	 0 |  erro	 0 |  drpi	 0 |  drpo       0 |
NET | kvm2004   0%  | pcki     161  | pcko    2865  | sp   10 Mbps  | si    6 Kbps  | so   25 Kbps  |              |  coll	 0 |  mlti	 0 |  erri	 0 |  erro	 0 |  drpi	 0 |  drpo       0 |
NET | eno1      0%  | pcki   12971  | pcko   16436  | sp 1000 Mbps  | si  372 Kbps  | so 2430 Kbps  |              |  coll	 0 |  mlti	37 |  erri	 0 |  erro	 0 |  drpi	 0 |  drpo       0 |
NET | kvm2041	0%  | pcki      56  | pcko    2849  | sp   10 Mbps  | si   10 Kbps  | so   23 Kbps  |              |  coll	 0 |  mlti	 0 |  erri	 0 |  erro	 0 |  drpi	 0 |  drpo	 0 |
NET | br0     ----  | pcki    3650  | pcko     295  | sp    0 Mbps  | si   24 Kbps  | so   18 Kbps  |              |  coll	 0 |  mlti	 0 |  erri	 0 |  erro	 0 |  drpi	 0 |  drpo	 0 |

  PID                             TID                             RDDSK                             WRDSK                           WCANCL                             DSK                            CMD      1/236
 5196                               -                            865.6M                                0K                               0K                             13%                            php-fpm
 1198                               -                            591.8M                                0K                               0K                              9%                            qemu-img
23863                               -                            414.8M                            19312K                               0K                              6%                            qemu-kvm
 8662                               -                            391.2M                             5080K                               0K                              6%                            qemu-kvm
22149                               -                            323.2M                                0K                               0K                              5%                            logrotate
 8454                               -                            319.3M                              328K                               0K                              5%                            qemu-kvm
 9230                               -                            285.6M                            23596K                               0K                              5%                            qemu-kvm
 6725                               -                            274.5M                              124K                               0K                              4%                            qemu-kvm
 6438                               -                            219.6M                            16012K                               0K                              3%                            qemu-kvm
 7824                               -                            230.0M                              336K                               0K                              3%                            qemu-kvm
 9007                               -                            227.1M                              140K                               0K                              3%                            qemu-kvm
 7483                               -                            207.1M                              472K                               0K                              3%                            qemu-kvm
 4432                               -                            207.0M                                0K                               0K                              3%                            virsh
 6993                               -                            192.5M                              692K                               0K                              3%                            qemu-kvm
  775                               -                            178.2M                                4K                               0K                              3%                            bash
 8858                               -                            175.3M                             2404K                               0K                              3%                            qemu-kvm
 2344                               -                            153.2M                                0K                               0K                              2%                            sshd

мало памяти?

page scan, емнип, поиск страниц ( кандидатов на удаление из памяти ). раз свопа нет, то остаётся удалять дисковый кэш и выкидывать из памяти исполняемый код

Т.е. последующее чтение к диску - это либо приложения читают свои данные обратно, либо ОСь подгружает в память исполняемые файлы и библиотеки, которые выкинула из памяти из-за нехватки памяти

router ★★★★★ ()