LINUX.ORG.RU

Проблема с производительностью дисковой подсистемы.

 , , ,


0

2

Добрый день, имеем

Total DISK READ :       0.00 B/s | Total DISK WRITE :    1794.15 K/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:       2.27 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                    
  406 be/3 root        0.00 B/s    3.57 K/s  0.00 % 40.46 % [jbd2/sda3-8]
11926 be/4 mysql       0.00 B/s  399.49 K/s  0.00 %  5.28 % mysqld --basedir=/usr --datadir=/var/lib/mysql~t=100000 --pid-file=/var/run/mariadb/mysqld.pid
11911 be/4 mysql       0.00 B/s  413.76 K/s  0.00 %  3.91 % mysqld --basedir=/usr --datadir=/var/lib/mysql~t=100000 --pid-file=/var/run/mariadb/mysqld.pid
11916 be/4 mysql       0.00 B/s  114.14 K/s  0.00 %  2.05 % mysqld --basedir=/usr --datadir=/var/lib/mysql~t=100000 --pid-file=/var/run/mariadb/mysqld.pid
11906 be/4 mysql       0.00 B/s   85.61 K/s  0.00 %  1.80 % mysqld --basedir=/usr --datadir=/var/lib/mysql~t=100000 --pid-file=/var/run/mariadb/mysqld.pid
11786 be/4 mysql       0.00 B/s  128.41 K/s  0.00 %  1.65 % mysqld --basedir=/usr --datadir=/var/lib/mysql~t=100000 --pid-file=/var/run/mariadb/mysqld.pid
11920 be/4 mysql       0.00 B/s  199.75 K/s  0.00 %  1.65 % mysqld --basedir=/usr --datadir=/var/lib/mysql~t=100000 --pid-file=/var/run/mariadb/mysqld.pid
11692 be/4 mysql       0.00 B/s   57.07 K/s  0.00 %  1.27 % mysqld --basedir=/usr --datadir=/var/lib/mysql~t=100000 --pid-file=/var/run/mariadb/mysqld.pid
11756 be/4 mysql       0.00 B/s  171.21 K/s  0.00 %  1.24 % mysqld --basedir=/usr --datadir=/var/lib/mysql~t=100000 --pid-file=/var/run/mariadb/mysqld.pid
11173 be/4 mysql       0.00 B/s   57.07 K/s  0.00 %  0.85 % mysqld --basedir=/usr --datadir=/var/lib/mysql~t=100000 --pid-file=/var/run/mariadb/mysqld.pid
11923 be/4 mysql       0.00 B/s   57.07 K/s  0.00 %  0.74 % mysqld --basedir=/usr --datadir=/var/lib/mysql~t=100000 --pid-file=/var/run/mariadb/mysqld.pid
11908 be/4 mysql       0.00 B/s   28.54 K/s  0.00 %  0.73 % mysqld --basedir=/usr --datadir=/var/lib/mysql~t=100000 --pid-file=/var/run/mariadb/mysqld.pid
11927 be/4 mysql       0.00 B/s   28.54 K/s  0.00 %  0.53 % mysqld --basedir=/usr --datadir=/var/lib/mysql~t=100000 --pid-file=/var/run/mariadb/mysqld.pid
11934 be/4 mysql       0.00 B/s    3.57 K/s  0.00 %  0.19 % mysqld --basedir=/usr --datadir=/var/lib/mysql~t=10


Zabbix сервер, место на диске есть, более 500 гигабайт свободно
/dev/sda3 on / type ext4 (rw,noatime,nodiratime,data=ordered)
База большая 380 гигов, куда посмотреть.

cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
innodb_use_native_aio=1
skip-log-bin
#datadir=/mnt/mysql
# socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
default_storage_engine=InnoDB
innodb_file_per_table = 1
#
tmpdir=/mytmp
#
# network
connect_timeout = 60
wait_timeout = 28800
max_connections = 2048
max_allowed_packet = 64M
max_connect_errors = 1000

# limits
tmp_table_size = 512M
max_heap_table_size = 256M
table_cache = 512

# logs
# log_error = /var/log/mysql-error.log
slow_query_log_file = /var/log/mariadb/mysql-slow.log
slow_query_log = 1
long_query_time = 20
#log-bin=mysql-bin
#log-bin=/var/lib/mysql_innodblog/mysql-bin
log-bin = /var/backups/mysql/mysql_innodblog/mysql-bin
log-bin-index = /var/backups/mysql/mysql_innodblog/mysql-bin.index

join_buffer_size=2048k
read_buffer_size=256k
read_rnd_buffer_size=256k

# innodb
default_storage_engine=InnoDB
innodb_file_per_table = 1
innodb_status_file = 1
innodb_additional_mem_pool_size = 128M
innodb_buffer_pool_size = 24G
innodb_buffer_pool_instances = 24
innodb_flush_method = O_DIRECT
innodb_io_capacity = 2000
innodb_flush_log_at_trx_commit = 2
innodb_support_xa = 0
innodb_log_buffer_size = 1024M
## innodb_data_file_path  = ibdata1:128M;ibdata2:128M:autoextend:max:4096M

# experimental
innodb_stats_update_need_lock = 0

# other stuff
event_scheduler = 1
query_cache_type = 0
query_cache_size = 536870912
join_buffer_size=1M
query_cache_limit=2M
sort_buffer_size=2M
read_buffer_size=2M
read_rnd_buffer_size=4M
key_buffer = 256M
key_buffer_size=64M
open_files_limit = 100000
thread_cache_size = 4

[mysqld_safe]
# log-error=/var/log/mysqld.log
pid-file=/var/run/mariadb/mysqld.pid

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d


Диски быстрые в 10 RAID, 600 gb 10000rpm, raid показывает что все норм с дисками и самим контроллером.

Ответ на: комментарий от Deleted
[root@localhost ~]# free
              total        used        free      shared  buff/cache   available
Mem:       36897024    31491956      553000     1971720     4852068     3092228
Swap:      52428796      288156    52140640
lumenok ()

innodb_flush_method = O_DIRECT

O_DSYNC, не? Хочешь скорости - придеться либо жертвовать надежностью либо вкладываться баблом. Например вынося логи транзакций на SSD

Pinkbyte ★★★★★ ()
Ответ на: комментарий от Pinkbyte

Лог транзакций на отдельном диске, кстати только заметил что батарея сдохла, а может дело в ней...?

hpssacli ctrl all show config detail

Smart Array P410i in Slot 0 (Embedded)
   Bus Interface: PCI
   Slot: 0
   Serial Number: 500143801181F640
   Cache Serial Number: PAAVPCN80ZO0ZKK
   Controller Status: OK
   Hardware Revision: C
   Firmware Version: 6.62
   Rebuild Priority: Medium
   Expand Priority: Medium
   Surface Scan Delay: 15 secs
   Surface Scan Mode: Idle
   Parallel Surface Scan Supported: No
   Queue Depth: Automatic
   Monitor and Performance Delay: 60  min
   Elevator Sort: Enabled
   Degraded Performance Optimization: Disabled
   Inconsistency Repair Policy: Disabled
   Wait for Cache Room: Disabled
   Surface Analysis Inconsistency Notification: Disabled
   Post Prompt Timeout: 0 secs
   Cache Board Present: True
   Cache Status: Temporarily Disabled
   Cache Status Details: Cache disabled; low batteries.
   Cache Ratio: 25% Read / 75% Write
   Drive Write Cache: Disabled
   Total Cache Size: 512 MB
   Total Cache Memory Available: 400 MB
   No-Battery Write Cache: Disabled
   Cache Backup Power Source: Batteries
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: Failed (Replace Batteries)
   SATA NCQ Supported: True
   Number of Ports: 2 Internal only
   Driver Name: hpsa
   Driver Version: 3.4.10
   Driver Supports HPE SSD Smart Path: True
   PCI Address (Domain:Bus:Device.Function): 0000:04:00.0
   Host Serial Number: CZJ0500GZ9
   Sanitize Erase Supported: False
   Primary Boot Volume: logicaldrive 1 (600508B1001C621D0D82BC612E02B0CC)
   Secondary Boot Volume: None


   Port Name: 1I
         Port ID: 0
         Port Connection Number: 0
         SAS Address: 500143801181F640
         Port Location: Internal

   Port Name: 2I
         Port ID: 1
         Port Connection Number: 1
         SAS Address: 500143801181F644
         Port Location: Internal

   Internal Drive Cage at Port 1I, Box 1, OK
      Power Supply Status: Not Redundant
      Drive Bays: 4
      Port: 1I
      Box: 1
      Location: Internal

   Physical Drives
      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 600 GB, OK)
      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 600 GB, OK)
      physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 600 GB, OK)


   Internal Drive Cage at Port 2I, Box 1, OK
      Power Supply Status: Not Redundant
      Drive Bays: 4
      Port: 2I
      Box: 1
      Location: Internal

   Physical Drives
      physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 600 GB, OK, spare)
      physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 146 GB, OK)
      physicaldrive 2I:1:7 (port 2I:box 1:bay 7, SAS, 146 GB, OK)

   Array: A
      Interface Type: SAS
      Unused Space: 0  MB (0.0%)
      Used Space: 2.2 TB (100.0%)
      Status: OK
      Array Type: Data       Spare Type: dedicated



      Logical Drive: 1
         Size: 1.1 TB
         Fault Tolerance: 1+0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 512 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001C621D0D82BC612E02B0CC
         Disk Name: /dev/sda 
         Mount Points: /boot 2.0 GB Partition Number 2, / 1.0 TB Partition Number 4
         OS Status: LOCKED
         Logical Drive Label: A4E942AA500143801181F6409C3D
         Mirror Group 1:
            physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, OK)
            physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 600 GB, OK)
         Mirror Group 2:
            physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 600 GB, OK)
            physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 600 GB, OK)
         Drive Type: Data
         LD Acceleration Method: Controller Cache

      physicaldrive 1I:1:1
         Port: 1I
         Box: 1
         Bay: 1
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 600 GB
         Drive exposed to OS: False
         Native Block Size: 512
         Rotational Speed: 10500
         Firmware Revision: C380
         Serial Number: 03V0DUYK
         Model: HGST    HUC101860CSS204
         Current Temperature (C): 44
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown
         Sanitize Erase Supported: False

      physicaldrive 1I:1:2
         Port: 1I
         Box: 1
         Bay: 2
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 600 GB
         Drive exposed to OS: False
         Native Block Size: 512
         Rotational Speed: 10500
         Firmware Revision: C380
         Serial Number: 03V0DUBK
         Model: HGST    HUC101860CSS204
         Current Temperature (C): 43
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown
         Sanitize Erase Supported: False

      physicaldrive 1I:1:3
         Port: 1I
         Box: 1
         Bay: 3
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 600 GB
         Drive exposed to OS: False
         Native Block Size: 512
         Rotational Speed: 10500
         Firmware Revision: C380
         Serial Number: 03V0ALZE
         Model: HGST    HUC101860CSS204
         Current Temperature (C): 42
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown
         Sanitize Erase Supported: False

      physicaldrive 1I:1:4
         Port: 1I
         Box: 1
         Bay: 4
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 600 GB
         Drive exposed to OS: False
         Native Block Size: 512
         Rotational Speed: 10500
         Firmware Revision: C380
         Serial Number: 03V0GZYK
         Model: HGST    HUC101860CSS204
         Current Temperature (C): 41
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown
         Sanitize Erase Supported: False

      physicaldrive 2I:1:5
         Port: 2I
         Box: 1
         Bay: 5
         Status: OK
         Drive Type: Spare Drive
         Interface Type: SAS
         Size: 600 GB
         Drive exposed to OS: False
         Native Block Size: 512
         Rotational Speed: 10500
         Firmware Revision: C380
         Serial Number: 03V0DUVK
         Model: HGST    HUC101860CSS204
         Current Temperature (C): 31
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown
         Sanitize Erase Supported: False

lumenok ()
Ответ на: комментарий от Pinkbyte

Не влезло в одно сообщение:

   Array: B
      Interface Type: SAS
      Unused Space: 0  MB (0.0%)
      Used Space: 273.4 GB (100.0%)
      Status: OK
      Array Type: Data 


      Logical Drive: 2
         Size: 136.7 GB
         Fault Tolerance: 1
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 35132
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001CB6B1838687F0171C35F5
         Disk Name: /dev/sdb 
         Mount Points: None
         Logical Drive Label: A5FC5ED9500143801181F6403400
         Mirror Group 1:
            physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 146 GB, OK)
         Mirror Group 2:
            physicaldrive 2I:1:7 (port 2I:box 1:bay 7, SAS, 146 GB, OK)
         Drive Type: Data
         LD Acceleration Method: Controller Cache

      physicaldrive 2I:1:6
         Port: 2I
         Box: 1
         Bay: 6
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 146 GB
         Drive exposed to OS: False
         Native Block Size: 512
         Rotational Speed: 10000
         Firmware Revision: HPDA
         Serial Number: D0A1PAC0GHRK1048
         Model: HP      EG0146FARTR
         Current Temperature (C): 34
         Maximum Temperature (C): 59
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown
         Sanitize Erase Supported: False

      physicaldrive 2I:1:7
         Port: 2I
         Box: 1
         Bay: 7
         Status: OK
         Drive Type: Data Drive
         Interface Type: SAS
         Size: 146 GB
         Drive exposed to OS: False
         Native Block Size: 512
         Rotational Speed: 10000
         Firmware Revision: HPDA
         Serial Number: D0A1PAC0GHJW1048
         Model: HP      EG0146FARTR
         Current Temperature (C): 32
         Maximum Temperature (C): 61
         PHY Count: 2
         PHY Transfer Rate: 6.0Gbps, Unknown
         Sanitize Erase Supported: False


   SEP (Vendor ID PMCSIERA, Model  SRC 8x6G) 250 
      Device Number: 250
      Firmware Version: RevC
      WWID: 500143801181F64F
      Vendor ID: PMCSIERA
      Model: SRC 8x6G

lumenok ()
Ответ на: комментарий от lumenok

батарея сдохла, а может дело в ней...

Естественно.

   Cache Status: Temporarily Disabled
   Cache Status Details: Cache disabled; low batteries.

bigbit ★★★★★ ()
Ответ на: комментарий от lumenok

Плюсую вышеотписавшегося, нормальный контроллер при сдохшей батарее вырубит write cache, чтобы не пролюбить данные при отключении питания

Pinkbyte ★★★★★ ()
Вы не можете добавлять комментарии в эту тему. Тема перемещена в архив.