• 高负载
    • 过多 IO 等待
    • 节点上部署了其它非 K8S 管理的服务
    • 参考资料

    高负载

    TODO 优化

    节点高负载会导致进程无法获得足够的 cpu 时间片来运行,通常表现为网络 timeout,健康检查失败,服务不可用。

    过多 IO 等待

    有时候即便 cpu ‘us’ (user) 不高但 cpu ‘id’ (idle) 很高的情况节点负载也很高,这是为什么呢?通常是文件 IO 性能达到瓶颈导致 IO WAIT 过多,从而使得节点整体负载升高,影响其它进程的性能。

    使用 top 命令看下当前负载:

    1. top - 19:42:06 up 23:59, 2 users, load average: 34.64, 35.80, 35.76
    2. Tasks: 679 total, 1 running, 678 sleeping, 0 stopped, 0 zombie
    3. Cpu(s): 15.6%us, 1.7%sy, 0.0%ni, 74.7%id, 7.9%wa, 0.0%hi, 0.1%si, 0.0%st
    4. Mem: 32865032k total, 30989168k used, 1875864k free, 370748k buffers
    5. Swap: 8388604k total, 5440k used, 8383164k free, 7982424k cached
    6. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    7. 9783 mysql 20 0 17.3g 16g 8104 S 186.9 52.3 3752:33 mysqld
    8. 5700 nginx 20 0 1330m 66m 9496 S 8.9 0.2 0:20.82 php-fpm
    9. 6424 nginx 20 0 1330m 65m 8372 S 8.3 0.2 0:04.97 php-fpm
    10. 6573 nginx 20 0 1330m 64m 7368 S 8.3 0.2 0:01.49 php-fpm
    11. 5927 nginx 20 0 1320m 56m 9272 S 7.6 0.2 0:12.54 php-fpm
    12. 5956 nginx 20 0 1330m 65m 8500 S 7.6 0.2 0:12.70 php-fpm
    13. 6126 nginx 20 0 1321m 57m 8964 S 7.3 0.2 0:09.72 php-fpm
    14. 6127 nginx 20 0 1319m 54m 9520 S 6.6 0.2 0:08.73 php-fpm
    15. 6131 nginx 20 0 1320m 56m 9404 S 6.6 0.2 0:09.43 php-fpm
    16. 6174 nginx 20 0 1321m 56m 8444 S 6.3 0.2 0:08.92 php-fpm
    17. 5790 nginx 20 0 1319m 54m 9468 S 5.6 0.2 0:17.33 php-fpm
    18. 6575 nginx 20 0 1320m 55m 8212 S 5.6 0.2 0:02.11 php-fpm
    19. 6160 nginx 20 0 1310m 44m 8296 S 4.0 0.1 0:10.05 php-fpm
    20. 5597 nginx 20 0 1310m 46m 9556 S 3.6 0.1 0:21.03 php-fpm
    21. 5786 nginx 20 0 1310m 45m 8528 S 3.6 0.1 0:15.53 php-fpm
    22. 5797 nginx 20 0 1310m 46m 9444 S 3.6 0.1 0:14.02 php-fpm
    23. 6158 nginx 20 0 1310m 45m 8324 S 3.6 0.1 0:10.20 php-fpm
    24. 5698 nginx 20 0 1310m 46m 9184 S 3.3 0.1 0:20.62 php-fpm
    25. 5779 nginx 20 0 1309m 44m 8336 S 3.3 0.1 0:15.34 php-fpm
    26. 6540 nginx 20 0 1306m 40m 7884 S 3.3 0.1 0:02.46 php-fpm
    27. 5553 nginx 20 0 1300m 36m 9568 S 3.0 0.1 0:21.58 php-fpm
    28. 5722 nginx 20 0 1310m 45m 8552 S 3.0 0.1 0:17.25 php-fpm
    29. 5920 nginx 20 0 1302m 36m 8208 S 3.0 0.1 0:14.23 php-fpm
    30. 6432 nginx 20 0 1310m 45m 8420 S 3.0 0.1 0:05.86 php-fpm
    31. 5285 nginx 20 0 1302m 38m 9696 S 2.7 0.1 0:23.41 php-fpm

    wa (wait) 表示 IO WAIT 的 cpu 占用,默认看到的是所有核的平均值,要看每个核的 wa 值需要按下 “1”:

    1. top - 19:42:08 up 23:59, 2 users, load average: 34.64, 35.80, 35.76
    2. Tasks: 679 total, 1 running, 678 sleeping, 0 stopped, 0 zombie
    3. Cpu0 : 29.5%us, 3.7%sy, 0.0%ni, 48.7%id, 17.9%wa, 0.0%hi, 0.1%si, 0.0%st
    4. Cpu1 : 29.3%us, 3.7%sy, 0.0%ni, 48.9%id, 17.9%wa, 0.0%hi, 0.1%si, 0.0%st
    5. Cpu2 : 26.1%us, 3.1%sy, 0.0%ni, 64.4%id, 6.0%wa, 0.0%hi, 0.3%si, 0.0%st
    6. Cpu3 : 25.9%us, 3.1%sy, 0.0%ni, 65.5%id, 5.4%wa, 0.0%hi, 0.1%si, 0.0%st
    7. Cpu4 : 24.9%us, 3.0%sy, 0.0%ni, 66.8%id, 5.0%wa, 0.0%hi, 0.3%si, 0.0%st
    8. Cpu5 : 24.9%us, 2.9%sy, 0.0%ni, 67.0%id, 4.8%wa, 0.0%hi, 0.3%si, 0.0%st
    9. Cpu6 : 24.2%us, 2.7%sy, 0.0%ni, 68.3%id, 4.5%wa, 0.0%hi, 0.3%si, 0.0%st
    10. Cpu7 : 24.3%us, 2.6%sy, 0.0%ni, 68.5%id, 4.2%wa, 0.0%hi, 0.3%si, 0.0%st
    11. Cpu8 : 23.8%us, 2.6%sy, 0.0%ni, 69.2%id, 4.1%wa, 0.0%hi, 0.3%si, 0.0%st
    12. Cpu9 : 23.9%us, 2.5%sy, 0.0%ni, 69.3%id, 4.0%wa, 0.0%hi, 0.3%si, 0.0%st
    13. Cpu10 : 23.3%us, 2.4%sy, 0.0%ni, 68.7%id, 5.6%wa, 0.0%hi, 0.0%si, 0.0%st
    14. Cpu11 : 23.3%us, 2.4%sy, 0.0%ni, 69.2%id, 5.1%wa, 0.0%hi, 0.0%si, 0.0%st
    15. Cpu12 : 21.8%us, 2.4%sy, 0.0%ni, 60.2%id, 15.5%wa, 0.0%hi, 0.0%si, 0.0%st
    16. Cpu13 : 21.9%us, 2.4%sy, 0.0%ni, 60.6%id, 15.2%wa, 0.0%hi, 0.0%si, 0.0%st
    17. Cpu14 : 21.4%us, 2.3%sy, 0.0%ni, 72.6%id, 3.7%wa, 0.0%hi, 0.0%si, 0.0%st
    18. Cpu15 : 21.5%us, 2.2%sy, 0.0%ni, 73.2%id, 3.1%wa, 0.0%hi, 0.0%si, 0.0%st
    19. Cpu16 : 21.2%us, 2.2%sy, 0.0%ni, 73.6%id, 3.0%wa, 0.0%hi, 0.0%si, 0.0%st
    20. Cpu17 : 21.2%us, 2.1%sy, 0.0%ni, 73.8%id, 2.8%wa, 0.0%hi, 0.0%si, 0.0%st
    21. Cpu18 : 20.9%us, 2.1%sy, 0.0%ni, 74.1%id, 2.9%wa, 0.0%hi, 0.0%si, 0.0%st
    22. Cpu19 : 21.0%us, 2.1%sy, 0.0%ni, 74.4%id, 2.5%wa, 0.0%hi, 0.0%si, 0.0%st
    23. Cpu20 : 20.7%us, 2.0%sy, 0.0%ni, 73.8%id, 3.4%wa, 0.0%hi, 0.0%si, 0.0%st
    24. Cpu21 : 20.8%us, 2.0%sy, 0.0%ni, 73.9%id, 3.2%wa, 0.0%hi, 0.0%si, 0.0%st
    25. Cpu22 : 20.8%us, 2.0%sy, 0.0%ni, 74.4%id, 2.8%wa, 0.0%hi, 0.0%si, 0.0%st
    26. Cpu23 : 20.8%us, 1.9%sy, 0.0%ni, 74.4%id, 2.8%wa, 0.0%hi, 0.0%si, 0.0%st
    27. Mem: 32865032k total, 30209248k used, 2655784k free, 370748k buffers
    28. Swap: 8388604k total, 5440k used, 8383164k free, 7986552k cached

    wa 通常是 0%,如果经常在 1 之上,说明存储设备的速度已经太慢,无法跟上 cpu 的处理速度。

    使用 atop 看下当前磁盘 IO 状态:

    1. ATOP - lemp 2017/01/23 19:42:32 --------- 10s elapsed
    2. PRC | sys 3.18s | user 33.24s | #proc 679 | #tslpu 28 | #zombie 0 | #exit 0 |
    3. CPU | sys 29% | user 330% | irq 1% | idle 1857% | wait 182% | curscal 69% |
    4. CPL | avg1 33.00 | avg5 35.29 | avg15 35.59 | csw 62610 | intr 76926 | numcpu 24 |
    5. MEM | tot 31.3G | free 2.1G | cache 7.6G | dirty 41.0M | buff 362.1M | slab 1.2G |
    6. SWP | tot 8.0G | free 8.0G | | | vmcom 23.9G | vmlim 23.7G |
    7. DSK | sda | busy 100% | read 4 | write 1789 | MBw/s 2.84 | avio 5.58 ms |
    8. NET | transport | tcpi 10357 | tcpo 9065 | udpi 0 | udpo 0 | tcpao 174 |
    9. NET | network | ipi 10360 | ipo 9065 | ipfrw 0 | deliv 10359 | icmpo 0 |
    10. NET | eth0 4% | pcki 6649 | pcko 6136 | si 1478 Kbps | so 4115 Kbps | erro 0 |
    11. NET | lo ---- | pcki 4082 | pcko 4082 | si 8967 Kbps | so 8967 Kbps | erro 0 |
    12. PID TID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD 1/12
    13. 9783 - 156 0.21s 19.44s 0K -788K 4K 1344K -- - S 4 197% mysqld
    14. 5596 - 1 0.10s 0.62s 47204K 47004K 0K 220K -- - S 18 7% php-fpm
    15. 6429 - 1 0.06s 0.34s 19840K 19968K 0K 0K -- - S 21 4% php-fpm
    16. 6210 - 1 0.03s 0.30s -5216K -5204K 0K 0K -- - S 19 3% php-fpm
    17. 5757 - 1 0.05s 0.27s 26072K 26012K 0K 4K -- - S 13 3% php-fpm
    18. 6433 - 1 0.04s 0.28s -2816K -2816K 0K 0K -- - S 11 3% php-fpm
    19. 5846 - 1 0.06s 0.22s -2560K -2660K 0K 0K -- - S 7 3% php-fpm
    20. 5791 - 1 0.05s 0.21s 5764K 5692K 0K 0K -- - S 22 3% php-fpm
    21. 5860 - 1 0.04s 0.21s 48088K 47724K 0K 0K -- - S 1 3% php-fpm
    22. 6231 - 1 0.04s 0.20s -256K -4K 0K 0K -- - S 1 2% php-fpm
    23. 6154 - 1 0.03s 0.21s -3004K -3184K 0K 0K -- - S 21 2% php-fpm
    24. 6573 - 1 0.04s 0.20s -512K -168K 0K 0K -- - S 4 2% php-fpm
    25. 6435 - 1 0.04s 0.19s -3216K -2980K 0K 0K -- - S 15 2% php-fpm
    26. 5954 - 1 0.03s 0.20s 0K 164K 0K 4K -- - S 0 2% php-fpm
    27. 6133 - 1 0.03s 0.19s 41056K 40432K 0K 0K -- - S 18 2% php-fpm
    28. 6132 - 1 0.02s 0.20s 37836K 37440K 0K 0K -- - S 11 2% php-fpm
    29. 6242 - 1 0.03s 0.19s -12.2M -12.3M 0K 4K -- - S 12 2% php-fpm
    30. 6285 - 1 0.02s 0.19s 39516K 39420K 0K 0K -- - S 3 2% php-fpm
    31. 6455 - 1 0.05s 0.16s 29008K 28560K 0K 0K -- - S 14 2% php-fpm

    在本例中磁盘 sda 已经 100% busy,已经严重达到性能瓶颈。按 ‘d’ 看下是哪些进程在使用磁盘IO:

    1. ATOP - lemp 2017/01/23 19:42:46 --------- 2s elapsed
    2. PRC | sys 0.24s | user 1.99s | #proc 679 | #tslpu 54 | #zombie 0 | #exit 0 |
    3. CPU | sys 11% | user 101% | irq 1% | idle 2089% | wait 208% | curscal 63% |
    4. CPL | avg1 38.49 | avg5 36.48 | avg15 35.98 | csw 4654 | intr 6876 | numcpu 24 |
    5. MEM | tot 31.3G | free 2.2G | cache 7.6G | dirty 48.7M | buff 362.1M | slab 1.2G |
    6. SWP | tot 8.0G | free 8.0G | | | vmcom 23.9G | vmlim 23.7G |
    7. DSK | sda | busy 100% | read 2 | write 362 | MBw/s 2.28 | avio 5.49 ms |
    8. NET | transport | tcpi 1031 | tcpo 968 | udpi 0 | udpo 0 | tcpao 45 |
    9. NET | network | ipi 1031 | ipo 968 | ipfrw 0 | deliv 1031 | icmpo 0 |
    10. NET | eth0 1% | pcki 558 | pcko 508 | si 762 Kbps | so 1077 Kbps | erro 0 |
    11. NET | lo ---- | pcki 406 | pcko 406 | si 2273 Kbps | so 2273 Kbps | erro 0 |
    12. PID TID RDDSK WRDSK WCANCL DSK CMD 1/5
    13. 9783 - 0K 468K 16K 40% mysqld
    14. 1930 - 0K 212K 0K 18% flush-8:0
    15. 5896 - 0K 152K 0K 13% nginx
    16. 880 - 0K 148K 0K 13% jbd2/sda5-8
    17. 5909 - 0K 60K 0K 5% nginx
    18. 5906 - 0K 36K 0K 3% nginx
    19. 5907 - 16K 8K 0K 2% nginx
    20. 5903 - 20K 0K 0K 2% nginx
    21. 5901 - 0K 12K 0K 1% nginx
    22. 5908 - 0K 8K 0K 1% nginx
    23. 5894 - 0K 8K 0K 1% nginx
    24. 5911 - 0K 8K 0K 1% nginx
    25. 5900 - 0K 4K 4K 0% nginx
    26. 5551 - 0K 4K 0K 0% php-fpm
    27. 5913 - 0K 4K 0K 0% nginx
    28. 5895 - 0K 4K 0K 0% nginx
    29. 6133 - 0K 0K 0K 0% php-fpm
    30. 5780 - 0K 0K 0K 0% php-fpm
    31. 6675 - 0K 0K 0K 0% atop

    也可以使用 iotop -oPa 查看哪些进程占用磁盘 IO:

    1. Total DISK READ: 15.02 K/s | Total DISK WRITE: 3.82 M/s
    2. PID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
    3. 1930 be/4 root 0.00 B 1956.00 K 0.00 % 83.34 % [flush-8:0]
    4. 5914 be/4 nginx 0.00 B 0.00 B 0.00 % 36.56 % nginx: cache manager process
    5. 880 be/3 root 0.00 B 21.27 M 0.00 % 35.03 % [jbd2/sda5-8]
    6. 5913 be/2 nginx 36.00 K 1000.00 K 0.00 % 8.94 % nginx: worker process
    7. 5910 be/2 nginx 0.00 B 1048.00 K 0.00 % 8.43 % nginx: worker process
    8. 5896 be/2 nginx 56.00 K 452.00 K 0.00 % 6.91 % nginx: worker process
    9. 5909 be/2 nginx 20.00 K 1144.00 K 0.00 % 6.24 % nginx: worker process
    10. 5890 be/2 nginx 48.00 K 692.00 K 0.00 % 6.07 % nginx: worker process
    11. 5892 be/2 nginx 84.00 K 736.00 K 0.00 % 5.71 % nginx: worker process
    12. 5901 be/2 nginx 20.00 K 504.00 K 0.00 % 5.46 % nginx: worker process
    13. 5899 be/2 nginx 0.00 B 596.00 K 0.00 % 5.14 % nginx: worker process
    14. 5897 be/2 nginx 28.00 K 1388.00 K 0.00 % 4.90 % nginx: worker process
    15. 5908 be/2 nginx 48.00 K 700.00 K 0.00 % 4.43 % nginx: worker process
    16. 5905 be/2 nginx 32.00 K 1140.00 K 0.00 % 4.36 % nginx: worker process
    17. 5900 be/2 nginx 0.00 B 1208.00 K 0.00 % 4.31 % nginx: worker process
    18. 5904 be/2 nginx 36.00 K 1244.00 K 0.00 % 2.80 % nginx: worker process
    19. 5895 be/2 nginx 16.00 K 780.00 K 0.00 % 2.50 % nginx: worker process
    20. 5907 be/2 nginx 0.00 B 1548.00 K 0.00 % 2.43 % nginx: worker process
    21. 5903 be/2 nginx 36.00 K 1032.00 K 0.00 % 2.34 % nginx: worker process
    22. 6130 be/4 nginx 0.00 B 72.00 K 0.00 % 2.18 % php-fpm: pool www
    23. 5906 be/2 nginx 12.00 K 844.00 K 0.00 % 2.10 % nginx: worker process
    24. 5889 be/2 nginx 40.00 K 1164.00 K 0.00 % 2.00 % nginx: worker process
    25. 5894 be/2 nginx 44.00 K 760.00 K 0.00 % 1.61 % nginx: worker process
    26. 5902 be/2 nginx 52.00 K 992.00 K 0.00 % 1.55 % nginx: worker process
    27. 5893 be/2 nginx 64.00 K 972.00 K 0.00 % 1.22 % nginx: worker process
    28. 5814 be/4 nginx 36.00 K 44.00 K 0.00 % 1.06 % php-fpm: pool www
    29. 6159 be/4 nginx 4.00 K 4.00 K 0.00 % 1.00 % php-fpm: pool www
    30. 5693 be/4 nginx 0.00 B 4.00 K 0.00 % 0.86 % php-fpm: pool www
    31. 5912 be/2 nginx 68.00 K 300.00 K 0.00 % 0.72 % nginx: worker process
    32. 5911 be/2 nginx 20.00 K 788.00 K 0.00 % 0.72 % nginx: worker process

    通过 man iotop 可以看下这几个参数的含义:

    1. -o, --only
    2. Only show processes or threads actually doing I/O, instead of showing all processes or threads. This can be dynamically toggled by pressing o.
    3. -P, --processes
    4. Only show processes. Normally iotop shows all threads.
    5. -a, --accumulated
    6. Show accumulated I/O instead of bandwidth. In this mode, iotop shows the amount of I/O processes have done since iotop started.

    节点上部署了其它非 K8S 管理的服务

    TODO 优化

    比如在节点上装了数据库,但不被 K8S 所管理,这是用法不正确,不建议在 K8S 节点上部署其它进程。

    参考资料

    • Linux server performance: Is disk I/O slowing your application: https://haydenjames.io/linux-server-performance-disk-io-slowing-application/