记一次Django reload导致 CPU飙升+大量stat系统调用的解决方案

背景：

发现app server在测试机经常CPU飙升到接近100%

strace发现大量的stat系统调用：

[root@app02 ~]#  strace -p 2205 -C
...
stat("/home/ouyang/storyapi/xb_handlers/support/system/get_pop_ups_handler.py", {st_mode=S_IFREG|0644, st_size=4094, ...}) = 0
stat("/home/ouyang/storyapi/xb_dao/support/share/base/user_share_newpull_dao.py", {st_mode=S_IFREG|0644, st_size=3695, ...}) = 0
stat("/usr/lib/python2.7/site-packages/urllib3/exceptions.py", {st_mode=S_IFREG|0644, st_size=6603, ...}) = 0
stat("/home/ouyang/storyapi/xb_dao/pay/comic/record/__init__.py", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
stat("/usr/lib/python2.7/site-packages/urllib3/packages/__init__.py", ^Cstrace: Process 6134 detached
 <detached ...>
% time     seconds  usecs/call     calls    errors syscall
  ---- ----------- ----------- --------- --------- ----------------
▽96.53   20.122361       17332      1161       252 futex
  2.95    0.614242           2    328764       477 stat
  0.35    0.073086         163       448           read
  0.06    0.013246         166        80           select
  0.04    0.008513           5      1721      1388 open
  0.02    0.004120           2      1780           getdents
  0.01    0.002841           5       531           fstat
  0.01    0.002252           3       890           openat
  0.01    0.001959           2      1226           close
  0.01    0.001300           1       883           lstat
  0.01    0.001283           7       174           munmap
  0.00    0.000485           3       191           mmap
  0.00    0.000161           3        51           lseek
  0.00    0.000080           2        34           brk
  0.00    0.000039          39         1           clone
  0.00    0.000014           3         5           mprotect
  0.00    0.000005           1         5         5 ioctl
  0.00    0.000000           0         6         6 access
  0.00    0.000000           0        12           fcntl
------ ----------- ----------- --------- --------- ----------------
100.00   20.845987                337963      2128 total
[root@app02 ~]#

分析：

1、业务代码并没有很多针对文件相关的操作。
2、看stat系统调用各个代码文件都有涉及到，怀疑是django的reload模块导致。

尝试：启动服务器时增加--noreload选项

[root@app02 ~]#    cat /etc/supervisord.d/app.ini 
[program:app]
user=root
environment= PATH="/usr/bin"
directory=/home/ouyang/storyapi
command=python manage.py runserver 0.0.0.0:8002 --nothreading --noreload
redirect_stderr=true
stdout_logfile=/home/ouyang/storyapi/log/super.log
stderr_logfile=/home/ouyang/storyapi/log/super.err
startsecs=1
stopwaitsecs=1
stopasgroup=true

结果：

1、问题解决，没有大量的stat系统调用了；
2、CPU稳定在1%。