使用pypiserver快速搭建内网离线pypi仓库实践

前言

本文介绍了如何快速搭建一个pypiserver，通过自建pypiserver，我们可以解决网络环境不好，或者离线无法安装python包的问题。如果结合最新的GitLab CI/CD和pipenv我相信各位还可以玩出更多的花样。

pypiserver - minimal PyPI server for use with pip/easy_install

更新记录

2018年04月12日 - 初稿

阅读原文 - https://wsgzao.github.io/post...

扩展阅读

pypiserver - https://github.com/pypiserver...

pypiserver 简介

pypiserver is a minimal PyPI compatible server for pip or easy_install. It is based on bottle and serves packages from regular directories. Wheels, bdists, eggs and accompanying PGP-signatures can be uploaded either with pip, setuptools, twine, pypi-uploader, or simply copied with scp.

pypiserver 服务端配置

如果你的Linux环境缺少Python 2.7可以参考我的文章直接离线升级至最新版本

Python 2.6 升级至 Python 2.7 的实践心得
https://wsgzao.github.io/post...

pypiserver > 1.2.x works with python 2.7 and 3.3+ or pypy. Older python-versions may still work, but they are not tested. For legacy python versions, use pypiserver-1.1.x series.

# 替换pip为阿里云，感概豆瓣的时代已经过去
tee ~/.pip/pip.conf <<-'EOF'
[global]
index-url = https://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host= mirrors.aliyun.com
EOF

# 直接在线安装pypiserver
pip install pypiserver
# 离线下载pypiserver
mkdir /tmp/pypiserver
pip install -d /tmp/pypiserver/ pypiserver

# Copy packages into this directory.
mkdir ~/packages
# Copy some packages into your ~/packages folder and then get your pypiserver up and running:
pypi-server -p 8080 ~/packages &

pypiserver 客户端配置

## Download and Install hosted packages.
pip install  --extra-index-url http://localhost:8080/simple/ ...

# or
pip install --extra-index-url http://localhost:8080

## Search hosted packages
pip search --index http://localhost:8080 ...

# 个人推荐的配置
tee ~/.pip/pip.conf <<-'EOF'
[global]
index-url = http://172.28.70.126/simple
extra-index-url = https://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host = 172.28.70.126
EOF

pypiserver 进阶配置

pypiserver Running as a systemd service

https://github.com/pypiserver...

Adjusting the paths and adding this file as pypiserver.service into your systemd/system directory will allow management of the pypiserver process with systemctl, e.g. systemctl start pypiserver.

More useful information about systemd can be found at https://www.digitalocean.com/...

# 安装需要的包
yum install nginx -y
pip install passlib pypiserver gunicorn

# 创建pypiserver服务方便服务启停管理
tee /usr/lib/systemd/system/pypiserver.service <<-'EOF'
[Unit]
Description=gunicorn daemon
After=network.target

[Service]
PIDFile=/run/pypiserver.pid
ExecStart=/usr/local/bin/gunicorn -w16 \
    --pid /run/pypiserver.pid \
    -b :10012 \
    'pypiserver:app(root="/var/www/pypi")'
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID

[Install]
WantedBy=multi-user.target 
EOF

# Warning: pypiserver.service changed on disk. Run 'systemctl daemon-reload' to reload units.
systemctl daemon-reload

# 启动pypiserver服务
systemctl enable pypiserver.service
systemctl start pypiserver.service
systemctl status pypiserver.service

# 停止pypiserver服务
systemctl disable pypiserver.service
systemctl stop pypiserver.service
systemctl status pypiserver.service

[root@centos7 run]# systemctl status pypiserver.service
● pypiserver.service - gunicorn daemon
   Loaded: loaded (/usr/lib/systemd/system/pypiserver.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-04-13 15:14:08 CST; 859ms ago
 Main PID: 22524 (gunicorn)
   CGroup: /system.slice/pypiserver.service
           ├─22524 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22530 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22531 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22532 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22533 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22534 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22535 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22536 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22537 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22538 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           ├─22539 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
           └─22540 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")

Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22531] [INFO] Booting worker with pid: 22531
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22532] [INFO] Booting worker with pid: 22532
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22533] [INFO] Booting worker with pid: 22533
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22534] [INFO] Booting worker with pid: 22534
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22535] [INFO] Booting worker with pid: 22535
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22536] [INFO] Booting worker with pid: 22536
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22537] [INFO] Booting worker with pid: 22537
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22538] [INFO] Booting worker with pid: 22538
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22539] [INFO] Booting worker with pid: 22539
Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22540] [INFO] Booting worker with pid: 22540
[root@centos7 run]# netstat -tnlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      1517/dnsmasq        
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      977/sshd            
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      978/cupsd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1383/master         
tcp        0      0 127.0.0.1:6011          0.0.0.0:*               LISTEN      19378/sshd: root@pt 
tcp        0      0 0.0.0.0:10012           0.0.0.0:*               LISTEN      22524/python        
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd           
tcp6       0      0 :::22                   :::*                    LISTEN      977/sshd            
tcp6       0      0 ::1:631                 :::*                    LISTEN      978/cupsd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1383/master         
tcp6       0      0 ::1:6011                :::*                    LISTEN      19378/sshd: root@pt 

# 检查pypiserver服务
cd /var/www/pypi
# 向仓库中添加python package
[root@centos7 pypi]# pip download pypiserver
Collecting pypiserver
  Downloading https://mirrors.aliyun.com/pypi/packages/d7/78/5772432dad2b9e754ab92f4d301fa507069b9decc8c943c1b18c2043ff4f/pypiserver-1.2.1-py2.py3-none-any.whl (83kB)
    100% |████████████████████████████████| 92kB 643kB/s 
  Saved ./pypiserver-1.2.1-py2.py3-none-any.whl
Successfully downloaded pypiserver

[root@centos7 pypi]# ll
total 84
-rw-r--r-- 1 root root 83529 Apr 13 14:55 pypiserver-1.2.1-py2.py3-none-any.whl

# 搜索刚才下载的package
[root@centos7 pypi]# pip search -i http://127.0.0.1:10012 pypiserver
pypiserver (1.2.1)  - 1.2.1
  INSTALLED: 1.2.1 (latest)

# 配置nginx做反向代理
tee /etc/nginx/conf.d/pypi.conf <<-'EOF'
upstream pypiserver {
  server 127.0.0.1:10012;
}

server {
  listen 10087;

  # disable any limits to avoid HTTP 413 for large package uploads
  client_max_body_size 0;

  location / {
    proxy_pass http://pypiserver/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # When setting up pypiserver behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_buffering off;
    proxy_request_buffering off;
  }

  location /packages/ {
    alias /var/www/pypi; # static file
  }
}
EOF

# 启动nginx
systemctl enable nginx
systemctl start nginx
systemctl status nginx

# 检查nginx服务
pip search -i http://172.28.79.126:10087 pypiserver
pypiserver (1.2.1)  - 1.2.1
  INSTALLED: 1.2.1 (latest)