avatar

目录
Ubuntu 安装Scrapy_Redis环境


打算用Ubuntu 装一个环境,测试一次 Scrapy_Redis 分布式爬虫

进入正文

先切到 root下

powershell
1
kb@kb:/home/kb# su root

更换apt源

powershell
1
root@kb:/home/kb# vim /etc/apt/sources.list

把文件中的系统源全部注释

切换为下面的阿里源

shell
1
2
3
4
5
6
7
8
9
10
deb http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse

在执行更新

powershell
1
root@kb:/home/kb# apt-get update

安装Python3.6

先接入ppa源

powershell
1
root@kb:/home/kb# sudo add-apt-repository ppa:deadsnakes/ppa

在执行更新

powershell
1
root@kb:/home/kb# sudo apt update

安装Python3.6

powershell
1
root@kb:/home/kb# sudo apt install python3.6

删除系统默认的python3.4路径

powershell
1
root@kb:/home/kb# sudo rm /usr/bin/python3

替换

powershell
1
root@kb:/home/kb# sudo ln -s /usr/bin/python3.6 /usr/bin/python3

安装setuptools

powershell
1
2
3
4
5
6
7
8
9
root@kb:/home/kb# wget --no-check-certificate https://pypi.python.org/packages/source/s/setuptools/setuptools-19.6.tar.gz#md5=c607dd118eae682c44ed146367a17e26

root@kb:/home/kb# tar -zxvf setuptools-19.6.tar.gz

root@kb:/home/kb# cd setuptools-19.6

root@kb:/home/kb/setuptools-19.6# python3 setup.py build

root@kb:/home/kb/setuptools-19.6# python3 setup.py install

安装pip

powershell
1
2
3
4
5
6
7
8
9
root@kb:/home/kb# wget --no-check-certificate  https://pypi.python.org/packages/source/p/pip/pip-8.0.2.tar.gz#md5=3a73c4188f8dbad6a1e6f6d44d117eeb

root@kb:/home/kb# tar -zxvf pip-8.0.2.tar.gz

root@kb:/home/kb# cd pip-8.0.2

root@kb:/home/kb/pip-8.0.2# python3 setup.py build

root@kb:/home/kb/pip-8.0.2# python3 setup.py install

更换Pip源

powershell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
root@kb:/home/kb# cd /
root@kb:/# mkdir ~/.pip
root@kb:/# cd ~/.pip
root@kb:~/.pip# vim pip.conf

点击“i”键,进入编辑模式,复制信息:
[global]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple
trusted-host = pypi.tuna.tsinghua.edu.cn

这个更换的是清华的源,清华的源5分钟同步官网一次。
清华大学 https://pypi.tuna.tsinghua.edu.cn/simple/
阿里云 http://mirrors.aliyun.com/pypi/simple/
中国科技大学 https://pypi.mirrors.ustc.edu.cn/simple/
豆瓣(douban) http://pypi.douban.com/simple/
中国科学技术大学 http://pypi.mirrors.ustc.edu.cn/simple/

点击:“ESC”切换到命令行模式,输入“:wq”保存离开

Ubuntu安装Python虚拟环境

更新pip版本

powershell
1
2
root@kb:~/.pip# cd/home/kb
root@kb:/home/kb# pip3 install --upgrade pip

安装pip six 模块

powershell
1
root@kb:/home/kb# sudo pip install six --upgrade --ignore-installed six
powershell
1
2
3
root@kb:/home/kb# pip3 install virtualenv  或者 sudo apt-get install python-virtualenv 

root@kb:/home/kb# pip3 install virtualenvwrapper

配置环境变量

powershell
1
root@kb:/home/kb# sudo vim ~/.bashrc

shift+g 到达文本最后插入

Code
1
2
3
4
export WORK_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
export VIRTUALENVWRAPPER_VIRTUALENV=/usr/local/bin/virtualenv
source /usr/local/bin/virtualenvwrapper.sh

执行 bashrc

powershell
1
root@kb:/home/kb# source ~/.bashrc

如果执行 bashrc 出现下图这个错误

那就修改环境变量

Code
1
2
3
4
vim /etc/profile
#在最后加上
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
export VIRTUALENVWRAPPER_VIRTUALENV=/usr/local/bin/virtualenv

安装Python虚拟环境

powershell
1
root@kb:/home/kb# mkvirtualenv -p /usr/bin/python3 scrapy_env

物理机环境下,导出项目库

powershell
1
pip freeze > 文本名字

如果库中有 pywin32 这些库。先手动删除。

把 导出的库文本。复制到 ubuntu中

进入 工作环境

powershell
1
root@kb:/home/kb# workon scrapy_env

ubuntu上安装scrapy之前,需要先安装以下依赖:

powershell
1
(scrapy_env) root@kb:/home/kb# sudo apt-get install python3-dev build-essential python3-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev

安装 cmake 不然安装 dlib库 会报错

powershell
1
2
3
4
(scrapy_env) root@kb:/home/kb# pip3 install cmake
两个都要装
(scrapy_env) root@kb:/home/kb# sudo apt-get install cmake
(我装了这两个还是安装dlib失败,琢磨了半个小时还是不行,后面就把dlib从文件中删除了,反正这次不用他。)

安装 requirements.txt 的库

powershell
1
(scrapy_env) root@kb:/home/kb# pip3 install -r requirements.txt

安装redis

powershell
1
(scrapy_env) root@kb:/home/kb# sudo apt-get install redis-server

查看是否启动

powershell
1
(scrapy_env) root@kb:/home/kb# ps -aux|grep redis

输出信息

Code
1
2
redis     14770  0.0  0.6  44048  6652 ?        Ssl  13:07   0:00 /usr/bin/redis-server 127.0.0.1:6379       
root 14812 0.0 0.2 15964 2068 pts/3 S+ 13:09 0:00 grep --color=auto redis

Windows安装redis

下载链接:

https://github.com/microsoftarchive/redis/releases

下载msi版本的。可以进行安装

下载好后,双击打开,一直Next,到了这里,把勾打上,这样子就可以添加为环境变量,然后一路Next就好了。

安装好后目录,那个Zip压缩包是我下载多了的

Windows下启动redis

Code
1
G:\redis>redis-server.exe redis.windows.conf

输出命令

Code
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
λ redis-server redis.windows.conf
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 3.2.100 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 4860
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'

[4860] 20 Jan 14:02:00.505 # Server started, Redis version 3.2.100
[4860] 20 Jan 14:02:00.506 * DB loaded from disk: 0.000 seconds
[4860] 20 Jan 14:02:00.506 * The server is now ready to accept connections on port 6379

测试

在打开一个cmd,输入

Code
1
2
C:\Users\Administrator>redis-cli
127.0.0.1:6379>

下载 Redis Desktop Manager

github地址:https://github.com/uglide/RedisDesktopManager/releases

百度网盘:http://pan.baidu.com/s/1kU8sY3P

文件名称:redis-desktop-manager-0.8.8.384.exe

一路傻瓜式安装就可以了

新建连接 Connect to Redis Server

文章作者: KeyboArd
文章链接: https://www.wrpzkb.cn/Ubuntu-Python/
版权声明: 本博客所有文章除特别声明外,均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自 KeyboArd's Blog
打赏
  • 微信
    微信
  • 支付寶
    支付寶

评论