CentOS6.8安装scrapy爬虫框架

背景

最近想搞一个个人项目,需要用到爬虫,所以接触了scrapy。

环境参数

  • OS环境
    CentOS6.8 x86_64
  • OS默认python版本
    python2.6

安装问题

在默认python版本环境(python2.6)下安装了pip,通过pip install scrapy 结果运的时候出现一下错误:

提示 scrapy1.4.0 依赖python2.7版本

[root@iZwz9e75q2nzsxqdr0ll5y) l 0 u s M M 5 cZ ~]# python
Python 2.6.6Q q 5 _ (r266:84292, Aug 18 2016, 15:13:37)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
Type "help", "copyright", "credits" or "licenseT & 5 M" for more information.
>>> import scrapy, % I f n
Scrapy 1.4.0 requires Python 2.7
[root@iZwz9e75q2nzsxqdr0ll5yZ} [  C P ~]#

简单,我们去官网看下,果然是的。

最新版(1.4.0)的scrapy需要python2.7及以上或python3.3以上的版本才支* . % { K K持。

CentOS6.8安装scrapy爬虫框架

解决方法

怎么解决?来,我们有两种方法解决这个问题。

方法一: 安装的时候选择版本

大不了我们不安装最新版本,安装旧版本不就了吗。
验了一Y U W 7 n下scrapy 1.0.0 及以上+ | E E B都依赖python2.7。

[root@iZwz9e75q2nzsxqdr0ll5yZe ( A 5 3 ] ) distribut* ; + ( + 1 F 2 (e-0.6.10]# pip install scrapy==0.9.0
You are using pip version 7.1.0, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' cy T ; N | z Xommand.
Collecting scrt d C Napy==0.9.0
Downloading http://mirrors.aliyun.com/pypi/packageX  & q ) V n ! Ks/1a/70/fc8948afda8349cfd6dbd099da59248fdfde1eeN n K ? W R f658af63fc046} P u43a928db3/Scrapy-0.9.tar.gz (79 ^ 6 P71kB)
100% |████████████████████████████████| 777 K K !4kB 2.2MB/s
Requirement already satisfied (use -y 0 h-upgrade to upgrade): TwO 3 F Misted>=2.5 in
...(省略)...
/usr/lib/python2.6/site-packages (from Automat>=0.3.0->Twisted>=2.5->scrapy==0.9.0)
Installing colq 8 [ U } 9lected packages: scrapy
Running setup.py install for scrapy
Successfully installed scrapy-0.9

安装成功,但是还是要import确认一下

[root@iZwz9e75q2nzsxqdr0ll5yZ distributB 9 he-0.6.10]# python
Python 2.6.6 (r266:84292, Aug 18 201` . P = V 6, 15:13:37)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
Type "help",  _ ? Z m L J u"copyrightM z Y", "credits" or "license" for more information.
&go M x & : r &t;>> import scrapyG t 2 z v
/usr/lib6} ; w ) S 3 t D4/python2.6/site-pac. o k J vkages/cryptography/__init__.py:26: Depr} L NecationWarning: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. The next version of cryp. k ftography will drop support for Python 2.6
Dv H A T j R 5eprecationWarning
>>>

OK,没问题; l | T e 3 z

只是有个提示,python2.6 快成为弃儿了,叫我们赶紧升级。= ? D c ^ g !

/usr/N V - W G %lib64/python2.6/site-packages/cryptogrV f Q M A 2 1 #aphy/init.py:2J t {6: DeprecationWarning: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. The next versj } ?ion of cryptogs ( L sra$ k ? U e sphy will drop support for Python 2.6
DeprecationWarning

方法二: 升级python到2.7版本

我个人使用软件包,一般情况下都会选择最新版,我认为最新版若R ! 1 & { S不是最好也是较好的,除非官方提示特殊情况。

来,那我们就看看升级python到2.7然后再安装scrapy1.4.0(最新版)的方法。

  • 安装python2.7

2.7.14 是python2 的最新版

[R P 0 [ L [ b ` 2root@iZwz9e75q2nzsxqdr0ll5yZ ~]# ht~ 3 w $ a J b xtps://www.python.org/ftp/python/2.7.14/Python-2.7.14.tgz
[root@iZwz9e75q2nzsx3 K ) ` 2 q {qdr0ll5yZ ~]# tar -zxvf Python-2.7.14.tg, ` 6 ; 2 ` * :z && cd Python-2.7.14 && ./configure && make all &&` ; p G x L n 7 make install
...(省略)...
[root@iZwz9e75q2nzsxqdr0ll5yZ ~]# cd /usr/bin/; rm -f python &s + 4  { Gamp;& ln -s /usr/local/bin/python2.7 python
(如果不创建软链接的话,运python2.7的时候要用python2.7+ + * w ? 3 * q命令,脚本头要写 #!/usr/bin/python2.7)
[root@iZwz9e75q2nzsxqdr0ll5yZ bin]# ll python
lrwxrwK d P n Nxrwx 1 root root 24 11月 23 14:42 pytho& [ D 7 5 B Xn ->P K & d j Z ` ? m /usr/local/bin/python2.7

注:1. 要把 /usr/bin/yum 中的 #!/usr/bin/python 改成 #!/usr/bin/@ T E u / R ] k fpython2.6

  • 安装 setuptools 最新版
[root@iZwz9e75q2nzsxqdr0ll5yZ ~, f w K x | * ?]# wget -O setuptools-37.0.0.zip  https://pypi.python.org/packages/7c/cb/bdfbb0b6a56459d5461768de824d4f40ec4c4v w ~ Wc778f3a8fb0b84c25f{ p a I w @ ; 3 _03b68/setuptools-37.0.0.zip#md5=f905ca70d2db37b7284c0f6314ab6814
[root@iZwz9e75q2nzsxqdr0ll5yZ ~]# unzip setuptools-37.0.0.zip && cd sQ  1 R 2 : G detuptools-37.0.0
[root@iZwz9e75q2nzsxqdr0ll5yZ setuptools-37.0.0]# python setup.py install
(或者python2.7 setup.py install)

注:安装成功之后,会看到以下目录

[root@iZwz9e75q2nzsxqdr0ll5yZ ~]# ls -d /usr/local/lib/pyt& - Q i $ m Vhon2.7/site-packages/setuptools*
/usr/local/lib/python2.7/siteN - G ? c E B e-packages/setuptools-37.0.0-py2.7.egg /usr/local/lib/python2.7/site-packages/setuptools.pth
[root@iZwz9e75q2nzsxqdr0ll5yZ ~]#

  • 安装pip最新版
[root@iZwM f  U $ M gz9e75q2nzsxqdr0ll5yZ ~]# wget -O pip-9.0.1.tar.gz https://pypi.python.org/packa+ . Cges/11/b6/abcb525026a4be042b486df439r ^ Y05d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz#md5=35f01da33009719497f01a4ba69d63c9
[root@iZwz9e75q4 5 [2nzsxqdr0ll5yZ ~]# tar -zxW g 2 T @ Q H Cvf pip-9.0.1.tar.gz && cd pip-k z * H : i # U e9.0.1
[root@iZwz9e75q2nzsxqdr0ll5yZ pip-9.0.1]# python setupn ? | * L  } 7 3.py install
(或者python2.7 setup.py ine - /stall)

注:安装成功之后,会看到以下目录

[root@iZwz9e75q2nzsxqdr0ll5yZ ~]# ls -d /usr/local/lib/pb * t 5 8 Tython2.7/site-packages/pip*
/usr/local/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg

  • 安装scrapy
[root@iZwz9e75q2nzsxqdr0ll5yZ ~]# pip install scrapy
Collecting Twisted>=13.E Y L g r g N1.0 (from scrapy)
Could not find a version that satisf# % ) B *ies the reg F g I o ^ Wquiremv | k q % W 8ent Twisted>=13.1.0 (from scrapy) (from versiE 7 E A } (ons: )N = . [ #
No matching distribution found for TwisteQ w rd>=13.1.0 (from scrH e I ` zapy)
wget https://twistedmatrix.com/ReleaB % p H Sses/Twisted/17.9/Twisted-17.9.0.tar.bz2

以上错误表示scrapy依赖Twisted 13H c i n b [ 9.1.0及以上的版本,但是pip 找不到这个版本p H . k(TwE C J 6 - * 5 a Visted是用Python实现的基于事件驱动的网络引擎框架),只能[ j l p Z o ! `手动下载这个版本的安装包来安装了。

[root@iZwz9e75q2nzsxqdr0ll5+ 5 @ i )yZ ~]# wg5 s z m X A 4 i ret -O Twisted-17.9.0.tar.bz2  https://pypi.pyb D OthV q bon.org/L u -packages/a2/3D : [ P7/298f9547606c453 k Q s s n %d75aa9792369302cJ 4 k g Tc63aa4bbcf7b5f607560180dd099d2/Twis6 f Yted-17.9.0.tar.bz2#md5=6dbedb918f0c7288a4| + J ) 0c670f59393ecf8
[root@iZwz9e75q2nl G J 5zsxqdr0ll5yZ ~]# tar -jxvf Twisted-17.9.0.tar.bz2 && cd Twisted-17.9.0
[root@iZ9 n @ 4 q m Rwz9e75q2nzsxqdr0ll5yZ Twisted-17.9.0]# python setup.py install
(或者python2.7 setup.py install)

然后再安装 scrapy

[root@iZwz9e75q2nzsxqdr0ll5yZ ~]# /usr/local/bin/po U + 8 % I O O oip install scrapy
Collecting scrapy
Downloading http://mirrors.alY M 2 o n i o j Iip h w & x D Zyun.com/pypi/packa~ V ! i / ) Bges/a8/969 X 6/3affe11cf53a5d2105536919113d5b453479038bb486f7387f4ce4a3b83f/Scrapy-1.4h ( c # W.0-py2.py3-none-any.whl (248kB)
100%O s b U |████████████████████████████████| 256kB 2.1MB/s
R9 ) q B H zequirement ag  q 0 H Y V B olready satisfied: queuelib in /usr/lr P ~ U k 2 }ocal/lib/python2.7/site-packages? # 7 9 s A : F (from scrapy)
...(省略)...
Requirement alB o Q X Y l w Y read Q j g 5 ~ Uy satisfied: pycparser in /usr/local/lib/python2.7/site-packages (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=1.9->pyOpenSSL->scrapy)
Installing collected packa| L r [ Z B 2ges: scrapy
Successfully installed scrapy-1.4.0
[root@i8 R RZwz9e75q2nzsxqdr0ll5yZ ~]#

验证import,很完美,没有任何告警

[root@iZwz9e75q2nzsxqdr0ll5yZ ~]# python
Python 2.7.14 (default, Nov 23 2017, 13:28:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux2
Type "help", "copyright", "credits" or "lice) z { {nse" for more informat, _ 5ion.
>>> impY z ;ort scrapy
>>>