Nutch-Hadoop-MongoDB搭建分布式爬虫

  • Nutch-Hadoop-MongoDB搭建分布式爬虫已关闭评论
  • 353 views
  • A+
所属分类:运维实战

一、实现目标

使用NutchHadoopMongoDB实现一个简单的分布式爬虫,在Hadoop上运行Nutch爬虫抓取网页,存储到MongoDB中。

二、实验环境

CentOS7 Linux x86_64

JDK 1.8.0_161

mongodb 2.6.12-6

hadoop 2.9.1

apache-ant-1.9.4

apache-nutch-2.3.1

三、安装Oracle JDK

可参考《Nutch-MongoDB-ElasticSearch搭建搜索引擎》:https://www.toutiao.com/i6539542640034054663/

四 、安装配置MongoDB

可参考《Nutch-MongoDB-ElasticSearch搭建搜索引擎》:https://www.toutiao.com/i6539542640034054663/

五、安装配置Hadoop

Nutch-Hadoop-MongoDB搭建分布式爬虫

5.1下载Hadoop 2.9.1:http://hadoop.apache.org/releases.html

5.2 设置环境变量

/etc/profile:

export HADOOP_HOME=/home/vminger/workspace/sysapp/hadoop/hadoop-2.9.1

export PATH=$PATH:$HADOOP_HOME/bin

export PDSH_RCMD_TYPE=ssh

source /etc/profile

eval "$(ssh-agent -s)"

ssh-add

5.3 配置HDFS

etc/hadoop/core-site.xml:

Nutch-Hadoop-MongoDB搭建分布式爬虫

etc/hadoop/hdfs-site.xml:

Nutch-Hadoop-MongoDB搭建分布式爬虫

5.4 初始化HDFS

bin/hdfs namenode -format

5.5 启动HDFS

sbin/start-dfs.sh

5.6 配置YARN

etc/hadoop/mapred-site.xml:

Nutch-Hadoop-MongoDB搭建分布式爬虫

etc/hadoop/yarn-site.xml:

Nutch-Hadoop-MongoDB搭建分布式爬虫

5.7 启动YARN

sbin/start-yarn.sh

5.8 jps查看HDFS和YARN进程

Nutch-Hadoop-MongoDB搭建分布式爬虫

六、安装配置Nutch

Nutch-Hadoop-MongoDB搭建分布式爬虫

6.1下载apache-ant-1.9.4-bin.tar.gz,并解压,下载地址:

https://archive.apache.org/dist/ant/binaries/binaries/apache-ant-1.9.4-bin.tar.gz

6.2 设置ant环境变量

/etc/profile:

export ANT_HOME=/home/vminger/workspace/sysapp/ant/apache-ant-1.9.4

export PATH=$PATH:$ANT_HOME/bin

source /etc/profile

6.3 下载apache-nutch-2.3.1-src.tar.gz,并解压,下载地址:

http://nutch.apache.org/downloads.html

6.4 设置nutch环境变量

/etc/profile:

export NUTCH_HOME=/home/vminger/workspace/sysapp/nutch/apache-nutch-2.3.1/runtime/local

export PATH=$PATH:$NUTCH_HOME/bin

source /etc/profile

6.5 配置nutch

conf/nutch-site.xml:

<configuration>

<property>

<name>storage.data.store.class</name>

<value>org.apache.gora.mongodb.store.MongoStore</value>

<description>Default class for storing data</description>

</property>

<property>

<name>http.agent.name</name>

<value>Hist Crawler</value>

</property>

</configuration>

ivy/ivy.xml:

<dependency org="org.apache.gora" name="gora-mongodb" rev="0.6.1" conf="*->default" />

conf/gora.properties:

gora.datastore.default=org.apache.gora.mongodb.store.MongoStore

gora.mongodb.override_hadoop_configuration=false

gora.mongodb.mapping.file=/gora-mongodb-mapping.xml

gora.mongodb.servers=vminger:27017

gora.mongodb.db=test1

gora.mongodb.login=root

gora.mongodb.secret=root

6.6 编译Nutch:ant runtime

6.7 设置抓取URL过滤规则:

conf/regex-urlfilter.txt:

+^http://([a-z0-9]*.)*sina.com.cn/

6.8 设置URL种子:

建立种子文件:urls/seed.ini

Nutch-Hadoop-MongoDB搭建分布式爬虫

将种子文件上传到HDFS上:hadoop dfs -put urls urls

6.9 进入runtime/deploy目录,开始抓取,id345,深度为3:

./bin/crawl urls id345 3

Nutch-Hadoop-MongoDB搭建分布式爬虫

6.10 mongodb中查看结果

db.id345_webpage.count();

Nutch-Hadoop-MongoDB搭建分布式爬虫

db.id345_webpage.find();

Nutch-Hadoop-MongoDB搭建分布式爬虫
  • 安卓客户端下载
  • 微信扫一扫
  • weinxin
  • 微信公众号
  • 微信公众号扫一扫
  • weinxin
avatar