hadoop单机版部署测试


下载

配置ssh无密码登录

# 测试ssh localhost, 默认需要密码
[root@node1 hadoop]# ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is SHA256:Ii9RadytomW4X2LEvMQwRxoOTeGgxfNbOgwXrc/wwZI.
ECDSA key fingerprint is MD5:bc:b5:ef:93:e6:fd:7c:cd:a3:4f:a7:f6:4c:24:c7:a7.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
root@localhost's password: 
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
# 测试免密登录,无需密码即可登录成功
[root@node1 hadoop]# ssh localhost
Last failed login: Thu May  7 11:44:07 CST 2020 from localhost on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Thu May  7 11:43:04 2020 from 192.168.41.1
[root@node1 ~]# 

配置java环境

#已经部署过了
[root@node1 hadoop]# echo $JAVA_HOME
/usr/local/bin/jdk1.8.0_112

安装hadoop

hadoop-2.7.7.tar.gz拷贝到/root/hadoop目录下并解压.

[root@node1 hadoop]# pwd
/root/hadoop
[root@node1 hadoop]# tar -zxvf hadoop-2.7.7.tar.gz 

/etc/profile或者~/.bash_profile配置环境变量HADOOP_HOME.

export HADOOP_HOME=/root/hadoop/hadoop-2.7.7
export PATH=$PATH:$HADOOP_HOME/bin

使配置生效

source /etc/profile

配置hadoop

进入$HADOOP_HOME/etc/hadoop目录,配置hadoop-env.sh等。涉及的配置文件如下:

hadoop-2.9.2/etc/hadoop/hadoop-env.sh
hadoop-2.9.2/etc/hadoop/yarn-env.sh
hadoop-2.9.2/etc/hadoop/core-site.xml
hadoop-2.9.2/etc/hadoop/hdfs-site.xml
hadoop-2.9.2/etc/hadoop/mapred-site.xml
hadoop-2.9.2/etc/hadoop/yarn-site.xml
  1. 配置hadoop-env.sh

    # The java implementation to use.
    export JAVA_HOME=${JAVA_HOME}
  2. 配置yarn-env.sh

    #export JAVA_HOME=/home/y/libexec/jdk1.6.0/
  3. 配置core-site.xml

    <configuration>
    <property>
      <name>fs.default.name</name>
      <value>hdfs://192.168.41.128:9000</value>
      <description>HDFS的URI,文件系统://namenode标识:端口号</description>
    </property>
    
  hadoop.tmp.dir   /root/hadoop/tmp   namenode上本地的hadoop临时文件夹 ```
  1. 配置hdfs-site.xml
    <configuration>
    <property>
      <name>dfs.name.dir</name>
      <value>/root/hadoop/name</value>
      <description>namenode上存储hdfs名字空间元数据 </description> 
    </property>
    
  dfs.data.dir   /root/hadoop/data   datanode上数据块的物理存储位置   dfs.replication   1   副本个数,配置默认是3,应小于datanode机器数量    ```
  1. 配置mapred-site.xml

    <configuration>
    <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
      </property>
    </configuration>
  2. 配置yarn-site.xml

    <configuration>
    <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
    </property>
    <property>
      <name>yarn.resourcemanager.webapp.address</name>
      <value>${yarn.resourcemanager.hostname}:8099</value>
    </property>
    </configuration>

启动hadoop

  1. 格式化hdfs文件系统

    bin/hadoop namenode -format
  2. 启动namenode

    sbin/hadoop-daemon.sh start namenode
  3. 启动datanode

    sbin/hadoop-daemon.sh start datanode
  4. 启动yarn

    sbin/start-yarn.sh
  5. 验证
    查看logs/目录下是否有错误日志,通过jps命令查看后台进程.

    [root@node1 hadoop-2.7.7]# jps
    17152 NodeManager
    17920 Jps
    16721 DataNode #数据节点
    16866 ResourceManager
    62190 HMaster # Hbase
    16623 NameNode #名称节点
  6. 查看UI

提交MapReduce作业

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.7.jar pi 2 100

hdfs使用

  1. 创建一个目录

    hadoop fs -mkdir /test
  2. 上传一个文件到指定目录

    hadoop fs -put README.txt /test

    或者

    hadoop fs -moveFromLocal README.txt /test
  3. 查看目录下文件

    [root@node1 hadoop-2.7.7]# hadoop fs -ls /test

更多命令参考


文章作者: wuzhiyong
版权声明: 本博客所有文章除特別声明外,均采用 CC BY 4.0 许可协议。转载请注明来源 wuzhiyong !
评论
 上一篇
springboot开发异步restful接口 springboot开发异步restful接口
SpringBoot中异步调用的使用 同步请求是会一直等待服务端相应的,需要返回结果给客户端的;而异步调用服务端会马上返回给客户端响应,完成这次整个的请求,至于异步调用的任务服务端后台自己慢慢跑就行,客户端不会关心。 开启异步调用,在启
2020-05-08
下一篇 
hadoop与hbase支持矩阵 hadoop与hbase支持矩阵
Hadoop版本支持矩阵: S=支持 X=不支持 NT=未测试 HBase的-1.1.x中 HBase的-1.2.x的 HBase的-1.3.x的 HBase的-2.0.x版本 Hadoop-2.0.x-alpha X X
2020-05-07
  目录