いろいろ参考にしながらwindowsで環境構築やってみたメモ書き
最近ちょっと時間がとれたのでいろいろなサイトを参考にしながら環境構築をしてみた。
vagrantで仮想環境構築
①VirtualBoxとvagrantのインストール
下記サイトからダウンロードしてインストール(ひたすら次へでOK)
VirtualBoxダウンロードサイトOracle VM VirtualBox
vagrantダウンロードサイトVagrant by HashiCorp
②boxを追加
$ vagrant box add CentOS6.6 https://github.com/tommy-muehle/puppet-vagrant-boxes/releases/download/1.0.0/centos-6.6-x86_64.box
③サーバー起動
$ mkdir centos $ cd centos $ vagrant init CentOS6.6 $ vagrant up
④サーバーに接続
$ vagrant ssh
⑤javaインストール
$ wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.rpm" -O jdk-7u79-linux-x64.rpm $ sudo rpm -ivh jdk-7u79-linux-x64.rpm Vorbereiten... ########################################### [100%] 1:jdk ########################################### [100%] Unpacking JAR files... rt.jar... jsse.jar... charsets.jar... tools.jar... localedata.jar... jfxrt.jar... $ java -version java version "1.7.0_79" Java(TM) SE Runtime Environment (build 1.7.0_79-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
⑥環境変数の設定
JAVA_HOMEの追記
$ sudo vi /etc/profile [ export JAVA_HOME=/usr/java/default ] $ source /etc/profile $ echo $JAVA_HOME /usr/java/default
mongodb
①10genリポジトリの追加
$ sudo vim /etc/yum.repos.d/10gen.repo [10gen] name=10gen Repository baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64 gpgcheck=0 enabled=0
②mongodbインストール
$ sudo yum install mongo-10gen-server.x86_64 mongo-10gen.x86_64 --enablerepo=10gen Geladene Plugins: fastestmirror Einrichten des Installationsprozess Determining fastest mirrors * base: ftp.yz.yamagata-u.ac.jp 中略 Installiert: mongodb-org.x86_64 0:2.6.11-1 mongodb-org-server.x86_64 0:2.6.11-1 Abhängigkeit installiert: mongodb-org-mongos.x86_64 0:2.6.11-1 mongodb-org-shell.x86_64 0:2.6.11-1 mongodb-org-tools.x86_64 0:2.6.11-1 Komplett!
③起動確認
$ sudo /etc/init.d/mongod start Starting mongod: [ OK ]
④起動時に自動立ち上げ設定
$ sudo chkconfig mongod on
rabbitMQ
①SSL証明書/認証局作成
$ cd /tmp $ wget http://sensuapp.org/docs/0.16/tools/ssl_certs.tar $ tar xvf ssl_certs.tar ssl_certs/ ssl_certs/sensu_ca/ ssl_certs/ssl_certs.sh ssl_certs/sensu_ca/openssl.cnf $ cd ssl_certs $ ./ssl_certs.sh generate Generating SSL certificates for Sensu ... Generating a 2048 bit RSA private key 中略 Write out database with 1 new entries Data Base Updated
②erlangインストール
$ sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm Retrieving http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm warning: /var/tmp/rpm-tmp.NeVGgn: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY Preparing... ########################################### [100%] 1:epel-release ########################################### [100%] $ sudo yum install erlang Loaded plugins: fastestmirror Setting up Install Process Loading mirror speeds from cached hostfile 中略 Dependency Installed: 中略 Complete!
③RabbitMQインストール
$ sudo rpm --import http://www.rabbitmq.com/rabbitmq-signing-key-public.asc $ sudo rpm -Uvh http://www.rabbitmq.com/releases/rabbitmq-server/v3.2.1/rabbitmq-server-3.2.1-1.noarch.rpm Retrieving http://www.rabbitmq.com/releases/rabbitmq-server/v3.2.1/rabbitmq-server-3.2.1-1.noarch.rpm Preparing... ########################################### [100%] 1:rabbitmq-server ########################################### [100%]
④RabbitMQ server起動
$ sudo chkconfig rabbitmq-server on $ sudo service rabbitmq-server start Starting rabbitmq-server: SUCCESS rabbitmq-server.
⑤SSL設定
$ sudo mkdir -p /etc/rabbitmq/ssl $ sudo cp -a sensu_ca/cacert.pem /etc/rabbitmq/ssl/ $ sudo cp -a server/cert.pem /etc/rabbitmq/ssl/ $ sudo cp -a server/key.pem /etc/rabbitmq/ssl/
⑥RabbitMQ SSL設定
$ sudo vi /etc/rabbitmq/rabbitmq.config [ {rabbit, [ {ssl_listeners, [5671]}, {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"}, {certfile,"/etc/rabbitmq/ssl/cert.pem"}, {keyfile,"/etc/rabbitmq/ssl/key.pem"}, {verify,verify_peer}, {fail_if_no_peer_cert,true}]} ]} ].
⑦RabbitMQ再起動
$ sudo service rabbitmq-server restart Restarting rabbitmq-server: SUCCESS rabbitmq-server.
Hadoop
①reposの登録
$ wget http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo --YYYY-MM-DD hh:mm:ss-- http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo 中略 YYYY-MM-DD hh:mm:ss (74.3 MB/s) - “cloudera-cdh3.repo” saved [295/295] $ sudo mv cloudera-cdh3.repo /etc/yum.repos.d/ $ sudo yum update yum Loaded plugins: fastestmirror 中略 Updated: yum.noarch 0:3.2.29-69.el6.centos Complete! $ yum search hadoop Loaded plugins: fastestmirror Determining fastest mirrors * base: ftp.riken.jp * epel: ftp.riken.jp * extras: ftp.riken.jp * updates: ftp.riken.jp cloudera-cdh3 75/75 ======================================================================================================= N/S Matched: hadoop ======================================================================================================= hadoop-0.20.noarch : Hadoop is a software platform for processing vast amounts of data hadoop-0.20-conf-pseudo.noarch : Hadoop installation in pseudo-distributed mode hadoop-0.20-datanode.noarch : Hadoop Data Node hadoop-0.20-debuginfo.i386 : Debug information for package hadoop-0.20 hadoop-0.20-debuginfo.x86_64 : Debug information for package hadoop-0.20 hadoop-0.20-doc.noarch : Hadoop Documentation hadoop-0.20-jobtracker.noarch : Hadoop Job Tracker hadoop-0.20-libhdfs.i386 : Hadoop Filesystem Library hadoop-0.20-libhdfs.x86_64 : Hadoop Filesystem Library hadoop-0.20-namenode.noarch : The Hadoop namenode manages the block locations of HDFS files hadoop-0.20-native.i386 : Native libraries for Hadoop Compression hadoop-0.20-native.x86_64 : Native libraries for Hadoop Compression hadoop-0.20-pipes.i386 : Hadoop Pipes Library hadoop-0.20-pipes.x86_64 : Hadoop Pipes Library hadoop-0.20-sbin.i386 : Binaries for secured Hadoop clusters hadoop-0.20-sbin.x86_64 : Binaries for secured Hadoop clusters hadoop-0.20-secondarynamenode.noarch : Hadoop Secondary namenode hadoop-0.20-source.noarch : Source code for Hadoop hadoop-0.20-tasktracker.noarch : Hadoop Task Tracker hadoop-hbase.noarch : HBase is the Hadoop database. Use it when you need random, realtime read/write access to your Big Data. This project's goal is the hosting of very large tables -- billions of rows X millions of columns -- : atop clusters of commodity hardware. hadoop-hbase-master.noarch : The Hadoop HBase master Server. hadoop-hbase-regionserver.noarch : The Hadoop HBase RegionServer server. hadoop-hbase-thrift.noarch : The Hadoop HBase Thrift Interface hadoop-hive.noarch : Hive is a data warehouse infrastructure built on top of Hadoop hadoop-zookeeper-server.noarch : The Hadoop Zookeeper server flume.noarch : Flume is a reliable, scalable, and manageable distributed log collection application for collecting data such as logs and delivering it to data stores such as Hadoop's HDFS. flume-ng.noarch : Flume is a reliable, scalable, and manageable distributed log collection application for collecting data such as logs and delivering it to data stores such as Hadoop's HDFS. hadoop-0.20-fuse.i386 : Mountable HDFS hadoop-0.20-fuse.x86_64 : Mountable HDFS hadoop-hbase-doc.noarch : Hbase Documentation hadoop-hbase-rest.noarch : The Apache HBase REST gateway hadoop-hive-hbase.noarch : Provides integration between Apache HBase and Apache Hive hadoop-hive-metastore.noarch : Shared metadata repository for Hive. hadoop-hive-server.noarch : Provides a Hive Thrift service. hadoop-pig.noarch : Pig is a platform for analyzing large data sets hadoop-zookeeper.noarch : A high-performance coordination service for distributed applications. hue-common.i386 : A browser-based desktop interface for Hadoop hue-common.x86_64 : A browser-based desktop interface for Hadoop hue-filebrowser.noarch : A UI for the Hadoop Distributed File System (HDFS) hue-jobbrowser.noarch : A UI for viewing Hadoop map-reduce jobs hue-jobsub.noarch : A UI for designing and submitting map-reduce jobs to Hadoop hue-plugins.noarch : Hadoop plugins for Hue hue-shell.i386 : A shell for console based Hadoop applications hue-shell.x86_64 : A shell for console based Hadoop applications oozie.noarch : Oozie is a system that runs workflows of Hadoop jobs. sqoop.noarch : Sqoop allows easy imports and exports of data sets between databases and the Hadoop Distributed File System (HDFS). Name and summary matches only, use "search all" for everything.
②Hadoopインストール
$ su $ yum install hadoop-0.20 -y Setting up Install Process Loading mirror speeds from cached hostfile 中略 Complete!
③1台で動かす設定追加
$ yum install hadoop-0.20-conf-pseudo -y Loaded plugins: fastestmirror 中略 Complete!
④hive,pig,hbaseインストール
$ yum install hadoop-hive -y Loaded plugins: fastestmirror Setting up Install Process 中略 Complete! $ yum install hadoop-pig -y Loaded plugins: fastestmirror Setting up Install Process 中略 Complete! $ yum install hadoop-hbase -y Loaded plugins: fastestmirror Setting up Install Process 中略 Complete!
⑤Hadoop起動
$ /etc/init.d/hadoop-0.20-datanode start Starting Hadoop datanode daemon (hadoop-datanode): starting datanode, logging to /usr/lib/hadoop-0.20/logs/hadoop-hadoop-datanode-localhost.localdomain.out datanode (pid 6243) is running... [ OK ] $ /etc/init.d/hadoop-0.20-namenode start Starting Hadoop namenode daemon (hadoop-namenode): starting namenode, logging to /usr/lib/hadoop-0.20/logs/hadoop-hadoop-namenode-localhost.localdomain.out [root@localhost vagrant]# [ OK ] $ /etc/init.d/hadoop-0.20-tasktracker start Starting Hadoop tasktracker daemon (hadoop-tasktracker): starting tasktracker, logging to /usr/lib/hadoop-0.20/logs/hadoop-hadoop-tasktracker-localhost.localdomain.out [root@localhost vagrant]# [ OK ] $ /etc/init.d/hadoop-0.20-jobtracker start Starting Hadoop jobtracker daemon (hadoop-jobtracker): starting jobtracker, logging to /usr/lib/hadoop-0.20/logs/hadoop-hadoop-jobtracker-localhost.localdomain.out [root@localhost vagrant]# [ OK ] $ /etc/init.d/hadoop-0.20-secondarynamenode start #これは最初は不要 Starting Hadoop secondarynamenode daemon (hadoop-secondarynamenode): starting secondarynamenode, logging to /usr/lib/hadoop-0.20/logs/hadoop-hadoop-secondarynamenode-localhost.localdomain.out [root@localhost vagrant]# [ OK ]
ここまででいろいろ入れてみたので時間を作って触りつつ何かつくりたいな−と。。。
参考にしたサイト
qiita.com www.task-notes.com qiita.com qiita.com yut.hatenablog.com