Record the process of installing CDH on Ubuntu 14.04, not guaranteed to work flawlessly on all machines...
The following commands are recommended to be run under su.
System Installation
Omitted
System Configuration
Static IP
Edit /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto p2p1
iface p2p1 inet static
address 172.16.1.68
netmask 255.255.255.0
gateway 172.16.1.1
Hosts
Set /etc/hosts, use names instead of IP addresses
172.16.1.68 master
172.16.1.69 data1
172.16.1.70 data2
Hostname
Add the future machine name to /etc/hostname on each machine.
The machine name should be the same as the name in the hosts file.
SSH Passwordless Communication
Go to ~/.ssh, generate keys with ssh-kengen (press Enter when prompted).
Send the key to other machines using ssh-copy-id [hostname]
For example: ssh-copy-id localhost to send it to yourself.
Disable Firewall
sudo ufw disable
sudo reboot
Time Synchronization
apt-get install ntp
Update Sources
This is about Cloudera's source, others are omitted.
cd /etc/apt/sources.list.d
sudo wget http://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/cloudera.list
sudo apt-get update
Get apt key
curl -s https://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/archive.key| sudo apt-key add -
Install Cloudera Server
On the Cloudera Manager machine
sudo apt-get -y --force-yes install cloudera-manager-daemons cloudera-manager-server
MySQL
Configure MySQL only on the master, apt-get install mysql-server
- Stop the MySQL service
- Add the configuration file, vim /etc/mysql/conf.d/mysql_cloudera_manager.cnf
Content:
[mysqld]
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0
key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
# For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
-
Restart the service, service mysql start
-
Log in to MySQL and create databases
create database amon DEFAULT CHARACTER SET utf8;
grant all on amon.* TO 'amon'@'localhost' IDENTIFIED BY 'amon_password';
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';
create database smon DEFAULT CHARACTER SET utf8;
grant all on smon.* TO 'smon'@'localhost' IDENTIFIED BY 'smon_password';
grant all on smon.* TO 'smon'@'%' IDENTIFIED BY 'smon_password';
create database rman DEFAULT CHARACTER SET utf8;
grant all on rman.* TO 'rman'@'localhost' IDENTIFIED BY 'rman_password';
grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';
create database hmon DEFAULT CHARACTER SET utf8;
grant all on hmon.* TO 'hmon'@'localhost' IDENTIFIED BY 'hmon_password';
grant all on hmon.* TO 'hmon'@'%' IDENTIFIED BY 'hmon_password';
create database hive DEFAULT CHARACTER SET utf8;
grant all on hive.* TO 'hive'@'localhost' IDENTIFIED BY 'hive_password';
grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';
create database oozie;
grant all privileges on oozie.* to 'oozie'@'localhost' identified by 'oozie';
grant all privileges on oozie.* to 'oozie'@'%' identified by 'oozie';
- Enable remote access to the database, comment out the line bind-host in my.cnf
JDK Installation
apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install oracle-j2sdk1.7
Master and agent both need to be installed, provided that the source has been added.
Cloudera Manager Database Initialization
/usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p --scm-host localhost scm scm scm_password
Start CM on the master
service cloudera-scm-server start
Deploy CDH using the web interface
The address is http://host:7180
The login credentials are both admin.