Category Archives: centos6

CentOS 6: Install Samba

Versions

CentOS 6.4
Samba 3.6.9
Windows 7

Install

sudo yum install samba

1	sudo yum install samba

Configure

Add linux user for samba access:

Note
Substitute $SAMBA_USER for samba username

Shell

sudo useradd $SAMBA_USER

1

sudo useradd $SAMBA_USER
Set linux user password:

Shell

sudo passwd $SAMBA_USER

1

sudo passwd $SAMBA_USER
Add linux user to samba (use same password)

Shell

sudo smbpasswd -a $SAMBA_USER sudo smbpasswd -e $SAMBA_USER

1
2

sudo smbpasswd -a $SAMBA_USER
sudo smbpasswd -e $SAMBA_USER
Optional: Add a path to share and change ownership to samba user:

Note
Substitute $PATH with the absolute path to the directory to share

Shell

mkdir $PATH chown -R $SAMBA_USER:$SAMBA_USER $PATH

1
2

mkdir $PATH
chown -R $SAMBA_USER:$SAMBA_USER $PATH

Note
home directories are shared by the default samba config
Optional: Set selinux context for directory to “samba_share_t” to permit sharing:

Shell

sudo chcon -t samba_share_t $PATH

1

sudo chcon -t samba_share_t $PATH
Optional: Set selinux bool to make home directories shareable:

Shell

sudo setsebool -P samba_enable_home_dirs on

1

sudo setsebool -P samba_enable_home_dirs on
Edit /etc/samba/smb.conf and change the workgroup to match the Windows 7 workgroup:

Note
Substitute $WG with the name of the workgroup

Default

workgroup = $WG

1

workgroup = $WG

Note
if not configured, Windows 7 defaults to the workgroup name “workgroup”
Add the following to the end of /etc/samba/smb.conf:

Note
Substitute $SHARE_NAME with the name of the share

Default

[$SHARE_NAME] path = $PATH writeable = yes browseable = yes valid users = $SAMBA_USER

1
2
3
4
5

[$SHARE_NAME]
       path = $PATH
       writeable = yes
       browseable = yes
       valid users = $SAMBA_USER
Start services

Shell

sudo service smb start sudo service nmb start

1
2

sudo service smb start
sudo service nmb start
Start on boot

Shell

sudo chkconfig smb on sudo chkconfig nmb on

1
2

sudo chkconfig smb on
sudo chkconfig nmb on
Open the following ports for samba:
- 137 udp
- 138 udp
- 139 tcp
- 445 tcp
  
  See
  Open a port for iptables

Test

On Windows 7, open command-line and map drive:

Note	Substitute $HOSTNAME for the hostname of the samba machine, $PASSWORD with the password of the samba user

net use z: \\$HOSTNAME\$SHARE_NAME /user:$SAMBA_USER $PASSWORD

1	net use z: \\$HOSTNAME\$SHARE_NAME /user:$SAMBA_USER $PASSWORD

Note

It may be necessary to run

net use * /delete

1	net use * /delete

to remove active connections and clear cached connection information if you have attempted to connect to the CentOS machine before.

CentOS 6.5: Virtualbox Guest Additions

3 Replies

Versions

CentOS 6.5
Virtualbox 4.3.12

Prerequisites

Install CentOS as Virtualbox Guest

Install Prereqs

sudo yum install gcc make kernel-devel kernel-devel kernel-headers

1	sudo yum install gcc make kernel-devel kernel-devel kernel-headers

Configure

Get the kernel version:

$ uname -r
2.6.32-431.el6.x86_64

1 2	$ uname -r 2.6.32-431.el6.x86_64

Note

VBoxLinuxAdditions.sh looks for the kernel source at $KERN_DIR/${uname -r}. The cp below is to make the directory structure meet its expectations (the default kernel sources folder includes the minor version)

export KERN_DIR=/usr/src/kernels
sudo cp -R /usr/src/kernels/2.6.32-431.17.1.el6.x86_64 /usr/src/kernels/2.6.32-431.el6.x86_64

1 2	export KERN_DIR=/usr/src/kernels sudo cp -R /usr/src/kernels/2.6.32-431.17.1.el6.x86_64 /usr/src/kernels/2.6.32-431.el6.x86_64

Install

sudo /media/VBOXADDITIONS_4.3.12_93733/VBoxLinuxAdditions.sh

1	sudo /media/VBOXADDITIONS_4.3.12_93733/VBoxLinuxAdditions.sh

Verify

Checking archive integrity... All good.
Uncompressing VirtualBox 4.3.12 Guest Additions for Linux............
VirtualBox Guest Additions installer
Copying additional installer modules ...
Installing additional modules ...
Removing existing VirtualBox non-DKMS kernel modules       [  OK  ]
Building the VirtualBox Guest Additions kernel modules     [  OK  ]
Building the main Guest Additions module                   [  OK  ]
Building the shared folder support module                  [  OK  ]
Building the OpenGL support module                         [  OK  ]
Doing non-kernel setup of the Guest Additions              [  OK  ]
Starting the VirtualBox Guest Additions                    [  OK  ]
Installing the Window System drivers
Installing X.Org Server 1.13 modules                       [  OK  ]
Setting up the Window System to use the Guest Additions    [  OK  ]
You may need to restart the hal service and the Window System (or just restart the guest system) to enable the Guest Additions.
Installing graphics libraries and desktop services componen[  OK  ]

Checking archive integrity... All good.

Uncompressing VirtualBox 4.3.12 Guest Additions for Linux............

VirtualBox Guest Additions installer

Copying additional installer modules ...

Installing additional modules ...

Removing existing VirtualBox non-DKMS kernel modules [ OK ]

Building the VirtualBox Guest Additions kernel modules [ OK ]

Building the main Guest Additions module [ OK ]

Building the shared folder support module [ OK ]

Building the OpenGL support module [ OK ]

Doing non-kernel setup of the Guest Additions [ OK ]

Starting the VirtualBox Guest Additions [ OK ]

Installing the Window System drivers

Installing X.Org Server 1.13 modules [ OK ]

Setting up the Window System to use the Guest Additions [ OK ]

You may need to restart the hal service and the Window System (or just restart the guest system) to enable the Guest Additions.

Installing graphics libraries and desktop services componen[ OK ]

Cleanup

sudo rm -rf /usr/src/kernels/2.6.32-431.el6.x86_64

1	sudo rm -rf /usr/src/kernels/2.6.32-431.el6.x86_64

Reboot

sudo reboot now

1	sudo reboot now

[DEPRECATED] CentOS 6: Install Datastax Cassandra OpsCenter Community Edition

Leave a reply

Versions

CentOS 6.4
Oracle Java JDK 1.6
OpsCenter Community 3.0.2

Prerequisites

Install

1. Edit /etc/yum.repos.d/datastax.repo

[datastax]
name= DataStax Repo for Apache Cassandra
baseurl=http://rpm.datastax.com/community
enabled=1
gpgcheck=0

[datastax]

name= DataStax Repo for Apache Cassandra

baseurl=http://rpm.datastax.com/community

enabled=1

gpgcheck=0

2. Install Opscenter Free

sudo yum install opscenter-free

1	sudo yum install opscenter-free

Configure

Note: by default opscener will only accept connections from 127.0.0.1:8888. This guide won’t change that setting but it can be changed by editing /etc/opscenter/opscenterd.conf

1. Start opscenter:

sudo service opscenterd start

1	sudo service opscenterd start

4. Start cassandra on boot:

sudo chkconfig opscenterd on

1	sudo chkconfig opscenterd on

Test

1. Reconnect ssh session and tunnel the default opscenter port (8888):

ssh <hostname> -L 8888:127.0.0.1:8888

1	ssh <hostname> -L 8888:127.0.0.1:8888

2. Connect to 127.0.0.1:8888 with a browser
3. Click “Use Existing Cluster”

4. Enter a seed cluster node hostname or IP
Note: you can enter a new-line separated list of all nodes, but this is unnecessary

5. Click “Save cluster”

Sources

http://www.datastax.com/docs/opscenter/index

[DEPRECATED] CentOS 6: Configure PAM authentication using Keroberos Tickets

Leave a reply

Versions

CentOS 6.4

Prerequsities

Install

yum install pam_krb5 krb5-libs krb5-workstation

1	yum install pam_krb5 krb5-libs krb5-workstation

Configure

authconfig --enablekrb5 --krb5kdc=<server> --krb5realm=<server> --enablekrb5kdcdns --enablekrb5realmdns --update

1	authconfig --enablekrb5 --krb5kdc=<server> --krb5realm=<server> --enablekrb5kdcdns --enablekrb5realmdns --update

Test

1. Add Kerberos authenticated user as an admin

useradd -G wheel <user>

1	useradd -G wheel <user>

2. Verify by logging in as the user.

Sources

http://www.linuxtopia.org/online_books/rhel6/rhel_6_deployment/rhel_6_deployment_ch-Authentication_Configuration.html#sect-The_Authentication_Configuration_Tool-Command_Line_Version

[DEPRECATED] CentOS 6: Install Single-node Hadoop from Cloudera CDH

Leave a reply

Overview

Guide for setting up a single-node Hadoop on CentOS using the Cloudera CDH repository.

Versions

CentOS 6.4
Oracle Java JDK 1.6
CDH 4
Hadoop 0.2

Prerequisties

Install Oracle JDK

Install

1. Download the yum repo file:

sudo wget -O /etc/yum.repos.d/cloudera-cdh4.repo http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/cloudera-cdh4.repo

1	sudo wget -O /etc/yum.repos.d/cloudera-cdh4.repo http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/cloudera-cdh4.repo

2. Install

sudo yum install hadoop-0.20-conf-pseudo

1	sudo yum install hadoop-0.20-conf-pseudo

Configure

1. Format the name node

sudo -u hdfs hdfs namenode -format

1	sudo -u hdfs hdfs namenode -format

Output:

...
13/03/26 15:24:01 INFO namenode.FSImage: Saving image file /var/lib/hadoop-hdfs/cache/hdfs/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
13/03/26 15:24:01 INFO namenode.FSImage: Image file of size 119 saved in 0 seconds.
13/03/26 15:24:01 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
13/03/26 15:24:01 INFO util.ExitUtil: Exiting with status 0
13/03/26 15:24:01 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1
************************************************************/

...

13/03/26 15:24:01 INFO namenode.FSImage: Saving image file /var/lib/hadoop-hdfs/cache/hdfs/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression

13/03/26 15:24:01 INFO namenode.FSImage: Image file of size 119 saved in 0 seconds.

13/03/26 15:24:01 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

13/03/26 15:24:01 INFO util.ExitUtil: Exiting with status 0

13/03/26 15:24:01 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1

************************************************************/

2. Start namenode/datanode services

sudo service hadoop-hdfs-namenode start
sudo service hadoop-hdfs-secondarynamenode start
sudo service hadoop-hdfs-datanode start

sudo service hadoop-hdfs-namenode start

sudo service hadoop-hdfs-secondarynamenode start

sudo service hadoop-hdfs-datanode start

3. Optional: Start services on boot

sudo chkconfig hadoop-hdfs-namenode on
sudo chkconfig hadoop-hdfs-secondarynamenode on
sudo chkconfig hadoop-hdfs-datanode on

sudo chkconfig hadoop-hdfs-namenode on

sudo chkconfig hadoop-hdfs-secondarynamenode on

sudo chkconfig hadoop-hdfs-datanode on

4. Create directories

sudo -u hdfs hadoop fs -mkdir /tmp
sudo -u hdfs hadoop fs -chmod -R 1777 /tmp
sudo -u hdfs hadoop fs -mkdir /user

sudo -u hdfs hadoop fs -mkdir /tmp

sudo -u hdfs hadoop fs -chmod -R 1777 /tmp

sudo -u hdfs hadoop fs -mkdir /user

5. Create map/reduce directories

sudo -u hdfs hadoop fs -mkdir -p /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
sudo -u hdfs hadoop fs -chmod 1777 /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
sudo -u hdfs hadoop fs -chown -R mapred /var/lib/hadoop-hdfs/cache/mapred

sudo -u hdfs hadoop fs -mkdir -p /var/lib/hadoop-hdfs/cache/mapred/mapred/staging

sudo -u hdfs hadoop fs -chmod 1777 /var/lib/hadoop-hdfs/cache/mapred/mapred/staging

sudo -u hdfs hadoop fs -chown -R mapred /var/lib/hadoop-hdfs/cache/mapred

6. Start map/reduce services

sudo service hadoop-0.20-mapreduce-jobtracker start
sudo service hadoop-0.20-mapreduce-tasktracker start

1 2	sudo service hadoop-0.20-mapreduce-jobtracker start sudo service hadoop-0.20-mapreduce-tasktracker start

7. Optional: Start services on boot

sudo chkconfig hadoop-0.20-mapreduce-jobtracker on
sudo chkconfig hadoop-0.20-mapreduce-tasktracker on

1 2	sudo chkconfig hadoop-0.20-mapreduce-jobtracker on sudo chkconfig hadoop-0.20-mapreduce-tasktracker on

8. Optional: Create a home directory on the hdfs for the current user

sudo -u hdfs hadoop fs -mkdir /user/$USER
sudo -u hdfs hadoop fs -chown $USER /user/$USER

1 2	sudo -u hdfs hadoop fs -mkdir /user/$USER sudo -u hdfs hadoop fs -chown $USER /user/$USER

9. Edit /etc/profile.d/hadoop.sh

export HADOOP_HOME=/usr/lib/hadoop

1	export HADOOP_HOME=/usr/lib/hadoop

10. Load into session

source /etc/profile.d/hadoop.sh

1	source /etc/profile.d/hadoop.sh

Test

1. Get a directory listing from hadoop hdfs

sudo -u hdfs hadoop fs -ls -R /

1	sudo -u hdfs hadoop fs -ls -R /

Output:

drwxrwxrwt   - hdfs supergroup              0 2012-04-19 15:14 /tmp
drwxr-xr-x   - hdfs supergroup              0 2013-03-26 15:38 /user
drwxr-xr-x   - hdfs     supergroup          0 2012-04-19 15:16 /var
drwxr-xr-x   - hdfs     supergroup          0 2012-04-19 15:16 /var/lib
drwxr-xr-x   - hdfs     supergroup          0 2012-04-19 15:16 /var/lib/hadoop-hdfs
drwxr-xr-x   - hdfs     supergroup          0 2012-04-19 15:16 /var/lib/hadoop-hdfs/cache
drwxr-xr-x   - mapred   supergroup          0 2012-04-19 15:19 /var/lib/hadoop-hdfs/cache/mapred
drwxr-xr-x   - mapred   supergroup          0 2012-04-19 15:29 /var/lib/hadoop-hdfs/cache/mapred/mapred
drwxrwxrwt   - mapred   supergroup          0 2012-04-19 15:33 /var/lib/hadoop-hdfs/cache/mapred/mapred/staging

drwxrwxrwt - hdfs supergroup 0 2012-04-19 15:14 /tmp

drwxr-xr-x - hdfs supergroup 0 2013-03-26 15:38 /user

drwxr-xr-x - hdfs supergroup 0 2012-04-19 15:16 /var

drwxr-xr-x - hdfs supergroup 0 2012-04-19 15:16 /var/lib

drwxr-xr-x - hdfs supergroup 0 2012-04-19 15:16 /var/lib/hadoop-hdfs

drwxr-xr-x - hdfs supergroup 0 2012-04-19 15:16 /var/lib/hadoop-hdfs/cache

drwxr-xr-x - mapred supergroup 0 2012-04-19 15:19 /var/lib/hadoop-hdfs/cache/mapred

drwxr-xr-x - mapred supergroup 0 2012-04-19 15:29 /var/lib/hadoop-hdfs/cache/mapred/mapred

drwxrwxrwt - mapred supergroup 0 2012-04-19 15:33 /var/lib/hadoop-hdfs/cache/mapred/mapred/staging

Note: results will vary based on user directories created

2. Navigate browser to http://<hostname>:50070

4. Navigate browser to http://<hostname>:50030

3. Run one of the examples

sudo -u hdfs hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 10 1000

1	sudo -u hdfs hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 10 1000

Output:

doop-0.20-mapreduce/hadoop-examples.jar pi 10 1000
Number of Maps  = 10
Samples per Map = 1000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
13/03/26 15:48:04 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/03/26 15:48:05 INFO mapred.FileInputFormat: Total input paths to process : 10
13/03/26 15:48:05 INFO mapred.JobClient: Running job: job_201303261534_0001
13/03/26 15:48:06 INFO mapred.JobClient:  map 0% reduce 0%
13/03/26 15:48:11 INFO mapred.JobClient:  map 20% reduce 0%
13/03/26 15:48:13 INFO mapred.JobClient:  map 40% reduce 0%
13/03/26 15:48:15 INFO mapred.JobClient:  map 60% reduce 0%
13/03/26 15:48:16 INFO mapred.JobClient:  map 80% reduce 0%
13/03/26 15:48:18 INFO mapred.JobClient:  map 100% reduce 26%
13/03/26 15:48:21 INFO mapred.JobClient:  map 100% reduce 100%
13/03/26 15:48:21 INFO mapred.JobClient: Job complete: job_201303261534_0001
13/03/26 15:48:21 INFO mapred.JobClient: Counters: 33
13/03/26 15:48:21 INFO mapred.JobClient:   File System Counters
13/03/26 15:48:21 INFO mapred.JobClient:     FILE: Number of bytes read=226
13/03/26 15:48:21 INFO mapred.JobClient:     FILE: Number of bytes written=2016361
13/03/26 15:48:21 INFO mapred.JobClient:     FILE: Number of read operations=0
13/03/26 15:48:21 INFO mapred.JobClient:     FILE: Number of large read operations=0
13/03/26 15:48:21 INFO mapred.JobClient:     FILE: Number of write operations=0
13/03/26 15:48:21 INFO mapred.JobClient:     HDFS: Number of bytes read=2390
13/03/26 15:48:21 INFO mapred.JobClient:     HDFS: Number of bytes written=215
13/03/26 15:48:21 INFO mapred.JobClient:     HDFS: Number of read operations=31
13/03/26 15:48:21 INFO mapred.JobClient:     HDFS: Number of large read operations=0
13/03/26 15:48:21 INFO mapred.JobClient:     HDFS: Number of write operations=3
13/03/26 15:48:21 INFO mapred.JobClient:   Job Counters 
13/03/26 15:48:21 INFO mapred.JobClient:     Launched map tasks=10
13/03/26 15:48:21 INFO mapred.JobClient:     Launched reduce tasks=1
13/03/26 15:48:21 INFO mapred.JobClient:     Data-local map tasks=10
13/03/26 15:48:21 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=19270
13/03/26 15:48:21 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=10129
13/03/26 15:48:21 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/03/26 15:48:21 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/03/26 15:48:21 INFO mapred.JobClient:   Map-Reduce Framework
13/03/26 15:48:21 INFO mapred.JobClient:     Map input records=10
13/03/26 15:48:21 INFO mapred.JobClient:     Map output records=20
13/03/26 15:48:21 INFO mapred.JobClient:     Map output bytes=180
13/03/26 15:48:21 INFO mapred.JobClient:     Input split bytes=1210
13/03/26 15:48:21 INFO mapred.JobClient:     Combine input records=0
13/03/26 15:48:21 INFO mapred.JobClient:     Combine output records=0
13/03/26 15:48:21 INFO mapred.JobClient:     Reduce input groups=2
13/03/26 15:48:21 INFO mapred.JobClient:     Reduce shuffle bytes=280
13/03/26 15:48:21 INFO mapred.JobClient:     Reduce input records=20
13/03/26 15:48:21 INFO mapred.JobClient:     Reduce output records=0
13/03/26 15:48:21 INFO mapred.JobClient:     Spilled Records=40
13/03/26 15:48:21 INFO mapred.JobClient:     CPU time spent (ms)=4110
13/03/26 15:48:21 INFO mapred.JobClient:     Physical memory (bytes) snapshot=2668306432
13/03/26 15:48:21 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=8668086272
13/03/26 15:48:21 INFO mapred.JobClient:     Total committed heap usage (bytes)=2210988032
13/03/26 15:48:21 INFO mapred.JobClient:   org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
13/03/26 15:48:21 INFO mapred.JobClient:     BYTES_READ=240
Job Finished in 16.69 seconds
Estimated value of Pi is 3.14080000000000000000

doop-0.20-mapreduce/hadoop-examples.jar pi 10 1000

Number of Maps = 10

Samples per Map = 1000

Wrote input for Map #0

Wrote input for Map #1

Wrote input for Map #2

Wrote input for Map #3

Wrote input for Map #4

Wrote input for Map #5

Wrote input for Map #6

Wrote input for Map #7

Wrote input for Map #8

Wrote input for Map #9

Starting Job

13/03/26 15:48:04 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

13/03/26 15:48:05 INFO mapred.FileInputFormat: Total input paths to process : 10

13/03/26 15:48:05 INFO mapred.JobClient: Running job: job_201303261534_0001

13/03/26 15:48:06 INFO mapred.JobClient: map 0% reduce 0%

13/03/26 15:48:11 INFO mapred.JobClient: map 20% reduce 0%

13/03/26 15:48:13 INFO mapred.JobClient: map 40% reduce 0%

13/03/26 15:48:15 INFO mapred.JobClient: map 60% reduce 0%

13/03/26 15:48:16 INFO mapred.JobClient: map 80% reduce 0%

13/03/26 15:48:18 INFO mapred.JobClient: map 100% reduce 26%

13/03/26 15:48:21 INFO mapred.JobClient: map 100% reduce 100%

13/03/26 15:48:21 INFO mapred.JobClient: Job complete: job_201303261534_0001

13/03/26 15:48:21 INFO mapred.JobClient: Counters: 33

13/03/26 15:48:21 INFO mapred.JobClient: File System Counters

13/03/26 15:48:21 INFO mapred.JobClient: FILE: Number of bytes read=226

13/03/26 15:48:21 INFO mapred.JobClient: FILE: Number of bytes written=2016361

13/03/26 15:48:21 INFO mapred.JobClient: FILE: Number of read operations=0

13/03/26 15:48:21 INFO mapred.JobClient: FILE: Number of large read operations=0

13/03/26 15:48:21 INFO mapred.JobClient: FILE: Number of write operations=0

13/03/26 15:48:21 INFO mapred.JobClient: HDFS: Number of bytes read=2390

13/03/26 15:48:21 INFO mapred.JobClient: HDFS: Number of bytes written=215

13/03/26 15:48:21 INFO mapred.JobClient: HDFS: Number of read operations=31

13/03/26 15:48:21 INFO mapred.JobClient: HDFS: Number of large read operations=0

13/03/26 15:48:21 INFO mapred.JobClient: HDFS: Number of write operations=3

13/03/26 15:48:21 INFO mapred.JobClient: Job Counters

13/03/26 15:48:21 INFO mapred.JobClient: Launched map tasks=10

13/03/26 15:48:21 INFO mapred.JobClient: Launched reduce tasks=1

13/03/26 15:48:21 INFO mapred.JobClient: Data-local map tasks=10

13/03/26 15:48:21 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=19270

13/03/26 15:48:21 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=10129

13/03/26 15:48:21 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0

13/03/26 15:48:21 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0

13/03/26 15:48:21 INFO mapred.JobClient: Map-Reduce Framework

13/03/26 15:48:21 INFO mapred.JobClient: Map input records=10

13/03/26 15:48:21 INFO mapred.JobClient: Map output records=20

13/03/26 15:48:21 INFO mapred.JobClient: Map output bytes=180

13/03/26 15:48:21 INFO mapred.JobClient: Input split bytes=1210

13/03/26 15:48:21 INFO mapred.JobClient: Combine input records=0

13/03/26 15:48:21 INFO mapred.JobClient: Combine output records=0

13/03/26 15:48:21 INFO mapred.JobClient: Reduce input groups=2

13/03/26 15:48:21 INFO mapred.JobClient: Reduce shuffle bytes=280

13/03/26 15:48:21 INFO mapred.JobClient: Reduce input records=20

13/03/26 15:48:21 INFO mapred.JobClient: Reduce output records=0

13/03/26 15:48:21 INFO mapred.JobClient: Spilled Records=40

13/03/26 15:48:21 INFO mapred.JobClient: CPU time spent (ms)=4110

13/03/26 15:48:21 INFO mapred.JobClient: Physical memory (bytes) snapshot=2668306432

13/03/26 15:48:21 INFO mapred.JobClient: Virtual memory (bytes) snapshot=8668086272

13/03/26 15:48:21 INFO mapred.JobClient: Total committed heap usage (bytes)=2210988032

13/03/26 15:48:21 INFO mapred.JobClient: org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter

13/03/26 15:48:21 INFO mapred.JobClient: BYTES_READ=240

Job Finished in 16.69 seconds

Estimated value of Pi is 3.14080000000000000000

Sources

https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode

CentOS 6: Open a Port for iptables

2 Replies

Verisons

CentOS 6.5

Configure

Edit /etc/sysconfig/iptables and add the following before COMMIT

Typical /etc/sysconfig/iptables:

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

# Firewall configuration written by system-config-firewall

# Manual customization of this file is not recommended.

*filter

:INPUT ACCEPT [0:0]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [0:0]

-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

-A INPUT -p icmp -j ACCEPT

-A INPUT -i lo -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited

-A FORWARD -j REJECT --reject-with icmp-host-prohibited

COMMIT

If opening TCP port, add the following line above the first reject statement:

Default

-A INPUT -m state --state NEW -m tcp -p tcp --dport <port> -j ACCEPT

1

-A INPUT -m state --state NEW -m tcp -p tcp --dport <port> -j ACCEPT

Note
if tcp or udp wasn’t specified assume tcp

If udp, instead add the following line above the first reject statement:

-A INPUT -m state --state NEW -m udp -p udp --dport <port> -j ACCEPT

1	-A INPUT -m state --state NEW -m udp -p udp --dport <port> -j ACCEPT

Example: Open port 666 for tcp

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 666 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

# Firewall configuration written by system-config-firewall

# Manual customization of this file is not recommended.

*filter

:INPUT ACCEPT [0:0]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [0:0]

-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

-A INPUT -p icmp -j ACCEPT

-A INPUT -i lo -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 666 -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited

-A FORWARD -j REJECT --reject-with icmp-host-prohibited

COMMIT

Restart iptables

Shell

service iptables restart

1

service iptables restart

Test from remote machine

Note	replace $HOSTNAME and $PORT below

telnet $HOSTNAME $PORT

1	telnet $HOSTNAME $PORT

CentOS 6: Install MongoDB

1 Reply

Versions

CentOS 6.5
MongoDB 2.4.10

Install

Edit /etc/yum.repos.d/10gen.repo

Default

[10gen] name=10gen Repository baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64 gpgcheck=0

1
2
3
4

[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64
gpgcheck=0
Install

Shell

sudo yum install mongo-10gen mongo-10gen-server

1

sudo yum install mongo-10gen mongo-10gen-server

Configure

Start service

Shell

sudo service mongod start

1

sudo service mongod start
Start on boot

Shell

sudo chkconfig mongod on

1

sudo chkconfig mongod on

Test

$ mongo
MongoDB shell version: 2.4.0
> use test
switched to db test
> db.foo.find()
> db.foo.save({a: 1})
> db.foo.find()
{ "_id" : ObjectId("514cd798c59449cc1f1344d3"), "a" : 1 }
> db.foo.update({a: 1}, {a: 5})
> db.foo.find()
{ "_id" : ObjectId("514cd798c59449cc1f1344d3"), "a" : 5 }
> db.foo.remove({a: 5})
> db.foo.find()

$ mongo

MongoDB shell version: 2.4.0

> use test

switched to db test

> db.foo.find()

> db.foo.save({a: 1})

> db.foo.find()

{ "_id" : ObjectId("514cd798c59449cc1f1344d3"), "a" : 1 }

> db.foo.update({a: 1}, {a: 5})

> db.foo.find()

{ "_id" : ObjectId("514cd798c59449cc1f1344d3"), "a" : 5 }

> db.foo.remove({a: 5})

> db.foo.find()

Optional: Open firewall port

Open port 27017 for mongodb See CentOS 6: Open a port for iptables

Sources

[DEPRECATED] CentOS 6: Install Hadoop from Apache Bigtop

3 Replies

WARNING

This guide is a work-in-progress and currently does not result in a fully working Hadoop. Please see CentOS 6: Install Single-node Hadoop from Cloudera CDH

Overview

Guide for setting up a single-node Hadoop on CentOS using the Apache Bigtop repo.

Versions

CentOS 6.3
Oracle Java JDK 1.6
Apache BigTop 0.5.0
Hadoop 2.0.2-alpha

Prerequisties

Install Oracle JDK

Install

1. Download the yum repo file:

sudo wget -O /etc/yum.repos.d/bigtop.repo http://www.apache.org/dist/bigtop/bigtop-0.5.0/repos/centos6/bigtop.repo

1	sudo wget -O /etc/yum.repos.d/bigtop.repo http://www.apache.org/dist/bigtop/bigtop-0.5.0/repos/centos6/bigtop.repo

2. Install

sudo yum install hadoop*

1	sudo yum install hadoop*

Configure

Separate where the namenode and datanode store their files

1. Edit /etc/hadoop/conf/hdfs-site.xml and change the following properties to the listing below:

dfs.namenode.name.dir
dfs.namenode.checkpoint.dir
dfs.datanode.data.dir

...
  <property>
     <name>dfs.namenode.name.dir</name>
     <value>file:///var/lib/hadoop-hdfs/namenode/${user.name}/dfs/name</value>
  </property>
  <property>
     <name>dfs.namenode.checkpoint.dir</name>
     <value>file:///var/lib/hadoop-hdfs/namenode/${user.name}/dfs/namesecondary\</value>
  </property>
  <property>
     <name>dfs.datanode.data.dir</name>
     <value>file:///var/lib/hadoop-hdfs/datanode/${user.name}/dfs/data</value>
  </property>
...

...

<name>dfs.namenode.name.dir</name>

<value>file:///var/lib/hadoop-hdfs/namenode/${user.name}/dfs/name</value>

</property>

<name>dfs.namenode.checkpoint.dir</name>

<value>file:///var/lib/hadoop-hdfs/namenode/${user.name}/dfs/namesecondary\</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>file:///var/lib/hadoop-hdfs/datanode/${user.name}/dfs/data</value>

</property>

...

Note: this step is not part of the official Apache BigTop instructions, but was required to avoid errors when running a datanode on the same machine as the namenode.

2. Format the name node

sudo -u hdfs hadoop namenode -format

1	sudo -u hdfs hadoop namenode -format

Output:

...
13/03/18 03:26:48 INFO namenode.FSImage: Image file of size 119 saved in 0 seconds.
13/03/18 03:26:48 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
13/03/18 03:26:48 INFO namenode.FileJournalManager: Purging logs older than 0
13/03/18 03:26:48 INFO util.ExitUtil: Exiting with status 0
13/03/18 03:26:48 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1
************************************************************/

...

13/03/18 03:26:48 INFO namenode.FSImage: Image file of size 119 saved in 0 seconds.

13/03/18 03:26:48 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

13/03/18 03:26:48 INFO namenode.FileJournalManager: Purging logs older than 0

13/03/18 03:26:48 INFO util.ExitUtil: Exiting with status 0

13/03/18 03:26:48 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1

************************************************************/

Note: formatting the datanode is not required, *however* if you have a previous install, you may have to to remove /var/lib/hadoop-hdfs/datanode to clear locks

3. Start hadoop namenode and datanode

sudo service hadoop-hdfs-namenode start
sudo service hadoop-hdfs-datanode start
sudo service hadoop-httpfs start
sudo service hadoop-mapreduce-historyserver start
sudo service hadoop-yarn-nodemanager start
sudo service hadoop-yarn-proxyserver start
sudo service hadoop-yarn-resourcemanager start

sudo service hadoop-hdfs-namenode start

sudo service hadoop-hdfs-datanode start

sudo service hadoop-httpfs start

sudo service hadoop-mapreduce-historyserver start

sudo service hadoop-yarn-nodemanager start

sudo service hadoop-yarn-proxyserver start

sudo service hadoop-yarn-resourcemanager start

TODO: figure out why hadoop-hdfs-zkfc doesn’t start
4. Start services on boot

sudo chkconfig hadoop-hdfs-namenode on
sudo chkconfig hadoop-hdfs-datanode on
sudo chkconfig hadoop-httpfs on
sudo chkconfig hadoop-mapreduce-historyserver on
sudo chkconfig hadoop-yarn-nodemanager on
sudo chkconfig hadoop-yarn-proxyserver on
sudo chkconfig hadoop-yarn-resourcemanager on

sudo chkconfig hadoop-hdfs-namenode on

sudo chkconfig hadoop-hdfs-datanode on

sudo chkconfig hadoop-httpfs on

sudo chkconfig hadoop-mapreduce-historyserver on

sudo chkconfig hadoop-yarn-nodemanager on

sudo chkconfig hadoop-yarn-proxyserver on

sudo chkconfig hadoop-yarn-resourcemanager on

5. Optional: Create a home directory on the hdfs

sudo -u hdfs hadoop fs -mkdir /user
sudo -u hdfs hadoop fs -mkdir /user/$USER
sudo -u hdfs hadoop fs -chown $USER /user/$USER

sudo -u hdfs hadoop fs -mkdir /user

sudo -u hdfs hadoop fs -mkdir /user/$USER

sudo -u hdfs hadoop fs -chown $USER /user/$USER

6. Edit /etc/profile.d/hadoop.sh

export HADOOP_HOME=/usr/lib/hadoop
export HADOOP_VERSION=2.0.2-alpha

1 2	export HADOOP_HOME=/usr/lib/hadoop export HADOOP_VERSION=2.0.2-alpha

7. Load into session

source /etc/profile.d/hadoop.sh

1	source /etc/profile.d/hadoop.sh

Test

1. Download the examples (they are missing 2.0.2-alpha for some reason)

sudo wget -O /usr/lib/hadoop/hadoop-examples.jar http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-examples/2.0.3-alpha/hadoop-mapreduce-examples-2.0.3-alpha.jar

1	sudo wget -O /usr/lib/hadoop/hadoop-examples.jar http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-examples/2.0.3-alpha/hadoop-mapreduce-examples-2.0.3-alpha.jar

2. Get a directory listing from hadoop hdfs

sudo -u hdfs hadoop fs -lsr /

1	sudo -u hdfs hadoop fs -lsr /

3. Run one of the examples

sudo -u hdfs hadoop jar /usr/lib/hadoop/hadoop-examples.jar pi 10 1000

1	sudo -u hdfs hadoop jar /usr/lib/hadoop/hadoop-examples.jar pi 10 1000

TODO: while the cluster appears to be working, this example hangs. :[

4. Navigate browser to http://<hostname>:50070

5. Click on “Live Nodes”

Sources

https://cwiki.apache.org/BIGTOP/how-to-install-hadoop-distribution-from-bigtop.html

[DEPRECATED] CentOS 6: Install wso2server.sh as a service that starts on boot

Leave a reply

Versions

CentOS 6.3
Oracle JDK 1.6

Configure

Note: Replace <product> with the abbreviated name of the WSO2 product (API Manager = ‘am’, Data Services Server = ‘dss’, etc) and replace <version> with the version number (’1.3.1′, ’4.0.6′, etc)

1. Edit /etc/init.d/wso2<product>

#!/bin/sh
#
# chkconfig: 2345 80 80
#
# Bash script to run wso2server.sh as a service
#

# Note: Normal env variables are discarded by service. So must export JAVA_HOME here, see http://serverfault.com/questions/374404/service-command-and-environment-variables
export JAVA_HOME=/usr/java/latest

WSO2_HOME=/usr/lib/<wso2 product>
WSO2_DAEMON=bin/wso2server.sh
WSO2_USER=root
START_OPTIONS=start
STOP_OPTIONS=stop

start() {
    echo "Starting WSO2 <product>..."
    su --preserve-environment $WSO2_USER -c "cd $WSO2_HOME && $WSO2_DAEMON $START_OPTIONS"
    return 0
}

stop() {
    echo "Stopping WSO2 <product>..."
    su --preserve-environment $WSO2_USER -c "cd $WSO2_HOME && $WSO2_DAEMON $STOP_OPTIONS"
    return 0
}

case "$1" in
  start)
    start
    ;;
  stop)
    stop
      ;;
  restart)
    stop
    start
    ;;
  *)
  echo $"Usage: $0 {start|stop|restart}"
  exit 1
esac

exit $?

#!/bin/sh

# chkconfig: 2345 80 80

# Bash script to run wso2server.sh as a service

# Note: Normal env variables are discarded by service. So must export JAVA_HOME here, see http://serverfault.com/questions/374404/service-command-and-environment-variables

export JAVA_HOME=/usr/java/latest

WSO2_HOME=/usr/lib/<wso2 product>

WSO2_DAEMON=bin/wso2server.sh

WSO2_USER=root

START_OPTIONS=start

STOP_OPTIONS=stop

start() {

echo "Starting WSO2 <product>..."

su --preserve-environment $WSO2_USER -c "cd $WSO2_HOME && $WSO2_DAEMON $START_OPTIONS"

return 0

}

stop() {

echo "Stopping WSO2 <product>..."

su --preserve-environment $WSO2_USER -c "cd $WSO2_HOME && $WSO2_DAEMON $STOP_OPTIONS"

return 0

}

case "$1" in

start)

start

;;

stop)

stop

;;

restart)

stop

start

;;

echo $"Usage: $0 {start|stop|restart}"

exit 1

esac

exit $?

Note: the script hard codes JAVA_HOME, but this can be deferred to the /etc/sysconfig/wso2<product> script (which you must create) if desired. Note that the normal /etc/profile.sh is *not* run for services.
TODO: find better way of setting JAVA_HOME, can’t execute /etc/profile or /etc/profile.d/java.sh directly

2. Make it executable

sudo chmod +x /etc/init.d/wso2<product>

1	sudo chmod +x /etc/init.d/wso2<product>

2. Start service on boot

sudo chkconfig wso2<product> on

1	sudo chkconfig wso2<product> on

Verify

1. Start service

sudo service wso2<product> start

1	sudo service wso2<product> start

Source

http://stackoverflow.com/questions/12596791/carbon-4-0-1-bam-2-0-as-a-linux-daemon

CentOS 6: Clear the yum cache

Leave a reply

Versions

CentOS 6.3

Overview

Ran into problems tonight after working installs of Hadoop of different versions where the yum installer would try to download the incorrect version. Required cleaning yum’s caches AND manually deleting the yum cache for the repo in /var/cache/yum.

Guide

Remove the repo

Note
replace $REPONAME below with the name of the repo to clear

Shell

sudo rm /etc/yum.repos.d/$REPONAME.repo

1

sudo rm /etc/yum.repos.d/$REPONAME.repo
Clean all

Shell

sudo yum clean all

1

sudo yum clean all
Delete the yum cache for the repo

Shell

sudo rm -rf /var/cache/yum/x86_64/6/$REPONAME

1

sudo rm -rf /var/cache/yum/x86_64/6/$REPONAME

Sources

http://www.centos.org/docs/5/html/yum/sn-yum-maintenance.html