Spark 1.6.1 standalone mode Installation on ubuntu 14.04
Apache Spark is an open source cluster computing framework. Originally developed at the University of California, Berkeley's AMPLab, the Spark codebase was later donated to the Apache Software Foundation, which has maintained it since. Spark provides an interface for programming entire clusters with implicit data parallelism and fault-tolerance.
1) A machine with Ubuntu 14.04 LTS operating system installed.
2) Apache Spark 1.6.1 Software (Download Here)
3) Scala 2.10.5 Software (Download Here)
Spark is not a replacement of Hadoop. Spark is a part of the hadoop eco system. Spark can use Hadoop's distributed file system (HDFS) and also submit jobs on YARN. In order to make use of hadoop's components, you need to install Hadoop first then spark (How to install Hadoop on Ubuntu 14.04). The downloaded spark must be compatible with hadoop version. Please notice hadoop version before downloading spark.
Spark Standalone Mode Installation
Spark provides a simple standalone deploy mode. You can launch a standalone cluster either manually, by starting a master and workers by hand, or use provided launch scripts. It is also possible to run these daemons on a single machine for testing.
Step 1 - Update. Open a terminal (CTRL + ALT + T) and type the following sudo command. It is advisable to run this before installing any package, and necessary to run it to install the latest updates, even if you have not added or removed any Software Sources.
Step 2 - Installing Java 7.
Step 3 - Install open-ssh server. It is a cryptographic network protocol for operating network services securely over an unsecured network. The best known example application is for remote login to computer systems by users.
Step 4 - Create a Group. We will create a group, configure the group sudo permissions and then add the user to the group. Here 'hadoop' is a group name and 'hduser' is a user of the group.
Step 5 - Configure the sudo permissions for 'hduser'.
Since by default ubuntu text editor is nano we will need to use CTRL + O to edit.
Add the permissions to sudoers.
Use CTRL + X keyboard shortcut to exit out. Enter Y to save the file.
Step 6 - Creating spark directory.
Step 7 - Change the ownership and permissions of the directory /usr/local/spark. Here 'hduser' is an Ubuntu username.
Step 8 - Creating scala directory.
Step 9 - Change the ownership and permissions of the directory /usr/local/scala. Here 'hduser' is an Ubuntu username.
Step 10 - Creating /app/spark/tmp directory.
Step 11 - Change the ownership and permissions of the directory /app/spark/tmp. Here 'hduser' is an Ubuntu username.
Step 12 - Switch User, is used by a computer user to execute commands with the privileges of another user account.
Step 13 - Change the directory to /home/hduser/Desktop , In my case the downloaded spark-1.6.1-bin-hadoop2.6.tgz file is in /home/hduser/Desktop folder. For you it might be in /downloads folder check it.
Step 14 - Untar the spark-1.6.1-bin-hadoop2.6.tgz file.
Step 15 - Move the contents of spark-1.6.1-bin-hadoop2.6 folder to /usr/local/spark
Step 16 - Untar the scala-2.10.5.tgz file. In my case the downloaded scala-2.10.5.tgz file is in /home/hduser/Desktop folder. For you it might be in /downloads folder check it.
Step 17 - Move the contents of scala-2.10.5 folder to /usr/local/scala
Step 18 - Edit $HOME/.bashrc file by adding the spark and scala path.
$HOME/.bashrc file. Add the following lines
Step 19 - Reload your changed $HOME/.bashrc settings
Step 20 - Change the directory to /usr/local/spark/conf
Step 21 - Copy the spark-env.sh.template to spark-env.sh
Step 22 - Edit spark-env.sh file
Step 23 - Add the below lines to spark-env.sh file. Save and Close.
Step 24 - Copy the spark-defaults.conf.template to spark-defaults.conf
Step 25 - Edit spark-defaults.conf file
Step 26 - Add the below line to spark-defaults.conf file. Save and Close.
Step 27 - Copy the slaves.template to slaves
Step 28 - Edit slaves file.
Step 29 - Add the below line to slaves file. Save and Close.
Step 30 - Change the directory to /usr/local/spark/sbin
Step 31 - Start Master and all Worker Daemons.
Step 32 - The JPS (Java Virtual Machine Process Status Tool) tool is limited to reporting information on JVMs for which it has the access permissions.
Once the spark is up and running check the web-ui of the components as described below
Step 33 - Stop Master and all Worker Daemons.
Please share this blog post and follow me for latest updates on
Previous Post Next Post
Labels : Spark Cluster Mode Installation Spark with YARN Configuration Spark WordCount Java Example Spark spark-submit Script Usage Spark Shell Usage Spark Shell Scala Example Spark WordCount Scala Example