#INSTALL SPARK UBUNTU 16.04 HOW TO#
#INSTALL SPARK UBUNTU 16.04 INSTALL#

$ sudo zookeeper-server-start /etc/kafka/zookeeper.properties Now let us make sure, our installation is fine by staring kafka and produce and consume messages.ДЎ.Start Zookeeper :- Since zookeepr is a long-running service, you should run it in its own terminal

usr/bin/ # Driver scripts for starting/stopping services, prefixed with names

This will complete the installation of kafka and the packages will be installed in the below locations.

#INSTALL SPARK UBUNTU 16.04 INSTALL#

$ sudo apt-get install confluent-platform-2.11 Run apt-get update and install the Confluent Platform: $ sudo add-apt-repository 'deb stable main' InstallationДЇirst install ConfluentвЂ™s public key, which is used to sign the packages in the apt repository.ДЄdd the repository to your /etc/apt/sources.list: Well documented and more examples available etcвЂ¦Д¬onfluent kafka installation instructions can be found from their documentation.

It is apache Kafka + some additional features.

It is completely open source (Its base version).

Initially I was using apache-kafka, later on I came to know about confluent kafka. Please find link I have referred for this here. Java(TM) SE Runtime Environment (build 1.8.0_91-b14) Once installation completed please make sure Java specified version is installed correctly by checking its version. $ sudo apt-get install oracle-java8-set-default To automatically set up the Java 8 environment variables, you can install the following package. It will download 180MB and install Java automatically. $ sudo apt-get install oracle-java8-installer $ sudo apt-add-repository ppa:webupd8team/java I have installed Oracle -Java 8 using below commands. You can install either openJDK or Oracle-Java. So please install that first incase it is not available on your machine already. Please find the major steps I have followed belowД«oth Kafka and Spark needs Java. It is applicable for both 14.04 LTS and 16.04 versions.

#INSTALL SPARK UBUNTU 16.04 HOW TO#

This blog post is my learning on how to set up Kafka & Spark on Ubuntu Linux.

All these for a simple development env with a stand-alone clusterДue to above reasons, thought of installing Kafka & Spark separately.

Ambari provided a wonderful admin / monitoring tool but it was heavy background process.

Network configurations personally really given me a hard time.

HDP required integration to other components like mysql etcвЂ¦.

Eventhough, HDP provides Amabari installation, it was hard on my standards.

I required only Zookeeper, Kafka and Spark but HDP stack got many.

Initially tried to use Hortonworks HDP stack and I have installed that. how I will try this out?вЂ¦ As a newbie in this filed, I have tried several options. I was recently started working on a stream analytics project and after a lots of googling I have zeroed in on Kafka + Spark as my tool of choices.

Install spark ubuntu 16.04