How to set up the spark in windows with 100% results

How to set up the spark in windows learn in detail and get rid of all errors, once and for all, here we will see how to set up a spark, and winutils for HDFS

Spark is an Apache-supported language which usually used for backend programming and maintaining the spark jobs in backend data, or we can say the bulk of data comes from different sources.

expand_less

Step 1: Download and Install, Java JDK 8, IntelliJ Idea Community edition and add the Hadoop path to run HDFS in lower Operating systems, which comes with a Spark assembly zip file you can download.

For Java 8 Download as shown below

For IntelliJ Idea download the community edition as shown below.

For Spark assembly, zip file download and extract it

Setting up the things for How to set up the spark in windows

After downloading Java 8 Install it as usual and once it is done, navigate to, C:\Program Files\Java, where we will see Jre and Jdk folder go inside the JDK folder and copy the root directory path example: C:\Program Files\Java\jdk1.8.0_281 , now set this path as environment variables with the label JAVA_HOME

Steps are: Click and type "Variables" you will get as shown below.

Now go to Edit the system environment and go to Environment Variables.

Inside System, variables make one new variable as JAVA_HOWE and add the JDK home path C:\Program Files\Java\jdk1.8.0_281 as shown below.

Similarly now go inside the bin folder of your Java 8 Jdk to setup Java compiler variables, which is C:\Program Files\Java\jdk1.8.0_281\bin this path should be set inside the Path variable in environment variables as shown below.

Click on the Path variable and inside Path make a new variable path and paste C:\Program Files\Java\jdk1.8.0_281\bin as shown below.

Now we are done with Java 🎉 to verify Java is successfully set up in your system, Just open the command prompt, don't know how to open it? simple just click + R, this opens the "run" type cmd inside run as shown below.

Now inside cmd type java -version you will see as shown below

How to setup Spark Hadoop HDFS in windows

After downloading the Spark assembly zip file unzip it and make one folder in C files as apps and copy the extracted folder. Now your folder something looks like this.

Now you should download winutils as well, it will differ for different versions mine is 3.2 as it shows in the above image spark-3.2.1 so I will download the 3.2 version of winutils.

Download the whole repository and unzip it, go to your exact version folder my folder is and copy all as shown below.

Copy all files and paste them into your spark folder which is C:\apps\spark-3.2.1-bin-hadoop3.2\bin copy and replace all as shown below.

Now we can see winutils.exe in the spark directory

Now get inside the spark folder and copy the current path I,e: C:\apps\spark-3.2.1-bin-hadoop3.2 again open environment variables follow the steps to open

Add new variables as HADOOP_HOME and SPARK_HOME as C:\apps\spark-3.2.1-bin-hadoop3.2 as shown below

And inside the Path variable add the path C:\apps\spark-3.2.1-bin-hadoop3.2\bin

We are done with spark, verify by opening cmd and type spark-shell

To start working with Spark using scala check the example here.

Conclusion:

In this article, we have learned How to set up the spark in windows, for more articles like this please comment to us we will add similar articles related to queries, and also please check out our example code for scala spark as well thank you.

Tags:

install apache spark on mac, Click to copy keyword, spark install windows, Click to copy keyword, spark installation on ubuntu, Click to copy keyword, install spark on windows, Click to copy keyword, spark installation on windows, Click to copy keyword, windows apache-spark, Click to copy keyword, apache-spark application on windows, Click to copy keyword, net for apache spark application,