How to set up the spark in windows learn in detail and get rid of all errors, once and for all, here we will see how to set up a spark, and winutils for HDFS
Spark is an Apache-supported language which usually used for backend programming and maintaining the spark jobs in backend data, or we can say the bulk of data comes from different sources.
Step 1: Download and Install, Java JDK 8, IntelliJ Idea Community edition and add the Hadoop path to run HDFS in lower Operating systems, which comes with a Spark assembly zip file you can download.
For Java 8 Download as shown below
For IntelliJ Idea download the community edition as shown below.
For Spark assembly, zip file download and extract it
Setting up the things for How to set up the spark in windows
After downloading Java 8 Install it as usual and once it is done, navigate to,
, where we will see Jre and Jdk folder go inside the JDK folder and copy the root directory path example:
, now set this path as environment variables with the label JAVA_HOME
Steps are: Click and type "Variables" you will get as shown below.
Now go to Edit the system environment and go to Environment Variables.
Inside System, variables make one new variable as
and add the JDK home path
as shown below.
Similarly now go inside the bin folder of your Java 8 Jdk to setup Java compiler variables, which is
this path should be set inside the Path variable in environment variables as shown below.
Click on the Path variable and inside Path make a new variable path and paste
as shown below.
Now we are done with Java 🎉 to verify Java is successfully set up in your system, Just open the command prompt, don't know how to open it? simple just click + R, this opens the "run" type cmd inside run as shown below.
Now inside cmd type
you will see as shown below
How to setup Spark Hadoop HDFS in windows
After downloading the Spark assembly zip file unzip it and make one folder in
files as apps and copy the extracted folder. Now your folder something looks like this.
Now you should download winutils as well, it will differ for different versions mine is 3.2 as it shows in the above image spark-3.2.1 so I will download the 3.2 version of winutils.
Download the whole repository and unzip it, go to your exact version folder my folder is and copy all as shown below.
Copy all files and paste them into your spark folder which is
copy and replace all as shown below.
Now we can see winutils.exe in the spark directory
Now get inside the spark folder and copy the current path I,e:
again open environment variables follow the steps to open
Add new variables as
as shown below
And inside the Path variable add the path
We are done with spark, verify by opening cmd and type
To start working with Spark using scala check the example here.
In this article, we have learned How to set up the spark in windows, for more articles like this please comment to us we will add similar articles related to queries, and also please check out our example code for scala spark as well thank you.
install apache spark on mac, Click to copy keyword, spark install windows, Click to copy keyword, spark installation on ubuntu, Click to copy keyword, install spark on windows, Click to copy keyword, spark installation on windows, Click to copy keyword, windows apache-spark, Click to copy keyword, apache-spark application on windows, Click to copy keyword, net for apache spark application,
1 thought on “How to set up the spark in windows with 100% results”
Pingback: How to set up and download python for windows working 100% | Solutiongigs