Windows上搭建Standalone模式的Spark环境

Java

安装Java8，设置JAVA_HOME，并添加 %JAVA_HOME%\bin 到环境变量PATH中

E:\java -version
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

Scala

下载解压Scala 2.11，设置SCALA_HOME，并添加 %SCALA_HOME%\bin 到PATH中

E:\ scala -verion
Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL

Spark

下载解压Spark 2.1, 设置SPARK_HOME，并添加 %SPARK_HOME%\bin 到PATH中，此时尝试在控制台运行spark-shell，出现如下错误提示无法定位winutils.exe。

E:\>spark-shell
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/06/05 21:34:43 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
        at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:379)
        at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:394)
        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:387)
        at org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:2327)
        at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:365)
        at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:229)
        at org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:991)
        at org.apache.spark.repl.Main$.createSparkSession(Main.scala:92)
        at $line3.$read$$iw$$iw.<init>(<console>:15)
        at $line3.$read$$iw.<init>(<console>:42)
        at $line3.$read.<init>(<console>:44)
        at $line3.$read$.<init>(<console>:48)
        at $line3.$read$.<clinit>(<console>)
        at $line3.$eval$.$print$lzycompute(<console>:7)
        at $line3.$eval$.$print(<console>:6)
        at $line3.$eval.$print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
        at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
        at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
        at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
        at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
        at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
        at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
        at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
        at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
        at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
        at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38)
        at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
        at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
        at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
        at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37)
        at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:105)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
        at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
        at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
        at org.apache.spark.repl.Main$.doMain(Main.scala:69)
        at org.apache.spark.repl.Main$.main(Main.scala:52)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

从错误消息中可以看出Spark需要用到Hadoop中的一些类库(通过HADOOP_HOME环境变量，因为我们之前并未设置过，所以文件路径null\bin\winutils.exe里面出现了null)，但这并不意味这我们一定要安装Hadoop，我们可以直接下载所需要的winutils.exe到磁盘上的任何位置，比如C:\winutils\bin\winutils.exe，同时设置 HADOOP_HOME=C:\winutils 。

现在我们再次运行spark-shell，又有一个新的错误：

java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
  at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:981)
  at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110)
  at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:878)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:878)
  at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
  at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
  at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
  at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
  at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
  at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:878)
  at org.apache.spark.repl.Main$.createSparkSession(Main.scala:96)
  ... 47 elided
Caused by: java.lang.reflect.InvocationTargetException: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
  at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:978)
  ... 58 more
Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':
  at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:169)
  at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:86)
  at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101)
  at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:101)
  at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100)
  at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157)
  at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32)
  ... 63 more
Caused by: java.lang.reflect.InvocationTargetException: java.lang.reflect.InvocationTargetException: java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: ---------
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
  at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:166)
  ... 71 more
Caused by: java.lang.reflect.InvocationTargetException: java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: ---------
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
  at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264)
  at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:358)
  at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:262)
  at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCatalog.scala:66)
  ... 76 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: ---------
  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
  at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:188)
  ... 84 more
Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: ---------
  at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:612)
  at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
  ... 85 more
<console>:14: error: not found: value spark
       import spark.implicits._
              ^
<console>:14: error: not found: value spark
       import spark.sql
              ^
Welcome to
      __              
     / /  ___ ___/ /
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.1
      /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_60)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

错误消息中提示零时目录 /tmp/hive 没有写的权限：

The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: ---------

所以我们需要更新E:/tmp/hive的权限(我在E盘下运行的spark-shell命令，如果在其他盘运行，就改成对应的盘符+/tmp/hive)。运行如下命令：

E:\>C:\winutils\bin\winutils.exe chmod 777 E:\tmp\hive

再次运行spark-shell，spark启动成功。此时可以通过 http://localhost:4040 来访问Spark UI

时间： 2024-11-05 14:59:44

Windows上搭建Standalone模式的Spark环境的相关文章

深入体验bash on windows，在windows上搭建原生的linux开发环境，酷！

1 安装 你必须安装开发者预览版本,才能使用windows的linux subsystem功能. 首先打开你的Windows 10的设置,在"更新和安全"选项中(我的机器是英文操作系统,但中文操作系统类似),选择"开发者",然后选择"开发者模式": 在Windows 更新和安全选项中,选择"Windows 更新",在"更新设置"中选择高级选项,你必须打开"内部预览版本"选项,并且将内部预

windows下搭建Apache+Mysql+PHP开发环境

原文:windows下搭建Apache+Mysql+PHP开发环境 要求 必备知识 熟悉基本编程环境搭建. 运行环境 windows 7(64位); Apache2.2;MySQL Server 5.5php-5.3 下载地址 环境下载 官方下载地址 Apache MySql PHP 至于我使用的版本已经上传到百度云提供大家下载了,这里就不多啰嗦了!! Apache Apache的安装和普通的应用程序安装没什么太大的区别,关键是配置.打开安装路径下的"conf\httpd.conf"文

《Spark Cookbook 中文版》一1.5　在集群上以独立模式部署Spark

1.5 在集群上以独立模式部署Spark 在分布式环境中的计算资源需要管理,使得资源利用率高,每个作业都有公平运行的机会.Spark有一个便利的被称为独立模式的自带集群管理器.Spark也支持使用YARN或者Mesos做为集群管理器. 选择集群处理器时,主要需要考虑延迟以及其他架构,例如MapReduce,是否共享同样的计算资源池.如果你的集群运行着旧有的MapReduce作业,并且这些作业不能转变为Spark作业,那么使用YARN作为集群管理器是个好主意.Mesos是一种新兴的.方便跨平台管理

Ubuntu上搭建PHP＋Mysql+Nginx环境（apt-get方式）

Ubuntu上搭建PHP+Mysql+Nginx环境(apt-get方式) ubuntu版本:Ubuntu 10.04 LTS 1.首先使用apt-get下载Nginx,php教程,mysql教程,phpmyadmin,spawn-fcgi. sudo apt-get install nginx php5-cgi php5-cli mysql-server-5.1 phpmyadmin  spawn-fcgi 期间可能要输入mysql的密码,按照提示一步一步安装就是了. OK后,你在Firefo

在windows上搭建镜像yum站的方法（附bat脚本）

方法一:支持rsync的网站  对于常用的centos.Ubuntu.等使用官方yum源在 http://mirrors.ustc.edu.cn 都存在镜像. 同时 http://mirrors.ustc.edu.cn 网站又支持 rsync 协议, 可以通过rsync实现 镜像yum源. _______________________________________________________________ | University of Science and Technology o

windows上搭建react native环境

一.配置环境 1.准备相关资源  AndroidStudio Android开发集成开发环境(推荐下载含SDK tools版)  JDK for Windows Java Develop Kit(java开发工具)  Nodejs nodejs环境 2.ReactNative环境配置步骤  1) 使用npm安装ReactNative 在cmd命令工具中执行以下命令,注意并非 install react-native,装了react-native在后面init项目的时候会报错,需要卸掉再重装. 1

使用docker toolbox 在windows上搭建统一环境

1.先下载docker toolbox 以下是下载地址: http://get.daocloud.io/#install-docker-for-mac-windows  2.下载安装 git windows 客户端   安装全都 忽咯了,一律允许. 点击    启动docker 如果看到以下界面就显示成功了:  第一次比较慢,要装default 这个系统,启动后,你可以打开你虚拟机看看   接着讲一下,如何将本地的工程目录放到docker 容器环境下:  事先约定好:我在E盘上面新建 了一个wo

请问如何在自己的电脑上搭建IBM WebSphere Commerce开发环境

问题描述 诸位达人好.我以前用过spring之类的框架做过javaweb的开发,最近找了一份工作,是做IBMWebSphereCommerce开发的.还没有入职.这几天我想自己先学习一下.可是没想到我在网上奋战一天,连怎么搭建IBMWebSphereCommerce的开发环境都没搞定,只好求助于大家.请大家不吝赐教,不要取笑:)到目前为止,我对IBMWebSphereCommerce的理解如下,不知道有多少是对的,请大家批评指正:IBMWebSphereCommerce7.0系列的软件运行时环境

在Windows下搭建React Native Android开发环境

安装JDK 从Java官网下载JDK并安装.请注意选择x86还是x64版本. 推荐将JDK的bin目录加入系统PATH环境变量. 安装Android SDK 可以单独安装Android SDK,也可以通过Eclipse ADT或者Android Studio一并安装.推荐使用Android Studio,以下说明会默认以Android Studio的方式说明.请注意选择x86还是x64版本. 为了加速下载,推荐从AndroidDevTools下载. 然后进入SDKManager(可通过Andro