来看看根据Kubernetes的Spark布置彻底攻略

本文是在Kubernets上建立Spark集群的操作攻略,一起供给了Spark测验使命及相关的测验数据,经过阅览本文,你能够实践从制造Spark镜像、建立Spark容器集群,到在集群上运转测验使命的完好流程。
Yarn曾经是Hadoop默许的资源编列办理渠道。但最近状况有所改变,特别是关于Hadoop中的Spark,因为其与S3等其他存储渠道集成得很好,而与Hadoop生态中其他组件反而没有太严密的相关,因而Kubernetes正敏捷代替Yarn,成为根据目标存储的Spark体系的默许编列办理渠道。在这篇文章中,咱们将深入研究如安在Kubernetes集群上构建和布置Spark容器。因为Spark的运转依赖于数据,咱们将装备Spark集群经过S3 API进行存储操作。
来看看根据Kubernetes的Spark布置彻底攻略来看看根据Kubernetes的Spark布置彻底攻略

构建Spark容器
在Kubernetes上布置运用的第一步,是创立容器。尽管有些项目会供给官方的容器镜像,但截止到写此文时,Apache Spark并没有供给官方镜像。因而咱们将自己创立Spark容器,让咱们从Dockerfile开端。

FROM java:openjdk-8-jdk

ENV hadoop_ver 2.8.2
ENV spark_ver 2.4.4

RUN mkdir -p /opt &&
cd /opt &&
curl http://archive.apache.org/dist/hadoop/common/hadoop-${hadoop_ver}/hadoop-${hadoop_ver}.tar.gz |

tar -zx &&  

ln -s hadoop-${hadoop_ver} hadoop &&
echo Hadoop ${hadoop_ver} installed in /opt

RUN mkdir -p /opt &&
cd /opt &&
curl http://archive.apache.org/dist/spark/spark-${spark_ver}/spark-${spark_ver}-bin-without-hadoop.tgz |

tar -zx &&  

ln -s spark-${spark_ver}-bin-without-hadoop spark &&
echo Spark ${spark_ver} installed in /opt

ENV SPARK_HOME=/opt/spark
ENV PATH=$PATH:$SPARK_HOME/bin
ENV HADOOP_HOME=/opt/hadoop
ENV PATH=$PATH:$HADOOP_HOME/bin
ENV LD_LIBRARY_PATH=$HADOOP_HOME/lib/native

RUN curl http://central.maven.org/maven2/org/apache/hadoop/hadoop-aws/2.8.2/hadoop-aws-2.8.2.jar -o /opt/spark/jars/hadoop-aws-2.8.2.jar
RUN curl http://central.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.3/httpclient-4.5.3.jar -o /opt/spark/jars/httpclient-4.5.3.jar
RUN curl http://central.maven.org/maven2/joda-time/joda-time/2.9.9/joda-time-2.9.9.jar -o /opt/spark/jars/joda-time-2.9.9.jar
RUN curl http://central.maven.org/maven2/com/amazonaws/aws-java-sdk-core/1.11.712/aws-java-sdk-core-1.11.712.jar -o /opt/spark/jars/aws-java-sdk-core-1.11.712.jar
RUN curl http://central.maven.org/maven2/com/amazonaws/aws-java-sdk/1.11.712/aws-java-sdk-1.11.712.jar -o /opt/spark/jars/aws-java-sdk-1.11.712.jar
RUN curl http://central.maven.org/maven2/com/amazonaws/aws-java-sdk-kms/1.11.712/aws-java-sdk-kms-1.11.712.jar -o /opt/spark/jars/aws-java-sdk-kms-1.11.712.jar
RUN curl http://central.maven.org/maven2/com/amazonaws/aws-java-sdk-s3/1.11.712/aws-java-sdk-s3-1.11.712.jar -o /opt/spark/jars/aws-java-sdk-s3-1.11.712.jar

ADD start-common.sh start-worker start-master /
ADD core-site.xml /opt/spark/conf/core-site.xml
ADD spark-defaults.conf /opt/spark/conf/spark-defaults.conf
ENV PATH $PATH:/opt/spark/bin
在这个Dockerfile中,咱们首先从官方地址下载Apache Spark和Hadoop,然后从Maven获取相关的jar包。当一切相关的文件都现已下载并解压到一个特定的目录后,咱们将这些重要的装备文件增加到镜像中。

在这个过程中,你能够很便利的增加自己环境特有的装备。

本来咱们能够越过以上过程,直接运用一个预先构建好的镜像,可是经过解读这些过程能够让咱们的读者看到Spark容器内部的内容,高档用户能够据此修正来满意他们特别的需求。

以上示例中运用到的Dockerfile和其他相关的装备文件,能够从这个GitHub库房中获取。假如要运用这个库房中的内容,请先运用以下指令将其克隆到本地:

git clone git@github.com:devshlabs/spark-kubernetes.git
现在,你能够根据需求在你的环境中进行任何更改,然后构建镜像,并上传到你运用的容器注册表中。在本文的示例中,我运用Dockerhub作为容器注册表,指令如下:

cd spark-kubernetes/spark-container
docker build . -t mydockerrepo/spark:2.4.4
docker push mydockerrepo/spark:2.4.4
记住将其间的mydockerrepo替换为你实践的注册表姓名。

在Kubernetes上布置Spark
至此,Spark容器镜像现已构建好,并能够拉取运用了。让咱们运用此镜像来布置Spark Master和Worker。第一步是创立Spark Master。咱们将运用Kubernetes ReplicationController创立Spark Master。在本文的示例中,我仅用单实例创立Spark Master。而在有HA需求的出产环境中,你或许需求将副本数设置为3或许以上。

kind: ReplicationController
apiVersion: v1
metadata:
name: spark-master-controller
spec:
replicas: 1
selector:
component: spark-master
template:
metadata:
labels:

component: spark-master 

spec:
hostname: spark-master-hostname
subdomain: spark-master-headless
containers:

- name: spark-master
image: mydockerrepo/spark:2.4.4
imagePullPolicy: Always
command: ["/start-master"]
ports:
- containerPort: 7077
- containerPort: 8080
resources:
requests:
cpu: 100m 

为了使Spark Worker节点能够发现Spark Master节点,咱们还需求创立headless服务
当你从GitHub库房完结克隆,并进入spark-kubernetes目录后,就能够发动Spark Master服务了,指令如下:

kubectl create -f spark-master-controller.yaml
kubectl create -f spark-master-service.yaml
现在,保证Master节点和一切的服务都正常运转,然后就能够开端布置Worker节点了。Spark Worker的副本数设置为2,你能够根据需求修正。Worker发动指令如下:
kubectl create -f spark-worker-controller.yaml
最终,经过以下指令承认是否一切服务都正常运转:
kubectl get all
履行以上指令,你应该能够看到相似下面的内容:

NAME READY STATUS RESTARTS AGE
po/spark-master-controller-5rgz2 1/1 Running 0 9m
po/spark-worker-controller-0pts6 1/1 Running 0 9m
po/spark-worker-controller-cq6ng 1/1 Running 0 9m

NAME DESIRED CURRENT READY AGE
rc/spark-master-controller 1 1 1 9m
rc/spark-worker-controller 2 2 2 9m

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/spark-master 10.108.94.160 7077/TCP,8080/TCP 9m
向Spark集群提交Job
现在让咱们提交一个Job,看看是否履行正常。不过在此之前,你需求一个有用的AWS S3账户,以及存有样本数据的桶存在。我运用了Kaggle下载样本数据,样本数据能够从https://www.kaggle.com/datasna ... s.csv获取,获取今后需求上传到S3的桶里。假定桶名是s3-data-bucket,那么样本数据文件则坐落s3-data-bucket/data.csv。
数据准备好今后,将其加载到一个Spark master pod中履行。以Pod名为spark-master-controller-5rgz2为例,指令如下:
kubectl exec -it spark-master-controller-v2hjb /bin/bash
假如你登录进入了Spark体系,能够运转Spark Shell:

export SPARK_DIST_CLASSPATH=$(hadoop classpath)
spark-shell
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://192.168.132.147:4040
Spark context available as 'sc' (master = spark://spark-master:7077, app id = app-20170405152342-0000).
Spark session available as 'spark'.
Welcome to
__
/ / _ ___/ /
/ / _ `/ __/ '_/
/___/ .__/_,_/_/ /_/_ version 2.4.4
/_/

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_221)
Type in expressions to have them evaluated.
Type :help for more information.

scala>
现在让咱们告知Spark Master,S3存储的详细信息,在上文所示的Scale提示符中输入以下装备:

sc.hadoopConfiguration.set("fs.s3a.endpoint", "https://s3.amazonaws.com")
sc.hadoopConfiguration.set("fs.s3a.access.key", "s3-access-key")
sc.hadoopConfiguration.set("fs.s3a.secret.key", "s3-secret-key")
现在,只需将以下内容粘贴到Scala提示符中,以提交Spark Job(请记住修正S3相关字段):

import org.apache.spark._
import org.apache.spark.rdd.RDD
import org.apache.spark.util.IntParam
import org.apache.spark.sql.SQLContext
import org.apache.spark.graphx._
import org.apache.spark.graphx.util.GraphGenerators
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.tree.DecisionTree
import org.apache.spark.mllib.tree.model.DecisionTreeModel
import org.apache.spark.mllib.util.MLUtils

val conf = new SparkConf().setAppName("YouTube")
val sqlContext = new SQLContext(sc)

import sqlContext.implicits._
import sqlContext._

val youtubeDF = spark.read.format("csv").option("sep", ",").option("inferSchema", "true").option("header", "true").load("s3a://s3-data-bucket/data.csv")

youtubeDF.registerTempTable("popular")

val fltCountsql = sqlContext.sql("select s.title,s.views from popular s")
fltCountsql.show()
最终,你能够运用kubectl patch command指令更新Spark布置。比方,你能够在负载较高时增加更多作业节点,然后在负载下降后删去这些作业节点。

原文来自:https://cloud.51cto.com/art/202003/612039.htm

本文地址:https://www.linuxprobe.com/kubernetes-spark.html修改:王华超,审核员:逄增宝

Linux指令大全:https://www.linuxcool.com/