Apache Spark

Spark application is just a type of yarn application.

Apache Spark is written in Scala and runs on any JVM as long as compatible scala runtime jars are presented on classpath.

A worker node requires HDFS and YARN client to work, and read / write files on a HDFS for job input and output.

Overview #

The user write a JVM bytecode program (either in Java or Scala), known as the driver program, and package into a foo.jar jar file. This jar file may include all dependencies (a.k.a a uber jar), or lack them in expectation of they are provided by worker node environment.

The driver program looks like:

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;

public class SimpleApp {
  public static void main(String[] args) {
    SparkSession spark = SparkSession.builder().appName("Simple Application").getOrCreate();
    Dataset<String> logData = spark.read().textFile("SOME_DIR/SOME_FILE.md").cache();

    long numAs = logData.filter(s -> s.contains("a")).count();
    long numBs = logData.filter(s -> s.contains("b")).count();

    System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);

    spark.stop();
  }
}

The user then should run this program by

./bin/spark-submit \
  --master 'spark://master-server-addr:port' \
  --class org.full.package.SimpleApp \
  foo.jar

which will immediately start a local JVM process and runs this program, all locally.

Then:

1. upon the create of SparkSession instance, this program (the driver) will establish TCP connection with the master node, saying "hi master, I need a few executor processes; please allocate for me the executors, and once they are ready, talk to me via TCP; this is my ip address and listening tcp port."
2. This driver program then blocks and listens for incoming connections from executors throughout its lifetime.

• The driver program doesn’t care about what exactly the master (a.k.a cluster manager) is, as long as the master understand the protocol, and in the future, some executors’ TCP connections comes in.

Misc #

1. Each instance of SparkContext are known as a Spark application. You can have multiple SparkContext in the same driver program that are completely agnoistic to each other.
2. Within the same spark application, multiple spark action a.k.a jobs can be run concurrently.

• The SparkContext is (should be) a singleton and can only be created once in a Spark application.
• The SparkSession, on the other hand, can be created multiple times within an application