Docker-based Hadoop/Spark/Hive Orchestration

Last update: 2026-05-12 21:11:23 +0800 CST
tags: Hadoop | Spark | Hive | Docker

I build a series of docker containers to play with the Hadoop product line. I find it uncomfortable to install everything (compatible JRE / Hadoop / Spark) on my native host OS since the whole Hadoop ecosystem is known to be fragile.

To begin with, we have a ro dir that will be mounted readonly:

sh 
ls -lF

# total 2764488
# drwxr-xr-x@ 15 -  -        480 May 12 14:20 apache-hive-4.0.1-bin/
# -rw-r--r--@  1 -  -  408225272 May  8 14:25 apache-hive-4.0.1-bin.tar.gz
# lrwxr-xr-x@  1 -  -         12 May 12 14:21 hadoop@ -> hadoop-3.4.1
# drwxr-xr-x@ 15 -  -        480 Oct 10  2024 hadoop-3.4.1/
# -rw-r--r--@  1 -  -  494173558 May  8 14:25 hadoop-3.4.1-lean.tar.gz
# lrwxr-xr-x@  1 -  -         21 May 12 14:20 hive@ -> apache-hive-4.0.1-bin
# lrwxr-xr-x@  1 -  -         40 May 12 14:21 jdk17@ -> zulu17.58.21-ca-jdk17.0.15-linux_aarch64
# lrwxr-xr-x@  1 -  -         30 May 12 14:21 spark@ -> spark-3.5.6-bin-without-hadoop
# drwxr-xr-x@ 17 -  -        544 May 23  2025 spark-3.5.6-bin-without-hadoop/
# -rw-r--r--@  1 -  -  314466745 May  8 14:25 spark-3.5.6-bin-without-hadoop.tgz
# drwxr-xr-x@ 14 -  -        448 Apr  4  2025 zulu17.58.21-ca-jdk17.0.15-linux_aarch64/
# -rw-r--r--@  1 -  -  198542235 May  8 14:25 zulu17.58.21-ca-jdk17.0.15-linux_aarch64.tar.gz

Minimal offline setup #

  • Stock JDK 17 + hadoop + spark + hive, mount readonly.
  • all bin/ from works, classpath properlly configured.
  • all config files are stock, unmodified.
  • HDFS and YARN is complete offline.

This should be enough for beginners to play with spark-shell. docker exec into the container after docker compoes up to play with the environment.

yaml 
# compose.yaml
services:
  base:
    image: debian:bookworm-slim
    entrypoint:
      - /bin/sh
      - /usr/local/bin/startup.sh
    environment:
      JAVA_HOME: /opt/java
      HADOOP_HOME: /opt/hadoop
      SPARK_HOME: /opt/spark
      HIVE_HOME: /opt/hive
      PATH: /opt/java/bin:/opt/hadoop/bin:/opt/spark/bin:/opt/hive/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    volumes:
      - ./startup.sh:/usr/local/bin/startup.sh:ro
      - ../ro:/opt/ro:ro

startup.sh will setup symlinks and SPARK_DIST_CLASSPATH whose value can only be determined at runtime.

sh 
#!/bin/sh

set -eu

mkdir -p /opt/ro

ln -sfn /opt/ro/zulu17.58.21-ca-jdk17.0.15-linux_aarch64 /opt/java
ln -sfn /opt/ro/hadoop-3.4.1 /opt/hadoop
ln -sfn /opt/ro/spark-3.5.6-bin-without-hadoop /opt/spark
ln -sfn /opt/ro/apache-hive-4.0.1-bin /opt/hive

echo 'export SPARK_DIST_CLASSPATH="$(hadoop classpath)"' >> /etc/bash.bashrc

exec sleep infinity

HDFS NameNode*1 + DataNode*3 #

This is a reasonably