Docker-based Hadoop/Spark/Hive Orchestration

Hadoop, Spark, and Hive have overlapping Java, Hadoop-client, and configuration requirements. Installing every distribution on a workstation also mixes cluster state with host files. A Docker Compose lab avoids both problems by mounting an immutable software tree at /opt/ro and placing daemon state in named volumes.

The guide builds the lab in five stages:

Start an offline toolbox with the JDK, Hadoop, Hive, and Spark commands but no cluster daemons.
Add one HDFS NameNode and three DataNodes.
Add one YARN ResourceManager and three NodeManagers.
Add HiveServer2 with an embedded Derby metastore.
Run Spark locally and on YARN.

Scope and Architecture #

The lab runs all roles on one Docker host:

Role	Hostname	Address	Published Port
Toolbox	`toolbox`	`172.28.0.10`	None
NameNode	`namenode`	`172.28.0.11`	`127.0.0.1:9870`
DataNode 1	`datanode1`	`172.28.0.12`	None
DataNode 2	`datanode2`	`172.28.0.13`	None
DataNode 3	`datanode3`	`172.28.0.14`	None
ResourceManager	`resourcemanager`	`172.28.0.20`	`127.0.0.1:8088`
NodeManager 1	`nodemanager1`	`172.28.0.21`	None
NodeManager 2	`nodemanager2`	`172.28.0.22`	None
NodeManager 3	`nodemanager3`	`172.28.0.23`	None
HiveServer2	`hiveserver2`	`172.28.0.30`	`127.0.0.1:10000`

Every container receives two read-only bind mounts:

/opt/ro contains extracted binary distributions and stable symlinks.
/opt/conf contains the shared environment script and XML configuration.

Named volumes hold HDFS, YARN, Hive, and toolbox state. Temporary files and daemon logs use container tmpfs mounts because Compose captures foreground-process output through docker compose logs.

The topology demonstrates process boundaries, HDFS replication, YARN scheduling, HiveServer2, and Spark submission. Three DataNodes on one physical host do not provide fault tolerance because a host failure still removes every replica.

Treat Version Compatibility as a Manifest Decision #

Do not infer compatibility from a version number or from a successful process launch. Before downloading a combination, verify all of the following against the official release notes and documentation for the exact releases:

The JDK version and CPU architecture supported by Hadoop, Spark, and Hive.
The Hadoop client line against which the Spark without-hadoop distribution was built and tested.
The Hadoop and execution-engine versions supported by Hive.
Any required JVM module flags, removed components, or changed configuration properties.
Whether the binary archives contain the example and client JARs used by the validation commands.

The paths below preserve one concrete example manifest. They are examples rather than a compatibility claim:

text

ro/
├── hadoop -> hadoop-3.5.0
├── hadoop-3.5.0/
├── hive -> apache-hive-4.2.0-bin
├── apache-hive-4.2.0-bin/
├── jdk17 -> zulu17.58.21-ca-jdk17.0.15-linux_aarch64
├── zulu17.58.21-ca-jdk17.0.15-linux_aarch64/
├── spark -> spark-4.1.2-bin-without-hadoop
└── spark-4.1.2-bin-without-hadoop/

Replace the symlink targets with a combination verified for the intended host architecture. Keep the stable symlink names because the remaining configuration refers only to /opt/ro/jdk17, /opt/ro/hadoop, /opt/ro/hive, and /opt/ro/spark.

The Compose image supplies a Linux userspace, not Java or Hadoop. An air-gapped host must already contain debian:bookworm-slim, for example through docker load, in addition to the files under ro/.

Directory Layout #

Create the following tree:

text

hadoop-lab/
├── compose.yaml
├── conf/
│   ├── init.sh
│   ├── hadoop/
│   │   ├── core-site.xml
│   │   ├── hdfs-site.xml
│   │   ├── mapred-site.xml
│   │   └── yarn-site.xml
│   └── hive/
│       └── hive-site.xml
└── ro/
    ├── hadoop -> hadoop-3.5.0
    ├── hadoop-3.5.0/
    ├── hive -> apache-hive-4.2.0-bin
    ├── apache-hive-4.2.0-bin/
    ├── jdk17 -> zulu17.58.21-ca-jdk17.0.15-linux_aarch64
    ├── zulu17.58.21-ca-jdk17.0.15-linux_aarch64/
    ├── spark -> spark-4.1.2-bin-without-hadoop
    └── spark-4.1.2-bin-without-hadoop/

Archive files may remain outside ro/; containers need only the extracted distributions. Create relative symlinks so moving the lab directory does not break them:

cd hadoop-lab/ro
ln -s hadoop-3.5.0 hadoop
ln -s apache-hive-4.2.0-bin hive
ln -s zulu17.58.21-ca-jdk17.0.15-linux_aarch64 jdk17
ln -s spark-4.1.2-bin-without-hadoop spark

Add the Shared Environment Script #

Save the following script as conf/init.sh:

bash

#!/usr/bin/env bash
set -euo pipefail

export JAVA_HOME=/opt/ro/jdk17
export HADOOP_HOME=/opt/ro/hadoop
export HADOOP_COMMON_HOME="$HADOOP_HOME"
export HADOOP_HDFS_HOME="$HADOOP_HOME"
export HADOOP_MAPRED_HOME="$HADOOP_HOME"
export HADOOP_YARN_HOME="$HADOOP_HOME"
export HADOOP_CONF_DIR=/opt/conf/hadoop
export YARN_CONF_DIR="$HADOOP_CONF_DIR"
export HIVE_HOME=/opt/ro/hive
export HIVE_CONF_DIR=/opt/conf/hive
export SPARK_HOME=/opt/ro/spark

export HADOOP_LOG_DIR=/var/log/hadoop
export YARN_LOG_DIR=/var/log/hadoop
export HADOOP_PID_DIR=/run/hadoop
export YARN_PID_DIR=/run/hadoop
export HIVE_LOG_DIR=/var/log/hive

# Hive must precede Spark because both distributions can contain "beeline".
export PATH="$HIVE_HOME/bin:$SPARK_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
export LD_LIBRARY_PATH="$HADOOP_HOME/lib/native${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

# A Spark "without-hadoop" archive needs the selected Hadoop client's JARs.
if [[ -z "${SPARK_DIST_CLASSPATH:-}" ]]; then
  export SPARK_DIST_CLASSPATH="$("$HADOOP_HOME/bin/hadoop" classpath)"
fi

wait_for_port() {
  local host="$1"
  local port="$2"
  local timeout="${3:-120}"
  local deadline=$((SECONDS + timeout))

  until (echo >"/dev/tcp/${host}/${port}") >/dev/null 2>&1; do
    if ((SECONDS >= deadline)); then
      echo "Timed out waiting for ${host}:${port}" >&2
      return 1
    fi
    sleep 2
  done
}

wait_for_hdfs() {
  local timeout="${1:-180}"
  local deadline=$((SECONDS + timeout))
  local status

  until status="$(hdfs dfsadmin -safemode get 2>/dev/null)" &&
    grep -q "Safe mode is OFF" <<<"$status"; do
    if ((SECONDS >= deadline)); then
      echo "Timed out waiting for writable HDFS" >&2
      return 1
    fi
    sleep 3
  done
}

wait_for_yarn_nodes() {
  local expected="${1:-1}"
  local timeout="${2:-180}"
  local deadline=$((SECONDS + timeout))
  local output
  local running

  while true; do
    output="$(yarn node -list -all 2>/dev/null || true)"
    running="$(awk '$0 ~ /RUNNING/ { count++ } END { print count + 0 }' <<<"$output")"
    if ((running >= expected)); then
      return 0
    fi
    if ((SECONDS >= deadline)); then
      echo "Timed out waiting for ${expected} YARN NodeManagers; found ${running}" >&2
      return 1
    fi
    sleep 3
  done
}

Every service sources the same file, so daemon and client processes use the same Java installation, Hadoop configuration, and binary ordering.

Configure Hadoop #

Save the following files under conf/hadoop/.

`core-site.xml` #

xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://namenode:8020</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/lib/hadoop/tmp</value>
  </property>
  <property>
    <name>io.file.buffer.size</name>
    <value>131072</value>
  </property>
</configuration>

`hdfs-site.xml` #

xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///var/lib/hadoop/name</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///var/lib/hadoop/data</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address</name>
    <value>namenode:8020</value>
  </property>
  <property>
    <name>dfs.namenode.http-address</name>
    <value>0.0.0.0:9870</value>
  </property>
  <property>
    <name>dfs.datanode.address</name>
    <value>0.0.0.0:9866</value>
  </property>
  <property>
    <name>dfs.datanode.http.address</name>
    <value>0.0.0.0:9864</value>
  </property>
  <property>
    <name>dfs.client.use.datanode.hostname</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.datanode.use.datanode.hostname</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
    <value>false</value>
  </property>
  <property>
    <name>dfs.permissions.enabled</name>
    <value>true</value>
  </property>
</configuration>

`yarn-site.xml` #

xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>resourcemanager</value>
  </property>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>resourcemanager:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>resourcemanager:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>resourcemanager:8031</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>resourcemanager:8033</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>0.0.0.0:8088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.recovery.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.resourcemanager.store.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore</value>
  </property>
  <property>
    <name>yarn.resourcemanager.fs.state-store.uri</name>
    <value>file:///var/lib/hadoop/yarn/rm-state</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  <property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/var/lib/hadoop/yarn/local</value>
  </property>
  <property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/var/lib/hadoop/yarn/log</value>
  </property>
  <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>1536</value>
  </property>
  <property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>2</value>
  </property>
  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>256</value>
  </property>
  <property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>1536</value>
  </property>
  <property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
  </property>
  <property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
  </property>
  <property>
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME,LD_LIBRARY_PATH,PATH,LANG,TZ</value>
  </property>
  <property>
    <name>yarn.application.classpath</name>
    <value>$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*,$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*</value>
  </property>
  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>false</value>
  </property>
</configuration>

Physical-memory enforcement is disabled because container and JDK accounting can otherwise reject small lab jobs before they start. Production deployments should enforce explicit container limits and retain YARN’s memory checks.

`mapred-site.xml` #

xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
  </property>
  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>512</value>
  </property>
  <property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx384m</value>
  </property>
  <property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>512</value>
  </property>
  <property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx384m</value>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.resource.mb</name>
    <value>512</value>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.command-opts</name>
    <value>-Xmx384m</value>
  </property>
</configuration>

Configure Hive #

Save the following file as conf/hive/hive-site.xml:

xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:derby:;databaseName=/var/lib/hive/metastore_db;create=true</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>org.apache.derby.jdbc.EmbeddedDriver</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>APP</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>mine</value>
  </property>
  <property>
    <name>hive.metastore.schema.verification</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/warehouse/tablespace/managed/hive</value>
  </property>
  <property>
    <name>hive.metastore.warehouse.external.dir</name>
    <value>/warehouse/tablespace/external/hive</value>
  </property>
  <property>
    <name>hive.server2.thrift.bind.host</name>
    <value>0.0.0.0</value>
  </property>
  <property>
    <name>hive.server2.thrift.port</name>
    <value>10000</value>
  </property>
  <property>
    <name>hive.server2.authentication</name>
    <value>NONE</value>
  </property>
  <property>
    <name>hive.server2.enable.doAs</name>
    <value>false</value>
  </property>
  <property>
    <name>hive.execution.engine</name>
    <value>mr</value>
  </property>
  <property>
    <name>hive.exec.scratchdir</name>
    <value>/tmp/hive</value>
  </property>
  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/tmp/hive</value>
  </property>
  <property>
    <name>hive.server2.logging.operation.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.server2.logging.operation.log.location</name>
    <value>/var/lib/hive/operation_logs</value>
  </property>
</configuration>

No hive.metastore.uris property is present because HiveServer2 opens Derby in-process. Derby embedded mode permits only one process to own the database, so do not run a second HiveServer2, hive CLI, or standalone metastore against the same volume. Beeline remains safe because it connects to HiveServer2 over JDBC rather than opening Derby.

The example selects MapReduce as Hive’s execution engine to reuse the YARN services already in the lab. Confirm that the chosen Hive release still ships and supports that engine; otherwise install and configure an execution engine documented for that release.

Compose the Lab #

Save the following file as compose.yaml:

yaml

name: hadoop-lab

x-common-environment: &common-environment
  HADOOP_USER_NAME: "${HADOOP_USER_NAME:-root}"

x-common: &common
  image: debian:bookworm-slim
  user: "${LAB_UID:-0}:${LAB_GID:-0}"
  read_only: true
  init: true
  working_dir: /tmp
  stop_grace_period: 30s
  environment: *common-environment
  tmpfs:
    - /tmp:rw,exec,nosuid,size=1g
    - /run:rw,nosuid,size=64m
    - /var/log/hadoop:rw,nosuid,size=256m
    - /var/log/hive:rw,nosuid,size=256m

x-datanode: &datanode
  <<: *common
  depends_on:
    namenode:
      condition: service_started
  environment:
    <<: *common-environment
    HADOOP_DATANODE_OPTS: "-Xms128m -Xmx512m"
  entrypoint: ["/bin/bash", "-lc"]
  command:
    - |
      source /opt/conf/init.sh
      mkdir -p /var/lib/hadoop/data /var/lib/hadoop/tmp
      wait_for_port namenode 8020 180
      exec hdfs datanode

x-nodemanager: &nodemanager
  <<: *common
  depends_on:
    resourcemanager:
      condition: service_started
  environment:
    <<: *common-environment
    YARN_NODEMANAGER_OPTS: "-Xms128m -Xmx512m"
  entrypoint: ["/bin/bash", "-lc"]
  command:
    - |
      source /opt/conf/init.sh
      mkdir -p /var/lib/hadoop/yarn/local /var/lib/hadoop/yarn/log
      wait_for_port resourcemanager 8031 180
      exec yarn nodemanager

services:
  toolbox:
    <<: *common
    hostname: toolbox
    working_dir: /work
    entrypoint: ["/bin/bash", "-lc"]
    command: ["source /opt/conf/init.sh; exec sleep infinity"]
    volumes:
      - ./ro:/opt/ro:ro
      - ./conf:/opt/conf:ro
      - toolbox-state:/var/lib/hadoop
      - toolbox-work:/work
    networks:
      lab:
        ipv4_address: 172.28.0.10

  namenode:
    <<: *common
    hostname: namenode
    environment:
      <<: *common-environment
      HADOOP_NAMENODE_OPTS: "-Xms256m -Xmx768m"
    entrypoint: ["/bin/bash", "-lc"]
    command:
      - |
        source /opt/conf/init.sh
        mkdir -p /var/lib/hadoop/name /var/lib/hadoop/tmp
        if [[ ! -f /var/lib/hadoop/name/current/VERSION ]]; then
          hdfs namenode -format -clusterid hadoop-lab -nonInteractive
        fi
        exec hdfs namenode
    volumes:
      - ./ro:/opt/ro:ro
      - ./conf:/opt/conf:ro
      - namenode-state:/var/lib/hadoop
    ports:
      - "127.0.0.1:9870:9870"
    networks:
      lab:
        ipv4_address: 172.28.0.11

  datanode1:
    <<: *datanode
    hostname: datanode1
    volumes:
      - ./ro:/opt/ro:ro
      - ./conf:/opt/conf:ro
      - datanode1-state:/var/lib/hadoop
    networks:
      lab:
        ipv4_address: 172.28.0.12

  datanode2:
    <<: *datanode
    hostname: datanode2
    volumes:
      - ./ro:/opt/ro:ro
      - ./conf:/opt/conf:ro
      - datanode2-state:/var/lib/hadoop
    networks:
      lab:
        ipv4_address: 172.28.0.13

  datanode3:
    <<: *datanode
    hostname: datanode3
    volumes:
      - ./ro:/opt/ro:ro
      - ./conf:/opt/conf:ro
      - datanode3-state:/var/lib/hadoop
    networks:
      lab:
        ipv4_address: 172.28.0.14

  resourcemanager:
    <<: *common
    hostname: resourcemanager
    environment:
      <<: *common-environment
      YARN_RESOURCEMANAGER_OPTS: "-Xms256m -Xmx768m"
    entrypoint: ["/bin/bash", "-lc"]
    command:
      - |
        source /opt/conf/init.sh
        mkdir -p /var/lib/hadoop/yarn/rm-state /var/lib/hadoop/tmp
        exec yarn resourcemanager
    volumes:
      - ./ro:/opt/ro:ro
      - ./conf:/opt/conf:ro
      - resourcemanager-state:/var/lib/hadoop
    ports:
      - "127.0.0.1:8088:8088"
    networks:
      lab:
        ipv4_address: 172.28.0.20

  nodemanager1:
    <<: *nodemanager
    hostname: nodemanager1
    volumes:
      - ./ro:/opt/ro:ro
      - ./conf:/opt/conf:ro
      - nodemanager1-state:/var/lib/hadoop
    networks:
      lab:
        ipv4_address: 172.28.0.21

  nodemanager2:
    <<: *nodemanager
    hostname: nodemanager2
    volumes:
      - ./ro:/opt/ro:ro
      - ./conf:/opt/conf:ro
      - nodemanager2-state:/var/lib/hadoop
    networks:
      lab:
        ipv4_address: 172.28.0.22

  nodemanager3:
    <<: *nodemanager
    hostname: nodemanager3
    volumes:
      - ./ro:/opt/ro:ro
      - ./conf:/opt/conf:ro
      - nodemanager3-state:/var/lib/hadoop
    networks:
      lab:
        ipv4_address: 172.28.0.23

  hiveserver2:
    <<: *common
    hostname: hiveserver2
    depends_on:
      namenode:
        condition: service_started
      resourcemanager:
        condition: service_started
    environment:
      <<: *common-environment
      HADOOP_HEAPSIZE: "1024"
    entrypoint: ["/bin/bash", "-lc"]
    command:
      - |
        source /opt/conf/init.sh
        mkdir -p /var/lib/hive/operation_logs /var/lib/hadoop/tmp
        wait_for_port namenode 8020 180
        wait_for_hdfs 180
        wait_for_port resourcemanager 8032 180
        wait_for_yarn_nodes 1 180
        hdfs dfs -mkdir -p /tmp /tmp/hive
        hdfs dfs -chmod 1777 /tmp /tmp/hive
        hdfs dfs -mkdir -p /warehouse/tablespace/managed/hive
        hdfs dfs -mkdir -p /warehouse/tablespace/external/hive
        hdfs dfs -chmod -R 1777 /warehouse/tablespace
        if ! /opt/ro/hive/bin/schematool -dbType derby -info; then
          /opt/ro/hive/bin/schematool -dbType derby -initSchema
        fi
        /opt/ro/hive/bin/schematool -dbType derby -info
        exec /opt/ro/hive/bin/hiveserver2
    volumes:
      - ./ro:/opt/ro:ro
      - ./conf:/opt/conf:ro
      - hiveserver2-state:/var/lib/hive
      - hiveserver2-hadoop-state:/var/lib/hadoop
    ports:
      - "127.0.0.1:10000:10000"
    networks:
      lab:
        ipv4_address: 172.28.0.30

networks:
  lab:
    name: hadoop-lab-net
    ipam:
      config:
        - subnet: 172.28.0.0/24

volumes:
  toolbox-state:
    name: hadoop-lab-toolbox-state
  toolbox-work:
    name: hadoop-lab-toolbox-work
  namenode-state:
    name: hadoop-lab-namenode-state
  datanode1-state:
    name: hadoop-lab-datanode1-state
  datanode2-state:
    name: hadoop-lab-datanode2-state
  datanode3-state:
    name: hadoop-lab-datanode3-state
  resourcemanager-state:
    name: hadoop-lab-resourcemanager-state
  nodemanager1-state:
    name: hadoop-lab-nodemanager1-state
  nodemanager2-state:
    name: hadoop-lab-nodemanager2-state
  nodemanager3-state:
    name: hadoop-lab-nodemanager3-state
  hiveserver2-state:
    name: hadoop-lab-hiveserver2-state
  hiveserver2-hadoop-state:
    name: hadoop-lab-hiveserver2-hadoop-state

depends_on controls container creation order only. DataNodes, NodeManagers, and HiveServer2 also use active readiness loops because a started container does not imply that its daemon socket or distributed service is ready.

The NameNode formats its volume only when current/VERSION is absent. Restarts therefore preserve the namespace rather than formatting it again.

Stage 1: Start the Offline Toolbox #

Start only the toolbox:

docker compose up -d toolbox
docker compose exec toolbox bash -lc '
  source /opt/conf/init.sh
  java -version
  hadoop version
  hive --version
  spark-submit --version
  command -v beeline
'

HDFS and YARN commands load their cluster configuration, but no daemon is running yet. The stage remains useful for inspecting classpaths and running computations that do not access HDFS:

docker compose exec toolbox bash -lc '
  source /opt/conf/init.sh
  spark-submit \
    --master "local[2]" \
    --class org.apache.spark.examples.SparkPi \
    "$SPARK_HOME"/examples/jars/spark-examples_*.jar \
    10
'

The command should print a Pi is roughly ... result from Spark’s example application. Failure before Spark creates its context generally indicates a Java, architecture, or classpath mismatch rather than an HDFS problem.

Stage 2: Start HDFS #

Start the NameNode and all three DataNodes:

docker compose up -d namenode datanode1 datanode2 datanode3

Wait for HDFS to leave safe mode, inspect the DataNode report, and perform a write/read cycle:

docker compose exec toolbox bash -lc '
  source /opt/conf/init.sh
  wait_for_hdfs 180
  hdfs dfsadmin -report
  hdfs dfs -mkdir -p /tmp /user/root/lab
  hdfs dfs -chmod 1777 /tmp
  printf "hdfs-ok\n" > /work/hdfs-check.txt
  hdfs dfs -put -f /work/hdfs-check.txt /user/root/lab/
  hdfs dfs -cat /user/root/lab/hdfs-check.txt
'

The report should list three live DataNodes. Open http://127.0.0.1:9870/ for the NameNode UI.

Stage 3: Start YARN #

Start the ResourceManager and all three NodeManagers:

docker compose up -d \
  resourcemanager \
  nodemanager1 nodemanager2 nodemanager3

Wait for all workers and submit the bundled MapReduce pi example:

docker compose exec toolbox bash -lc '
  source /opt/conf/init.sh
  wait_for_yarn_nodes 3 180
  yarn node -list -all
  hadoop jar \
    "$HADOOP_MAPRED_HOME"/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar \
    pi 2 10
'

Open http://127.0.0.1:8088/ to inspect the application, containers, and NodeManagers. A process that runs entirely in the toolbox does not validate YARN; the ResourceManager UI or yarn application -list -appStates ALL must show the submitted application.

Stage 4: Start HiveServer2 #

Start the single HiveServer2 process:

docker compose up -d hiveserver2

HiveServer2 waits for writable HDFS and at least one running NodeManager, creates warehouse directories, initializes the Derby schema when necessary, and then starts its Thrift endpoint. Wait for the endpoint before invoking Beeline:

docker compose exec toolbox bash -lc '
  source /opt/conf/init.sh
  wait_for_port hiveserver2 10000 240
  beeline \
    -u jdbc:hive2://hiveserver2:10000/default \
    -n root \
    --silent=true \
    -e "
      CREATE TABLE IF NOT EXISTS lab_numbers (n INT);
      INSERT INTO lab_numbers VALUES (1), (2), (3);
      SELECT SUM(n) AS total FROM lab_numbers;
    "
'

The final query should return 6. DDL success validates HiveServer2, Derby, and the HDFS warehouse. The insert and aggregation also validate the configured execution path. Consult the HiveServer2 and YARN logs if DDL succeeds but the query engine fails.

Stage 5: Run Spark on YARN #

Run the same Spark example through YARN:

docker compose exec toolbox bash -lc '
  source /opt/conf/init.sh
  wait_for_yarn_nodes 3 180
  spark-submit \
    --master yarn \
    --deploy-mode client \
    --conf spark.executor.instances=2 \
    --conf spark.executor.cores=1 \
    --conf spark.executor.memory=512m \
    --conf spark.driver.memory=512m \
    --class org.apache.spark.examples.SparkPi \
    "$SPARK_HOME"/examples/jars/spark-examples_*.jar \
    10
'

YARN should allocate an ApplicationMaster and two executors. Confirm completion independently:

docker compose exec toolbox bash -lc '
  source /opt/conf/init.sh
  yarn application -list -appStates ALL
'

Understand `SPARK_DIST_CLASSPATH` and `PATH` Precedence #

A Spark without-hadoop archive omits the Hadoop client JARs. Spark reads SPARK_DIST_CLASSPATH, so the environment script populates it from the selected Hadoop distribution:

export SPARK_DIST_CLASSPATH="$(hadoop classpath)"

The setting lets the Spark submission client load YARN and HDFS classes while preserving a single Hadoop installation under /opt/ro/hadoop. Spark uploads or references its own application artifacts through YARN; NodeManagers inherit the same JAVA_HOME and Hadoop environment from their daemon process.

Some Spark distributions also contain a beeline launcher, but a without-hadoop archive may not contain the Hive classes expected by that launcher. Placing $HIVE_HOME/bin before $SPARK_HOME/bin ensures that an unqualified beeline command uses Hive’s client:

command -v beeline
# /opt/ro/hive/bin/beeline

Spark’s own scripts remain unambiguous because spark-submit, spark-shell, and their internal relative launchers come from $SPARK_HOME.

Persistence and Targeted Reset #

docker compose down removes containers and the dedicated network but retains every named volume:

docker compose down

Starting the services again reuses the HDFS namespace, DataNode blocks, ResourceManager recovery state, NodeManager local state, Derby database, and toolbox files.

Reset only the subsystem whose state is invalid:

# Reset Hive metadata and HiveServer2-local Hadoop scratch state.
docker compose stop hiveserver2
docker compose rm -f hiveserver2
docker volume rm \
  hadoop-lab-hiveserver2-state \
  hadoop-lab-hiveserver2-hadoop-state

# Reset YARN state without deleting HDFS or Hive.
docker compose stop resourcemanager nodemanager1 nodemanager2 nodemanager3
docker compose rm -f resourcemanager nodemanager1 nodemanager2 nodemanager3
docker volume rm \
  hadoop-lab-resourcemanager-state \
  hadoop-lab-nodemanager1-state \
  hadoop-lab-nodemanager2-state \
  hadoop-lab-nodemanager3-state

# Reset HDFS as one consistency unit.
docker compose stop namenode datanode1 datanode2 datanode3
docker compose rm -f namenode datanode1 datanode2 datanode3
docker volume rm \
  hadoop-lab-namenode-state \
  hadoop-lab-datanode1-state \
  hadoop-lab-datanode2-state \
  hadoop-lab-datanode3-state

Resetting only the NameNode or only a DataNode can leave incompatible cluster IDs or orphaned block state. Remove all four HDFS volumes together when a fresh filesystem is required. None of the commands above deletes unrelated Docker volumes or host directories.

Troubleshooting #

UID and Volume Permissions #

The Compose file defaults to UID and GID 0 because a stock read-only Debian image has no matching lab account and Docker creates named volumes as root. The containers remain isolated from host bind-mounted write paths, but root inside a container is still unsuitable for production.

To run with another numeric identity, set both variables and make each service volume writable by that identity. A narrowly scoped repair for an existing NameNode volume is:

LAB_UID=1000 LAB_GID=1000 \
docker compose run --rm --user 0:0 --entrypoint bash namenode \
  -lc 'chown -R 1000:1000 /var/lib/hadoop'

Repeat the operation only for the affected service volumes. A purpose-built image with a real passwd entry for the chosen UID is preferable because Hadoop and YARN may query operating-system user and group information.

CPU Architecture Mismatch #

The JDK under /opt/ro must be a Linux build for the container architecture. A macOS JDK cannot run in a Linux container, and an AArch64 JDK cannot run in an AMD64 container without matching emulation:

docker compose run --rm --entrypoint bash toolbox -lc '
  uname -m
  dpkg --print-architecture
  /opt/ro/jdk17/bin/java -version
'

An Exec format error indicates a binary for the wrong architecture or operating system. Select archives for the same architecture or set an explicitly supported Docker platform and accept the cost and limitations of emulation.

Hostname and Configuration Errors #

Errors such as UnknownHostException, DataNode registration failures, or YARN clients connecting to localhost indicate inconsistent service names or configuration mounts. Inspect resolution and active configuration from the toolbox:

docker compose exec toolbox bash -lc '
  source /opt/conf/init.sh
  getent hosts namenode resourcemanager hiveserver2
  hdfs getconf -confKey fs.defaultFS
  yarn getconf -confKey yarn.resourcemanager.hostname
'

Keep Compose service names, hostname values, fixed addresses, and XML hostnames aligned. Recreate containers after changing a read-only configuration file:

docker compose up -d --force-recreate

Memory and Stalled Allocations #

Three NodeManagers advertise 1536 MiB each, while Hadoop, Hive, Spark, and Docker also require memory. Allocate enough memory to the Docker runtime, then inspect pending requests:

docker compose exec toolbox bash -lc '
  source /opt/conf/init.sh
  yarn node -list -all
  yarn application -list -appStates ACCEPTED,RUNNING
'

Reduce Spark executor count or memory before increasing YARN’s advertised capacity. Advertised YARN memory does not create physical memory.

Logs and Failed Containers #

Foreground daemons send their primary output to Compose:

docker compose ps
docker compose logs --tail=200 namenode
docker compose logs --tail=200 nodemanager1
docker compose logs --tail=200 hiveserver2

Application diagnostics are also available through YARN:

docker compose exec toolbox bash -lc '
  source /opt/conf/init.sh
  yarn application -list -appStates ALL
  yarn logs -applicationId APPLICATION_ID
'

Replace APPLICATION_ID with the identifier reported by YARN.

Derby Schema Initialization #

An interrupted first start can leave a partial Derby schema. Inspect the error before deleting anything:

docker compose run --rm --no-deps --entrypoint bash hiveserver2 -lc '
  source /opt/conf/init.sh
  schematool -dbType derby -info
'

If the database contains no needed metadata and schema initialization cannot recover, reset only hadoop-lab-hiveserver2-state as described above. Do not start another process against the same embedded Derby directory.

Java Compatibility #

UnsupportedClassVersionError, module-access failures, removed Java APIs, or native-library errors indicate an unsupported release combination. Capture the exact versions:

docker compose exec toolbox bash -lc '
  source /opt/conf/init.sh
  java -version
  hadoop version
  hive --version
  spark-submit --version
'

Resolve the mismatch by selecting a combination documented by the component maintainers. Do not add broad JVM --add-opens flags or replace JARs until the relevant release documentation requires that change.

Security and Production Boundaries #

The lab deliberately omits controls required by a production deployment:

Hadoop uses simple authentication rather than Kerberos.
HiveServer2 accepts unauthenticated connections and does not use TLS.
HDFS and YARN authorization policies are not hardened.
HiveServer2 impersonation is disabled.
Derby provides no concurrent metastore service or external backup strategy.
Daemons run as root by default inside containers.
Published ports bind to loopback, but every container on hadoop-lab-net can reach the services.
HDFS replicas and YARN workers share one Docker host.
Images and mounted archives are not pinned by digest or verified by checksum in Compose.
Resource limits, monitoring, secret management, high availability, and rolling upgrades are absent.

Use the topology for local integration work and protocol inspection. A production cluster requires authenticated services, separate hosts or failure domains, a supported external metastore database, non-root identities, durable logging, resource isolation, and a compatibility matrix maintained with each upgrade.