MODE 01 · STANDALONE
FIRST RUN
One container
All six Hadoop daemons, one lower-cost place to start.
NameNodeDataNodeResourceManager+3
Hadoop 3.4.1 · local-first · open source
Run the cluster. Learn the system.
Hadoop Lab turns setup, service observation, MapReduce, node failure, and recovery into a repeatable learning path. Start with one container, then reveal the three-node roles when the lesson calls for them.
Built for classes, self-study, and local experiments—not as a production Hadoop platform.
One entrypoint
./hadoop-lab
doctor → up → lesson → status
- preflight checks
- evidence-based status
- narrow, safe recovery
Choose by learning goal
MODE 02 · CLUSTER
3 NODES
Roles separated
Three containers make service placement and failure observable.
hadoop1hadoop2hadoop3
Observe locally
HDFS :9870
YARN :8088
History :19888
Entrypoint
One lifecycle CLI
Check · run · observe · recover
Learning modes
Standalone + cluster
Concepts first, roles second
Guided labs
Seven evidence-led lessons
From first run to failure recovery
The learning path
From machine checks to safe recovery.
Each step produces something you can inspect. The same CLI used in a lesson checks readiness, explains failures, and keeps reset scope visible.
-
PREPARE
Check the machine
Create a local configuration without overwriting existing choices, then check Docker, memory, ports, and Compose before starting anything.
init · doctor -
LAUNCH
Start the right mode
Pull the published multi-architecture image, select standalone or cluster, and wait for the expected services to become healthy.
up standalone · up cluster -
RUN
Produce a real result
Submit the included WordCount job or start a guided lesson. Outputs stay under lesson-specific HDFS paths instead of erasing unrelated work.
demo wordcount · lesson start -
OBSERVE
Inspect the evidence
Compare Docker health, expected JVM processes, HDFS, YARN, JobHistory, and lesson checks instead of treating a running container as proof.
status · open · logs · lesson check -
RECOVER
Reset only what you mean to
Stop while preserving named volumes, recreate runtime state, or explicitly confirm a mode-scoped data reset when the lesson requires a clean slate.
stop · reset · reset --data
Two modes, one model
Start small. Reveal the roles when they matter.
Standalone removes the hardware cost of the first lesson without hiding the core HDFS and YARN concepts. Cluster mode then makes process placement, SSH, and DataNode failure visible.
Read the architecture notesChoose the layout by lesson—not by prestige
standalone
Six daemons in one container. Lower memory, faster feedback, same observation surfaces.
cluster
Three containers. Separate service roles, cross-node SSH, and recoverable node failure.
NameNode · DataNode
Owns HDFS namespace metadata and also contributes local block storage.
ResourceManager · NodeManager · DataNode
Schedules YARN work, executes containers, and contributes HDFS storage.
SecondaryNameNode · JobHistory · DataNode
Makes checkpointing, completed jobs, and a recoverable DataNode failure observable.
Seven guided labs
Lessons that go beyond command copying.
Every lab names an objective, expected evidence, explanation, automated check, and the narrowest useful reset. Students leave with something they can explain—not just a terminal transcript.
00Standalone
First run
Separate container health, Java daemons, and browser observation surfaces.
01Any mode
HDFS basics
Understand the HDFS namespace instead of confusing it with host files.
02Any mode
MapReduce WordCount
Connect input, a YARN application, reducers, and the final HDFS output.
03Any mode
YARN observation
Distinguish live scheduling in ResourceManager from completed JobHistory.
04Cluster
Cluster roles
Locate six services across three nodes and explain why they are separated.
05Cluster
Failure recovery
Take a DataNode offline, inspect degraded evidence, and restore it safely.
06Any mode
Configuration experiment
Trace a value from .env and XML templates into the effective runtime config.
Teach it in 90 minutes.
A suggested classroom sequence connects the same labs into one evidence-led session.
Open the teaching guideObservable by design
One image, two layouts, evidence at every layer.
The lifecycle CLI drives Docker Compose, runtime role settings select daemons, and the same checks connect container state to actual Hadoop processes and learning outcomes.
Read operations and recoveryhadoop-lab CLI
Turns readiness, lifecycle, lessons, logs, diagnosis, and reset into one discoverable interface.
Docker Compose
Selects the standalone or three-node layout while keeping configuration and state ownership explicit.
HDFS · YARN · MapReduce
Runtime roles start the expected daemon set from the same published Hadoop 3.4.1 image.
Checks · UIs · diagnostics
Status, lesson checks, browser UIs, logs, and redacted diagnostic bundles make behavior inspectable.
Diagnostics exclude .env and full container environment values, but every bundle should still be reviewed before sharing.
The honest boundary
A teaching lab that does not pretend to be production.
Keeping the boundary explicit makes destructive experiments safer and the learning surface smaller. Production Hadoop requires a different architecture and operating model.
No Kerberos or identity platform
The lab does not model production authentication, authorization, or enterprise identity integration.
No NameNode HA
There is no active/standby control plane, failover controller, or production availability claim.
No multi-host orchestration
Three containers model roles on one local Docker host; they are not a distributed production deployment.
No backup, capacity, or SLA
Backups, upgrades, capacity planning, monitoring operations, and runtime service levels remain out of scope.
Source available now
Reach the first MapReduce result in four steps.
Use Docker Desktop, or Docker Engine with Compose v2. The normal path pulls the public multi-architecture image; build locally only when changing the image itself.
$ git clone https://github.com/QianQIUlp/docker-hadoop-cluster.git
$ cd docker-hadoop-cluster
$ ./hadoop-lab init
$ ./hadoop-lab doctor
$ ./hadoop-lab up standalone
$ ./hadoop-lab demo wordcount