Docker Misc Q&A

Fun things and memorandum about Docker.

Refer to the host machine’s IP address from container #

• For MacOS host + Docker Desktop

  the name docker.for.mac.host.internal resolves to the IP address of the virtual NIC installed on MacOS, automatically without any need of configuration. Of course the VM layer (https://github.com/linuxkit/linuxkit/blob/master/examples/docker-for-mac.md) still exist, but at least you have the IPv4 routing working delivering real TCP.

• For Linux host + Docker engine

  the docker run and create command accepts a --add-host=name:<ip> flag to add a <ip> name line in container’s /etc/hosts. A special literal value host-gateway can be used for <ip> to refer to the IP address on the virtual NIC in the host’s Linux kernel. (the Docker engine will set it up). So eventually you get a line like 172.0.0.1 name.

An example of allowing containerized GUI to talk to host’s X11 server socket:

docker run --name gugu -it --entrypoint bash --user jose --add-host=hostgg:host-gateway -v $AA:$AA -v $BB:$CC -e DISPLAY=:0 -e XAUTHORITY=$CC kk

# in the container
getent hosts hostgg
172.17.0.1      hostgg

Reuse component in docker compose yaml #

There are 2 ways practicing the DRY principle in docker compose yaml file:

1. use the extends keywords
2. Use the YAML anchor/merge syntax for reusable pieces of configuration in docker compose yaml

networks:
  gugunet:
    driver: bridge
    enable_ipv6: false
    ipam:
      driver: default
      config:
        - subnet: 99.99.99.0/24
          gateway: 99.99.99.1

x-hdfs-commons: &hdfs-commons
  # hadoop launcher scripts requires bash, don't use alpine.
  image: ubuntu:latest
  volumes:
    - ../rofiles/zulu17.58.21-ca-jdk17.0.15-linux_aarch64:/opt/jdk:ro
    - ../rofiles/hadoop-3.4.1:/opt/hadoop:ro
    - ./core-site.xml:/opt/hadoop/etc/hadoop/core-site.xml:ro
#    - ./hdfs-site.xml:/opt/hadoop/etc/hadoop/hdfs-site.xml:ro
    - ./hadoop-logs:/tmp/hadoop-logs
  environment:
    - JAVA_HOME=/opt/jdk
    - PATH=$PATH:/opt/jdk/bin:/opt/hadoop/bin
    - HADOOP_LOG_DIR=/tmp/hadoop-logs


services:
  hdfs-name-node:
    <<: *hdfs-commons
    networks:
      gugunet:
        ipv4_address: 99.99.99.100
    command: bash -c "hdfs namenode -format && hdfs --daemon start namenode && sleep infinity"
  hdfs-data-node-1:
    <<: *hdfs-commons
    command: bash -c "hdfs --daemon start datanode && sleep infinity"
    networks:
      gugunet:
        ipv4_address: 99.99.99.101

Rules for Variable substitution in Dockerfile #

• Dockerfile uses both $VAR and ${VAR} syntax.
• VAR must be either a previous ENV (defined in self or base images’s Dockerfile) or ARG
• reference to any shell variable is not possible: neither from guest’s shell nor host’s shell (isolation by design)

Sometimes you can indeed refer to $PATH. That’s because your base image wrote ENV PATH in its Dockerfile, not because the guest’s shell init rc is evaluated.

Any dynamic power that requires shell eval in the guest’s shell is not possible. That means if your value of your envvar must be computed by a command in the guest, ENV won’t help. See below.

Setting Envvars #

Rule of thumbs:

• Read (Rules for Variable substitution in Dockerfile)[#var-rule]. If that’s okay for you, stick with (1) and (2).
• otherwise, if you want envvar see by your entrypoint process and its subprocesses, escalate to 3 and 4.
• otherwise, you want the docker container hang forever and docker exec into it for fun, go with 5.

1. Dockerfile ENV command

   this method make containerd to add the specified env var upon fork-exec of any process that runs as part of the container. So works for both entrypoint process and any docker exec into a running container, and has nothing to do with a shell nor any shell rc.

   The only variable substitutions supported by ENV is ARG and other ENV, so $PATH won’t work (neither do any other well-known unix env vars)

2. docker run -e or compose.yaml environment

   The mechanism is identical to the Dockerfile ENV command - non-shell entrypoint see the envvar, non-shell exec also see the envvar - except that

   • it’s valid for a one-shot container and all later docker exec into it.
   • All expansion is done by the host’s shell. No variable allowed except the host shell’s ones.

3. Dockerfile RUN command to modify shell rc

pros:

• we are in a real guest machine’s shell environment, values of env vars can be the output of a process e.g. adding a line export foo="$(...)" to rc. This was never possible with ENV in 1 and 2. (here for an example).
• docker exec command and container entrypoint, if both shell, now sees the same envvar.

cons:

• you need to build a new image
• this won’t work if your entrypoint or exec command is not a shell.

Also docker images of most linux distros uses non-login bash. Be sure you modify the right system or user level rc, not the profiles.

4. export in entrypoint init script

This is just a weakened version of 3. The exported env vars can only be observed subprocess of that init script; and docker exec sees nothing.

Limiting the visibility of variable is always a good thing though.

5. export in a standalone script prepared for docker exec -it ... script.sh

   This setup only makes sense if the entrypoint just sleep infinity and you are about to docker exec into the container to play with the isolated environment by typing shell commands; All subprocess of the init script won’t see the env var (only you source it), but again, make sense for sleep infinity.

Docker Volumes Myth #

A Docker volume is a persistent storage managed by dockerd.

1. volume is not deleted upon the exit or deletion of any container; they are simply orthogonal.
2. you can attach any volume to any container when creating the container. But, by design, once the container is created, adding / removing of volumes is not possible.

Most confusion about volume comes from the -v flag to the docker run|create command. It’s indeed a terrible design. The syntax is -v [LHS:]RHS:

1. if LHS: is omitted, this flag creates an anonymous volume and mount it to the path RHS within the container.
2. otherwise, if LHS is a string start with / or ./, it’s interpreted as a path on the host machine, absolute or relative to PWD when the docker run command is typed, and the dir or file there is bind mount to the RHS path within the container. if LHS resolve to a path that does not exist on the host, an empty dir is created there. THIS DOES NOT CREATE NEW DOCKER VOLUME.
3. otherwise, if LHS matches the name of an existing docker volume (as reported by docker volume ls), that volume mounted to RHS within the container.
4. otherwise, A new docker volume is created and given the name LHS, then mounted to RHS within the container.

Only 1 and 4 creates new volume, 3 reuse old volumes, and 2 has nothing to do with docker volumes. Yet they are all done by the -v flag. Modern docker practice recommends replacing -v with -mount type=..., flag since it’s being more explicit

The followings are NOT volumes / will not create a volume:

1. docker run -v LHS:RHS if LHS starts with / or ./
2. docker run --mount type=bind,...
3. creating a new container creates a writing layer internal to docker, which is not a volume. You can inspect its location on the host machine in docker inspect <container> which on linux is usually under /var/lib/docker/overlay2/. That storage is not a volume, it’s meant to be docker implementation details, over.

The following involves volumes:

1. explicit create a new one: docker volume create
2. docker run -v LHS:RHS if LHS does not start with / nor ./; or LHS: is omitted.
3. docker run --mount type=volume,src=...,dest=...

tmpfs on /tmp #

Docker by default does not give /tmp in container a in-memory tmpfs. You must explicit say so if you want one:

• for compose file, use tmpfs keyword under service
• for docker run / create command, use the repeatable --tmpfs /MOUNTING/POINT flag.

read only container rootfs #

To create a container whose rootfs is readonly, use

• --read-only flag to docker run / create command; or
• read_only: true in compose file’s service

This is useful whe you want to be prudent to observe the behavior of a certain software, or limit your user if your container provides remote ssh.

This only makes rootfs readonly, while other mounted fs e.g. tmpfs on /proc and sysfs on /sys are read-write as normal, which makes sense since they must be mutable for the kernel to work. But that should be sufficient. Notably /tmp is also read-only; explicitly mount a tmpfs on /tmp if you want a writable /tmp

Mount your own docker volume if you want a given path to be read-write in a --read-only container.