Hands on Windows Container Internals

Illustration by Murat Kalkavan from Icons8

After having used Windows containers for a while I wanted to better understand how they work under the hood. What functionality is provided by Docker, what comes from Windows? Is there a Windows low level container runtime like runc on Linux? What components do I have to take into account when debugging problems?

I think a good starting point to get a better understanding for Windows containers is to run a simple Windows container entirely without Docker. Let’s do that. It’s easier than it sounds.

The following is an extremely simplified example. Our container will be very basic (no networking, etc). If you want to work with Windows containers for any other reason then writing your own container tool higher-level tools like Docker or containerd’s ctr (which both build on top of hcsshim for Windows containers) are a much better choice.

Some background first

This a hands-on article, so just some brief background infos (you can read more details about the architecture here):

• Windows provides public (but unfortunately largely undocumented) C APIs for running containers (HCS — Host Compute Service) and container networking (HNS — Host Network Service).
• to make these APIs better consumable in container tools like Docker Microsoft provides golang bindings for these APIs in the MIT-licensed project hcsshim. (I think in many aspects hcsshim (and particularly its cmd tool runhcs) can be compared to Linux container runtimes like runc).
• hcsshim has the usual Go package docs and provides a set of cmdline tools that, amongst other things, demonstrate how to use the API. We’re going to use these tools to run our simple container. (In addition there’s a closed-source tool called hcsdiag.exe that is a “swiss army knife” for debugging Windows containers. We’ll use it, too).

Let’s run a simple Windows container without Docker

Assuming a working Golang installation, we install two of the hcsshim tools:

go install github.com/Microsoft/hcsshim/cmd/wclayer
go install github.com/Microsoft/hcsshim/cmd/runhcs

Assuming a working Docker installation* (in this example I’m using a minimal non-HyperV Docker installation as described here), we’ll first need to find a Windows container base image that matches our Windows version.

(*I know, I promised that we wouldn’t use Docker to run the container. We’ll still need and use it as a convenient tool to pull the container base image from Microsoft’s container registry. But that’s really the only thing we’re using it for: As a downloader).

First we check our Windows version with winver (for me: Windows 10 2004) and then use Docker to pull the matching servercore image:

$ docker image pull mcr.microsoft.com/windows/servercore:2004
2004: Pulling from windows/servercore
295f12394c4f: Pull complete
b1cbf80244af: Pull complete
Digest: sha256:fc0517098cb71e946af157b98b091bcd726c47b11edefaf574b7fba219d60f30
Status: Downloaded newer image for mcr.microsoft.com/windows/servercore:2004
mcr.microsoft.com/windows/servercore:2004

As you can see this Docker image consists of two layers. They both got downloaded and extracted into a subfolder here: %programdata%\docker\windowsfilter. (If you started with a blank Docker installation you should now find two layer folders there. If you had previously other images pulled you might see more image layer folders there).

We can also see the layers when we inspect the image:

$ docker image inspect mcr.microsoft.com/windows/servercore:2004
...
"GraphDriver": {
            "Data": {
                "dir": "C:\\ProgramData\\docker\\windowsfilter\\9ed7a02c267f9c8ec6db412de3d9fb40477b520c777b14f63d849e6c072d09be"
            },
            "Name": "windowsfilter"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:345550b5cacc31efae770e24e66ca3c6be85cbc95fbebe657b430e2f13ac7786",
                "sha256:dbc52acedded3319b1c5ad7e8eae6295bd7145d6c0250d405b6ec27a2c89fc45"
            ]
        },
...

But how can we follow the layer chain without the help of Docker? Let’s look at the path displayed above as Data.dir. This is the path to the highest layer of this image. Every image layer folder has a layerchain.json file. Its content points to the path of the next layer in the chain. This is how we can travel down the chain to the first layer — let’s call it scratch layer here. The content of layerchain.json for a scratch layer is null as it doesn’t depend on any other layer.

OK, now we know that we have two layers and their order in the chain. Equipped with that info we can create a volume for our container that consists of the scratch layer, any additional layers on top (in our case one) plus a new writeable layer.

First, let’s create a temp folder:

mkdir c:\temp
cd c:\temp

Then we’ll create the volume.

For this command we need to specify the paths to all layers, in the order of highest layer first (the path from Data.dir) and lowest layer (scratch layer) last.

wclayer create --layer "C:\ProgramData\docker\windowsfilter\9ed7a02c267f9c8ec6db412de3d9fb40477b520c777b14f63d849e6c072d09be" --layer "C:\ProgramData\docker\windowsfilter\2542025381c1a79c72ca8ef4d8041be85f8ccda8fd0117cf3b8b99b9875274a1" mybundle

This should have now created a new folder mybundle that contains a sandbox.vhdx file. This file represents the writeable layer of our container.

To be able to create and start a container we’ll create a config file mybundle\config.json , where we can i.e. specify the command that our container should run (here: just cmd.exe). Here’s the contents:

(For layerFolders: the first value is the path to the scratch layer, the second one is the path to the bundle folder. We don’t need to specify the paths to the in between layers).

{
    "ociVersion": "1.0.1",
    "process": {
        "env": [],
        "cwd": "c:\\",
        "args": [
            "cmd"
        ]
    },
    "windows": {
        "layerFolders": [
            "C:\\ProgramData\\docker\\windowsfilter\\2542025381c1a79c72ca8ef4d8041be85f8ccda8fd0117cf3b8b99b9875274a1",
            "C:\\temp\\mybundle"
        ]
    }
}

Now we can create a container using our bundle:

runhcs create -b c:\temp\mybundle mycontainer

If there was no error we should be able to see our newly created container listed with:

$ runhcs list
ID            PID         STATUS      BUNDLE              CREATED                             OWNER
mycontainer   8888        created     c:\temp\mybundle   2020-08-15T08:24:21.5599597+02:00

We can also list it with hcsdiag.exe (which is included with Windows and is super handy to also debug Docker containers):

$ hcsdiag list
mycontainer
    Windows Server Container,   Created,            , runhcs

So that container has been created but not yet started. Let’s start it:

runhcs start mycontainer

Our container should now be running in the background. We may see its terminal output in the shell where we’ve created it, but we can’t interact with its console. To open an interactive shell to our container we can use hcsdiag:

$ hcsdiag console mycontainer
$$ echo %username%
ContainerAdministrator

Success! We have started a Windows container and opened an interactive console to it. 🎉 🥳

We can now poke around inside of the container. Like…

Networking? Nope. (We knew before).

$ ipconfig /all
Windows IP Configuration
Host Name . . . . . . . . . . . . : WIN-4IAO0GIB886
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No

Do we have a working registry? Yes!

$ reg query HKLM
HKEY_LOCAL_MACHINE\Hardware
HKEY_LOCAL_MACHINE\SAM
HKEY_LOCAL_MACHINE\Security
HKEY_LOCAL_MACHINE\SOFTWARE
HKEY_LOCAL_MACHINE\SYSTEM

When we’re done playing around in our container we can exit the console with exit. Now let’s stop and delete the container:

runhcs kill mycontainer
wclayer unmount mybundle
runhcs delete mycontainer

If we want we can also finally delete the bundle (backup the config file if you want to repeat things later):

wclayer remove mybundle

That’s it — I hope you’ve enjoyed this little experiment as much as I did. 👋