Debugging Go with Delve and Tilt

If you've been following my tutorial series as a handful of you have been, you'll probably have run into a few WTF moments here and there; the kind of moments where you just want to give up and go play non-violent video games, or sing sad karaoke songs by yourself, or whatever it is that people do when they are upset. But it's ok to feel feelings. You just have to eventually put those feelings aside, because we have work to do!

I'm going to make the (potentially unfounded) assumption that you've used a debugger before. That makes my job easier, because I don't have to explain breakpoints, or stepping through code, or the sense of godlike power that courses through your veins when you finally find the bug after many hours of banging your head against the desk. But if you've never experienced that feeling, here's a good tutorial on how to actually use the Go debugger.

The tutorial above is super helpful for actually learning how to debug in Go, so I recommend reading it for the general gist of how to use the debugger itself. I also recommend reading it because I'm too lazy to regurgitate any of it. What I didn't find on the internet though was a concise explanation of how to debug Go via Kubernetes and Tilt, or more generally how to debug Go within a running Docker container. That's the point of this tutorial, and it's (hopefully) short and sweet.

Anyway, as you might have determined from the link above, the Go debugger is called Delve. You can download it thusly:

$ go get github.com/go-delve/delve/cmd/dlv

If you've been following along with my tutorial, you can debug the app in your terminal like so:

$ dlv debug cmd/api.go

But that is assuming we are running and debugging the app via the terminal. In this set of tutorials, we are using a tool called Tilt which helps you manage a Kubernetes cluster for local development. So instead of running dlv debug in our terminal, we need to run our app via the debug server within our Docker container. Once we've set that up correctly, we should be able to connect to the debug server either via the terminal or via our favorite IDE. I'll show you how to do that via the terminal as well as with VSCode, my current favorite editor.

First things first: we need to modify our Dockerfile to run the dlv debug server. I did this by copying the existing Dockerfile to Dockerfile.debug, a file that is going to match the existing one except for a couple of lines:

...
COPY go.mod .

RUN go mod download

# ADDED THIS LINE BELOW
RUN go get github.com/go-delve/delve/cmd/dlv

COPY . .

# MODIFIED THE go build COMMAND HERE
RUN go build -gcflags "-N -l" -o /app/build/api /app/cmd

...

The first line installs Delve in our docker container, and the second line disables compiler optimizations, which will make it easier for us to see what we are debugging.

Now that we've got our special Dockerfile for debugging, we need to modify our Tiltfile to use the new Dockerfile and run the Delve debug server. If you've been following the tutorial, you can add something like this to your Tiltfile:

docker_build_with_restart('dating-app/api', '.',
    dockerfile='./Dockerfile.debug',
    entrypoint='$GOPATH/bin/dlv --listen=:40000 --api-version=2 --headless=true exec /app/build/api',
    ignore=['./Dockerfile', '.git'],
    live_update=[
        sync('.', '/app'),
        run('go build -gcflags "-N -l" -o /app/build/api /app/cmd'),
    ]
)
...

This will replace the earlier call to docker_build_with_restart() that we had in there before. The difference between the earlier one and this one is:

• Using the new dockerfile dockerfile='./Dockerfile.debug'
• Running the Delve server instead of the app by itself: entrypoint='$GOPATH/bin/dlv --listen=:40000 --api-version=2 --headless=true exec /app/build/api'
• Changing the go build command to run without compiler optimizations, same way we had it in the new Dockerfile

Here we run the Delve server on port 40000, so we need to make sure that that port is exposed properly so that we can access it from outside the Kubernetes cluster. I've got my exposed ports listed in the deployments/api.yaml file, so go ahead and add the appropriate port in there:

spec:
  containers:
    - name: api
      image: dating-app/api
      command: ["/app/build/api"]
      ports:
        - containerPort: 3000
        - containerPort: 40000 # <-- Add this

You'll also need to make sure the port is exposed on the Kubernetes resource in Tilt, so go ahead and add it there as well:

# Change port_forwards to include 40000
k8s_resource('api', port_forwards=[3000, 40000])

Once you've done that and Tilt has done its thing, you should be able to connect the debugger in your favorite IDE. Mine is VSCode, so I'll throw in an example launch config. If you're using VSCode, add the following to your launch.json config:

    ...

    "configurations": [
    {
      "name": "Attach to Delve",
      "type": "go",
      "request": "attach",
      "mode": "remote",
      "port": 40000
    }
  ]

Once that's done, you should be able to run the "Attach to Delve" command in the VSCode debugger tab, and once you've connected, be sure to continue the app execution so that it actually starts the web server we are trying to debug.

If you don't have or want to use an IDE for debugging, you can just as easily connect to the running Delve server via the terminal:

$ dlv connect localhost:40000 --api-version=2

And that's that. Be sure to set a breakpoint before you run continue, try this for instance:

(dlv) break github.com/kriskelly/dating-app-example/internal/graph.Signup

That way, when you run continue start the server execution, you can then run the Signup mutation in the GraphQL playground and it should hit the breakpoint in the resolver.

Congratulations, now you have enough info to debug your Go application. Go forth and correct all the mistakes I've inadvertently put into the example code in these tutorials. And good luck!