Skip to main content

Troubleshooting

Start here, always:

docker logs --tail 50 neurogrid-agent

The agent stopped after I changed networks​

Symptom. Heartbeats were fine, then every one fails. Or a job loads the model and then dies opening its tunnel, always after about the same number of seconds. Your internet works fine everywhere else.

Cause. Docker gives a container its DNS configuration when the container is created, and never updates it. Move the machine to a different network — a laptop leaving a hotspot, a new DHCP lease, a router swap — and the container keeps asking a resolver that no longer exists. Routing still works, so nothing looks offline: only names fail, each one hanging until it times out.

Fix. Reconnecting the Wi-Fi does not help. Recreate the container, or just restart it:

docker restart neurogrid-agent

The agent says so itself when this happens — look for Cannot resolve the backend's hostname in the logs.

Prevention. Install with --dns 1.1.1.1 --dns 8.8.8.8, as in the Linux and Windows instructions. Those resolvers exist on every network, so the container stops caring which one the machine is on.

nvidia-smi works on the host but not in Docker​

The NVIDIA Container Toolkit is missing or the daemon has not picked it up.

# Linux
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

On Windows, this usually means Docker Desktop is not on the WSL2 backend, or the Windows driver is older than 575.

Registration fails with an invalid token​

The log says the backend does not recognise the token. Almost always one of:

  • It was truncated when copied. Machine tokens are long; copy the whole string.
  • It was already used by another machine. A token claims one GPU. Get a fresh one from the console.
  • The listing was cancelled in the console before the agent checked in.

It says the config file is writable and refuses to start​

Deliberate. That file sets which server the agent talks to, so anyone who can write it can redirect your machine and collect its work.

docker exec neurogrid-agent chmod 600 /data/config/config.yaml
docker restart neurogrid-agent

Jobs fail right after the model loads​

Almost always the tunnel, and almost always DNS — see above. If DNS is healthy, check that outbound HTTPS on port 443 is not blocked; some corporate and campus networks filter it.

Disk filled up​

Model weights are large and the cache grows with every distinct model you serve.

docker system df                 # where it went
docker image prune -f            # untagged images
docker volume rm neurogrid-hf-cache   # the model cache, only while idle

Never remove neurogrid-data — that is your registration.

The GPU shows offline but the container is running​

The agent is alive but its heartbeats are not landing. Check the logs for gRPC error sending heartbeat, then:

docker exec neurogrid-agent getent hosts api.neurogrid.cc

No output means DNS — restart the container. Output but still offline means the connection is being blocked somewhere between you and us.

A job failed and I do not know why​

docker logs neurogrid-agent 2>&1 | grep -iE "error|failed"

Job containers are removed after they finish, so their own logs are gone by the time you look. The agent's log keeps the reason.

Still stuck​

Open an issue at github.com/NeuroGrid-AI-exchange with the output of:

docker logs --tail 100 neurogrid-agent
docker inspect neurogrid-agent --format '{{.Config.Image}}'
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv