Rootless Podman Container Drops Its Persistent Volume After a Reboot: A Quick Fix for My HomeLab 🚀

Running containers without root is a lifesaver, but I hit a snag the other night. I had a small database container that wrote data to a named volume, and the next time I rebooted my machine the data was gone. Turns out the volume directory was being created under a transient mount that didn’t survive shutdown.

Why the Volume Vanishes

In rootless mode Podman keeps everything under ~/.local/share/containers/storage. When you do podman volume create, it records the mount point in a JSON file there. On reboot the user‑session systemd instance that owns the rootless runtime comes back up, but systemd doesn’t automatically remount the user‑created volumes unless you tell it to. The result? The container starts, and the volume looks like a fresh empty directory.

[Read More]

Taming Container Logs with Podman and systemd-journald

Introduction to Container Logs

I’ve been running a homelab with various self-hosted services for a while now, and managing container logs has been a challenge. With containerization making it easy to spin up multiple containers, keeping track of their logs can be overwhelming. In my experience, using Podman and systemd-journald has been a game-changer for taming container logs.

What are Container Logs?

Container logs are essentially the output of a container’s stdout and stderr streams. They contain valuable information about the container’s execution, such as errors, warnings, and debug messages. By default, container logs are stored in the container’s filesystem, which can lead to log rotation issues and make it difficult to manage logs across multiple containers. I’ve seen this go wrong when logs start filling up the container’s disk space, causing all sorts of issues.

[Read More]

Troubleshooting Podman Container Networking Issues with ss and nftables

Introduction to Podman Container Networking

I’ve been working with Podman for a while now, and I’ve found that networking issues can be a real pain point. When containers can’t communicate with each other or the host machine, it’s frustrating and can cause downtime. In this article, I’ll show you how to use the ss and nftables commands to troubleshoot Podman container networking issues.

Understanding Podman Networking

Before we dive into troubleshooting, it’s essential to understand how Podman handles networking. By default, Podman uses a bridge network, which allows containers to communicate with each other and the host machine. You can verify this by running podman network ls, which lists all available networks.

[Read More]

Troubleshooting Podman Container Networking Issues with rootless Containers and FirewallD

Introduction to Podman Container Networking

I’ve been experimenting with containers for a while now, and Podman has become my go-to tool for managing rootless containers. However, when it comes to networking, things can get complicated quickly. I’ve seen this go wrong when the container’s network stack is isolated from the host’s network stack, making it difficult to troubleshoot issues. In this article, I’ll share my experience with troubleshooting Podman container networking issues, focusing on rootless containers and FirewallD.

[Read More]