HPC

·

·

7 min read

The file permissions problem that nobody warns you about

Alex Hatfield

CEO, Juno Innovations

Terminal output full of permission denied errors next to a directory listing

ON THIS PAGE

The first person to feel it is usually whoever launches a second workload on the same files.

Say a researcher spins up a new AI agent, then opens a VS Code IDE to work on the same codebase. The files they need are there. They can see them. They cannot edit them, because the permissions are wrong. The files are owned by a UID from the container that created them, and the IDE container runs as a different UID. Standard Linux permissions do the rest.

The cluster has a shared filesystem. It does not have a shared workspace.

This is one of those problems that does not exist until more than one person uses a cluster, and then it shows up constantly. Every research computing team running multi-tenant Kubernetes on shared NFS storage hits some version of it. The files are there, the permissions are wrong, someone files a ticket, IT changes the ownership, and the next person hits the same wall.

It happens because Kubernetes containers run as whatever UID the container image says. That UID is baked into the image at build time and has nothing to do with the person who launched the container. On a single-user system, nobody notices. On a multi-tenant research cluster with shared NFS storage, every workload writes files as a different owner, and the Linux permission model enforces that correctly in the most annoying way possible.

What the fix looks like

The fix is to make the container run as the researcher’s actual POSIX UID instead of the UID the image picked.

That sounds simple. To do it, the platform has to know who is launching the workload, look up their UID and GID from the directory service, and inject both into the container’s security context before it starts. Then every file the container creates belongs to the right person from the very first write.

In Orion this is handled by a service called titan, part of Orion Admin. When a researcher launches a workload through Orion Workspace, titan catches the launch request, resolves their POSIX UID and GID from the logged-in session, and injects both into the Helm values that kuiper uses to generate the Kubernetes resources. The container’s security context is set to the researcher’s real identity before the pod spec ever reaches the cluster.

So a Jupyter notebook, a VS Code IDE, an AI agent, and a web terminal launched by the same researcher all create files with the same owner. A collaborator added to the project group can read and edit those files from any workload, on any node, without anyone fixing permissions by hand. The NFS server sees the researcher’s actual campus UID. Standard Linux group permissions handle access control. IT never gets the ticket.

What the workspace looks like in production

Our shared workspace mounts from a central NFS server to every node in the cluster through a ReadWriteMany storage class. Every workload in a project gets the same workspace at the same path, with the researcher’s correct identity applied.

Now a researcher launches an AI agent from the catalog and the workspace is already mounted with their files. The VS Code IDE they open next has the same files and permissions, and so does the terminal and the second agent they start for a different task. None of those workloads had to be told where the files were. The storage layout is defined once in the project, and Orion’s mount map makes sure every workload in that project gets the right storage attached.

When a new team member joins a project group, the system maps their UID to the project group and applies the correct mount options. Their first workload launches into the same shared workspace with the right permissions already in place. Nobody files an onboarding ticket or fixes permissions after the fact.

The one thing this does not fix is the mess that is already there. Files written before the identity layer was in place keep their old owners, so most teams do a one-time ownership cleanup when they move a shared filesystem onto Orion.

Why this matters for research computing

Most research computing documentation treats storage as an infrastructure problem. You configure the storage class, create the PVC, and mount it in the pod spec. That is correct at the Kubernetes level, and it falls well short in a multi-tenant research environment.

The people using the cluster have identities the cluster does not know about, and those identities have to map correctly to the files those people create. That mapping has to happen before the container starts, because fixing ownership after the files exist is the ticket queue all over again.

For institutions running LDAP or Active Directory, Orion’s POSIX identity layer maps directory identities to container UIDs. Researchers log in the same way they do everywhere else on campus. The cluster knows who they are, and every workload they launch runs as them.

For institutions that export their shared filesystem over NFS, the backend stays the same and so does the network. The only thing Orion adds is the identity layer that makes shared storage behave correctly for everyone using the cluster.

The other side of the problem is letting researchers use the cluster without knowing anything about Kubernetes. The self-service post covers that part.

See how Orion handles multi-tenant research storage. Book a demo


WRITTEN BY

Alex Hatfield

Alex co-founded Juno to fix how enterprise compute gets done. He leads product vision and customer strategy, and spends most of his time working directly with infrastructure and research teams pushing the limits of what their hardware can do.