Life Sciences

·

·

7 min read

Local AI for research that cannot leave the building

Alex Hatfield

CEO, Juno Innovations

A dark research lab bench with a lab instrument, sample racks and a monitor

ON THIS PAGE

AI help is now normal in software development, writing, and most knowledge work. In regulated research, meaning life sciences, pharmaceutical development, and clinical research, the same tools other teams use every day are often off the table.

Data governance is what keeps them off.

When your research data includes patient information, unreleased clinical trial results, proprietary compound structures, or anything covered by patient-privacy or institutional data governance rules, sending it to an external API is a governance question. In most regulated environments the answer is no.

The way around that is local inference. The model runs on hardware you control, on a network you control, and the data never leaves your environment.

What running it locally looks like

The hardware that makes this realistic is the NVIDIA DGX Spark. Its GB10 Grace Blackwell chip has 128GB of unified memory shared between the CPU and GPU, so models that would need several separate GPUs on traditional hardware fit on one desktop box.

On our own Spark we run a Qwen Mixture-of-Experts model with about 35 billion total parameters and roughly 3 billion active per token, quantized to FP8. The Mixture-of-Experts design keeps each request light, because the model only wakes up a fraction of its parameters for any given token. FP8 halves the memory the weights need compared with 16-bit, and for most tasks the quality loss is small.

We give half of the 128GB to the KV cache, the memory that holds each active conversation. That works out to roughly 885,000 tokens of cache and a maximum context of 262,144 tokens per session. That is enough for an entire research paper, a big codebase or a long run log in one context, so nobody has to summarize a document before the model can read it.

vLLM serves the model through the OpenAI-compatible API, so any tool built for the standard AI APIs connects by swapping the base URL. On Orion, vLLM installs as an Orion Apps plugin. An administrator fills in the model name, a Hugging Face token if the model needs one, the GPU requirement and the memory target, and the plugin pulls the weights and puts the endpoint at a stable internal address. A department that needs a different model installs a second instance.

Once you own the hardware, inference costs you power plus your Orion subscription, and neither goes up when someone sends another request. If the bigger problem is getting a whole team onto one Spark, we wrote that up in a separate post.

A working model is only the start for regulated research.

What regulated research AI needs

“Run it locally” is the start of the requirements list, and the list is longer than that.

Audit trails matter. Every query to the model, every output, and every action an automated tool takes usually has to be logged in enough detail to reconstruct what happened. Validated environments also expect computational processes to be validated and their outputs traceable.

Access controls have to be granular. Different researchers have access to different data, and the AI system has to respect that. For example, a clinical researcher should not be able to use an AI assistant to pull data from a study they are not on, even indirectly through a shared model endpoint.

The model has to be version-controlled and validated. If AI is part of a regulated workflow, you need to document the exact model version used for each analysis. In some contexts, a model update in the middle of a study counts as a protocol deviation.

Workloads have to be isolated from each other. A researcher querying sensitive patient data should not share compute with another workload in any way that could leak data.

General-purpose cloud AI services are not built around these requirements. Local infrastructure gives you a place to meet them. Some of the pieces come with the infrastructure and some you build yourself, and it helps to know which is which.

What Orion handles

At the center is a local inference endpoint running on hardware inside the regulated environment. Orion is customer-hosted, so the whole stack runs on the institution’s hardware and network, and Orion is built to run air-gapped. People sign in with their existing directory accounts (LDAP or Active Directory), and role-based permissions decide what each of them can launch.

Each research team or study gets its own Orion project, with its own compute quotas, storage access, and project network policies that keep traffic inside the project. One study’s workloads cannot reach another study’s services, because the network policy will not route the traffic.

The POSIX identity layer makes file permissions follow the researcher. Files a workload creates while running in a researcher’s context are owned by that researcher. A project’s storage mounts are only reachable by users in that project’s group. Both controls work below the application, so a misbehaving app cannot talk its way around them.

Model versions are handled the same way as workload templates. A specific model version is pinned in the template configuration. Moving to a new model means creating a new template version instead of editing the old one. The old version stays available, so a study that needs to reproduce results with the original model can relaunch it.

On a normal server you just upgrade the thing. Here, being able to put the old version back exactly is the whole point.

What you build yourself

Orion does not keep an audit log of prompts and answers. It shows the container logs for each workload, which helps with debugging but is not an audit trail. If your rules require a record of every prompt, answer and model version, you put a logging proxy in front of the model endpoint and send its records wherever your audit records already live.

Validation works the same way. Orion can pin an environment and keep the old version available, and your quality team still does the validating.

GPU sharing needs a decision too. Orion can time-slice a GPU between containers, which is great for utilization, but NVIDIA’s time-slicing does not wall off one workload’s GPU memory from another’s. If a study needs that separation, give its workloads whole GPUs.

Where agents fit

We run four agents against our own Spark’s endpoint, each in its own container with its own workspace and file permissions, and they hand work to each other through shared files. In regulated research, hold agents to tighter rules than that. They should not change study data without explicit researcher approval or query across study boundaries, and their outputs should be marked as AI-generated and reviewed by a person before anything enters a regulated workflow.

On Orion an agent is another workload inside the study’s project, so it gets the same network policy and the same file permissions as the researcher who launched it. The rules about what it may do are yours to write.

The air-gapped case

Some regulated environments need complete network isolation, with no external connectivity at all. That includes some clinical trial environments and high-security research facilities.

Orion is built to run air-gapped, so this case needs no special setup. The container images live in a local registry and the model weights sit on local disks, so nothing reaches outside at runtime.

Updates are the hard part in an air-gapped environment. Security patches, model updates, and software updates all have to cross the air gap. The usual pattern is a staging environment with external connectivity where updates are tested and validated, then packaged and moved into the air-gapped environment through approved transfer methods.

Orion’s deployment model fits that pattern. Updates ship as new container images or Helm chart versions. You validate them in staging and carry them into the air-gapped environment as a bundle. The update never needs the air-gapped side to reach out.

What this takes to set up

The pieces listed under what Orion handles are part of Orion today. The harder part is organizational. You have to decide what the access control model should be, which workload types need audit logging and at what level of detail, how model updates get approved and rolled out, and who owns the infrastructure versus who uses it.

Those questions have answers, and the answers are different at every institution. The infrastructure can support several configurations, and the real work is picking the right one for your regulatory context. A single study makes a good first scope, because it forces every one of those questions without putting the whole institution on the line.

If you are working out what local AI infrastructure looks like for your research environment, we are happy to go through the specifics. Book a demo

WRITTEN BY

Alex Hatfield

Alex co-founded Juno to fix how enterprise compute gets done. He leads product vision and customer strategy, and spends most of his time working directly with infrastructure and research teams pushing the limits of what their hardware can do.