CrucibleCooking Houdini HDAs Inside Karma, and Why the Bake Should Not Happen Before Render Time

Context

A Houdini Digital Asset is how a studio packages a procedural effect so it can be reused and art-directed rather than rebuilt per shot. Everything valuable about it comes from staying procedural: parameters that mean something, behaviour that responds, a setup that can be changed late.

Every pipeline then throws that property away at a fixed point, because renderers want geometry. The bake is the moment a live asset becomes a mesh, and it has three properties worth stating plainly. It is irreversible, so the parameters stop existing downstream of it. It is eager, so it computes whether or not the result is looked at. And it happens early, which means it happens again on every change.

The usual response is to make baking faster. This publication argues for a different one: move the bake to the last possible moment, which is render time, and let the renderer pull it.

Crucible is our implementation of that position. It is an open-source Hydra scene index plugin that cooks HDAs when Karma asks for them. Author a CrucibleProcedural prim pointing at an HDA, and when Karma renders, Crucible cooks the referenced asset on Houdini's main thread, injects the resulting mesh into the Hydra scene, and hides the input geometry, all at the scene index level. There is no pre-cooking step, no LOPs recook, and no HAPI session to manage.

---

Why This Matters

If you do not write pipeline code, here is the short version.

A Houdini Digital Asset is how a studio packages a procedural effect: a destruction setup, a crowd system, a growth simulation, built once so it can be reused and directed rather than rebuilt. Getting one into a final render normally means baking it down to geometry first. That bake is one-way, and it has to be repeated every single time an artist changes their mind.

Crucible removes the bake as a separate step. The effect stays live and procedural right up to the moment Karma renders it, so an artist adjusts the setup while looking at the finished frame instead of a proxy.

That is the whole reason we build tooling in-house. It shortens the distance between a creative decision and seeing its consequence, and it leaves the judgment exactly where it belongs. The tool does not decide what the shot should look like. It stops making the person who does decide wait. Crucible was written by the same practitioners who run our visual effects supervision and animation work, which is why it solves a problem that actually shows up on a Tuesday afternoon in production rather than one that looks good in a paper.

---

What It Looks Like

A CrucibleProcedural prim cooking in Solaris. Editing the HDA and its exposed parameters re-cooks the mesh live in the Karma viewport, with no LOPs recook or render restart.

---

The Core Idea

A CrucibleProcedural is authored as a USD GenerativeProcedural prim. All of its configuration lives in the primvars: namespace, so the Hydra scene index layer reads it through the standard HdPrimvarsSchema, with no custom schema translator required:

def GenerativeProcedural "myProc" (
    prepend apiSchemas = ["HydraGenerativeProceduralAPI"]
) {
    token primvars:hdGp:proceduralType = "CrucibleProcedural"
    asset  primvars:Recipe             = @/absolute/path/to/myEffect.hda@
    string primvars:InputPrim          = "/root/sourceGrid"
}

Using stock USD and Hydra machinery rather than a custom schema is a deliberate constraint. It means a stage containing these prims remains a valid USD stage that other tools can open, and it keeps the plugin's surface area small enough to reason about.

The HDA contract is deliberately minimal: a SOP-context asset with a single geometry output that optionally accepts input geometry on port 0. Any parameter you want to drive from USD is overridden by name through a child params Scope, where each primvars:* attribute maps directly to an HDA parm token.

---

How It Works

Crucible inserts two scene indices between the USD stage and Karma. Neither modifies USD; they operate purely at the Hydra data model level. Data is pulled top-down by Karma, and dirty notifications propagate bottom-up.

Scene Index Chain

• CrucibleOverlaySceneIndex: re-types the procedural prim so the resolving scene index recognises it, forces the InputPrim source geometry invisible so it does not appear in the beauty render, and synthesises dirty notifications to drive live updates.
• HdGpGenerativeProceduralResolvingSceneIndex: Hydra's stock resolving index, which instantiates CrucibleGenerativeProcedural and calls Update() to evaluate the asset.

The Cook

When Karma evaluates the procedural, Crucible reads the HDA path, the input prim, and every parameter override, then dispatches the cook through a Python C API bridge into a standalone crucible_houdini.cook module. Because Houdini's object model (hou.*) is not thread-safe, the cook is marshalled onto Houdini's main thread via dispatch_async, while the render thread waits on a semaphore, with a timeout configurable via CRUCIBLE_COOK_TIMEOUT.

The cook installs the HDA, wires the input mesh read directly from the live USD stage, applies parameter overrides, and triggers the SOP cook. The returned geometry is deserialised into Hydra data sources and emitted as child mesh prims (myProc/mesh_0, myProc/mesh_1, and so on) that Karma renders.

Keeping the cook logic in Python means the evaluation behaviour can be changed with no recompile of the C++ plugin.

Live Updates

Two triggers converge on a single in-place re-cook:

• Parameter edits: changing a primvar in the params Scope fires a USD dirty notification. The overlay index detects that the dirty prim is a child of a registered procedural and synthesises an additional dirty entry for the procedural itself.
• HDA file saves: a background thread polls the modification time of every watched HDA once per second and re-cooks when the file changes.

In both cases, the resolving index diffs the new cook result and sends targeted PrimsDirtied notifications, so Karma updates vertex buffers in place: no mesh teardown, no GPU buffer recreation, no render restart.

---

What It Costs

Deferring the bake is not free, and the costs are structural rather than incidental. They follow directly from the decision.

The cook is a serialisation point. hou.* is not thread-safe, so every cook must land on Houdini's main thread while the render thread blocks on a semaphore. That is the price of talking to a live Houdini session rather than a headless geometry library, and it means cooks do not parallelise across a render the way an ordinary Hydra prim's data access would. For scenes with many procedurals, this is the first thing that will bite.

CRUCIBLE_COOK_TIMEOUT exists for a reason. A cook that hangs would otherwise hang the render. The timeout is a guard, not a solution, and its presence is an honest admission that an arbitrary user-authored HDA is arbitrary user-authored code running inside a render.

Polling has a floor. HDA file-save detection is a background thread checking modification times once per second. That is a fixed background cost and a latency floor on that particular trigger. Parameter edits do not go through it, so the interactive path is unaffected, but the mechanism is a poll rather than a watch.

Lazy is not free either. Pulling work into render time means the cost appears during rendering, where it is less visible and harder to attribute than a bake step that announces itself. The argument is that the total is lower because unviewed work is never done. That argument holds better for iteration than for a farm submission where everything is viewed exactly once, which is the case lop_expand below exists to serve.

---

Beyond Render Time

The cook backend is a standalone Python module usable independently of the Karma pipeline:

• cook_hda: cook an HDA with parameter overrides and an optional input prim, returning a plain mesh dictionary. Useful for baking, batch processing, and Python Script LOPs.
• lop_expand: bake every CrucibleProcedural in a stage to a real UsdGeom.Mesh, for render farm submission or USD interchange without the plugin installed.
• Debug LOP: a built-in LOP HDA that cooks at author time rather than render time.
• CLI introspection: create_procedural_prim.py introspects an HDA and writes a complete .usda layer, exposing every parameter tagged for USD in the Houdini parameter editor.

lop_expand is worth dwelling on, because it is the concession the argument requires. Deferring the bake is right for iteration and wrong for a farm, so the design keeps a way to put the bake back where it was.

---

Limitations and Validity

• No published performance measurements. We have not benchmarked Crucible and are not going to imply figures we do not have. Every claim above is architectural: about where work happens and what stays reversible, not about how fast it is. The interactive benefit is one we observe in use, which is not the same as one we have measured.
• The HDA contract is narrow. SOP context, a single geometry output, and optionally a single input on port 0. Assets outside that shape are out of scope.
• A live Houdini session is required. This cooks through hou, not through a headless library, which is what makes the main-thread constraint unavoidable rather than a fixable detail.
• Arbitrary code runs during render. An HDA is user-authored and can do anything, including hang. The timeout bounds the damage; it does not remove the exposure.
• Platform and version. Houdini 21.0.559 or later, CMake 3.21 or later, macOS and Linux.

---

Open Questions

What is the crossover point? The design bets that lazy evaluation wins over eager baking during iteration. Where that stops being true, as procedural count rises and the main-thread serialisation dominates, is the measurement we most want and do not have.

Can the serialisation be relaxed? The main-thread constraint comes from Houdini's object model rather than from our design. Whether a headless cook path could avoid it, and what it would cost in HDA compatibility, is open.

Should file watching replace polling? A one-second poll is a pragmatic choice that works. A real filesystem watch would be better and is not yet done.

---

Availability

Open source under MIT, for production use, modification, and redistribution.

GitHub: github.com/plattipus/crucible

Dependency	Version
Houdini	21.0.559 or later, provides Karma, USD, Python 3.11
CMake	3.21 or later
Platform	macOS / Linux
rez	Optional; recommended for managed pipelines
License	MIT

---

From the Lab

Crucible is one output of our open research practice: problems hit during live production get solved properly, documented, and published rather than patched and forgotten, including the parts still marked open.

It shares an argument with houdini_usd_gsplat, which holds that a format should be a first-class citizen of the scene graph rather than a guest, and with our work on NVIDIA Warp in Houdini, where measurement showed that the expensive part of a pipeline is rarely the computation but the crossings between systems. A bake is a crossing. Everything we learn building these tools feeds back into the visual effects and animation work that raised the problem in the first place.