Data-oriented workflow with OpenUSD Hydra

An entity is just a uint32_t. Its entity_signature is a std::bitset<256>: each component type reserves one bit. That gives you cheap set tests between "what this entity has" and "what a consumer wants".

using entity_t          = uint32_t;
static inline constexpr uint32_t max_signatures = 256;
using entity_signature  = std::bitset<max_signatures>;

template <typename SUBCLASS>
class signature_idx
{
public:
    static entity_signature sig()
    { return (size_t)1 << entity_manager::mgr_nodev().signature_bit<SUBCLASS>(); }
};

Components live in dense arrays with swap-and-pop removal, so storage stays contiguous but the order changes.

On the other side, managers store the components – instances, indices, submesh, materials, primvars. They are the producers: each owns its component data, and each announced itself onto the world. A single component write fanned out like this:

%%{init: {"themeVariables": {"fontSize": "40px"}}}%%
flowchart LR
    A["manager::set(...)<br/>component written"]
    B["entity_manager::add_signature(entity, sig)"]
    C["<b>signature_events</b> (GLOBAL)<br/>on_sig_changed / on_sig_dirty"]
    D["every listener filters<br/>(bits & related).any()"]
    E["per-consumer pending upload to GPU<br/>"]
    A --> B --> C --> D --> E

The ECS also separates CPU work from GPU work with a resolve pass. When entities are dirtied, the steps happen in batches:

1. the world broadcasts on_entities_resolve;
2. every producer does its CPU→GPU work and hands back a {signature, future} pair;
3. then on_entities_processed broadcasts those same futures;
4. runtime stages wait on the futures during update(). That part survived the rewrite untouched, and it is why the renderer never has to call vk::QueueWaitIdle.

In OpenUSD Hydra 2.0, there are concepts of HdSceneIndexPrim and HdDataSource. A prim is a lot like an entity, and its data sources are a lot like components. The prim is a stable id (an SdfPath); its data sources are the typed attributes hanging off it – xform, visibility, mesh topology, and primvars. The difference is storage: Hydra keeps a sparse, lazily-sampled tree, while our ECS keeps dense, GPU-facing arrays. The bridge between them is really an ECS-to-ECS translation, not a scene-graph flattening, and that framing is what made the redesign click.

The data flows through an interface called HdSceneIndex. You observe one by implementing HdSceneIndexObserver, with three callbacks:

• _PrimsAdded – prims appeared; you get Paths and their types.
• _PrimsDirtied – prims changed; each entry carries an HdDataSourceLocatorSet, so you are told which attributes moved (the transform? the visibility? the mesh topology? one primvar?).
• _PrimsRemoved – prims disappeared.

A static scene trickles a few entries per frame. A load or a reload dumps tens of thousands at once.

Note the push/pull split. The changes are pushed at you – the scene index calls your callbacks. The values are pulled – once you are told where something moved, you reach back into the data sources and sample what you need. Push-notify, pull-sample.

The prim schemas page is the catalog of what can show up under each prim type – one entry per prim type, each listing the attributes it can carry:

• Prim – carried by every prim:

  • xform (HdXformSchema)
  • visibility (HdVisibilitySchema)
  • purpose (HdPurposeSchema)
  • extent (HdExtentSchema)

• Gprim – carried by prims where HdPrimTypeIsGprim(type) is true:

  • materialBindings (HdMaterialBindingsSchema)
  • primvars (HdPrimvarsSchema)
  • instancedBy (HdInstancedBySchema)

• Mesh:

  • mesh (HdMeshSchema), whose topology holds faceVertexCounts, faceVertexIndices, subdivisionScheme, …
  • geomSubsets (HdGeomSubsetsSchema)
  • per-purpose materialBindings and primvars

• Material:

  • material (HdMaterialSchema)

For us the consequence is a stream of "prim X changed under these locators", which we have to turn into edits on a component system whose storage is dense, GPU-facing, and completely unlike the scene graph.

We consume it with a filtering scene index (our resolve_observer) that owns a bi-directional path↔entity map, mints one ECS entity per prim, and re-samples the changed sub-data-source when a prim is dirtied.

Before the payload rework, that bridge handed data straight to the managers, one prim at a time:

%%{init: {"themeVariables": {"fontSize": "16px"}}}%%
flowchart LR
    A["Hydra data source<br/>xform / visibility / mesh / primvars"]
    B["<b>manager::set(...)</b><br/>per prim"]
    C["<b>entities_resolve</b>"]
    D["<b>entities_processed</b>"]
    A --> B --> C --> D

Each of those three hurt for a different reason:

• Too many locks. The manager singleton took a global mutex on every mgr(dev) lookup, and producers called it inside per-entity loops. Each manager then added its own – indices_manager two mutexes, submesh_manager a std::shared_mutex on every get and set, range_allocator one per allocation. The count was never the point: nobody owned the data, so everybody defended their slice of it.

  sp::entity_t e = sp::entity_manager::mgr_nodev().new_entity();  // lock
  
  sp::instance_manager::mgr(dev).add_entity(e);                   // lock
  sp::instance_manager::mgr(dev).set_entity_xform(e, xform);      // lock
  /// other interface as well, mgr::set_indices(e, ...), mgr::set_submeshes(e, ...),
  ///   mgr::set_material(e...)
  

• OOP interfaces on managers. Every manager hand-declared the same ebus_handler mixins and re-implemented the same virtual callbacks, so the shape of the flow – which entities, which fields, in what order – travelled through type-erased virtual dispatch instead of through data.

  // prim_observer::sync -- one prim, through each manager's own typed setter
  auto& mgr = instance_manager::mgr(dev);
  mgr.add_entity(record.entity_id);
  mgr.set_entity_xform(record.entity_id, xform);
  mgr.set_entity_visible(record.entity_id, visibility);
  mgr.set_entity_bbox(record.entity_id, bbox);
  _sync(dev, record);

• Duplicated membership and a broadcast storm. Each batch kept its own slots map (a std::map with a free-list and compaction) plus a std::set of pending entities, while a global signature_events bus announced every entity's signature change to every listener to filter. A bulk load was N entities × M listeners of virtual calls, each with a manager-map lock underneath.

  // simple_batches: every batch heard every change, then masked it itself
  void on_sig_dirty(entity_t e, signature_t const& bits) override
  {
      if ((bits & self().related_sigbits()).any())
          self().pending_entity(e);      // this batch's own m_pendings
  }

BTW, not just my code did this, you can check out how Pixar's storm implemented their OOP zoo 😬.

To improve the efficiency with data oriented design, we need to "group the data close where they are processed". That is instead of process every entity, inside it we send dirty data source to every component managers. We should have component array process all the array at once. In this way we can even parallelized the processing (if no dependency).

A producer no longer pokes managers entity by entity. It accumulates a batch and hands it over once. The unit is a payload: a type-erased, index-aligned pair of columns. One column is the entities, the other is the typed data with the entity id removed (it lives in the entity column now, not duplicated per row):

template <typename UNIT>
class typed_entity_payload : public entity_payload_base
{
public:
    std::span<const entity_t> entities() const { return m_entities; }
    std::span<const UNIT>     units()    const { return m_units; }

protected:
    void emplace(entity_t entity, UNIT&& unit)   // the single writer
    {
        m_entities.push_back(entity);
        m_units.push_back(std::move(unit));
    }

    std::vector<entity_t> m_entities;
    std::vector<UNIT>     m_units;
};

Batches of payloads are applied in priority order, then materials are bound last:

void
entities_payload_batch::apply(device_data& dev)
{
    // sort by PAYLOAD::payload_priority ...
    for (auto* entry : ordered) { entry->payload->apply(dev); }
    m_payloads.clear();
}

Soon later we realize the actual consumers often need multiple components: eg. a draw call needs index, vertex, instance buffer to be useful. We quickly run into the problem of membership management. Popular ECS system like skypjack/entt provides interface like view for quick update.

auto view = registry.view<const Position, Velocity>();
for(auto [entity, pos, vel] : view.each()) {
    // Standard iteration
}

what it effectively does is selecting an intersection set of the components array, the entities that contains all the required components. The view in entt is quite light weight, works like std::view::filter() so no actual data is stored.

This is more suitable if need to update all the involved entities at once like doing physics simulation. For our use case here(processing dirty prims) However it is still too expensive, looping through the entire array (even if the smallest one) for just a few dirty entries leaves some bitter taste in my mouth. So I have to come up with something my own.

using dirty_group_t = dirty_group<membership_t, std::unordered_set<entity_t>>;

class entities_group :
    public ebus_handler<entity_events>,          // GLOBAL: destruction
    public ebus_handler<entities_group_events>   // GROUP: dirties, keyed by id
{
    // ...
    entity_signature m_interests;                // membership gate AND routing key
    membership_t     m_entities;                 // unordered_map + dense vector
    dirty_group_t    m_dirty{m_entities};
};

auto view() const
{
    auto const entities = ref.entities();
    auto everything = std::views::iota(size_t{0}, entities.size()) |
                      std::views::transform([entities](size_t i) {
                          return std::pair{entities[i], i};   // {entity, slot}
                      });
    auto pending = const_filter_view{pendings, /* still a member? */} |
                   std::views::transform([this](entity_t e) {
                       return std::pair{e, ref.entity_idx(e)};
                   });
    return either_view{pendings.contains(invalid_entity_id), everything, pending};
}

void
entities_groups_manager::dirty_sig(std::span<const entity_t> entities,
                                   entity_signature const&   sig)
{
    for (auto const& [group_sig, group_id] : m_group_ids)
    {
        if ((group_sig & sig).any())                 // .any() routes the dirty
        {
            ebus<entities_group_events>::multicast(
                group_id, &entities_group_events::on_sig_updated, entities, sig);
        }
    }
}

The whole bridge is now five phases with clear ownership.

%%{init: {"themeVariables": {"fontSize": "40px"}}}%%
flowchart LR
    A["Hydra notices<br/>_PrimsAdded / _PrimsDirtied / _PrimsRemoved"]
    B["observer produces <b>payloads</b> into <b>batches</b>"]
    C["managers::apply(<b>payload</b>) in priority order<br/>"]
    D["manager::<b>apply(...)</b> updating membership/dirty states<br/>"]
    E["entities_group::<b>on_sig_updated</b><br/>== gate, mark dirty"]
    F["<b>on_entities_resolve</b>:: <br/>producers do CPU to GPU work,<br/>batches upload dirty members,<br/>each hands back a (signature, <b>future</b>)"]
    G["<b>on_entities_processed</b><br/>the same <b>futures</b> are broadcast"]
    H["stages wait on the <b>futures</b> in update()"]
    A --> B --> C --> D --> E --> F --> G --> H

1. Produce. Hydra notices arrive. The observer mints entities and pushes rows into typed payloads; apply() is not called yet, so nothing touches managers.
2. Apply. payloads.apply(dev) runs. Each payload's manager consumes its column in one pass and ends by calling entities_groups_manager::add_or_dirty_sig(entities, sig()).
3. Dirty. add_or_dirty_sig records membership truth (add_signature, idempotent) and issues one GROUP-bus multicast. Each group gates admission (==) and records the pending entities.
4. Resolve. emit_entities_resolve() broadcasts; producers gather upstream futures and append their own; on_entities_processed broadcasts the result. Stages wait on those futures in update().
5. Upload. The batch reads its dirty set through view(), runs its three-clause dispatch (whole-membership sentinel / capacity changed / data changed), calls take_dirty(), builds coalesced regions, and records a single copyBuffer.