Data platform
The Commercial Data Engine
One trusted gold copy of your reference data — vendor-neutral, fully auditable, and yours to run.
Asset managers, asset servicers, asset owners and sovereign wealth funds all depend on the same raw material: accurate securities, entity and benchmark data from a handful of commercial vendors. Yet almost every firm assembles that data the same painful way — a tangle of destructive ETL jobs that overwrite what came before, bake in one vendor's identifiers, and leave nobody able to answer the question that actually matters: what did we believe about this security on the day the investment was made, and why?
The Commercial Data Engine is built to answer exactly that. It ingests data from multiple commercial vendors, blends it into a single trusted gold copy, and distributes that gold copy to the systems and people who consume it — order management platforms such as Charles River and SimCorp Dimension, and analytical users over a REST API. Every value it serves is explainable, re-derivable, and reproducible to any point in the past.
The core idea
Your data is a recipe, not a rewrite
Most master-data platforms build a "golden record" by destructive match-and-merge. Vendor inputs go in, a survivorship engine picks winners, and the raw context is thrown away. When a rule changes or a vendor restates a figure, you cannot cleanly reconstruct what you previously held — the reasoning is gone.
The Commercial Data Engine takes the opposite approach. The gold copy is treated as a pure, deterministic function of immutable inputs and versioned logic:
Nothing authoritative is ever mutated in place. Raw vendor data is retained verbatim, exactly as it arrived. The gold copy is derived from it — and can always be re-derived. To change an output, you change the recipe (the mappings, the survivorship rules, the canonical schema, the symbology snapshot), never the underlying data. Audit targets the recipe; replay regenerates any past state by re-applying that recipe to inputs as they stood at the time.
The practical payoff is trust you can prove. When a regulator, a portfolio manager or an auditor asks how a number came to be, the answer is not a shrug at an opaque merge engine — it is a precise, reproducible lineage back to source.
Separation of concerns
The loaf and the sandwich
Producing trusted data and customising it for individual consumers are two different jobs, and mixing them is where most platforms come undone. The Commercial Data Engine separates them with a hard, deliberate boundary.
The loaf
The shared, vendor-neutral gold copy. It is computed once, blended across every vendor, and identical for everyone who consumes it — its entire value is that everybody gets the same trusted version. It is a strict function of vendor inputs, with no consumer-specific adjustment folded in.
The sandwich
Where per-consumer customisation lives: reshaping, overrides and adjustments layered on top of the persisted loaf, owned by each consuming team. One team can override a coupon while another keeps the vendor's original — and neither team's choices are ever visible to the other.
Because the baseline loaf is always retrievable un-overridden, a team that wants the raw, unadjusted truth simply consumes the baseline directly. This boundary holds up both across clients and across teams inside a single client — strong evidence that it sits in exactly the right place.
Vendor independence
Engineered into the foundations
Vendor neutrality is not a marketing line here — it is designed into the physical key structure. Many platforms quietly key their entire data model on one vendor's identifiers, turning that vendor into a hard runtime dependency. The Commercial Data Engine keeps a vendor-neutral internal surrogate key as its canonical spine, for both instruments and legal entities. Every external identifier — FIGI, ISIN, CUSIP, SEDOL, CINS, LEI, Bloomberg company IDs — is demoted to an entry in a first-class, point-in-time symbology cross-reference, never a primary key and never on the critical path of loading data.
The result: an instrument can exist in the platform even when a given feed lacks a particular identifier. No vendor sits on the insert path. No vendor can hold your data model hostage. And when a vendor corrects an identifier mapping, that correction becomes a new dated entry in the cross-reference — so historical replays still resolve correctly, exactly as they should.
Point-in-time by design
Bitemporal at the core
The canonical data carries two independent time axes. Valid time records when a fact was true in the world — typically the vendor's effective date. Knowledge time records when the platform learned it. Together they make vendor restatements completely non-destructive: a correction is a new row at a new knowledge time, never an overwrite.
This is what lets the platform answer "what did we send the consuming system on the morning of a given date" — and prove it. The gold copy distributed to each consumer is content-hashed as it is sent, so a replay can be demonstrated to match exactly what a downstream system actually received. Full as-of reconstruction and audit come essentially for free, rather than being bolted on after the fact.
Fit for purpose
The right shape for every kind of data
A close analysis of the Bloomberg Data License catalogue — 46,438 fields across 71 functional categories — produced the central modelling insight behind the platform: this "one" catalogue is really several very different domains wearing a single label, and roughly half of it is not instrument reference data at all. Forcing one storage paradigm across all of it is wrong in both directions.
So the Commercial Data Engine matches storage shape to each domain's natural grain:
- a wide, typed reference spine for instruments and entities — the loaf's backbone;
- a wide, partitioned time-series for pricing that retains every source, with the blended gold price written back alongside them;
- a long fact store for fundamentals, where thousands of ragged financial line-items would never fit typed columns;
- slowly-changing history for ratings;
- and a normalised cluster for classification.
Each domain stays natural, typed where consumers need typing, and flexible where vendors keep evolving their fields — instead of everything being crushed into a single ill-fitting shape.
Blending you can reason about
Decisions at cell grain
At the heart of the platform is a survivorship engine that always operates at cell grain — one decision per instrument-and-attribute. "One vendor wins the whole record" is simply the degenerate case where every cell resolves to the same source. That design choice keeps attribute-level blending — equity from one vendor, fixed income from another; environmental scores from one ESG provider, governance from another — permanently within reach at zero structural cost. Switching between record-level and attribute-level survivorship changes only which rules you author, never the engine and never the schema.
Manual stewardship corrections never contaminate this shared blend. Because an override is net-new information — not something derivable from vendor data plus rules — it lives downstream in the sandwich layer, keeping the shared loaf pure and override-free for everyone else.
Built for independence, sized for reality
Correctness over machinery
This is a master-data problem — modest in volume, high in complexity — not a big-data or streaming problem. Volumes are bounded: typically under ~200,000 securities, with the largest reference set seen (ESG issuer data) around three million rows. Data at that scale fits comfortably on a single node, so the platform deliberately favours correctness, explainability and independence over distributed machinery it would never actually need.
The technology choices follow directly from that:
- PostgreSQL as the bitemporal system of record and serving store — relational integrity, point-in-time queries, and typed wide serving to Charles River, SimCorp and Python users alike.
- MinIO (open-source, S3-compatible object storage) for immutable verbatim raw artifacts — self-hostable per client, with no lake-provider lock-in, and cloud object stores a configuration swap away when wanted.
- DuckDB, with Polars where columnar logic reads more cleanly, as the blend engine — embedded libraries running inside the process, with no cluster and no managed control plane for a vendor to charge rent on or revoke. Their SQL keeps the survivorship logic readable and auditable.
Heavyweight distributed frameworks were deliberately left out: at reference-data scale they add operational cost and a gravitational pull toward exactly the managed lock-in this platform exists to avoid.
Deployed for you, and only you
An isolated stack per client
Each client receives an isolated stack — its own raw storage, its own compute, its own database. This is not a shared multi-tenant platform, and that is a deliberate, load-bearing decision. Commercial vendor redistribution terms make commingling client data a genuine contractual and commercial hazard, so isolation is a licensing necessity rather than a premium add-on. It also removes the hardest and riskiest multi-tenant problems — cross-tenant partitioning and leakage — entirely.
Inside that isolated stack, one shared pure loaf still serves many per-team sandwiches, so teams get independence from each other without any duplication of the underlying truth.
Governance built in, not bolted on
Entitlements enforced at delivery
In a platform defined by vendor redistribution terms, knowing what each client is contractually allowed to receive is not optional. The Commercial Data Engine carries field-level entitlement metadata from day one — every field mapped to its vendor, its license category, and whether it may be redistributed. The catalogue already flags thousands of fields as not downloadable. The distribution layer checks these entitlements before emitting any field to any consumer, because a redistribution breach is a commercial and legal event, not a bug to be patched later.
From source to consumer
A clean, auditable spine
Land
Raw vendor data lands verbatim and immutably into object storage, addressable as-of, with a queryable manifest and per-feed completeness checks that only release data once it is genuinely complete.
Type
Schema-on-read gives the data structure without ever touching the raw bytes.
Resolve
Every vendor’s identifiers resolve to the stable internal key through the point-in-time symbology cross-reference.
Blend
The cell-grain survivorship engine blends the inputs to produce the trusted gold copy.
Persist
The loaf is persisted as versioned, bitemporal tables — fast to read as-of, with the recipe retained for audit and regeneration.
Serve
It is served to Charles River, SimCorp Dimension and analytical users over REST API and file export, gated by field-level entitlements — with the un-overridden baseline always available beneath any per-team customisation.
Why it matters
A foundation you can defend
The Commercial Data Engine gives asset managers, servicers, owners and sovereign wealth funds a commercial-data foundation that replaces the opaque, destructive, lock-in-prone data plumbing most firms live with today — with a platform whose entire design philosophy is that the truth should always be reconstructable.
Trusted
One gold copy, blended once, shared by all.
Explainable
Every value re-derivable to source, every restatement preserved.
Independent
No vendor baked into the keys, no cluster to rent, self-hostable per client.
Defensible
Point-in-time answers you can prove, and redistribution entitlements enforced at the point of delivery.