How BlackForge measures the market

Updated August 2026 · Download as PDF

Nine venues, measured the same way

BlackForge collects order books and trades for every pair on nine spot exchanges: Binance, OKX, Bybit, Coinbase, Bitget, Kraken, Gate.io, MEXC and KuCoin. That is roughly ~11,800 pairs, and the exact figure drifts by a few pairs a day as venues list and delist.

The transport is WebSocket push, not REST polling. For each pair we maintain the live order book from the venue’s diff stream, and we run the trade tape on a separate connection so a book resync never interrupts the trade tally. REST is used only to seed and reconcile the book: venues are compared against a deep snapshot (up to the venue’s maximum, 500 levels for most, 5,000 on the venues that serve it) and aligned by sequence number, not wall clock. The result is a book we maintain, not a snapshot we poll.

How a five-minute window closes

The native window is five minutes. A row’s timestamp is the UTC wall-clock five-minute boundary that ends the window, derived arithmetically, so any writer produces the identical key and duplicate rows collapse to the latest version in storage. A pair being re-seeded still always emits its row: an absent row means the collector was down, never that the market was quiet.

The nine collectors do not flush at the same instant. Each computes its columns at a deterministic per-venue offset inside the window, so nine flush spikes never land on the database together. The window label is unaffected; only the moment the book is read shifts.

When each column is measured

This is the part most vendors leave undocumented, and it matters for cross-venue work. Two different clocks are in play, and BlackForge records both honestly rather than pretending everything happened at the window label.

The trade columns are bucketed by the exchange’s own event time. A trade that printed at 12:04:59 lands in the 12:00 to 12:05 window wherever it arrives, so taker volumes and counts are directly comparable across venues. The order-book columns are different: the book is read at the venue’s deterministic flush offset, and each reading carries its own observed-at timestamp. Measured on production, that offset ranges from about ten seconds after the boundary on the fastest venue to about 150 seconds on the slowest, so two venues’ depth columns for the same window label can be observed up to roughly 150 seconds apart.

The row-quality bitmask

The answer to “bought data turned out to be garbage” is that every row tells you, in the data itself, which of its measurements to trust. Each row carries a quality field: a bitmask, the names of the bits that are set, and the column families each set bit contaminates. If a bit is set, the columns in its families are placeholders for that window, not measurements, and you should drop or grey them out. No competitor ships this.

On a response, the quality object looks like this. Here the row is clean apart from one flag, and it names exactly which families to distrust:

"quality": {
  "raw": 2048,
  "all": 2048,
  "flags": ["DEPTH_BEYOND_COVERAGE"],
  "contaminates": ["bookWalls", "orderLadders"],
  "observedAt": 1786581011523
}

The repairs we have run, with measured results

Data quality is not a claim; it is a programme with measured before-and-after numbers. Two repairs carry a clean acceptance test against an independent reference:

Alongside those, a set of labelling and backfill passes made the archive self-describing rather than changing values:

• A base-asset backfill filled every historical row: 45,296,738 rows, zero left unfilled on all nine venues.
• A coin-identity pass labelled 84,831 rows and repaired 12,518 where a symbol had resolved to more than one candidate coin.
• 18,893,904 rows where the price had been carried from the book mid or the last trade were flagged with the PRICE_FROM_MID bit, with zero false positives, so a derived price can never be mistaken for a traded one.
• 38,362 empty-book windows were labelled rather than left looking like real, zero-depth books.

Kept forever

The earliest retained window is 14 July 2026, and retention is infinite: nothing has aged out. A plan’s history window is an entitlement over that archive, not a limit on how long the data is kept. This is the one thing an exchange API and a self-recorder cannot give you at any price, because their history starts the day you start recording. More on that in free versus paid, recording it yourself, and whether order book liquidity is real.

API documentation · Every column, by family · Coverage by venue · This page as a PDF