Raw Journal
A raw journal captures what a market data provider or exchange originally sent before an internal system turns it into normalized events, parsed deltas, analytics rows, or a final reconstructed book. For WebSocket streams and other real-time feeds, it usually preserves the original payload, the receive timestamp, the connection or session context, sequence values if the provider includes them, and replay metadata. This matters because order book reconstruction depends on small details that can be lost when data is converted too early. In L3 and L4 workflows, one missed delete, duplicate SET event, reconnect gap, or misapplied sequence value can change the book state. Hyperliquid L4 reconstruction is especially sensitive because engineers may need to reason about executable depth, parent/child orders, conditional orders, and state-machine behavior rather than only price-level totals.
A raw journal gives developers and quants a source of truth they can return to when the normalized output looks wrong. It is different from normalized events because normalized events have already been mapped into a common schema. It is different from snapshots because a snapshot shows a point-in-time view, while the journal records the message stream that led there. It is different from analytics tables because analytics tables are derived and usually discard transport-level details. It is also different from the final reconstructed book state because the final state is the result, while the raw journal is the evidence used to reproduce and test that result.
Why It Matters
Raw journals help market-data engineers prove whether an order book issue came from the provider feed, the network connection, the parser, or the reconstruction logic. They are essential when investigating crossed books, missed deletes, duplicate SET events, reconnect behavior, schema changes, parsing bugs, and exchange or provider incidents.
What is a raw journal in market data systems?
In market data systems, a raw journal is an append-only record of incoming provider messages saved as close to the original form as practical. It may contain JSON messages, binary frames, headers, channel names, subscription identifiers, receive times, sequence values, and connection metadata. The goal is not to make the data easier to query immediately; the goal is to preserve enough context to replay and inspect the feed later. Teams commonly design raw journals with compression, time-based partitioning, clear retention policies, and schema-version metadata so large message histories remain usable. Clock discipline is also important because receive timestamps are only useful when clocks are synchronized and monitored. Privacy and security controls matter too, especially when logs can contain API identifiers, account-related context, or provider-specific metadata. If the raw message is correct but the parsed event is wrong, the parser or schema mapping is likely the place to investigate.
Why do raw journals matter for L3 and L4 order book reconstruction?
L3 and L4 order book reconstruction often depends on order-level identity, update ordering, connection-local sequence handling, idempotent updates, and state-machine transitions. A normalized delta may say that an order was added, changed, or deleted, but the raw journal can show exactly which provider message caused that interpretation. This is important for Hyperliquid L4 workflows where parent/child orders, conditional orders, and executable depth can affect how engineers understand the book. When a reconstructed book becomes crossed or fails validation, the raw journal lets the team replay the same messages through a corrected reconstructor. It also supports regression tests because the same recorded stream can be used after parser changes or schema-version updates. Deterministic replay tooling turns the journal into a repeatable test fixture, which helps teams confirm that a fix changes reconstruction behavior for the right reason. Without a raw journal, teams may only have the broken final state and not the message path that produced it.
How is a raw journal different from normalized events or snapshots?
A raw journal stores provider messages before transformation, while normalized events are already converted into a common internal format. Parsed deltas are more convenient for reconstruction, but they may omit connection context, unused fields, provider-specific flags, or schema details that become important during an incident. Snapshots show the order book at a point in time, but they do not explain every message that changed the book before or after that point. Analytics tables are designed for research and reporting, so they often aggregate, filter, or reshape data. The final reconstructed book state is the output of a reconstruction process, not the original evidence. A raw journal is the record teams use when they need to audit, replay, and validate how that output was produced.
Real-World Example
A market-data engineer notices that a reconstructed Hyperliquid L4 book becomes crossed after a reconnect. The normalized events show two active orders that should not both be executable at those prices, but they do not explain whether the issue came from the provider stream or the parser. The engineer opens the raw journal for the affected session and finds the original reconnect messages, receive timestamps, sequence values, and subscription metadata. During replay, the team discovers that one duplicate SET event was not handled idempotently after the connection resumed. They update the state machine, rerun the journal through deterministic replay tooling, and confirm that the crossed book no longer appears.
Raw Journal and Market Data API
Raw journals are most relevant to CoinAPI's Market Data API when teams need reliable diagnostics around real-time and historical market data ingestion. Developers can use raw-message retention alongside normalized CoinAPI data workflows to validate parsers, replay incidents, and test order book reconstruction logic. For advanced L2, L3, and L4 engineering, a raw journal helps connect provider-level evidence with the normalized data used by trading systems, research platforms, and monitoring tools.