Pagination
Pagination is a technique for retrieving large API result sets in smaller chunks (“pages”) rather than in a single response. It helps clients control response sizes, reduce timeouts, and process data incrementally. Pagination is common for endpoints that return long lists, such as symbol catalogs or historical event streams.
Why It Matters
Market data can be high volume: a single day of trades for an active symbol can contain millions of events, and a symbol catalog can contain many thousands of instruments. Pagination lets you iterate through results predictably and build robust ingestion pipelines. It also helps you resume ingestion after failures by restarting from a known cursor or offset.
Common pagination patterns
APIs typically paginate using one of these patterns:
- Offset + limit: request page N using an offset and a fixed page size.
- Cursor-based: request the next page using a cursor token from the previous response.
- Time-slicing: request data in time windows (not always called pagination, but often used for event streams). Cursor-based pagination is usually more stable for changing datasets, while offset pagination is simpler but can be brittle if the underlying list changes.
Pitfalls and best practices
Choose a page size that balances throughput and reliability. Treat pagination as an ordered iteration: do not assume the full dataset fits in memory. Handle duplicates and gaps by using stable ordering keys (time and unique identifiers) and idempotent storage. For historical market data, combine pagination with coverage windows so you don’t page through empty ranges.
When should you use cursor-based pagination instead of offset pagination?
Use cursor-based pagination when the underlying data can change between requests or when you need stable iteration without skipping/duplicating due to inserts. Cursor pagination is also typically faster at scale because it avoids deep offsets. Offset pagination is fine for static datasets and simple UI paging.
How do you prevent missing or duplicated events when paginating time series data?
Use a stable sort order (usually by timestamp, then a unique event ID if available) and persist your checkpoint. When you request the next page, start from the last-seen key and allow a small overlap, then deduplicate on write. This approach is resilient to late-arriving events and retries.
What is the relationship between pagination and rate limits?
Pagination increases the number of API calls, which can bring you closer to rate limits. Use batching, backoff, and concurrency controls to stay within limits. If the API supports larger page sizes or bulk exports, those may be more efficient than making many small requests.
Real-world example
You ingest all symbols for multiple exchanges. Instead of one huge request, you request 1,000 symbols per page and store each page. If the job fails halfway, you resume from the last successful page token, rather than starting over.
Pagination CoinAPI Market Data API
CoinAPI endpoints that return large lists or event streams can require pagination to retrieve complete results. Use the documented paging parameters (such as limits and continuation tokens, where applicable) and persist your last cursor/offset so ingestion can resume reliably. Combine pagination with time-bounded requests for large historical datasets.