January 04, 2024

A Guide to Using Flat Files for Crypto Historical Data

featured image

Did you know that CoinAPI offers a simple way to access large volumes of historical crypto market data?

CoinAPI Flat Files provides bulk historical cryptocurrency data through S3-compatible access, with additional delivery options including Snowflake and MCP. Instead of collecting and storing years of exchange data yourself, you can access organized historical datasets including trades, quotes, order books, and OHLCV.

For analytics firms, quant teams, researchers, and ML developers, Flat Files provides a practical way to move from selecting historical data to integrating it into existing data pipelines.

Flat Files is CoinAPI’s bulk historical market data product, designed for efficient retrieval of large cryptocurrency datasets.

The platform offers a range of data types, including quotes, trades, order book data, and OHLCV, organized in structured formats for integration and analysis.

For data scientists, quantitative researchers, financial analysts, trading firms, ML teams, and market-data engineers, Flat Files provides historical datasets that can support everything from backtesting to liquidity research.

Flat Files is a cloud-based historical data solution that uses a pull-based, S3-compatible API for accessing data.

This allows users to retrieve datasets on demand using familiar S3-compatible tools, SDKs, and libraries.

Instead of requesting large amounts of historical data through individual REST API calls, teams can work directly with files organized for bulk retrieval.

Flat Files is designed to handle extensive historical datasets, making it useful for both focused research projects and large-scale data pipelines.

For real-time streaming data, CoinAPI’s Market Data API provides access through WebSocket and FIX. Flat Files is designed specifically for bulk historical data.

Using Flat Files offers several benefits:

  • Access historical datasets on demand: Users can retrieve available historical data when needed, subject to dataset coverage, plan access, and availability.
  • Efficient data retrieval: S3-compatible access makes it possible to list and download large historical datasets using familiar tools and workflows.
  • Flexible delivery options: Flat Files supports S3-compatible access, with Snowflake and MCP also available for supported workflows. Enterprise customers can discuss additional delivery requirements with CoinAPI.
  • Usage-based access: Teams can retrieve the historical datasets they need without building and maintaining their own market-data archive infrastructure.
  • Efficient data formats: Most Flat Files datasets are delivered as gzip-compressed CSV files, with selected datasets also available in formats such as Parquet where supported.
  • Data quality and normalization: CoinAPI applies normalization and quality-control processes to provide reliable historical datasets across supported exchanges and assets.

Flat Files is designed for easy access to cryptocurrency historical data.

Think of it as a large archive of crypto market history. Instead of collecting years of exchange data yourself, you can retrieve historical datasets and use them directly in your own research, models, analytics systems, or trading infrastructure.

Flat Files provides normalized historical market datasets derived from exchange feeds, including trades, quotes, order book data, and OHLCV.

Whether you need tick-level trade records, quote data, historical order book updates, or OHLCV aggregates, Flat Files lets you explore different parts of the crypto market’s history.

This kind of information can be valuable for trading research, backtesting, machine learning, market analysis, and other data-intensive applications.

Flat Files combines large-scale historical coverage with structured data delivery. Here are some of its key features:

Most datasets are stored as gzip-compressed CSV files, helping reduce storage and transfer requirements while remaining easy to process with common data tools.

Selected datasets may also be available in formats such as Parquet where supported.

Flat Files uses predictable S3-style paths to make historical data easier to locate and retrieve.

File organization depends on the dataset. Data may be partitioned by date, exchange, symbol, and/or timeframe, rather than following one structure for every data type.

This allows teams to retrieve the parts of the historical archive relevant to their specific research or analysis.

Flat Files uses S3-compatible access, allowing developers to work with common S3 clients, SDKs, and data-engineering tools.

Users need the appropriate Flat Files credentials and sufficient plan or credit access for the datasets they want to retrieve.

For current authentication, endpoint, and access configuration, refer to the latest CoinAPI Flat Files documentation.

Where available, sample files can help users evaluate the schema, file format, and data quality before integrating a dataset into their workflow.

This can be particularly useful when planning larger historical downloads or building new data pipelines.

Getting started with Flat Files follows a straightforward process:

  1. Create a CoinAPI/API BRICKS account if you don’t already have one.
  2. Create or retrieve your Flat Files API credentials in the Customer Portal.
  3. Make sure your plan or available credits provide access to the datasets you need.
  4. Choose an S3-compatible client, SDK, or data tool.
  5. Configure it with the CoinAPI endpoint and your credentials.
  6. Browse the available objects and start retrieving historical market data.

The exact files available depend on the exchange, symbol, dataset, date range, and your access level.

For detailed authentication, billing, path structure, and API usage information, refer to the Flat Files documentation.

Flat Files provides several types of historical cryptocurrency market data.

These include:

  • Trades — historical executed trade events.
  • Quotes — historical bid and ask quote information.
  • Full limit order book data — detailed historical order book updates where available.
  • limitbook_snapshot_50 — snapshots of the top 50 bid and ask levels at one-minute intervals.
  • limitbook_depth_bands — summarized historical order book depth information.
  • OHLCV — historical open, high, low, close, and volume aggregates across supported periods.

OHLCV datasets are organized by date, exchange, and timeframe depending on dataset availability.

This range of data allows researchers to move from relatively compact OHLCV analysis to more detailed trade, quote, and order book research.

The right CoinAPI product depends on how you need to consume the data.

Use Flat Files when you need large volumes of historical data for backtesting, quantitative research, machine learning, archival, liquidity analysis, or order book reconstruction.

Use the Market Data API when you need current or real-time cryptocurrency market data through REST, WebSocket, or FIX.

In many professional workflows, the two can also complement each other: Flat Files can provide the historical archive for research and model development, while Market Data API can provide current data for production systems.

If your team needs historical crypto data for backtesting, quantitative research, machine learning, liquidity analysis, or order book reconstruction, Flat Files gives you access to large-scale datasets without requiring you to build and maintain your own historical archive.

Choose the datasets you need, connect through S3-compatible access, and integrate the data into your existing research or analytics pipeline.

New users can start testing CoinAPI with $25 in free credits through the API BRICKS Console.

→ Explore CoinAPI Flat Files

→ View Flat Files Documentation

Recent Articles

Crypto API made simple: Try now or speak to our sales team