Data Types and File Structure Overview

The Flat Files S3 API provides access to various types of cryptocurrency market data. This document provides an overview of the available data types and the general file structure used across the system.

Available Data Types

Our API offers the following types of market data:

1. Quotes
2. Trades
3. Full Limit Order Book
4. Limit Order Book Snapshot 50
5. Limit Order Book Depth Bands
6. OHLCV (Open, High, Low, Close, Volume)
7. Hyperliquid Oracle Prices
8. Hyperliquid TWAP Statuses
9. Hyperliquid Misc Events
10. Hyperliquid System Events

Each data type is documented in detail in its respective article, including field descriptions, data formats, and usage examples. You will find the articles in left-side menu.

Buckets

Flat Files are served from two S3 buckets. Starting from 2026-06-09, new Trades, Quotes, and Full Limit Order Book data in coinapi is published as hourly partitions (D-YYYYMMDDHH) instead of daily partitions. Daily partitions published before that date remain in coinapi and are not moved to coinapi-daily-tail. Update integrations that fetch new data to use hourly partition paths in coinapi.

The coinapi-daily-tail bucket is a rolling window for the previous day's daily files only. It is not a historical archive: existing daily files are not moved into it, and data is not retained beyond 24 hours. The date directory always reflects yesterday's date (UTC) and changes each day. This bucket does not include OHLCV.

Daily partitions published before 2026-06-09 remain in the coinapi bucket. For longer historical access to data published on or after that date, use the hourly partitions in the coinapi bucket.

File Structure

Within each bucket, data is organized according to the following structure:

/
├── T-TRADES/
│   └── D-{YYYYMMDD | YYYYMMDDHH}/
│       └── E-[EXCHANGE]/
│           └── IDDI-[IDENTIFIER]+SC-[COINAPI_SYMBOL_ID]+S-[EXCHANGE_SYMBOL].csv.gz
├── T-QUOTES/
│   └── D-{YYYYMMDD | YYYYMMDDHH}/
│       └── E-[EXCHANGE]/
│           └── IDDI-[IDENTIFIER]+SC-[COINAPI_SYMBOL_ID]+S-[EXCHANGE_SYMBOL].csv.gz
├── T-LIMITBOOK_FULL/
│   └── D-{YYYYMMDD | YYYYMMDDHH}/
│       └── E-[EXCHANGE]/
│           └── IDDI-[IDENTIFIER]+SC-[COINAPI_SYMBOL_ID]+S-[EXCHANGE_SYMBOL].csv.gz
├── T-LIMITBOOK_SNAPSHOT_50/
│   └── D-{YYYYMMDD | YYYYMMDDHH}/
│       └── E-[EXCHANGE]/
│           └── IDDI-[IDENTIFIER]+SC-[COINAPI_SYMBOL_ID]+S-[EXCHANGE_SYMBOL].csv.gz
├── T-LIMITBOOK_DEPTH_BANDS/
│   └── D-{YYYYMMDD | YYYYMMDDHH}/
│       └── E-[EXCHANGE]/
│           └── IDDI-[IDENTIFIER]+SC-[COINAPI_SYMBOL_ID]+S-[EXCHANGE_SYMBOL].csv.gz
├── T-HLORACLEPRICES/
│   └── D-{YYYYMMDD | YYYYMMDDHH}/
│       └── E-HYPERLIQUIDL4/
│           └── IDDI-[IDENTIFIER]+SC-[COINAPI_SYMBOL_ID]+S-[EXCHANGE_SYMBOL].csv.gz
├── T-HLTWAPSTATUSES/
│   └── D-{YYYYMMDD | YYYYMMDDHH}/
│       └── E-HYPERLIQUIDL4/
│           └── IDDI-[IDENTIFIER]+SC-[COINAPI_SYMBOL_ID]+S-[EXCHANGE_SYMBOL].csv.gz
├── T-HLMISCEVENTS/
│   └── D-{YYYYMMDD | YYYYMMDDHH}/
│       └── E-HYPERLIQUIDL4/
│           └── *.csv.gz
├── T-HLSYSTEMEVENTS/
│   └── D-{YYYYMMDD | YYYYMMDDHH}/
│       └── E-HYPERLIQUIDL4/
│           └── *.csv.gz
└── T-OHLCV_ACTIVE_CONSOLIDATED/   # `coinapi` bucket only
    └── D-{YYYYMMDD | YYYYMMDDHH}/
        └── E-[EXCHANGE]/
            └── TP-[TF].csv.gz

Where:

• T-{DATA_TYPE} defines the data type (e.g. T-TRADES, T-QUOTES, T-LIMITBOOK_FULL, T-HLORACLEPRICES)
• D-{YYYYMMDD | YYYYMMDDHH} defines the time partition:
  • YYYYMMDD in coinapi → legacy daily partitions published before 2026-06-09
  • YYYYMMDD in coinapi-daily-tail → previous day only, retained 24 hours
  • YYYYMMDDHH in coinapi → hourly partitions for longer historical access to data published on or after 2026-06-09
• [EXCHANGE] is the identifier for the specific exchange
• [IDENTIFIER] is a unique identifier for the data file
• [COINAPI_SYMBOL_ID] is the CoinAPI symbol identifier
• [EXCHANGE_SYMBOL] is the symbol as used by the exchange
• T-HLMISCEVENTS and T-HLSYSTEMEVENTS are exchange-consolidated streams and do not include symbol-specific file keys

Additional Notes:

• OHLCV, Limit Order Book Snapshot 50, and Limit Order Book Depth Bands are available only in the coinapi bucket
• coinapi-daily-tail is not a historical archive; daily files published before 2026-06-09 remain in coinapi
• Daily partitions in coinapi-daily-tail are replaced each day and are not kept beyond 24 hours
• All time partitions use UTC

File Format

All data files are stored in CSV (Comma-Separated Values) format and compressed using gzip compression. This approach balances human readability with efficient storage and transfer sizes.

To use these files:

1. Download the .csv.gz file
2. Decompress the file using a tool that supports gzip (e.g., gzip, 7-zip)
3. Open the resulting CSV file in a spreadsheet application or process it with your preferred data analysis tool

Data Consistency and Synchronization

All timestamp fields across different data types are synchronized to ensure consistency when analyzing data from multiple sources. We use high-precision time synchronization to maintain accuracy across our data collection infrastructure.

Best Practices for Data Retrieval

1. Choose the correct bucket: coinapi for hourly historical access, legacy daily files before 2026-06-09, OHLCV, Limit Order Book Snapshot 50, and Limit Order Book Depth Bands; or coinapi-daily-tail for the previous day's daily files only (24-hour retention).
2. Use date-based partitioning to efficiently retrieve data for specific time periods.
3. Leverage the prefix functionality in S3 listing operations to filter data by exchange, symbol, or date.
4. Implement parallel downloads for large datasets to improve retrieval speed.
5. Consider implementing local caching for frequently accessed data to reduce API calls and improve application performance.

For detailed information on each data type, please refer to the individual data type documentation.