Release Process

This document describes how SpiderRock publishes and maintains its v8 historical datasets. It focuses on the structure of a release and the lifecycle of a published file, so that data consumers understand when data becomes available, which location to download for a given use case, and how corrections are incorporated over time.

Overview

SpiderRock v8 historical datasets are delivered as Parquet files. Each dataset is processed on a defined schedule and published to a stable, predictable location. A given file is published once and then maintained according to the lifecycle described below.

Release cadence

v8 datasets are processed on a regular schedule following the close of the relevant trading session, and are generally available the following morning, local market time. The coverage start date for each dataset is published in the data dictionary.

Data versions

Each dataset is exposed through two top-level locations. The appropriate choice depends on whether you require a stable canonical view or the most current available version.

Location	Contents	Recommended use
v8/ (published)	The canonical published version of every file. This is the default location for most consumers.	A stable, consistent view of the data.
v8_latest/	The most recent version of every file, including corrections, as soon as they are produced.	The earliest possible access to corrections and restatements.

For most workflows, v8/ is the recommended location. If your pipeline must incorporate corrections at the earliest opportunity, download from v8_latest/.

Lifecycle: corrections and restatements

Occasionally, a previously published file is corrected — for example, to apply a data-quality fix or to reflect an upstream restatement. Corrections are managed so that the canonical history remains stable while fixes are made available promptly:

1. A correction is published to v8_latest/ as soon as it is produced. Consumers downloading from v8_latest/ therefore receive the corrected version immediately.
2. After a 30-day stabilization period, a correction that has remained stable is promoted into the canonical v8/ location, replacing the prior version of that file. This promotion runs on a weekly cycle.

In summary:

• v8_latest/ always reflects the newest version of every file.
• v8/ reflects the canonical, stabilized history, and is updated for a given file once a correction to that file has remained stable for 30 days.
• Consumers requiring an exact, reproducible snapshot should download from v8/; consumers requiring corrections at the earliest opportunity should download from v8_latest/.

Only files whose contents have genuinely changed are re-published into v8/ during a promotion, so repeated downloads of v8/ do not incur unnecessary change.