A case study on solving a real infrastructure problem fast — from zero to full analytical access in under 3 months.

The Problem

SpinCity Solutions was a B2C iGaming operator. When I joined as Data Architect, their BI Manager had a straightforward problem with no straightforward solution: the company had accumulated over 6 TB of historical data across its production databases, and nobody could query it.

The databases were operational systems — built to serve the live platform, not to handle analytical workloads. A complex query across the full history would either time out, degrade platform performance, or both. The BI team had effectively lost access to their own history.

That meant no trend analysis, no year-over-year comparisons, no way to answer questions like "how did player behaviour change after we launched this feature six months ago?" The BI Manager was making decisions on whatever recent slice of data could be pulled without crashing the system.

What I Built

I designed and delivered an ETL pipeline that moved the full 6+ TB dataset off the production databases and onto Amazon S3, structured for analytical access — then put a query engine on top so the BI team could run SQL against the whole history without managing any infrastructure.

The shape of it:

• Extraction — incremental pulls from the source MySQL databases, run against a read-replica so the live platform was never touched.
• Storage — landing the data on S3 as CSV, partitioned by tenant, day, and type.
• Query layer — DuckDB first to prove the data was portable and readable, then a self-hosted ClickHouse warehouse the BI team could query directly.

Here's how it actually came together.

Extraction

I built a NiFi flow that read the source schema and table definitions and used them to generate dynamic SQL — so the extraction logic didn't have to be hand-written per table. Those prepared statements fed an ExecuteSQL Record processor that pulled the data in batches and wrote it out, and the result sets were uploaded to S3, partitioned by tenant, day, and type.

The hard constraint was that none of this could affect the live platform. So rather than read from production, a read-replica was created to act as the migration source — extraction ran against the replica in the background while production carried on untouched. For roughly two and a half years of history, the full pull ran over a single weekend.

Storage: why CSV, deliberately

The result sets landed on S3 as CSV — and that was a deliberate call, not the end state.

CSV isn't what you'd choose for a permanent warehouse; a columnar format like Parquet is the textbook answer. But the team didn't have much hands-on experience with Parquet yet, and adopting it properly would have meant spending more time exploring it before anyone saw value. The priority was proving the approach worked — that the data could come off production cleanly and be queried at all — and CSV got us there fastest. It's a format every engine reads, it was trivial to produce, and it let us validate the pipeline end-to-end in days rather than weeks. Parquet stayed on the table as a known, deliberate next step once the value was proven, rather than a prerequisite that delayed it.

Query layer: DuckDB, then ClickHouse

With the data on S3, the first job was simply to prove it was usable. DuckDB was used for that — pointing it straight at the S3 files and querying the raw data. The point wasn't only "can we read it"; it was demonstrating that the same data on S3 could be read by different engines without being tied to any one of them. That portability mattered, and I'll come back to it.

Once that was proven, the client wanted a proper warehouse they could query directly rather than spinning up DuckDB locally each time. We adopted a single node self-hosted ClickHouse instance. The BI team started by querying S3 through ClickHouse's S3 integration, then used materialized views to move the data into ClickHouse itself — eliminating the per-query network round-trip to S3 and bringing query times down to where interactive analysis was actually comfortable.

The architecture

What it cost

The economics were the whole point for an operator this size, so they're worth being concrete about. Order of magnitude:

• Storage — 6 TB on S3 Standard runs roughly 140/month; pushing colder partitions to Infrequent-Access storage takes that lower.
• Compute — a single self-hosted ClickHouse machine (ClickHouse compresses the raw CSV down hard, so the working set is a fraction of 6 TB) sits in the low hundreds per month.
• Query — once data is materialized into ClickHouse, queries hit local disk, so ongoing S3 request and transfer cost is negligible.

All-in, the platform ran comfortably under €300/month. An equivalent always-on managed analytics platform at this scale would have been an order of magnitude more — the difference between a few-hundred-dollar line item and a multi-thousand-dollar one, which for a smaller operator is the difference between viable and not.

What a query actually looks like

Here's the kind of cross-historical question that simply could not run before — a full scan across the entire 2.5-year history, multi-tenant, on ClickHouse:

-- Monthly active players, total wagered, and gross gaming revenue
-- per tenant, across the full history.
SELECT
    tenant_id,
    toStartOfMonth(bet_placed_at) AS month,
    uniqExact(player_id)          AS active_players,
    sum(stake)                    AS total_wagered,
    sum(stake - payout)           AS gross_gaming_revenue
FROM bets
WHERE bet_placed_at >= '2022-01-01'
GROUP BY tenant_id, month
ORDER BY tenant_id, month;

On the operational MySQL instance, a query like this either timed out or threatened the live platform. On ClickHouse it returns in minutes, against billions of rows.

The Outcome

The BI team went from being unable to query historical data at all to having full analytical access across the entire 6+ TB dataset, with response times under 20 minutes for complex cross-historical analyses — queries that literally could not run before.

The BI Manager went from making educated guesses to making data-informed decisions with full historical context. And because the storage layer is just S3 with on-demand querying on top, they only pay for what they use rather than maintaining always-on analytical infrastructure.

What I Learned

Speed matters more than perfection in startup environments. SpinCity needed this in weeks/months, not quarters. I evaluated tooling deliberately, avoided over-engineering, and optimised for getting the BI team productive fast. The CSV decision is the clearest example: the "correct" format was Parquet, but the right call was the one that proved value first and left Parquet as a deliberate next step.

The portability was the point — and it's an anti-lock-in argument. Validating with DuckDB before committing to ClickHouse wasn't busywork. It proved the data on S3 was engine-agnostic: the same files could be read by DuckDB, ClickHouse, or anything else that speaks S3. For a smaller operator, that's insurance. The data isn't trapped inside a proprietary platform's storage, and the warehouse engine can be swapped without re-migrating anything. I recommended open-source tooling specifically to avoid creating a dependency on an expensive proprietary platform — and the S3-as-source-of-truth design is what makes that promise real rather than rhetorical.

The before/after story is the most powerful metric. "6 TB migrated" is a number. "The BI Manager went from unable to query to full historical access" is a story. When I describe this project, the transformation is what resonates — not the ETL internals.

I'm Julian Calleja, focused on real-time data platforms in iGaming. Get in touch if you want to talk about data infrastructure or iGaming.