fantasy football · dilleystone

data pipeline

bronze

minio

raw json.gz from espn api. partitioned by source, season, week, date.

events/ players/ stats/ teams/

silver

clickhouse

standardized, normalized, cleaned. structured schema.

games players playergamestats teamgamestats teams

gold

clickhouse

aggregated. feature-engineered. model-ready.

playergame playergame_features

serve

fastapi

training data, testing data, model evaluation endpoints.

/training-data /testing-data /evaluate

api — dilleystone.com:8000/docs ↗

GET /training-data gold layer · causal features

GET /testing-data gold layer · no causal info

POST /evaluate score model predictions

airflow dags — github ↗

@daily espn → minio params: year, week

get_events

→

get_stats.expand(game_id=ids)

get_events fetches event list + writes events/season={y}/week={w}/data.json.gz · get_stats fans out per game_id → stats/{y}/{w}/{id}/data.json.gz

@daily espn → minio params: batch_size max_active_tasks: 5

delimit_pages

→

extract_refs.expand

→

load_refs.expand

→

extract_players.expand

→

load_players.expand

→

cleanup

delimit_pages calculates page count from player_count // batch_size · players fetched concurrently with ThreadPoolExecutor(max_workers=10) per page · writes espn/raw/players/date={today}/refs/page={n}/data.json.gz

@daily minio → clickhouse params: year, week

fetch_games

→

extract.expand(game_path)

→

transform.expand

→

load.expand

transform pulls espn_id, slug, season, week, home/away team ids + scores, game_date · weather_condition + temperature hardcoded (sunny, 70°F) — a known gap

@daily minio → clickhouse

fetch → extract.expand → transform.expand → load.expand

same pattern as silver_games — reads stats json.gz from bronze, maps to playergamestats schema, writes to clickhouse silver layer

@daily clickhouse silver → gold

load (INSERT_GOLD)

→

build_features ⚠ wip

load runs INSERT_GOLD SQL to aggregate silver tables into gold.playergame · build_features reads playergame → fantasy points calc (passing_tds×6, rushing/receiving yards×0.1, etc.) is commented out pending implementation

service registry

service image ports link

minio minio/minio 9000 (s3 api) · 9001 (ui) :9001 ↗

clickhouse clickhouse/clickhouse-server 8123 (http) · 9000 (native) internal

airflow apache/airflow 8080 (webserver) :8080 ↗

fastapi python · uvicorn 8000 (http) :8000/docs ↗

k3s rancher/k3s 6443 (api-server) internal