Fee Data — Quality & Normalization Design

The headline (read this first)

The amount decomposition Jane Street needs already exists — but in the wrong place. The delivered JSON (schema_version 2.0) already ships fee_amount_typefee_amount_valuefee_amount_min_valuefee_amount_max_valuefee_amount_signfee_amount_unitfee_amount_currency — but the database stores only the raw fee_amount text. Those typed fields are parsed on the fly at delivery time, unvalidated, and never persisted.

1 · What the data actually looks like (measured)

1.1 The amount field is the core debt — but it's 88.6% clean

1.2 It's multi-currency — the USD assumption is wrong for ~7%

Also in the wild: 2,375 bps-denominated amounts and 228 compound expressions (min/max/greater-of/€6k /100m). The 2.8% "unknown" bucket is the dangerous one — those are shipping today with a best-guess currency and no guard.

1.3 Enums already exist — but two are dominated by the "Other" catch-all

1.4 Sparse free-text columns (null rates)

fee_basis 50.7% null · fee_monthly_volume 66.1% · fee_monthly_volume_criteria 68.1% · fee_conditions 56.5% · fee_action_details 36.0%. The two monthly_volume* columns mix units wildly (contracts, users, %, ADV, shares) — tier data trapped in prose.

1.5 Baseline (delivered) cut — the numbers that ship to Jane Street

The figures above are all-versions. The delivery ships baseline only (view_fee_schedule_results, is_baseline=true) — 75,426 rows / 45 venues. Delivery-accurate stats:

2 · The delivery contract (what must not break)

Jane Street consumes, per exchange/day: detailed_fee_<date>.json = { metadata, fees[] }, plus an index.json manifest.

// index.json
{ "schema_version": "2.0", "generated_at": "…Z", "total_records": 558,
  "amount_types": { "flat": 548, "range": 0, "formula": 0, "varies": 10, "not_applicable": 0 },
  "suppliers": [ { "exchange_name": "…", "record_count": 311,
                   "confidence": { "average": 99.5, … } } ] }
// each fees[] record (34 keys) — already carries the decomposition + confidence
record_id, exchange_name, scraped_time, version_id, fee_type,
fee_category, fee_charge_type,
fee_amount, fee_amount_type, fee_amount_value, fee_amount_min_value,
fee_amount_max_value, fee_amount_sign, fee_amount_unit, fee_amount_currency,
fee_action, fee_action_details, fee_basis, fee_participant, fee_participant_details,
fee_monthly_volume, fee_monthly_volume_criteria, fee_symbol_classification,
fee_symbol_type, fee_trade_type, fee_symbol, fee_excluded_symbols,
fee_conditions, fee_exclusions, fee_notes, fee_extra_info,
changed, confidence, relationships

3 · Target model — persist & constrain the decomposition

Add stored columns on fee_schedules_result (keep raw fee_amount forever for audit). Every one already has a delivery-side equivalent, so the serializer just switches from "parse now" to "read column".

Existing enums stay; two get work: fee_action — add the real actions hiding in Other (59%) by mining fee_charge_type (e.g. routing, surcharge, cross, marketing); fee_participant — resolve the 19% Other via a value→enum alias map (e.g. Professional Customer, DPM, Lead Market Maker).

4 · Validation constraints (each ties to a measured anomaly)

Direction rule (locked — matches what goes to Jane Street)

• fee_amount_sign is derived from the amount form (the source's own parentheses / Fee/(Rebate) columns). It is the field to trust.
• fee_charge_type is a descriptive label — NOT authoritative, NOT a tiebreaker. The EDGA equities rows proved it can be wrong in delivered data.
• A disagreement means an extraction-defect candidate on our side — V8 flags it for source verification before ship; fee_charge_type never overrides the sign.

V8 in practice — the EDGA sweep (run 2026-07-31, 75,426 delivered rows)

Jane Street found 6 EDGA rows ((0.00270)/(0.15000%) parenthesised = rebate, but fee_charge_type='Fee'). Verified in prod: the sign is correct (traces to Cboe's "Fee/(Rebate)" columns; code W removes = plain 0.00300 charge confirms the bracket convention is Cboe's own) and fee_charge_type='Fee' is the mislabel. Running V8 across all delivered rows surfaces the siblings before the client does:

• 7,217 total sign-vs-label disagreements across 24 venues (candidates, not all defects — e.g. Distribution Fee Waiver ($3,500) is a correct credit whose label just contains "fee").
• ~109 in the EDGA-layout family (EDGA 29 / EDGX 32 / BZX 42 / BYX 6) — the exact venues sharing EDGA's split-header template.
• Two sub-classes: 573 parenthesised-but-labeled-charge (the EDGA class) and 6,644 bare-positive-but-labeled-Rebate/Credit — candidates for the "sign echoing printed text" defect (a rebate shipped as a positive charge; the next grounding-sweep target).

5 · Confidence-gated delivery

The scores already ride in the delivery as confidence. Make them an operational gate on the serializer:

• Green — correctness ≥ 90 & completeness ≥ 95 & 0 validation failures ⇒ auto-deliver.
• Amber — 60–90 or isolated V1/V4 flags ⇒ deliver with a flagged-rows manifest in index.json.
• Red — currency unknown (V2), completeness < 80 (V7), value/raw mismatch (V4), or correctness < 60 ⇒ hold + route to review before it reaches Jane Street.

6 · Rollout — backwards-compatible, JS never sees a break

1. Add the stored columns (nullable) + enums alongside the raw text. Non-breaking DDL.
2. Backfill with a deterministic amount parser over all 163,624 live rows; diff the result against today's delivery-time parse — same JSON keys, so any divergence is a real parser bug to fix, not a contract change.
3. Shadow-validate V1–V7 in report-only for one delivery cycle; tune currency detection + participant/action alias maps against the residuals (esp. the 2.8% unknown-currency + 59% action=Other).
4. Flip the serializer to read stored columns instead of re-parsing. Byte-compatible output ⇒ schema_version stays 2.0.
5. Enforce CHECK constraints + wire the confidence gate into the SFTP delivery step. Additive changes (new enum values, new fields) ⇒ bump to 2.1 under an additive-only contract rule; never remove/rename a delivered key.

7 · Priority order (highest ROI first)

1. Persist + constrain fee_amount_value + fee_amount_sign — kills the 15.6% sign-in-punctuation risk, the biggest correctness exposure.
2. Stored fee_amount_currency enum + V2 — stops the 2.8% unknown / 6.8% non-USD from silently shipping.
3. V4 raw↔parsed reconciliation — the guard that makes the delivery parse trustworthy.
4. Promote fee_category to enum — trivial, zero-loss, immediate.
5. Re-derive fee_action (59% Other) + participant alias map — the biggest queryability upgrade.
6. Structured tiers from fee_monthly_volume* — last, highest effort, mostly-null today.

8 · Implementation effort & phasing

Smaller than it looks — the hard parts half-exist (delivery already parses amounts; 5 enums already there).

Phase 1 — The hardening ("Jane Street can't break")

Persist + validate the amount decomposition, currency safety, raw↔parsed guard, promote fee_category, confidence gate, flip the serializer to read stored columns.

≈ 10–16 person-days → ~2–3 weeks (1 engineer), ~1.5 weeks (2 engineers). Plus one delivery cycle of shadow-validation calendar time (waiting, not work).

Phase 2 — Enum enrichment (fee_action 59% Other, participant aliases)

Bigger/fuzzier — the fix lives in the extraction pipeline (the AI assigns these): new taxonomy + prompt/logic changes + re-run. ≈ 4–6 days + ongoing tuning. Needs Muizz/Haris's eyes.

Phase 3 — Structured tiers (fee_monthly_volume)

Lowest ROI (66% null), highest parse effort. ≈ 3–4 days. Optional / defer.