Alembicdata distiller
Distill messy data into training-grade datasets, on your Mac.
Alembic is a private data-preparation workspace: import real-world files, compose a deterministic pipeline of cleaning, deduplication, privacy and chunking steps, inspect every decision, and export retrieval-ready collections or training schemas — with the same input always producing the same output.
A pipeline you can trust, replay and audit.
Every step is deterministic and inspectable: disable, reorder and replay any part of the pipeline. Real tokenization (cl100k BPE), MinHash near-duplicate detection, semantic chunking and decontamination — the workflow professionals run in scripts, in one native app.
What it does
Every step a real script would run — visible and reversible.
From raw files to datasets you'd stake a training run on.
Import real-world data
CSV, TSV, JSON, JSONL, SQLite, Markdown, HTML, plain text, code, or entire document folders — with structure, encoding and delimiter detection up front.
Clean deterministically
Normalization, null handling, type inference and quality filtering as explicit, replayable pipeline steps.
Deduplicate properly
Exact and MinHash near-duplicate removal so your model doesn't memorize the same paragraph fifty times.
Protect privacy
PII scrubbing and benchmark decontamination before anything leaves your workspace.
Chunk and tokenize
Semantic chunking with real cl100k BPE token counting for retrieval and context budgeting.
Export for your stack
Eight schemas for fine-tuning and RAG, plus dataset splits — the pipeline itself is a saved, reusable recipe.
Inside the app
A workspace built for inspection.
Pipelines, previews and reports — every decision visible.
The pipeline
Composable steps with enable/disable, reorder and duplicate — the recipe is the undo history.
Inspect everything
Preview the data at any stage and see exactly what each step changed.
Reports
Quality and processing reports that tell you what was dropped and why.
How to use Alembic
From folder of files to finished dataset.
A repeatable distillation you can run again next month.
Import sources
Point Alembic at files or folders; it detects structure and flags problems before processing.
Compose the pipeline
Add cleaning, dedupe, privacy, chunking and schema steps in the order you need.
Preview at any stage
Inspect intermediate results and tune thresholds with live feedback.
Scrub and decontaminate
Remove PII and benchmark contamination as explicit, auditable steps.
Split and export
Produce train/val/test splits in the schema your stack expects.
Save the recipe
The pipeline is a reusable recipe — re-run it deterministically on next month's data.
Private by design
Sensitive data stays on your machine.
All processing is local. Optional model-assisted augmentation is strictly bring-your-own-key and clearly separated from the deterministic pipeline.
- Deterministic engine — identical input produces identical output
- Real cl100k tokenization for honest token budgets
- MinHash near-dedupe beyond simple exact matching
- PII scrubbing and decontamination as first-class steps
- Recipes as pipelines reusable, replayable, auditable
- No account, no cloud processing — your data stays put
- Works on
- macOS 14+ (Apple Silicon and Intel)
- Inputs
- CSV, TSV, JSON, JSONL, SQLite, MD, HTML, text, code, folders
- Processing
- Cleaning, dedupe, PII, chunking, tokenization, splits
- Exports
- 8 training/retrieval schemas
- Privacy
- Local processing; optional BYO-key augmentation only
Support the project
Support the project and download Alembic.
Alembic turns the most tedious part of AI work — preparing data you can trust — into a visible, deterministic, repeatable pipeline. Donate what you like to support the project and download the full app straight away. The download is the complete, fully activated app — every feature included. Each download link works once.
AlembicNative MacDistill first. Train with confidence.
Get the software
Support the project and download a build.
Donate what you like, pick your product and your computer type, and download it right after checkout. You always receive the complete, fully activated app. Each download link works once and expires after a while.