Universal Adaptive Trainer
PhD research platform that turns a textbook into an adaptive Python course: RAG question generation against a topic taxonomy, deterministic sandboxed evaluation plus an LLM-as-judge layer aligned to professor feedback with GEPA, and Bayesian Knowledge Tracing that picks each student's next question from their weak topics.
- FastAPI
- Next.js
- RAG
- LLM-as-Judge
- BKT
- Python
How it works
UAT connects four stages that are usually separate tools: ingestion, retrieval-grounded generation, automated evaluation, and adaptive delivery. A book goes in as source material; a versioned, evaluated question bank comes out; students train against it through an adaptive loop.
Retrieval-grounded generation
- PDF ingestion reads a book's own outline and page text into structured chapter / section / provenance metadata.
- Dense-embedding retrieval (
text-embedding-3-small, cosine similarity over an in-memory matrix; no vector DB needed at this scale) maps each subtopic to its supporting sections before generation. The retrieval query representation was tuned on a taxonomy benchmark: hit@1 of 0.85 vs. 0.65 for the subtopic name alone. BM25 and a reranker were tested; neither beat dense retrieval. - Multiple-choice, Parsons, and code-completion questions are generated against the retrieved chunk.
Evaluation and judge alignment
- Deterministic validation: generated code runs sandboxed with a timeout, plus static type checks.
- Four advisory LLM judges (issues, subtopic fit, difficulty, generatability) run at temperature 0, with async batch re-judging of the whole bank at roughly half the token cost of judging one at a time.
- A read-only calibration step pairs every judge verdict against the professor's review, covering per-metric agreement, difficulty confusion matrices, and quadrant analysis. It also documents a negative result: the disagreement signal meant to drive judge self-improvement sat within verdict noise.
Two learning loops
Grounded in prompt-optimization practice (GEPA, MIPROv2, OPRO): the generator's instructions for each question type are rewritten from professor reviews, with rules accumulating onto the shipped prompt rather than being regenerated each round. Each judge's prompt is rewritten from its disagreements with the professor, behind an acceptance gate, with a held-out split excluded so a judge is never scored on its own fitting.
Adaptive delivery
Students join a frozen question set with just a name. No account is needed. Bayesian Knowledge Tracing topic mastery, subtopic weakness, and question-priority rotation pick each next question toward the weakest subtopics, with mastery updating per answer in real time.