ISSUE NO. 4 · WEEKEND TECH & RESEARCH BROADSHEETWEEKEND BROADSHEET · NOVEMBER 1, 2026PAGE 1 OF 4
AI WEEKEND
THE GRAPHICAL SYSTEMS & RESEARCH NEWSPAPER FOR AI ENGINEERS
VOL. 1, NO. 4 · THE DATA DISTILLATION ISSUE★ EDITOR: HAIFENG GONG ★WHO DISTILLS WHOM × THE FOUR DEPTHS OF DARK KNOWLEDGE
🎭 2 IDENTITIES
Every frontier lab is distiller (🏢 flagship → ⚡ flash/mini) AND distillee (its 🌐 public API is a behavior surface)
🔬 4 DEPTHS
🍽️ Outputs → 🧠 chain-of-thought → 📊 scores → ⚙️ internals: access depth, not cleverness, caps transfer fidelity
🌡️ T > 1
Hinton '15 dark knowledge: softened logits carry ✨ inter-class geometry that one-hot labels destroy
📈 72% / 60%
OpenRouter tokens on open-weight models (Oct '26) / share carried by 🐉 5 Chinese labs: distillation economics in action
🏛️ 20 YEARS
Model compression (Caruana '06) → ImageNet KD ('15) → 🏭 trace factories ('26): distillation is older than the hype
PLATE 1A · ⚗️ INTERNAL VS. EXTERNAL DISTILLATION🎭 EVERY LAB PLAYS BOTH ROLES
PLATE 1B · EDITORIAL: TWO STILLS, ONE TECHNIQUEPEN & INK
⚗️ The hardware is identical on both sides of the fence; only the narrative changes jurisdiction. The still that ships your beloved ⚡ flash tier is the still the lawyers call 🏴☠️ larceny when it runs across the property line. The dispute worth having is about acquisition — never about the copper.
PLATE 1C · 🔬 THE FOUR DEPTHS OF DARK KNOWLEDGE: WHAT YOU CAN TAP DEPENDS ON WHERE YOU STAND🍽️ RESTAURANT RULE: TASTE → 👨🍳 WATCH → 📋 SCORECARD → 🔑 KITCHEN
AI WEEKENDSECTION II · THE SPECIES MAPVOL. 1, NO. 4 · NOVEMBER 1, 2026
KNOWLEDGE FLOWS FROM ANY BEHAVIOR SURFACE TO ANY CAPACITY
PLATE 2A · THE ONE-ROOM SCHOOLHOUSEPEN & INK
🏫 Cross-species distillation flips size & roles both ways: (Top) Small specialist teachers — 📄 OCR Expert, 🎯 Vision/SAM & 🎨 Image Gen — lecture a giant 🧠 General Multimodal LLM student; (Bottom) the giant 🧠 LLM Teacher distills cart semantics into a tiny ⚡ 5ms RecSys & 7B Flash student.
PLATE 2B · 🏭 THE TRACE FACTORY: MODERN LLM→LLM PIPELINE🎯 THE DISTILLER SUPPLIES THE INPUTS
PLATE 2C · 🧬 THE CROSS-SPECIES TEACHER→STUDENT MAP: SAME-SPECIES IS THE SPECIAL CASE, NOT THE RULE🔀 HETEROGENEOUS = THE NORM
AI WEEKENDSECTION III · COMMON MISCONCEPTIONSVOL. 1, NO. 4 · NOVEMBER 1, 2026
COMMON MISCONCEPTIONS:SIX POPULAR MYTHS ABOUT DISTILLATION
WHAT PEOPLE GET WRONG ABOUT WHAT CAN BE DISTILLED, WHO WINS & WHAT THE LAW SAYS
PLATE 3A · 🧪 MYTH №1 — "EVERYTHING CAN BE DISTILLED"🧱 THE INPUT-DISTRIBUTION WALL
PLATE 3B · EDITORIAL: THE SELF-WRITTEN EXAMPEN & INK
📝 The exam paper, not the teacher, is the bottleneck: 💬 Chat LLMs print prompts for free; 🤖 Agent Policies connect at step t=0 then drift off-manifold at t>1; 🚗 Embodied AI teachers can grade real sensor logs, but collecting those physical-world inputs costs 10M fleet miles.
PLATE 3C · 🏆 MYTH №2 — "STUDENT IS ALWAYS WEAKER"
PLATE 3D · 🕰️ MYTH №3 — "DISTILLATION IS AN LLM THING"
PLATE 3E · ⚡ RAPID-FIRE FACT-CHECK — MYTHS №4–6
AI WEEKENDSECTION IV · ANTI-DISTILLATIONVOL. 1, NO. 4 · NOVEMBER 1, 2026
THE CUSTOMS HOUSE:WATERMARKS, POISON & TRAFFIC FORENSICS
EVERY DEFENSE RAISES COST · NO DEFENSE CLOSES THE SURFACE
PLATE 4A · 🛡️ THE DEFENSE STACK — AND WHERE EACH LAYER LEAKS🔍 DEFENSE-IN-DEPTH, HOLES INCLUDED
🐱🐭 Every defense spawns its own counter-contraption: 💧 watermarks meet paraphrase laundering, ☠️ honeypot poison meets cross-teacher verification filters, and 📡 MITM radar meets obfuscated residential proxy relays.
PLATE 4C · ⚖️ THE DISTILLATION-WARS DOSSIER: A DISPUTE ABOUT ACQUISITION, NEVER ABOUT THE COPPER🎭 TWO LANES: COURTROOM NARRATIVE VS. LAB SHIPPING FLOOR
PLATE 4D · 🏰 THE REAL MOAT: OUTPUTS CAN BE DISTILLED; INPUT DISTRIBUTIONS CANNOT🎯 CLOSING ARGUMENT · ECHOES MYTH №1
COMPLETE TECHNICAL ARCHIVE · VOL. 1, NO. 4
Data Distillation: Foundations, Pipelines, Heterogeneous Transfer & Defenses — Primary Literature
FOUNDATIONS · 2006 → 2026
Model Compression — Bucilă, Caruana & Niculescu-Mizil (KDD 2006). Ensembles label unlabeled data; one small net learns the lot. The original playbook.
Distilling the Knowledge in a Neural Network — Hinton, Vinyals & Dean (arXiv:1503.02531). Soft targets at temperature T>1; "dark knowledge" coined.
Born-Again Neural Networks — Furlanello et al. (ICML 2018). Same-size students outscore their teachers (Page 3, Exhibit A).
A Survey on Knowledge Distillation of LLMs — Xu et al. (arXiv:2402.13116). The modern taxonomy: from logits to synthetic trace corpora.
What is Dataset Distillation Learning? — Yang et al. (arXiv:2406.04284). The OTHER "distillation": compressing datasets into synthetic exemplars — a separate research line (Page 3 fine print).
THE TRACE FACTORY · PIPELINES & EFFICIENCY
DeepSeek-R1 — DeepSeek-AI (arXiv:2501.12948). 800K curated traces distill the reasoner into Qwen/Llama students 1.5B–70B (Page 2, Quadrant A).
Sequence-Level Knowledge Distillation — Kim & Rush (EMNLP 2016). Depth-1 distillation formalized for generation.
GKD: On-Policy Distillation of Language Models — Agarwal et al. (arXiv:2306.13649). Train on the STUDENT's own samples with teacher feedback — the DAgger fix, ported to LLMs (Page 3, Myth №1 fine print).
Sparse Logit Sampling — Anshumann et al. (arXiv:2503.16870). Depth-3 distillation at pretraining scale without storing full vocabularies.
User-Centric Knowledge Distillation — (arXiv:2604.21536). Sequential recommenders aligned to LLM-written user profiles; zero LLM calls at serving (Page 2, Quadrant B).
SelfDR: Self-Distillation from Reasoning — (arXiv:2609.03313). LLM-based recommenders distilling their own rationales.
Progressive Distillation for Fast Sampling of Diffusion Models — Salimans & Ho (ICLR 2022). 50 steps → 4: the step-count distillation that made real-time image generation shippable (Page 2, Quadrant C).
COLLAPSE, IMITATION LIMITS & DEFENSES
AI Models Collapse When Trained on Recursively Generated Data — Shumailov et al. (Nature 631, 2024). The closed-loop failure mode (Myth №5's kernel of truth).
Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification — Feng et al. (arXiv:2406.07515). Verification injects information; the spiral stops.
A Reduction of Imitation Learning to Structured Prediction (DAgger) — Ross, Gordon & Bagnell (AISTATS 2011). Why offline trajectory cloning drifts — and why the fix needs live teacher queries (Myth №1).
A Watermark for Large Language Models — Kirchenbauer et al. (ICML 2023). Green-list token watermarking: Layer 1 of the defense stack, paraphrase caveat included.
