The library

Every interactive paper explainer on Rudrite Research, filterable by field.

• Attention Is All You Need — The 2017 paper behind every LLM you use — watch attention decide what matters.
• FlashAttention — Exact attention, made fast by never writing the big matrix to memory.
• PagedAttention (vLLM) — Serve far more requests by paging the KV cache like an operating system.
• Megatron-LM — Split a model across GPUs along the matrix — and train billions of parameters.
• DeepSeek-R1 — Reasoning that emerges from reinforcement learning, not imitation.
• GPT-3: Language Models are Few-Shot Learners — Scale a language model until it learns new tasks from a few examples.
• ZeRO: Zero Redundancy Optimizer — Partition a model across GPUs instead of replicating it — and train toward a trillion parameters.
• Mixtral of Experts — Grow capacity without growing per-token cost — route each token to two of eight experts.
• Training Compute-Optimal Large Language Models — Given a fixed compute budget, double the model and double the data — in equal proportion.
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Let a state-space model read what it's reading — and a recurrence outruns attention.
• BERT: Pre-training of Deep Bidirectional Transformers — Read the whole sentence at once — pre-train by filling in the blanks, then fine-tune anywhere.
• DeepSeek-V3 — A 671B mixture-of-experts that activates only 37B — via latent-KV attention and loss-free routing.
• Qwen3 — One family, dense and MoE — with a unified thinking / non-thinking switch.
• OLMo 2 — A fully-open model, stabilized by moving the norms to the output and clamping QK.
• MiniMax-01 — Near-linear attention at 456B — lightning attention, with a softmax layer every eighth block.
• Gemma 4 — Five sizes, one design — interleaved local/global sliding-window attention, now with MoE.
• Scaling Laws for Neural Language Models — Loss falls as a clean power law in size, data, and compute — and tells you how to spend the budget.
• Adam: A Method for Stochastic Optimization — A per-parameter adaptive learning rate from two moving averages of the gradient.
• Deep Residual Learning for Image Recognition — Add the input back — the identity skip that made 152-layer nets trainable.
• Denoising Diffusion Probabilistic Models — Add noise to an image, then learn the reverse — the recipe behind modern diffusion.
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — Send each token to a single expert — and scale a model to a trillion parameters.
• LoRA: Low-Rank Adaptation of Large Language Models — Freeze the model, learn its change as two skinny matrices — 10,000× fewer trainable weights, zero added latency.
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism — Split a giant model across chips and pipeline micro-batches to keep them all busy
• GSPMD: General and Scalable Parallelization for ML Computation Graphs — Annotate a few tensors; the compiler shards the trillion-parameter rest.
• Pathways: Asynchronous Distributed Dataflow for ML — One controller, thousands of accelerators — parallel dispatch makes single-controller ML as fast as SPMD.
• Ring Attention with Blockwise Transformers for Near-Infinite Context — Shard one sequence across a ring of devices, rotate the KV blocks — context scales with device count.
• Efficiently Scaling Transformer Inference — Chop a 540B model across a TPU pod: 29ms/token, 76% MFU, 32x longer context
• Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Schedule the KV cache, not the GPU: disaggregated prefill/decode serving that survives overload.
• Fast Inference from Transformers via Speculative Decoding — A small model guesses ahead, the big one verifies in parallel — same output, 2–3× faster.
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Add worked examples to the prompt — and reasoning emerges in big models, no training
• Training language models to follow instructions with human feedback — RLHF: align GPT-3 from human feedback — a 1.3B model beats the 175B on preference
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Skip the reward model and the RL — one cross-entropy loss aligns the policy directly from preferences.
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — A 7B open model hits 51.7% on MATH — by web-mining 120B math tokens and inventing GRPO.
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — Think longer on hard prompts — and let difficulty decide how to spend the compute.
• Constitutional AI: Harmlessness from AI Feedback — Train a harmless, non-evasive assistant from a written constitution — zero human harm labels.
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale — Four named techniques turn DeepSeek-style RL into a reproducible run to AIME 50.
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models — Wrap a frozen GPT-4 in tree search — branch, self-evaluate, prune. Game of 24: 4% to 74%.
• ReAct: Synergizing Reasoning and Acting in Language Models — A frozen LLM that thinks, acts, and reads results in one loop — the blueprint for every agent.
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — Rebuilds attention for Hopper — async warps + FP8 — for 740 TFLOPs/s, 1.5-2.0x over FA-2.
• Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — Selective SSMs and masked attention are one structured matrix, computed two ways.
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — 236B MoE, 21B active per token — MLA folds the whole KV cache into one latent vector
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty — Draft one layer down: autoregress on features, not tokens — 2.7–3.5× faster, losslessly.
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Find the 1% of weights that matter by watching activations, then scale to protect them at INT4.
• RoFormer: Enhanced Transformer with Rotary Position Embedding — Encode position by rotating Q and K, so attention sees only the relative offset m−n.
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — Cut an image into 16×16 patches, call each a word, feed a plain Transformer.
• Learning Transferable Visual Models From Natural Language Supervision — Match captions to images, and you get a classifier for any concept you can name.
• High-Resolution Image Synthesis with Latent Diffusion Models — Move diffusion into a compact latent space — cheaper, and the architecture behind Stable Diffusion.
• Scalable Diffusion Models with Transformers — Drop the U-Net: a plain transformer on latent patches whose quality scales with Gflops.
• Robust Speech Recognition via Large-Scale Weak Supervision — 680k hours of weak supervision → one Transformer that transcribes the real world, zero-shot
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Trainable, hardware-aligned sparse attention: 3 gated branches, 11.6x decode, beats dense
• Group Sequence Policy Optimization — Reward lands on the whole sequence — so the importance ratio should too, not per token
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Split a request's timeline into prefill and decode GPU pools — 4.48x more requests under SLO.
• CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Reuse every retrieved chunk's KV cache anywhere, then recompute the ~15% of tokens that stitch cross-attention back.
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — Top-2 experts per token + an SPMD compiler: a 600B model trained in 4 days.
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — One dial from MQA to MHA — near-MHA quality at near-MQA decode speed, retrofitted cheaply.
• YaRN: Efficient Context Window Extension of Large Language Models — Extend a RoPE model to 128k by reshaping frequencies per wavelength — for a tenth of the tuning
• Efficient Streaming Language Models with Attention Sinks — Pin 4 "attention-sink" tokens + a rolling window — stream 4M tokens, no fine-tuning.
• Generative Adversarial Networks — Two networks duel — a forger and a detective — until the fakes pass for real.
• Segment Anything — Point at anything, get a clean mask back in milliseconds — segmentation as a foundation model.
• Visual Instruction Tuning — A blind GPT-4 writes the lessons; one matrix turns sight into tokens — the open VLM template.
• s1: Simple test-time scaling — 1,000 curated examples + one word — “Wait” — buy o1-style test-time scaling in 26 GPU-minutes.
• Tülu 3: Pushing Frontiers in Open Language Model Post-Training — The full post-training recipe in the open — SFT, DPO, and RL with verifiable rewards.
• Let's Verify Step by Step — Reward every correct step, not just the answer — process supervision trains the stronger verifier.
• Self-Consistency Improves Chain of Thought Reasoning in Language Models — Sample many reasoning paths, take the majority answer — accuracy jumps with no training at all.
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Bolt a retriever to a generator and train them together — knowledge moves out of the weights.
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — 2,294 real GitHub issues as the exam — can a model patch an actual repository?
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits — Every weight is −1, 0, or 1 — matching full precision while multiplication becomes addition.
• KAN: Kolmogorov–Arnold Networks — Put the learnable functions on the edges, not the nodes — splines replace fixed activations.
• Differential Transformer — Two softmax maps subtracted — attention noise cancels like a differential amplifier.
• Mixture-of-Depths: Dynamically allocating compute in transformer-based language models — A fixed compute budget, spent unevenly — tokens route around blocks they don't need.
• RWKV: Reinventing RNNs for the Transformer Era — An RNN trained like a Transformer — constant-state inference at GPT scale.
• Titans: Learning to Memorize at Test Time — A neural memory that learns at test time — surprise decides what's worth remembering.
• Byte Latent Transformer: Patches Scale Better Than Tokens — No tokenizer — bytes group into entropy-sized patches, and patches scale better than tokens.
• The Llama 3 Herd of Models — The 405B herd report — data, scale, and infrastructure for an open frontier model.
• Mistral 7B — Sliding-window attention + GQA — the 7B that beat 13B and started the small-model race.
• Phi-4 Technical Report — Synthetic data as the main course, not the garnish — a 14B that punches at reasoning.
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Re-cutting the same tiles across warps — ~2× faster by fixing the work partition, not the math.
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Extra decoding heads draft ahead; tree attention verifies — speedup with no draft model.
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis — Rectified flow meets a multimodal DiT — straighter paths, two streams, one per modality.
• Flow Matching for Generative Modeling — Train the velocity field directly — diffusion-quality generation from straight probability paths.
• Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty — Add a calibration reward to RLVR so a reasoning model states how sure it is — and means it.
• Rewarding Doubt: Calibrated Confidence Expression of LLMs — RL on a proper scoring rule teaches an LLM to express calibrated confidence in its answers.
• Why Language Models Hallucinate — Hallucinations are the predictable result of training and grading that reward confident guessing.
• τ-bench: Tool-Agent-User Interaction in Real-World Domains — A benchmark for tool-using agents talking to a simulated user — and the reliability cliff at pass^k.
• ToolRL: Reward is All Tool Learning Needs — Tool use learned by RL with a decomposed reward — format plus correctness beats SFT imitation.
• Group-in-Group Policy Optimization for LLM Agent Training — Group-in-group advantages give long-horizon LLM agents step-level credit without a critic.
• MiniMax-M1: Scaling Test-Time Compute with Lightning Attention — Clip the importance weight, not the update — efficient RL for a hybrid-attention reasoning model.
• ProRL: Prolonged RL Expands Reasoning Boundaries — Prolonged RL with KL resets expands what a reasoning model can do, not just sharpens it.
• The Entropy Mechanism of RL for Reasoning Language Models — Why RL entropy collapses, the law that predicts it, and two covariance-clipping fixes.
• Spurious Rewards: Rethinking Training Signals in RLVR — On Qwen, even random or wrong RLVR rewards lift math accuracy — what the signal really does.
• GenPRM: Generative Process Reward Models — A process reward model that reasons and runs code to verify each step — a 7B beats a 72B.
• From Hard Refusals to Safe-Completions — Train safety on the output, not the request: graded safe-completions over hard refusals.
• Proximal Policy Optimization Algorithms — The clipped-objective RL algorithm under RLHF — stable policy gradients without trust-region overhead.
• Efficiently Modeling Long Sequences with Structured State Spaces — A state-space layer that models 16k-long sequences — the origin of the line that leads to Mamba.
• Auto-Encoding Variational Bayes — The reparameterization trick turns a variational lower bound into a trainable autoencoder.
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — Cast every NLP task as text-to-text — one model, one objective, one format.
• Toolformer: Language Models Can Teach Themselves to Use Tools — A model self-supervises where to call APIs — keeping only the calls that lower its own loss.
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Quantize a 175B model to 3–4 bits in a few GPU-hours with a one-shot, Hessian-aware solver.
• Muon is Scalable for LLM Training — An orthogonalizing optimizer that beats Adam on the matrix parameters — scaled to LLM training.
• Consistency Models — Map any point on the diffusion trajectory straight to its origin — generation in a single step.
• Large Language Diffusion Models — Generate text by un-masking, not left-to-right — an 8B diffusion LM that rivals autoregression.
• Mean Flows for One-step Generative Modeling — Train on average velocity, not instantaneous — one-step generation from scratch, no distillation.
• Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion — Roll out the model on its own outputs during training — real-time streaming video on one GPU.
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction — Predict the next resolution, not the next token — autoregression that beats diffusion on ImageNet.
• Emerging Properties in Unified Multimodal Pretraining — One model that understands, generates, and edits — multimodal reasoning that emerges as data scales.
• Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling — Split the visual pathway in two — separate encoders for seeing and for drawing, in one transformer.
• Moshi: a speech-text foundation model for real-time dialogue — Full-duplex spoken dialogue — both voices as parallel token streams, with a 200ms inner monologue.
• π0.5: a Vision-Language-Action Model with Open-World Generalization — A robot policy that co-trains on web data and many robots — and cleans a kitchen it has never seen.
• Cosmos World Foundation Model Platform for Physical AI — Pretrained world models — diffusion and autoregressive — that learn to simulate the physical world.
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models — DeepSeek Sparse Attention — a lightning indexer picks the tokens that matter, dropping O(L²) toward O(Lk).
• Kimi Linear: An Expressive, Efficient Attention Architecture — Kimi Delta Attention, interleaved 3:1 with full attention — the first linear hybrid to beat softmax.
• Kimi K2: Open Agentic Intelligence — A 1T-parameter MoE with 32B active, trained stably by MuonClip — built for agentic tool use.
• DeepSeek-OCR: Contexts Optical Compression — Render long text as an image and read it back — ~10× context compression at 97% decode accuracy.
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free — One sigmoid gate after attention adds non-linearity, kills the attention sink, and stabilizes training.
• LongCat-Flash Technical Report — Zero-computation experts spend a token budget unevenly — 560B params, ~27B active, 100+ tokens/s.
• FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling — Attention rebuilt for Blackwell — async pipelines and software-emulated exp, up to 1.3× over cuDNN.
• Pretraining Large Language Models with NVFP4 — Pretrain in 4-bit floating point — micro-block scaling and Hadamard transforms match FP8 over 10T tokens.
• Absolute Zero: Reinforced Self-play Reasoning with Zero Data — A model proposes its own tasks and a code executor grades them — reasoning RL with no human data.
• Agent Learning via Early Experience — No rewards, no imitation — agents learn from the futures their own early actions create.
• Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity — Ask for a distribution, not an answer — a training-free prompt that restores the diversity alignment flattened.
• Orpheus TTS — TTS as plain next-token prediction — a Llama-3.2-3B emits flattened SNAC codec tokens, decoded to 24 kHz in real time.
• Fish Audio S2 — Why one autoregressive stream is not enough — a slow Qwen3-4B semantic backbone, a fast depth-wise head, and an RVQ codec, without the N× flatten.
• IndexTTS2 — Disentangle emotion from identity — hold a cloned voice fixed and dial its feeling on a separate axis, in a cascaded AR-semantic + flow-matching TTS.
• CosyVoice 2 — Streaming and offline speech synthesis in one model — an FSQ semantic tokenizer, a Qwen2.5-0.5B text-speech LM, and a chunk-aware causal flow-matching mel decoder.
• Higgs Audio v2 — Naturalness at scale via DualFFN — a Llama-3.2-3B with a forked audio FFN emits interleaved text + delay-patterned RVQ codes, pretrained on 10M hours with no post-training.
• Chatterbox — A single scalar dials emotion — one number becomes one conditioning token in a 0.5B Llama LM — plus classifier-free guidance at the AR token stage, an S3Gen flow-matching codec, and a watermark on every clip.
• Spark-TTS — One LM, one token stream, no acoustic model — a Qwen2.5-0.5B autoregresses a single flat stream of BiCodec tokens, and a GAN decoder turns them straight into 16 kHz audio.
• Kokoro — TTS without any of this — an 82M non-autoregressive StyleTTS2 + ISTFTNet model turns phonemes and a split style vector into 24 kHz audio in one feed-forward pass; no LLM, no diffusion, no codec, no tokens.
• Gorilla: Large Language Model Connected with Massive APIs — Stop hallucinating APIs: a LLaMA-7B fine-tuned on machine-generated instruction–API pairs, optionally reading retrieved docs at inference, and graded by AST sub-tree matching — out-calling GPT-4 on APIBench while cutting hallucinated calls.
• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs — Teaching an open model to drive 16,464 real REST APIs: ToolBench built from RapidAPI with no human labels, a depth-first search that lets the model back out of dead ends (DFSDT), an API retriever, and ToolEval to grade it all.
• Voyager: An Open-Ended Embodied Agent with Large Language Models — An agent that writes its own tools: GPT-4 proposes tasks, writes executable code against the game API, verifies it works, and archives every verified program into an ever-growing skill library — skills that persist, compound, and transfer to unseen worlds.
• Agent Workflow Memory — Agents that learn the recipe, not the run: induce reusable workflows from past trajectories — offline from a training set, or online with no labels at all — feed them back into memory, and solve new tasks in fewer steps.