Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Ren, Yi (Joshua)
Doctor of Philosophy - PhD
University of British Columbia
This thesis investigates the learning dynamics of deep learning systems through a local, physics-inspired analytical lens. Motivated by the need for fine-grained insights into model behavior, we begin with the step-wise influence that a single training example exerts on a specific observing example during learning. Central to our approach is the proposed AKG decomposition, which dissects this influence into three interpretable components: similarity (K), normalization (A), and prediction gap (G). This decomposition enables an analogy with classical force analysis: the force originates from G, is shaped by AK, and is ultimately applied to the target object, e.g., to the model confidence, output, hidden representations, or parameters. Building upon this foundation, we gradually scale the analysis from individual interactions to cumulative effects over time, akin to tracking an objectâs motion under multiple forces. We apply it to the following problems. Supervised classification: We study the learning trajectories of examples with varying difficulty and reveal an interesting "zig-zag" pattern that emerges during optimization. Our analysis explains this behavior and inspires a novel knowledge distillation method, Filter-KD, which improves supervision signals for student models. Large language model (LLM) finetuning: We extend the framework to account for the autoregressive nature of LLMs and the presence of negative gradients. The unified perspective explains behaviors across finetuning methods such as SFT, DPO, and GRPO. We also highlight the critical role of negative gradients. In particular, we identify the "squeezing effect": a counterintuitive phenomenon caused by improperly applied gradient ascent. Representation learning: We explore the dynamics of hidden features, revealing how adaptation energy and directions influence the feature drift. Our analysis leads to a provable pattern of feature adaptation in a head-probing then finetuning pipeline, offering insights and inspiring several practical strategies. Simplicity bias and compositional learning: Revisiting foundational questions about why structured representations are learned faster, we apply our framework to a compositional learning setting. Our findings align with principles such as Occamâs Razor and the idea of "compression for AGI," offering a novel dynamical explanation rooted in compression and learning speed.
Learning Dynamics of Deep Learningâ Force Analysis of Deep Neural NetworksbyYi (Joshua) RenM.S., University of Edinburgh, UK, 2019B.A., Tongji University, China, 2013A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFDOCTOR OF PHILOSOPHYinThe Faculty of Graduate and Postdoctoral Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)September 2025© Yi (Joshua) Ren 2025The following individuals certify that they have read, and recommend tothe Faculty of Graduate and Postdoctoral Studies for acceptance, the thesisentitled:Learning Dynamics of Deep Learning â Force Analysis of DeepNeural Networks,submitted by Yi (Joshua) Ren in partial fulfillment of the requirementsfor the degree of Doctor of Philosophy in Computer Science.Examining Committee:Danica J. Sutherland, Assistant Professor, Department of Computer Sci-ence, UBCSupervisorMark Schmidt, Professor, Department of Computer Science, UBCSupervisory Committee MemberTrevor Campbell, Associate Professor, Department of Statistics, UBCSupervisory Committee MemberLeonid Sigal, Professor, Department of Computer Science, UBCUniversity ExaminerMuhammad Abdul-Mageed, Associate Professor, School of Information andDepartment of Linguistics, UBCUniversity ExaminerQuanquan Gu, Associate Professor, Department of Computer Science, Uni-versity of California, Los AngelesExternal ExaminerAdditional Supervisory Committee Members:Jeff Clune, Professor, Department of Computer Science, UBCSupervisory Committee MemberAaron Courville, Professor, Department of Computer Science and Opera-tions Research, Universite´ de Montre´al and MilaSupervisory Committee MemberiiAbstractThis thesis investigates the learning dynamics of deep learning systemsthrough a local, physics-inspired analytical lens. Motivated by the needfor fine-grained insights into model behavior, we begin with the step-wiseinfluence that a single training example exerts on a specific observing exam-ple during learning. Central to our approach is the proposed AKG decom-position, which dissects this influence into three interpretable components:similarity (K), normalization (A), and prediction gap (G). This decompo-sition enables an analogy with classical force analysis: the force originatesfrom G, is shaped by AK, and is ultimately applied to the target object, e.g.,to the model confidence, output, hidden representations, or parameters.Building upon this foundation, we gradually scale the analysis from individ-ual interactions to cumulative effects over time, akin to tracking an objectâsmotion under multiple forces. We apply it to the following problems.â¢ Supervised classification: We study the learning trajectories ofexamples with varying difficulty and reveal an interesting âzig-zagâpattern that emerges during optimization. Our analysis explains thisbehavior and inspires a novel knowledge distillation method, Filter-KD, which improves supervision signals for student models.â¢ Large language model (LLM) finetuning: We extend the frame-work to account for the autoregressive nature of LLMs and the pres-ence of negative gradients. The unified perspective explains behaviorsacross finetuning methods such as SFT, DPO, and GRPO. We alsohighlight the critical role of negative gradients. In particular, we iden-tify the âsqueezing effectâ: a counterintuitive phenomenon caused byimproperly applied gradient ascent.â¢ Representation learning: We explore the dynamics of hidden fea-tures, revealing how adaptation energy and directions influence thefeature drift. Our analysis leads to a provable pattern of feature adap-tation in a head-probing then finetuning pipeline, offering insights andiiiAbstractinspiring several practical strategies.â¢ Simplicity bias and compositional learning: Revisiting foun-dational questions about why structured representations are learnedfaster, we apply our framework to a compositional learning setting.Our findings align with principles such as Occamâs Razor and theidea of âcompression for AGI,â offering a novel dynamical explanationrooted in compression and learning speed.ivLay SummaryThis thesis explores how deep learning models learn over time, using ideasinspired by force analysis. Specifically, we zoom in on the modelâs trainingprocedure to see how one training example affects another during learning,like analyzing how forces move objects. We break this influence into twoparts: how similar the two examples are, and how strong the updating forceis. This framework helps us understand a wide range of the modelâs behav-iors in different real systems. For example, it explains why certain exampleshave non-trivial learning paths, why (and why not) some LLM finetuningmethods work, and why simpler, more structured patterns tend to be learnedmore easily. We apply this approach to various learning tasks and uncovernew strategies for improving model training. While the method is still de-veloping, it offers a new way to interpret modelsâ behaviors systematically.vPrefaceThe work presented in this dissertation has been included in several publi-cations. We list them as below:1. âBetter Supervisory Signals by Observing Learning Paths.âYi Ren, Shangmin Guo, Danica J. SutherlandInternational Conference on Learning Representations (ICLR), 2022(Ren et al. 2022)Contributions: We formulated the problem together. Danica introduced theNTK into the framework, which made the decomposition more theoreticallysolid and interpretable. We all contributed to the writing.2. âHow to Prepare Your Task Head for Finetuning.âYi Ren, Shangmin Guo, Wonho Bae, Danica J. SutherlandInternational Conference on Learning Representations (ICLR), 2023(Ren, S. Guo, et al. 2023)Contributions: We formulated the problem together. Shangmin helped withcoding. Wonho did the image segmentation part. Danica checked the theo-retical derivations carefully. We all contributed to the writing.3. âImproving Compositional Generalization using Iterated Learning andSimplicial Embeddings.âYi Ren, Samuel Lavoie, Mikhail Galkin, Danica J. Sutherland, Aaron CourvilleConference on Neural Information Processing Systems (NeurIPS), 2023(Ren, Lavoie, et al. 2023)Contributions: We formulated the problem together. Samuel helped withthe SEM part. Mikhail helped with the graph network part. Danica andAaron checked the theory and results. We all contributed to the writing.4. âlpNTK: Better Generalisation with Less Data via Sample InteractionDuring LearningâShangmin Guo, Yi Ren, Stefano V Albrecht, Kenny SmithviPrefaceInternational Conference on Learning Representations (ICLR), 2024(S. Guo, Ren, et al. 2024)Contributions: We formulated the problem together. Shangmin did most ofthe work; I helped with the theory part. We all contributed to the writing.5. âBias Amplification in Language Model Evolution: An Iterated LearningPerspective.âYi Ren, Shangmin Guo, Linlu Qiu, Bailin Wang, Danica J. SutherlandConference on Neural Information Processing Systems (NeurIPS), 2024(Ren et al. 2024)Contributions: We formulated the problem together. Linlu helped with thegame design. Danica helped with the theoretical and experimental checks.We all contributed to the writing.6. âLearning Dynamics of LLM Finetuning.âYi Ren, Danica J. SutherlandInternational Conference on Learning Representations (ICLR), 2025(Ren and Sutherland 2025)Contributions: We formulated the problem together. Danica helped with thetheory and reorganization of the paper. We all contributed to the writing.7. âOn the Effect of Negative Gradient in Group Relative Deep Reinforce-ment Optimization.âWenlong Deng, Yi Ren, Muchen Li, Danica J Sutherland, Xiaoxiao Li,Christos ThrampoulidisConference on Neural Information Processing Systems (NeurIPS), 2025(W. Deng, Ren, M. Li, et al. 2025)Contributions: We formulated the problem together. Wenlong did most ofthe work; I helped with the theory part. We all contributed to the writing.8. âUnderstanding Simplicity Bias towards Compositional Mappings viaLearning Dynamics.âYi Ren, Danica J. SutherlandWorkshop on Compositional Learning at NeurIPS, 2024(Ren and Sutherland 2024)Contributions: We formulated the problem together. Danica helped with thetheory and reorganization of the paper. We all contributed to the writing.viiPrefaceChapter 3 is largely based on papers 1 and 4; Chapter 4 is largely basedon papers 1; Chapter 5 is largely based on papers 5, 6, and 7; Chapter 6 islargely based on papers 2; Chapter 7 is largely based on papers 8 and 3.Disclosure of Generative AI Usage: A generative AI tool, specifically ChatGPT,was used solely for grammar checking and rephrasing. No AI-generated ideaswere included in the research or analysis presented in this dissertation.viiiContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viContents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivList of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . xviiiList of Symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiiiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Math-like and Physics-like DL Theory . . . . . . . . . . . . . 21.2 Rapidly Evolving Algorithms Call for Physics-like Theories . 31.3 Publications and Related Chapters . . . . . . . . . . . . . . 51.4 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . 102 An Overview of Progress in Deep Learning . . . . . . . . . 132.1 History of Deep Learning . . . . . . . . . . . . . . . . . . . . 132.2 More Local and More Microscopic . . . . . . . . . . . . . . . 153 Fundamentals of Learning Dynamics . . . . . . . . . . . . . 173.1 Warm-up with Supervised Learning . . . . . . . . . . . . . . 18ixContents3.1.1 Problem: Start from Supervised Classification . . . . 183.1.2 Learning: Gradient Descent . . . . . . . . . . . . . . 203.1.3 Evaluation: Loss, Accuracy, and More . . . . . . . . . 213.2 One-Step Influence . . . . . . . . . . . . . . . . . . . . . . . . 223.2.1 The AKG Decomposition . . . . . . . . . . . . . . . . 223.2.2 Role of A . . . . . . . . . . . . . . . . . . . . . . . . . 233.2.3 Role of K . . . . . . . . . . . . . . . . . . . . . . . . . 243.2.4 Role of G . . . . . . . . . . . . . . . . . . . . . . . . . 243.2.5 Demonstration using an MNIST Example . . . . . . . 253.3 Accumulated Influence . . . . . . . . . . . . . . . . . . . . . 263.3.1 Relative Stable K and Pairing Effect . . . . . . . . . 263.3.2 Evolution of G and Zig-zag Learning Path . . . . . . 283.4 Summary: Force Analysis of DNN . . . . . . . . . . . . . . . 324 Understanding and Improving Knowledge Distillation . . 334.1 Background and Related Work . . . . . . . . . . . . . . . . . 344.1.1 One-hot Supervision is not Optimal . . . . . . . . . . 344.1.2 Teacher as Better Supervision in KD . . . . . . . . . 344.2 Insights From Learning Dynamics . . . . . . . . . . . . . . . 354.2.1 Better Supervisory, Better Generalization . . . . . . . 354.2.2 Good Supervision Emerges during Training . . . . . . 364.3 Proposed Method: Filter-KD . . . . . . . . . . . . . . . . . . 394.3.1 The Algorithm . . . . . . . . . . . . . . . . . . . . . . 394.3.2 Evaluation of Filter-KD . . . . . . . . . . . . . . . . . 394.3.3 Auto Noisy-label Fixing Ability . . . . . . . . . . . . 414.4 Conclusion and Future Works . . . . . . . . . . . . . . . . . 425 Understanding LLM Finetuning . . . . . . . . . . . . . . . . 445.1 Introduction and Background . . . . . . . . . . . . . . . . . . 465.1.1 Finetuning LLM Unlocks Potential . . . . . . . . . . 465.1.2 Finetuning LLMs Comes with Instability . . . . . . . 475.2 Learning Dynamics of SFT . . . . . . . . . . . . . . . . . . . 495.2.1 Theory: Tackling the Auto-regressive Problem . . . . 495.2.2 Prepare the Probing Dataset . . . . . . . . . . . . . . 525.2.3 Explanation: Force Analysis on Different Responses . 535.2.4 Explanation: Why a Specific Hallucination Emerges . 575.2.5 Summary of SFT Part . . . . . . . . . . . . . . . . . 595.3 Method Involving Gradient Ascent: DPO . . . . . . . . . . . 595.3.1 Theory: Unified Framework on Different Methods . . 605.3.2 Theory: Gradient Ascent and Squeezing Effect . . . . 63xContents5.3.3 Theory: Positive Gradients Scatter, Negative Unite . 675.3.4 Explanation: Unexpected Behaviors of DPO . . . . . 695.3.5 Algorithm: A Simple yet Effective Pipeline . . . . . . 725.4 RL-based Finetuning: GRPO . . . . . . . . . . . . . . . . . . 765.4.1 Theory: Decompose the Gradient of GRPO . . . . . 785.4.2 Theory: Grouping Tokens via Influence Scores . . . . 805.4.3 Explanation: Lazy Likelihood Displacement . . . . . 815.4.4 Algorithm: Negative Token Hidden Reward . . . . . . 845.5 Conclusion and Discussions . . . . . . . . . . . . . . . . . . . 865.5.1 Summary of each Section . . . . . . . . . . . . . . . . 875.5.2 More on Negative Gradient . . . . . . . . . . . . . . . 885.5.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . 906 Feature Adaptation in Transfer Learning . . . . . . . . . . . 936.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 946.2 Learning Dynamics and Feature Adaptation . . . . . . . . . 956.2.1 Background and Motivation . . . . . . . . . . . . . . 956.2.2 Formalizing the Feature Adaptation . . . . . . . . . . 976.2.3 Factors that Matter: Energy and Direction . . . . . . 986.2.4 How Features Evolve under Different Settings . . . . 1006.2.5 Proof Sketch of the Trends . . . . . . . . . . . . . . . 1026.3 Strategies to Control Adaptation . . . . . . . . . . . . . . . . 1046.4 Conclusion and Future Works . . . . . . . . . . . . . . . . . 1077 Simplicity Bias and Compositionality . . . . . . . . . . . . . 1087.1 Papers Covered in this Chapter . . . . . . . . . . . . . . . . 1097.1.1 The Platonic Representation Hypothesis . . . . . . . 1097.1.2 Compression for AGI . . . . . . . . . . . . . . . . . . 1107.1.3 Related Discussions in Our Works . . . . . . . . . . . 1117.2 Compositionality and Compressibility . . . . . . . . . . . . . 1127.2.1 The Data Generating Assumption . . . . . . . . . . . 1127.2.2 The Ladder of Compositionality . . . . . . . . . . . . 1137.2.3 Kolmogorov Complexity of Different Bijections . . . . 1157.3 Simplicity Bias and Learning Dynamics . . . . . . . . . . . . 1167.3.1 Toy256 Dataset and Different Mappings . . . . . . . . 1167.3.2 The Experimental Setups . . . . . . . . . . . . . . . . 1177.3.3 Simpler Mappings are Learned Faster . . . . . . . . . 1187.3.4 Force Analysis of Different Mappings . . . . . . . . . 1217.4 Conclusion and Discussions . . . . . . . . . . . . . . . . . . . 122xiContents8 Conclusion and Discussion . . . . . . . . . . . . . . . . . . . . 124Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127AppendicesA Proofs and Calculations . . . . . . . . . . . . . . . . . . . . . . 140A.1 Risk Estimate Variance Bound in Section 4.2 . . . . . . . . . 140A.2 Calculation of the GtSFT(Ïu) in Equation (5.3) . . . . . . . . 141A.3 The Inner Product Calculation in GRPO (Proof of Equa-tion (5.14) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142A.4 Proof of Kolmogorov Complexity Bound in Section 7.2 . . . 144B Other Metrics Used in the Thesis . . . . . . . . . . . . . . . 146B.1 Calculation of Expected Calibration Error . . . . . . . . . . 146C More Details about the Experiments . . . . . . . . . . . . . 147C.1 The Probing Dataset in Chapter 5 . . . . . . . . . . . . . . . 147xiiList of Tables4.1 Quantitative comparison of generalization performance forResNet18 self-distillation (mean value of 5 runs, bold num-bers are the best method). ECE is the expected calibra-tion error (C. Guo et al. 2017), which measures how well themodelâs prediction quality. Please also refer to Appendix B.1for the formal definition. . . . . . . . . . . . . . . . . . . . . . 414.2 Noisy-label learning results on TinyImageNet dataset. . . . . 415.1 Win-rate to baseline. . . . . . . . . . . . . . . . . . . . . . . . 745.2 Ranking correlation experiment. . . . . . . . . . . . . . . . . 845.3 Results across selected math benchmarks for different Qwen2.5models and methods. The proposed method consistently pro-vides average performance gains on various models. . . . . . . 866.1 Question marks in the last column mean we cannot accuratelypredict its change. But it usually decreases in experiments. . 1046.2 How lsHP and larger heads influence the feature adaptation.The models are pretrained using SimCLR (Sim, T. Chen etal. 2020) or supervised (Sup) classification on IN1K. Numbersin blue (red) represent a decrease (increase) compared withtheir counterparts in the baseline. . . . . . . . . . . . . . . . . 1067.1 The statistical correlation between learning speed and sim-plicity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121xiiiList of Figures1.1 The evolution of xPO methods. The losses of DPO-like meth-ods can be found in Table 3 of the SimPO paper (Meng et al.2024), and the GRPO-like summary is in Table 9 of the GPGpaper (Chu et al. 2025). . . . . . . . . . . . . . . . . . . . . . 31.2 Outline of the thesis and papers covered in each part. Chap-ters 1 to 3 provide the necessary fundamentals for all thefollowing four applications. Hence, it is fine to focus only onChapters 3 and 5 if the reader only feels interested in LLMfinetuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.1 A general pipeline and notation for a typical supervised clas-sification task using Vision Transformer (ViT, Dosovitskiy etal. 2021). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.2 The per-step influence in an MNIST experiment. A largelearning rate is applied to make the difference more obvious. . 263.3 First three rows: Results showing the relative stability ofâ¥Ktuoâ¥F for learning a specific number (title of each panel)on other numbers (labeled by the colorful digits near thelines). Last row: change of train/test loss, mean of eNTK,and Eu,o[â¥Ktuo âKtâ1uo â¥F ]. (Ktuo is just Kt(xo, xu) for nota-tion conciseness.) . . . . . . . . . . . . . . . . . . . . . . . . . 273.4 Accumulated influence (pairing effect) in an MNIST experi-ment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.5 Left: a multi-variate Gaussian dataset with V = 3. Right:three typical sample groups in the Toy-Gaussian and noisy-CIFAR-10 dataset (Krizhevsky, Hinton, et al. 2009). . . . . . 293.6 Learning path of samples with different base difficulty. Cor-ners correspond to one-hot vectors. Colors represent trainingtime: transparent at initialization, dark blue at the end oftraining. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30xivList of Figures3.7 Demonstration of how the âzig-zagâ learning path emerges ina hard example: evolution of Gt is the key. . . . . . . . . . . . 324.1 Normalized (divided byâ2) distance between output distri-bution p and ground truth qâ during the one-hot training indifferent stages (left to right: initial, early stop, convergence).The bar plots show the density along the corresponding axis.The code for this experiment can be found in the notebook,and the animation of this figure is in the git repo. . . . . . . . 364.2 Learning paths on different problems. In the first two columns,we record pt for each batch, while in the last column, werecord it for each epoch. The first row is training a ResNet18on CIFAR-10, the second row is ResNet18 on CIFAR-100,and the last row is LSTM on AGNews. . . . . . . . . . . . . . 384.3 Test accuracy under different noise ratio Ï (this notation isnot softmax function here). Solid lines are the means, whileshaded regions are the standard errors for 3 runs with differentrandom seeds (shaded range is the standard error). The lastpanel compares the influence of different temperatures. Eachthin rectangle plot represents a different Ï = {0, 0.1, ..., 0.8},in which we plot the results with different Ï = {0.5, 1, 2, 4, 10}. 425.1 The teacher forcing mechanism and the causal mask. . . . . . 515.2 The 2-D plane for different probing responses. . . . . . . . . . 525.3 Left: the learning curves of y+u and other related responses;right: the force analysis of them. . . . . . . . . . . . . . . . . 545.4 Learning curves and force analysis on ungrammatical responses. 555.5 Comparison of the inflection point of different response groups(calculated using SciPy on the smoothed curves). Differentpoints in the scatter plot represent the results of differentmodels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565.6 Example of a type-B hallucination in Ravichander et al. 2025. 575.7 Learning curves of y+u ,y+test and y+j Ì¸=u and their force analysis.We visualize them together because they are all âpreferredresponse.â . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585.8 Force analysis for SFT and DPO. . . . . . . . . . . . . . . . . 635.9 Demonstration, intuition, and verification of the squeezingeffect. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655.10 Left: accumulated influence on positive and negative gradi-ents. Right: verification on a 2-gram experiment. . . . . . . . 68xvList of Figures5.11 Left: training curves of ÏÎ¸t(y+u ) and ÏÎ¸t(yâu ). Right: forceanalysis on y+u in one epochâs training (the dataset containsN problems). . . . . . . . . . . . . . . . . . . . . . . . . . . . 705.12 More evidence supporting the squeezing effect. . . . . . . . . 725.13 Illustration of the proposed method and baseline. âEâ is shortfor the âdataset extensionâ. . . . . . . . . . . . . . . . . . . . 735.14 Learning dynamics of the baseline and the proposed methodwith training data extension. . . . . . . . . . . . . . . . . . . 745.15 Prompt used for evaluating modelâs response (from Rafailovet al. 2023), an example feedback from GPT3.5-turbo, andtwo examples of the âdegenerateâ effect described in Holtz-man et al. 2020. (Sorry for the small fonts, since the degen-erated messages are usually nonsense. Please zoom in if youwant the details.) . . . . . . . . . . . . . . . . . . . . . . . . . 755.16 Averaged positive response confidence change after one update. 835.17 Highlight the tokens in the harmful group, i.e., those shouldnot be penalized in the negative response. . . . . . . . . . . . 845.18 Impact of NTHR on Gap(x) in different models. . . . . . . . . 855.19 Force analysis for some DPO variants. . . . . . . . . . . . . . 895.20 A brief summary of this chapter. . . . . . . . . . . . . . . . . 926.1 The training pipeline considered in this chapter. . . . . . . . 956.2 A transfer learning experiment on ReNet18 (pretrained onImageNet-1K and finetuned on STL10 (Coates et al. 2011). . 966.3 Left: statistics of [p0]y and [ey]y under different HP epochs.Right: approximated change of the âdirectionâ term in Equa-tion (6.2) under different Ï . Toy+MLP means pretrain a4-layer MLP on full MNIST, then transfer to a distortedsubset of MNIST. Res50-IN1K(Sup)-STL means pretrain aResNet50 on ImageNet-1k using supervised classification, thentransfer to a downstream task on STL10. . . . . . . . . . . . 996.4 How hT changes(assume full-tuning T epochs to converge)compare with h0 when head-prob Ï epochs. . . . . . . . . . . 1006.5 Up: A vector-space illustration of how features adapt dur-ing finetuning. Bottom: Empirical verification of the vector-based intuition using a toy example. From left to right:strong, mild, and tiny adaptation. . . . . . . . . . . . . . . . 1016.6 Illustrations of different metrics if they are 1-D quadraticfunctions. The curves in Figure 6.4 are a summarization ofthem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104xviList of Figures6.7 Sweeping Ï from 0 to 200 (in 2n fashion). The valid accuracyof the full-tuning-only setting is the leftmost point in eachpanel. The first 4 columns are on the image classificationtask, the fifth one is on the graph task, and the last one is onthe image segmentation task. . . . . . . . . . . . . . . . . . . 1057.1 The data-generating assumption, details in Section 7.2. . . . . 1107.2 The ladder of compositionality stating the requirements of zusing the entropy-related measurement. H(Â·) is the ShannonEntropy of a distribution. Details in Appendix A of our paper(Ren, Lavoie, et al. 2023). . . . . . . . . . . . . . . . . . . . . 1147.3 Left: four groups of mappings in our Toy256 example. Right:their roles, reproducing Figure 7 of Huh et al. (2024). . . . . 1167.4 Left: The learning curves of all 256 mappings. Middle: Thecorrelation between the learning speed and the minimum de-scription length of all mappings. Right: the compression ratein LLMâs training. The figure comes from Touvron et al. 2023and is discussed in Raeâs talk (Rae 2023). . . . . . . . . . . . 1187.5 Force analysis of two types of mappings. . . . . . . . . . . . . 1228.1 Summary and highlights of each chapter. . . . . . . . . . . . 126C.1 The prompts used to generate y+selfr, y+gpts, and y+gptf. Therephrases of rejected samples are generated similarly. Theself-rephrase template comes from Zhaorui Yang et al. 2024. . 149C.2 Example of all possible responses for one x in our probingdataset. Note that the pure random token is generated byfirst creating a random sentence, then randomly permutingits tokens. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149xviiList of AbbreviationsAGI Artificial General IntelligenceAIE Average Initial EnergyCNN Convolutional Neural NetworkCE Cross EntropyDL Deep LearningDPO Direct Preference OptimizationERM Empirical Risk MinimizationEMA Exponential Moving AverageFT Finetuning (Full-tuning in Chapter 6)GAN Generative Adversarial Networks(S)GD (Stochastic) Gradient DescentGNN Graph Neural NetworkGRPO Group Relative Policy OptimizationHP Head ProbingKD Knowledge DistillationK-complexity Kolmogorov ComplexityKL-distance KullbackâLeibler divergenceLLM Large Language ModelsLLDisp Lazy Likelihood DisplacementMSE Mean Square ErrorxviiiList of AbbreviationsMLP Multilayer PerceptronNTHR Negative Token Hidden RewardOOD Out of DistributionOPM Overparameterized ModelPT PretrainingPAC Probably Approximately CorrectPPO Proximal Policy OptimizationRL Reinforcement LearningRLHF Reinforcement Learning with Human FeedbackRLVR Reinforcement Learning with Verifiable RewardsSFT Supervised FinetuningTRPO Trust Region Policy OptimizationUFM Unconstrained Feature ModelVC-dim Vapnik-Chervonenkis DimensionViT Vision TransformerxixList of SymbolsA normalizing term in our decompositionAn, An,l the advantage used in GRPOâ¥aâ¥2 L2-norm of aâ¥aâ¥op operation-norm of aâ¥aâ¥F Frobenius Norm of aâ¨a,bâ© inner product or tensors multiplicationaâ¤ matrix or vector transposec â a scalar multiplicationa · b inner product or matrix multiplicationbâ a element-wise multiplicationbâ a element-wise divisionB the backbone parameters in OPM, the role of θ in hθb the column-wise reparameterization of B, a long vectorc, c usually a constant, or a constant vectorD datasetd¯ usually the estimated distanceE expectationey one-hot labelfθ : X â Y a function we want to learnG updating term in our decompositionxxList of SymbolsG the ground truth semantical generating factorsh the feature vector in UFM or OPMI+, Iâ the index of positive (negative) responses in one roll-outK similarity term in our decomposition, eNTKKuo the simplified notation for K(xu, xo)L the length of the sequence, or the number of labelsL loss functionM : G â Z a mapping from the ground truth factors to representationsm, v number of factors, the index of the attributesN+, Nâ the number of positive (negative) responses in one roll-outN Gaussian distributionO the task-irrelavant factorsp = Ï(z) probability predictionspi or pi the i-th element of this vectorq, q another notation for probability distributionRËn the empirical riskR real numberR the expected riskSv a symmetric group with v elementst, T usually the training time, or the stopped training timeÏ sometimes the training time, sometimes the temperatureu a uniform distribution vectorVar varianceV usually the vocabulary size or the number of classesw the readout layer in UFM or OPM, the role of Ï in gÏx, x â X the input data or featuresxxiList of Symbolsxo we generally use subscript âoâ for observing elementsxu we generally use subscript âuâ for updating elementsÏ = [x, y] the concatenated input used in LLM partÏ+u ,Ïâu the positive (negative) sequences, they share xuy, y â Y the supervision label or labelsz logits of the outputÎ±Ë+n,l, Î±Ëân,l NTHR score (with positive or negative advantage) for GRPOβ regularity strength in DPO, sometimes a pre-defined ratioγn,l the ratio of the l-th token in the n-th response in GRPOκ the eNTK of all data samplesÏ(y | x) predicting probability in the LLM part, replace pÏ+, Ïâ the ratio of positive (negative) responses in one roll-outµ mean of the Gaussian distributionÏ â¦ gÏ â¦ hθ(·) a general decomposition of our function fÏ(Â·) Softmax or Sigmod functionÏ variance of the Gaussian distributionθ parameters of the backboneÏ parameters of the task-headη learning rate, sometimes a pre-defined ratioxxiiAcknowledgementsThis thesis is the result of years of curiosity, frustration, discovery, andreflection. But more than that, it is a testament to the support, guidance,and kindness Iâve received from many people along the way. I am deeplygrateful to everyone who has walked this path with me.First and foremost, I would like to express my deepest gratitude to mysupervisor, Professor Danica J. Sutherland, for her invaluable guidance, un-wavering support, and insightful feedback throughout my Ph.D. journey.Her constant encouragement, the cheerful and open atmosphere she fostered,and the intellectual freedom she generously offered have been essential to thedevelopment of my research and my growth as an independent researcher.I would also like to thank the members of my committee, Prof. Mark Schmidt,Prof. Jeff Clune, Prof. Trevor Campbell, and Prof. Aaron Courville, for theirconstructive comments and inspiring discussions.I am grateful to my labmates and friends in Sutherlab. Special thanks to Dr.Wonho Bae, the first Ph.D. graduate in our lab. Whether warm start or coldstart, you have already embarked on a new and exciting start at your newjob! Thanks to Hamed Shirzad, the âsocial expertâ in our lab â wishingyou all the best in tackling important biological problems with your belovedgraph models. I also thank Dr. Bingshan Hu, Zheng He, Nathaniel Xu,Aaron Wei, Kuang Yao, and Mohamad Bazzi â wishing you all enjoyabletimes ahead in the lab (perhaps in a newly upgraded room!).Special thanks go to my collaborators and mentors across the world. ToShangmin Guo, who first introduced me to the intersection of cognitivescience and deep learning. To Wenlong Deng, your passion for reinforcementlearning and LLMs made working with you a real joy and a fruitful process.To Prof. Xiaoxiao Li and Prof. Christos Thrampoulidis, your instructionsand discussions were truly enlightening. I also thank Linlu Qiu and Dr.Bailin Wang for our exciting Bayesian iterated learning project. Thanks toQi Yan, Muchen Li, and other UBC peers; discussion with you is inspiring.xxiiiAcknowledgementsI am grateful to my mentors at Borealis AI, Dr. Tristan Sylvain and Dr.David Evans â thank you for your guidance and for welcoming me sowarmly to the group. I am especially thankful for the opportunity to visitand work with Prof. Aaron Courvilleâs group at Mila. Their expertise andopen-minded research environment were deeply inspiring. Thanks to mycollaborators there, Samuel Lavoie, Mikhail Galkin, and Prof. Courville, foryour critical insights and ideas, which were key to our workâs success.I also wish to thank Prof. Shay Cohen and Prof. Simon Kirby. Your teachingsignited my interest in natural language processing and iterated learning âinterests that ultimately led me into machine learning.Thanks to my dear friends and small group members â Kevin, Oleg, andRay. Your support helped carry me through many difficult times during thisjourney. I am also deeply grateful to the other members of our home group,Jungsun, Ricky, Mitsi, Joe, Sophia, and others, for creating such a warmand welcoming space. Every time we gathered, you made me feel at home.Thanks as well to pickleball and video games, my unexpected companionson this journey. They comforted me whenever I found âmy machine refusedto learn.â When experiments failed and proofs got stuck, at least I knewI was improving at something: either my pickleball skills or my World ofWarcraft gear. :-)To my parents and other family members, thank you for your endless love,patience, and belief in me, even in times when I doubted myself. Thoughwe had many misunderstandings about my career path and future plans, weendured a long, harsh winter together. Thankfully, we made it through, andthings have begun to bloom again. Your unwavering support has been myanchor throughout this journey.And finally, my deepest and most heartfelt thanks go to my best friend of thepast several years, my closest confidante, the rock of our family, and the loveof my life â my wife, Yezi (Charlotte) Xia. You were there from the verybeginning, when I left my first job to chase my dreams. Weâve traveled theworld together, seen friends come and go, but through it all, weâve alwayshad each other. Youâve stood by me in every anxious, helpless, or painfulmoment, and celebrated with me in every joyful one. Youâve witnessed everymilestone of the past several years and shared every quiet detail of our dailylife. Thank you for making me a better person, for building a shared life fullof memories with me on this winding road, and for making my life excitingand beautiful. I look forward to a lifetime of adventures together.xxivDedicationTo my family.xxv.To close this chapter, I would like to share a few poetic lines that capturewhat I feel about this journey â about what the world has given me:The world bestowed on me a life, and the grace to love it back;A tiny window to the mystery, and a gentle guide along the track.It gave me a name to grow into, and years that softly rise;A quiet poem in my hands, unfolding calm and wise.â adapted from the lyrics of âThe Treasures Presented by the WorldâxxviChapter 1IntroductionDeep learning (DL) has experienced rapid growth over the past decade, rev-olutionizing a wide range of fields including computer vision, natural lan-guage processing, robotics, and scientific discovery. Its success can largelybe attributed to the remarkable ability of deep neural networks to automat-ically extract useful representations from high-dimensional data, enablingbreakthroughs in tasks that were once considered intractable. From pow-ering superhuman-level agents like AlphaGo (Silver et al. 2016) to drivingrecent advances in generative models and large language models (LLMs),deep learning has become a central force in modern artificial intelligence.Despite its empirical success, our theoretical understanding of DL remainslimited. The behaviors of these models sometimes exceed what traditionallearning theory would predict (in both beneficial and harmful ways), raisingfundamental questions about generalization, optimization, and the natureof the learned representations. As deep learning systems grow increasinglycomplex and widely deployed, gaining deeper insights into their learningbehavior becomes not only intellectually intriguing but also practically im-portant for improving robustness, efficiency, and interpretability.This section begins by contrasting two major paradigms of deep learningtheory: the âmathematical style,â which emphasizes formal guarantees andproofs, and the âphysics style,â which focuses on behaviors and averagetrends. We then use the evolution of reinforcement learning (RL) inspiredLLM finetuning algorithms as a case study to illustrate why rapidly evolvingsystems demand physics-like theories.A brief overview of different chapters is also provided. In essence, this workaims to advance our understanding of deep learning systems by studyingtheir learning dynamics. Through a combination of theoretical insights andempirical analyses, we investigate how learning different data samples in-fluences the modelâs behavior and how these influences are accumulated toshape both the modelâs generalization behavior and its internal representa-11.1. Math-like and Physics-like DL Theorytional structures. We argue that the proposed force analysis and learningdynamics framework offers a valuable and flexible toolkit for interpretingand monitoring the behavior of modern deep learning systems. It has thepotential to generate novel insights and guide the development of more prin-cipled and efficient algorithms.1.1 Math-like and Physics-like DL TheoryIn the study of deep learning, researchers often adopt two distinct theoreti-cal paradigms: math-like theory and physics-like theory. Although sharingmany overlaps and similarities, these paradigms differ in their goals, method-ologies, and views on inevitable assumptions.Math-like theory aims for a rigorous and formal understanding. It focuseson providing provable guarantees, typically under well-defined assumptions,regarding generalization, optimization, or expressivity. These results are of-ten elegant and general, but may rely on idealized conditions that do notfully capture the complexity of real-world deep learning systems. For exam-ple, convergence proofs for gradient descent or bounds on VC dimension fallunder this category (Blumer et al. 1989). The strength of this approach liesin its clarity and reliability. However, its scope is sometimes limited by thegap between theoretical assumptions and practical realities.In contrast, physics-like theory draws inspiration from empirical sciences.Rather than seeking absolute guarantees, it emphasizes building intuitive,often approximate models that help explain and predict observed behaviorsof neural networks. This approach treats deep learning systems as complexdynamical entities, akin to physical systems, and tries to identify governingprinciples, such as implicit bias, learning dynamics, or phase transitions,through a combination of empirical evidence, analytical approximations, andsimplified models. Examples include the information bottleneck (Tishby etal. 1999; Tishby and Zaslavsky 2015), which combines information theoryand a general phase transition of deep modelsâ training, the study of localelastics on neural networks (H. He and W. Su 2020), loss landscapes, circuithypothesis of the Transformer (Olsson et al. 2022), and so on.The physics-like approach accepts that a perfect model may be unattainable,but a good enough approximation can still yield powerful insight and guidepractice. While this comes at the cost of mathematical rigor, it allows theoryto stay closer to the empirical frontier, particularly for large-scale modelswhere traditional assumptions no longer apply.21.2. Rapidly Evolving Algorithms Call for Physics-like TheoriesIn summary, math-like theory prioritizes analytical expressions and rigor,while physics-like theory values intuition and empirical alignment. Both areessential: one provides a solid foundation, and the other helps us navigatethe messy reality of deep learning systems. Their interplay continues toshape the evolving landscape of theoretical machine learning. The frame-works studied in this thesis lean more towards the physics-like theory.1.2 Rapidly Evolving Algorithms Call forPhysics-like TheoriesTRPO(2015)PPO(2017)DPO(2023)GRPO(2024)KTOSimPOIPOR-DPOCPOSLiC2023~2025ORPO2024~2025DAPO ORZ-repoDrGRPOGVPO GRPO-LeadSRPOFigure 1.1: The evolution of xPO methods. The losses of DPO-like methodscan be found in Table 3 of the SimPO paper (Meng et al. 2024), and theGRPO-like summary is in Table 9 of the GPG paper (Chu et al. 2025).Many deep learning systems are highly complex in different ways, whichmakes math-like theories hard to track. For instance, many such theo-ries focus on convergence to a global optimum or assume full training tocompletion. In practice, however, models are often early-stopped based onperformance on a held-out validation set, and convergence to a local opti-mum (or even stopping at the vicinity of a good local optimum) is usuallysufficient for deployment. Similarly, while theoretical analyses often assumedatasets are drawn from well-behaved distributions, real-world models mightencounter some outlier examples, which can strongly influence the modelâsbehavior: the actual dataset might be way smaller than the required sizefor the theoryâs assumption. Moreover, the architectures of modern systemslike LLMs are often too complicated for a precise analytical description. Inshort, while we aspire for theory to be rigorous and comprehensive, practicalalgorithms typically operate under far-from-ideal conditions.31.2. Rapidly Evolving Algorithms Call for Physics-like TheoriesThis gap between theory and practice is further enlarged by the rapid evo-lution of the focusing tasks and settings of the algorithms. For example,the Trust Region Policy Optimization (TRPO, Schulman et al. 2015) algo-rithm provides strong theoretical guarantees within the classical reinforce-ment learning (RL) framework. Its successor, Proximal Policy Optimization(PPO, Schulman et al. 2017), retains similar theoretical backing and signif-icantly improves its efficiency.However, when applying these RL algorithms to the field of LLM finetuning,maintaining strict theoretical guarantees has become increasingly difficult.For example, while Direct Preference Optimization (DPO, Rafailov et al.2023) offers valuable theoretical insights in its equivalence to PPO in theideal setting, its extension to LLM finetuning requires more considerations,e.g., whether modeling context-token as a state-action pair is appropriate, orwhether the samples are generated in an on-policy way. Due to its simplic-ity and empirical effectiveness, DPO inspired numerous follow-up methods.These variants prioritize theoretical intuition more over theoretical guaran-tees, as the scenarios they consider are more practical and subtle.For example, SimPO points out the inaccuracy of the token-level rewardand hence designs a new reward using the mean log-probability (Meng etal. 2024). SLiC calibrates the likelihood of ranked sequences and appliesa threshold to the margin (Yao Zhao et al. 2023). Other extensions, e.g.,KTO (Ethayarajh et al. 2024), CPO (H. Xu et al. 2024), IPO (Azar et al.2024), R-DPO (Park et al. 2024), and ORPO (Hong et al. 2024), are alsoproposed following a similar methodology. Although these methods sacrificesome theoretical rigor compared to PPO and DPO, they demonstrate prac-tical effectiveness across diverse settings: understanding why the algorithmswork, although many assumptions are violated, is interesting and important.A parallel trend is seen in Group Relative Policy Optimization (GRPO, Shaoet al. 2024) and its derivatives. For instance, DAPO independently adjustsGRPOâs dual thresholds to balance exploration and exploitation (Q. Yuet al. 2025); DrGRPO removes the normalization term when computingadvantages (Z. Liu et al. 2025); and GPG omits both the clipping term andKL regularization entirely (Chu et al. 2025). Other variants, such as ORZ-Repo (J. Hu et al. 2025), GVPO (K. Zhang et al. 2025), GRPO-Lead (J.Zhang and C. Zuo 2025), and SRPO (Xiaojiang Zhang et al. 2025), furtherdeviate from the original theoretical assumptions while still achieving strongempirical results under different settings.Also striking is the pace of the development of the algorithms: all these41.3. Publications and Related Chaptersvariants emerged between 2024 and 2025, within a single year of DPOand GRPOâs first introduction. Such a rapid growth of different practi-cal methods focusing on distinct scenarios is also common in other deeplearning fields, e.g., computer vision, vision language models, graph neuralnetworks, etc. This rapid evolution inevitably restrains the exploration ofthose strict guarantees and simultaneously underscores the need for fast-adapting physics-like theories: for more on capturing qualitative patternsand dynamics rather than on deriving strict guarantees, and if possible, aunified view for most of them.1.3 Publications and Related ChaptersThis subsection presents the set of research papers that form the founda-tion of this thesis, to which I have contributed. The papers are listed inchronological order to illustrate how the central ideas, particularly learningdynamics and force analysis in deep neural networks, have gradually de-veloped over time. For each paper, I include a brief summary of its maincontent, key takeaways, and specific contributions of each author.As this thesis spans a broad range of topics, some conceptual connectionsmay not be immediately clear upon first reading. I recommend revisiting thispart after completing the thesis to gain a more comprehensive understandingof how learning dynamics serve as a unifying framework across diverse deeplearning systems. The relationships between these works and their respectiveroles in different chapters are also illustrated in Figure 1.2.0. âCompositional Languages Emerge in a Neural Iterated Learning Model.âYi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, Simon KirbyInternational Conference on Learning Representations (ICLR), 2020This paper was completed during my Masterâs studies, but I believe it de-serves inclusion here, as it directly inspired the development of the learningdynamics framework central to this thesis. The work extends the iteratedlearning framework, originally studied in evolutionary linguistics (Kirby etal. 2015), to the domain of deep learning. In essence, Kirby et al. (2015)argue that the evolution of human language toward compositionality can beexplained by the simultaneous pressures of expressivity (language should beuseful and unambiguous) and compressibility (language should be as efficientas possible), which are then gradually amplified over generations.51.3. Publications and Related ChaptersIn our study, we adapt this idea to a simplified emergent communicationsetting, modeled as a two-agent reinforcement learning scenario involving asender and a receiver. We observe that the simplicity bias favoring compo-sitional mappings is manifested as a learning speed advantage.To understand the origin of this simplicity bias, we initially considered for-malizing it using the Vapnik-Chervonenkis dimension (VC-dim, Blumer etal. 1989) within the Probably Approximately Correct (PAC, Valiant 1984)framework. However, we found that such classical generalization theoriesstruggled to adequately capture the dynamics we observed. This limitationled us to shift our focus to a more fine-grained, micro-level perspective, i.e.,examining how learning one updating example influences the modelâs behav-ior on another observing example. This line of thinking laid the foundationfor the force analysis framework later developed and expanded in this thesis.This paper is covered by Chapters 3 and 7.1. âBetter Supervisory Signals by Observing Learning Paths.âYi Ren, Shangmin Guo, Danica J. SutherlandInternational Conference on Learning Representations (ICLR), 2022In this work, we begin to formalize the interaction between different trainingexamples within a fundamental supervised learning setting. We introducethe AKG decomposition, where G(xu) quantifies the distributional gap fromthe modelâs prediction to the supervisory distribution of the updating exam-ple xu; and K(xo, xu) captures how the learning of xu influences the modelâsconfidence on an observing example xo. Notably, we find that K correspondsexactly to the empirical Neural Tangent Kernel (eNTK, Jacot et al. 2018)of the backbone network, allowing us to integrate a broad range of existingNTK-related theoretical results into our analysis.The most interesting contribution of this paper lies in applying the proposedforce analysis framework to explain the âzig-zagâ learning trajectories ob-served in certain difficult examples. These dynamics arise from the interplaybetween the similarity structure encoded in K and the evolving gradientmagnitude governed by G. This analysis offers a mechanistic understandingof how the modelâs predictions gradually evolve.As a practical byproduct of our framework, we also propose âFilter-KD,â avariant of knowledge distillation designed to improve robustness, particularlyin scenarios where labels may be noisy. By selectively filtering examplesbased on their learning dynamics, this method helps stabilize training and61.3. Publications and Related Chaptersenhance generalization. This paper is covered by Chapters 3 and 4.2. âHow to Prepare Your Task Head for Finetuning.âYi Ren, Shangmin Guo, Wonho Bae, Danica J. SutherlandInternational Conference on Learning Representations (ICLR), 2023This paper extends our learning dynamics framework to a broader represen-tation learning setting, where we model the adaptation of hidden featuresrather than just the modelâs output. The initial motivation stemmed fromthe idea of generalizing the iterated learning framework to deep networks.Since iterated learning typically involves a sender and a receiver, we reinter-pret a standard neural network by splitting it at the hidden representations:the backbone acts as the sender, the task head as the receiver, and thehidden representation as the message passed between them.Although this parallelism directly hints at a multi-generation training forrepresentation learning, this paper does not pursue that direction in depth.Instead, we uncover a particularly interesting dynamical interaction betweenthe adaptation of features and the initialization of the task head. As a result,the paper shifts focus to understanding how to choose a task head that bestfacilitates feature adaptation. The analysis of feature learning dynamicsthus becomes a secondary yet insightful by-product of this investigation.We will elaborate on this work in Chapter 6.3. âImproving Compositional Generalization using Iterated Learning andSimplicial Embeddings.âYi Ren, Samuel Lavoie, Mikhail Galkin, Danica J. Sutherland, Aaron CourvilleConference on Neural Information Processing Systems (NeurIPS), 2023This work successfully extends the iterated learning framework to more gen-eral representation learning problems and can be viewed as a direct follow-up to our ICLR 2020 paper. We generalize the framework to address thesystematic generalization problem across different domains, including bothvisual inputs and molecular graph inputs.In this study, force analysis and learning dynamics are employed as explana-tory tools to account for the observation that mappings containing more cor-rect structural guesses are learned more efficiently. Thus, the paper servesnot only as a practical application of iterated learning and force-based anal-ysis to real-world problems, but also as a bridge linking learning dynamicsto another fundamental theme in deep learning: simplicity bias. This con-71.3. Publications and Related Chaptersnection is further explored and elaborated in Chapter 7 of the thesis.4. âlpNTK: Better Generalisation with Less Data via Sample InteractionDuring Learning.âShangmin Guo, Yi Ren, Stefano V Albrecht, Kenny SmithInternational Conference on Learning Representations (ICLR), 2024This paper is a joint side project with Shangmin. Here, we focus on the Kterm in our AKG decomposition. Since K can be interpreted as a similaritymeasurement between examples, we explore its application to the core-setselection problem in supervised learning.Our analysis reveals that the relationships between different training exam-ples can be broadly categorized into three types: 1.) Interchangeable pairs,where the examples contain highly similar information, such that retainingonly one of them is sufficient; 2.) Orthogonal pairs, where the updates fromeach example have minimal influence on the other; 3.) Contradictory pairs,where learning from one example negatively impacts the modelâs confidenceon the other, and vice versa.These insights provide a richer understanding of the interactions amongtraining examples, offering valuable perspectives on phenomena such as gen-eralization, overfitting, and data redundancy. This work demonstrates howthe K term can serve as a powerful tool for data-efficient learning.5. âBias Amplification in Language Model Evolution: An Iterated LearningPerspective.âYi Ren, Shangmin Guo, Linlu Qiu, Bailin Wang, Danica J. SutherlandConference on Neural Information Processing Systems (NeurIPS), 2024This work represents our initial attempt to extend the iterated learningframework to large language models (LLMs). Motivated by the growingpopularity of self-interaction among LLM agents, we observe that their it-erative behaviors can often be well-characterized by the Bayesian iteratedlearning framework. While our learning dynamics framework does not playa central role in this study, the observed phenomenon of self-amplifying be-havior in LLM self-improvement strongly motivates a deeper investigationinto their evolution during finetuning. This direction ultimately leads to oursubsequent projectâLearning Dynamics of LLM Finetuning, which servesas the central focus and highlight of this thesis.81.3. Publications and Related Chapters6. âLearning Dynamics of LLM Finetuning.âYi Ren, Danica J. SutherlandInternational Conference on Learning Representations (ICLR), 2025In this work, we extend our previously developed learning dynamics frame-work to the finetuning of LLMs. The AKG decomposition is also adapted tothe next-token prediction setting, with the help of some standard assump-tions commonly considered in LLM training.By applying force analysis to the modelâs training, we offer a unified mech-anistic explanation for several intriguing behaviors observed across differentfinetuning methods. These include the unexpected increase in confidence forless-preferred answers during Supervised Finetuning (SFT), the emergenceof specific hallucinations, the exacerbated ârepeaterâ phenomenon, and thesimultaneous reduction in confidence for both chosen and rejected responsesduring Direct Preference Optimization (DPO, Rafailov et al. 2023).A particularly notable and counter-intuitive phenomenon revealed by ouranalysis is the âsqueezing effect,â which arises when negative gradients areinappropriately imposed in the loss function. This effect offers insight intohow confidence mass can be unintentionally redistributed during finetuning,sometimes harming the modelâs alignment and reasoning capabilities.The key results and analysis from this paper are reorganized into a moreaccessible structure in Chapter 5 of this thesis. Readers may also find thefoundational background provided in Chapters 3 and 4 helpful for under-standing the full context of this work. Some new findings like âpositivepressure scatters, negative pressure unitesâ are also proposed in this thesis.7. âOn the Effect of Negative Gradient in Group Relative Deep Reinforce-ment Optimization.âWenlong Deng, Yi Ren, Muchen Li, Danica J Sutherland, Xiaoxiao Li,Christos ThrampoulidisConference on Neural Information Processing Systems (NeurIPS), 2025This is a recent project led by Wenlong, in which I participated actively. Inthis work, we extend our learning dynamics analysis to a more complex rein-forcement learning-based LLM finetuning algorithm, namely Group RelativePolicy Optimization (GRPO, Shao et al. 2024). Specifically, we adapt theAKG decomposition to a token-wise formulation, allowing for fine-grainedanalysis of gradient dynamics during training.91.4. Thesis OutlineOur study reveals that certain inappropriately applied negative gradientscan still degrade model performance, even in this on-policy setup. Throughempirical analysis, we find that such harmful negative gradients are oftenimposed on semantically correct tokens (i.e., the correct part in an incorrectresponse), which can undermine the modelâs intended behavior.To address this, we introduce the concept of Negative Token Hidden Reward(NTHR), a metric designed to assess whether the negative gradient on aspecific token is constructive or detrimental. By selectively masking outtokens with high NTHR during training, we are able to stabilize GRPOâstraining process and enhance the modelâs reasoning ability. This paper isalso discussed in Chapter 5.8. âUnderstanding Simplicity Bias towards Compositional Mappings viaLearning Dynamics.âYi Ren, Danica J. SutherlandWorkshop on Compositional Learning at NeurIPS, 2024This is the final paper discussed in this thesis, which returns to the foun-dational question that originally motivated our study of learning dynamics:Can the interactions among training examples help explain the emergenceof simplicity bias? This question lies at the heart of representation learn-ing, and connects to broader philosophical and theoretical themes such asOccamâs Razor and the pursuit of Artificial General Intelligence (AGI).This study represents an initial step toward understanding this problem.We show that representations that are better aligned with the underlyingground-truth factors tend to be learned faster, providing empirical evidencefor a learning speed bias favoring simpler, more structured solutions.These findings resonate with several contemporary discussions, includingthe Platonic Representation Hypothesis (Huh et al. 2024) and Compressionfor AGI (Rae 2023). We hope that our proposed framework offers a newlens to understand this fundamental phenomenon, and that it can inspiremore practical methods for improving deep learning systems by leveraginginsights from learning dynamics. We discuss this paper in Chapter 7.1.4 Thesis OutlineRecognizing the need for a physics-like theory to understand various deeplearning systems, we adopt the following perspective: instead of focusing on101.4. Thesis Outlineglobal or asymptotic behavior, we emphasize a local and microscopic viewof model dynamics. Specifically, we begin by analyzing the modelâs one-stepbehavior after learning a single training example. Once this foundationalbehavior is understood, we carefully scale up our analysis to consider theaccumulated influence over multiple updates or across different stages oftraining. This progressive methodology offers a clearer picture of how com-plex behaviors and emergent phenomena arise in deep learning systems.Our main analytical tool is a form of âforce analysisâ applied to the modelâsoutput or hidden parameters, theoretically grounded in an influence functiondecomposition. The central topic studied here is the learning dynamics ofthe model, i.e., how its outputs and internal states evolve over time underthe influence of training signals. This approach resonates with classicalmechanics, where understanding the force acting on an object reveals itsfuture trajectory. For this reason, the thesis is titled âLearning Dynamicsof Deep Learning.â The remainder of this thesis is organized as follows:â¢ Chapter 2 reviews the historical development and recent trends in theo-retical deep learning, highlighting a renewed interest in physics-like ap-proaches. This context motivates our core contribution: a microscopicand dynamical analytical framework centered on learning dynamics.â¢ Chapter 3 introduces the foundational elements of our framework. Start-ing from a basic supervised classification setting, we present key conceptssuch as the AKG decomposition, the one-step, and accumulated influ-ences, and the dynamical evolution of the force subsequently. The chapterconcludes by drawing parallels between our framework and force analy-sis in Newtonian mechanics to help readers intuitively grasp the gist andimplications of our framework.â¢ Chapter 4 applies the framework to a knowledge distillation (KD) setting.We show how better supervisory signals can emerge during training, andpropose a practical method called Filter-KD, which is especially robustunder noisy labeling conditions.â¢ Chapter 5 is the highlight of this thesis. Here, we extend our force anal-ysis framework to LLM finetuning, offering novel insights into severalcounterintuitive behaviors observed across different settings. Topics in-clude hallucination, model degeneration, misalignment, confidence decayin preferred responses, and the underestimated influence of negative gra-dients. We study a range of algorithms, including SFT, DPO, and GRPO,and emphasize that our framework is broadly applicable to other methods111.4. Thesis Outlinethat rely on gradients of model log probabilities. We hope this frameworkoffers clearer interpretations of the pros and cons of different algorithms,and hence inspires more effective future algorithm designs.â¢ Chapter 6 expands the framework by changing the observed targets fromoutputs to hidden representations. In the context of feature adaptation,we theoretically predict how internal features evolve under different ini-tialization conditions. We also highlight core physical concepts such asforce, direction, and adaptation energy. Inspired by the analysis, we pro-pose strategies to guide adaptation in a controlled and efficient manner.â¢ Chapter 7 switches to a more fundamental fact in machine learning: un-derstanding the simplicity bias in deep learning, i.e., answering why mod-els tend to favor simpler mappings in accordance with Occamâs Razor.Using force analysis, we connect this bias to mutual influence betweentraining examples across different mappings, offering a perspective alignedwith several prominent theories in the machine learning literature.â¢ Chapter 8 summarizes the contributions of the thesis. The limitationsof this framework and potential future directions are also discussed. Theassociated publications for each chapter are listed in Figure 1.2.Chapt. 1Introduction Chapt. 3Fundamentals of Learning Dynamics1. Ren et al., ICLR 20220. Ren et al., ICLR20204. Shangmin, Ren et al., ICLR2024Chapt. 5UnderstandingLLMâs finetuning6. Ren et al., ICLR 20255. Ren et al., NeurIPS 20247. Wenlong, Ren et al. NeurIPS2025 Chapt. 6Feature Adaptationin Transfer Learning2. Ren et al., ICLR 2023Chapt. 7Simplicity Biasand Compositionality8. Ren et al., NeurIPS-W 20243. Ren et al., NeurIPS 20230. Ren et al., ICLR2020Chapt. 8ConclusionChapt. 2OverviewChapt. 4For BetterKnowledge Distillation1. Ren et al., ICLR 2022Figure 1.2: Outline of the thesis and papers covered in each part. Chapters1 to 3 provide the necessary fundamentals for all the following four applica-tions. Hence, it is fine to focus only on Chapters 3 and 5 if the reader onlyfeels interested in LLM finetuning.12Chapter 2An Overview of Progress inDeep Learning2.1 History of Deep LearningThe history of deep learning traces back to the early development of artificialneural networks in the 1940s, beginning with the McCulloch-Pitts neuronmodel, which laid the foundation for the usage of simple binary thresholdunits (McCulloch and Pitts 1943). This was followed by the perceptron algo-rithm introduced by Rosenblatt (1957), which demonstrated the feasibilityof learning from labeled examples, but was soon critiqued for its inabilityto solve non-linearly separable tasks such as XOR (Minsky and Papert 1969).The 1980s witnessed a revival of interest with the development of backprop-agation, a gradient-based optimization algorithm enabling the training ofmulti-layer networks (MLP, Rumelhart et al. 1986). This period also sawthe introduction of convolutional neural networks (CNNs) for image recog-nition tasks, notably the Neocognitron (Fukushima 1980) and later LeNet(LeCun et al. 1989).Despite these advances, neural networks remained relatively niche for decades,in part due to computational limitations and the dominance of alternativemachine learning models such as support vector machines (SVMs) and de-cision trees. The resurgence of deep learning in the 2000s and 2010s wasenabled by three key factors: access to large labeled datasets, advances incomputing hardware (especially GPUs), and algorithmic innovations. A ma-jor turning point came in 2012 with the success of AlexNet (Krizhevsky et al.2012) in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC,J. Deng et al. 2009), which dramatically reduced the error rate and demon-strated the power of scale.Since then, deep learning has rapidly expanded across domains, includingspeech recognition (Hinton, L. Deng, et al. 2012), natural language pro-132.1. History of Deep Learningcessing (NLP, Sutskever et al. 2014), and reinforcement learning (RL, Mnihet al. 2015). Transformer-based architectures (Vaswani et al. 2017) in par-ticular have become foundational to modern NLP systems, culminating inlarge-scale models like BERT (Devlin et al. 2019) and GPT (Radford et al.2018).Throughout the history of deep learning, both the models and the problemsthey address have grown increasingly complex along multiple dimensions.One notable example is model size, typically measured by the number of pa-rameters. For instance, the state-of-the-art sequence-to-sequence (seq2seq)model for natural language processing in 2014, based on LSTMs (Hochre-iter and Schmidhuber 1997a) and introduced by Sutskever et al. (2014),contained approximately 100 million parameters. This number rose to 340million in 2018 with the introduction of BERT-large. Today, with the riseof ChatGPT and the validation of scaling laws for large language models(LLMs), the scale of modern deep learning systems has expanded dramati-cally, reaching hundreds of billions of parameters. The estimated parametercount for GPT-4, for example, is around 1 trillion (1000B), which is 10,000times larger than the early seq2seq model. This rapid increase in model sizehas not only significantly enhanced the capabilities of LLMs but also madethe analysis of their behavior vastly more challenging.Another dimension of increasing complexity lies in the data and tasks mod-ern models are expected to handle. As the capabilities of language modelshave advanced significantly, the focus of natural language processing hasshifted: from relatively simple tasks like grammatical parsing to more chal-lenging ones such as reading comprehension and mathematical reasoning.Moreover, the widespread adoption of Transformer architectures across dif-ferent modalities has made multimodal tasks not only feasible but increas-ingly common. In these settings, models must integrate and reason over het-erogeneous inputs, such as text, images, and audio, posing a much greaterchallenge than single-modality tasks.These developments have not only redefined state-of-the-art performanceacross tasks but also prompted a renewed focus on understanding the learn-ing behavior and generalization properties of deep models. As deep learningsystems continue to grow in scale and application, there is an increasing needfor theoretical frameworks that can capture their dynamics, interpretabil-ity, and robustness, motivating the exploration of a unified and effectiveexplanatory framework pursued in this thesis.142.2. More Local and More Microscopic2.2 More Local and More MicroscopicAs models continue to grow rapidly in size and capability, we are witnessinga subtle but significant shift in the direction of deep learning theory tokeep pace with this remarkable progress. In particular, there is a growinginterest in behavioral analysis and physics-inspired theoretical frameworks.The focus of such theories is also shifted from a global and macroscopicperspective to a more local and microscopic viewpoint. Below, we presentseveral illustrative examples.For example, classical generalization theory has traditionally relied on globalcapacity measures, including VC-dimension, Rademacher complexity, anduniform convergence bounds across the entire hypothesis space. However,recent research emphasizes the local geometry of decision boundaries arounddata points. The norm-based and margin-based generalization bounds fordeep networks introduced by Neyshabur et al. (2017) and Bartlett et al.(2017), for instance, analyze local quantities such as the Lipschitz constantnear training points. These localized analyses help explain why overpa-rameterized networks can still generalize well, despite their large hypothesisclasses.A similar shift is observed in the study of the loss of landscapes. Earlierwork focused on understanding the global non-convex optimization land-scape, such as identifying the number and nature of critical points or badlocal minima. In contrast, more recent analyses emphasize local curvatureproperties, such as the Hessian near the learned solution or throughout train-ing. A notable example is the flat minima hypothesis proposed by Keskaret al. (2016), which suggests that flatter local regions in the loss landscapeare associated with better generalization.The neural tangent kernel (NTK, Jacot et al. 2018) hypothesis, a central the-oretical tool in this thesis, can also be interpreted as an instance of local andmicroscopic analysis. Its derivation relies heavily on repeated applications ofthe first-order Taylor expansion, which not only provides an effective linearapproximation of the modelâs behavior but also shifts the analytical focusto higher-level phenomena, such as model architecture, interactions amongdifferent components of the model, or phase transitions during training.This theory has also been applied to analyze generalization (Arora, Du, WeiHu, Z. Li, and R. Wang 2019) and convergence rates (Zou et al. 2020), sometypical questions in classical learning theory. Furthermore, NTK-based anal-ysis has been extended to different network architectures, e.g., convolutional152.2. More Local and More Microscopicnetworks (Arora, Du, Wei Hu, Z. Li, Salakhutdinov, et al. 2019), residualnetworks (J. Lee et al. 2019), and even graph neural networks (Du et al.2019). Despite its limitations in capturing feature learning in finite-widthsettings, NTK remains a powerful tool for probing the local geometry of thelearning process and motivating further theoretical developments.This local perspective also extends to the temporal dimension of training.Different from the classical theoretical analysis, which focuses more on theasymptotic convergence behavior of models, more work about understand-ing the training dynamics over time, particularly phenomena such as phasetransitions and emergent behaviors (J. Wei et al. 2022), has emerged.Collectively, these developments reflect a growing recognition that to trulyunderstand how deep models learn, adapt, and generalize, it is crucial tostudy the dynamics of local updates, gradient flows, and pointwise behaviors,because the global, static analyses often miss these nuances.In line with this paradigm shift, the learning dynamics framework proposedin this thesis can be viewed as a local extension of the influence functionframework introduced by Koh and Liang (2017). While that work traces amodelâs test-time predictions back to influential training examples over theentire training trajectory, our framework zooms in further: we begin by ana-lyzing the one-step influence between pairs of examples, and then graduallyscale up to accumulated influence over time. Instead of using the gradientflow, which assumes the training time t to be infinitely small and focuses onthe change of the observing example, we use the one-step gradient update,which focuses more on the mutual influence between the updating and ob-serving examples. However, we believe that these two framing frameworksare inherently identical under mild assumptions.16Chapter 3Fundamentals of LearningDynamicsAs discussed in earlier chapters, the growing complexity of models and tasksin modern deep learning has made it increasingly difficult for traditionalanalytical approaches, which usually study the system as a whole, to provideaccurate global behavioral guarantees. In response, this thesis adopts abottom-to-up methodology, i.e., the one that emphasizes localized analysisand aligns more closely with a physics-inspired theoretical style. Specifically,we narrow down our scope in the following two aspects:â¢ Time-level: rather than the global convergence analysis, which consid-ers training a randomly initialized model to its convergence, we startfrom the modelâs behavioral change after one (or few) update(s);â¢ Sample-level: rather than analyzing the modelâs performance on a fulldataset, we start from the modelâs behavioral change on one exampleafter learning one training sample.Combining the two principles, we define the learning dynamics here asMeasuring how the modelâs confidence on an observing examplechanges after learning an updating example.In the remainder of this thesis, we will define the problem more formallyunder corresponding settings. For now, we only need to know that the modelparameters are updated using the gradient from an âupdating example,âand we track the resulting confidence change in one or more âobservingexamples.â Our analysis centers on the proposedAKG decomposition, whichbreaks down this one-step change into three interpretable terms. We alsoextend the analysis to accumulated influence over time, illustrated by atypical âzig-zagâ learning trajectory. The chapter concludes with a summaryand discussion of limitations.173.1. Warm-up with Supervised Learning3.1 Warm-up with Supervised Learning3.1.1 Problem: Start from Supervised ClassificationSupervised learning is one of the most established and widely used paradigmsin machine learning. It refers to the task of learning a function f : X â Yfrom a labeled dataset of input-output pairs {(xi, yi)}ni=1, where xi â X de-notes the input (e.g., a feature vector) and yi â Y denotes the correspondinglabel or target for the model to learn.The goal is to find a hypothesis f â F , from a chosen function class F (e.g.,neural networks, decision trees, etc.), that generalizes well to new data drawnfrom the same (but unknown) data-generating distribution D(x, y).The learning process aims to minimize the expected risk:R(f) = E(x,y)â¼D[L(f(x), y)]where L : YÃY â R is a loss function that quantifies the error between pre-diction and ground truth (e.g., 0-1 loss, mean squared error, cross-entropy).Since D is usually unknown, we minimize the empirical risk instead:RËn(f) = 1nnâi=1L(f(xi), yi).This leads to the Empirical Risk Minimization (ERM) principle, a centralconcept in statistical learning theory.For most of the deep learning systems considered in this thesis, the outputwould be a probability distribution over all possible classes. We hence de-compose the function f into several parts and parameterize them separately.In general, we have f(x) = Ïâ¦gÏâ¦hÎ¸(x), where hθ(·) is usually the backboneof the network parameterized by θ, and gÏ(Â·) is the task head parameterizedby Ï. We usually call the output of the backbone part h = hθ(x) âhiddenembeddingsâ or âhidden features.âThen, we define the logits z â RV as z = gÏ â¦ hθ(x), which is a length-Vvector, and V is the number of all possible classes. Since we usually wantour model to give a probability distribution prediction over all possible V ,a softmax layer Ï(Â·) is usually attached to the logits z, i.e.,Ïi(z) â Softmaxi(z) =eziâVj=1 ezj.183.1. Warm-up with Supervised Learningð¥ ðð½ ð³ð¡ ðð ð(â ) ð© ð(ð©, ð)Input BackboneHidden-embdfeatures Task head Logits Softmax Prediction LossViT EncoderCLS TokenInput patchesMLPCan also be complexLength-Vvectorðð§ð ð=1ð ððð§Length-VsimplexDepending onthe taskFigure 3.1: A general pipeline and notation for a typical supervised classi-fication task using Vision Transformer (ViT, Dosovitskiy et al. 2021).Note that the separation of hθ and gÏ can be arbitrary, depending on whathidden features we want to observe (as will be discussed in detail in Chap-ter 6). As a warm-up example, we simplify the model by considering g asan identity function. In other words, we havep(x) = f(x) = Ï(hÎ¸(x)) = Ï(z),where p(x) = [p(y = 1 | x), . . . , p(y = V | x)] is the vector form of thepredicted distribution. Please refer to Figure 3.1 for an example of thenotations and their roles in a deep learning system. We will largely stick tothis definition throughout this thesis.We then discuss the target we want to optimize. Depending on the problemwe are facing, the loss functions can be diverse. Throughout this thesis, wemainly focus on the following two types of losses:â¢ Cross-Entropy Loss (CE, for classification with probabilities):Lce (f(x), y) = âVâv=1yv log(p(y = v | x)) = âeâ¤y log p(x), (3.1)where yv is the indicator of ground-truth label and ey is its vectorform (i.e., yv is one for the correct label v and is zero otherwise).p(x) â [p(y = 1 | x), . . . , p(y = V | x)] is the vector form of the modelâsprediction. If the supervisory signal is not one-hot but a distribution,e.g., in the knowledge distillation setting we will discuss in Chapter 4,we can have a cross-entropy loss like Lce = âqâ¤ log p, where q isanother probability vector with the same size as p.â¢ Mean Squared Error (MSE, usually for regression):Lmse (f(x), y) = (f(x)â y)2 , (3.2)193.1. Warm-up with Supervised Learningwhere both f(x) and Y â R are scalars. Note that it is also possiblefor us to use MSE loss for classification problems. For example, if ourmodel generates probabilistic predictions p(x) and we still want to useMSE loss, we can use Lmse = â¥p â eyâ¥22, where â¥ · â¥2 is the L2-normof a vector. Hui (2021) provides more discussions about the pros andcons between MSE and CE when applied to a classification task.3.1.2 Learning: Gradient DescentRecall our goal is to find a good function f that performs well on the datadistribution D. Since the ground truth D is usually intractable, we insteadminimize the empirical risk RËn using a training dataset sampled from D.Depending on the task we are facing, an appropriate loss function is chosento measure how well our parameterized model fθ performs on this dataset.Then, the next question is: how could we find the best fÎ¸â with the help ofa training set Dtrain?Although numerous closed-form solutions for fÎ¸â exist under various assump-tions, optimization-based methods have become the primary workhorse inthe deep learning era, owing to their simplicity and scalability. Among these,gradient descent (GD, see e.g. Ruder 2016) is particularly popular, as it onlyrequires the existence of first-order gradients and is applicable across deeplearning systems of varying scales.Specifically, we want to solve the following optimization problem:Î¸â = argminÎ¸âÎRËn(f).The idea of GD is to iteratively update the parameters θ in the direction ofthe gradient of the function, which points toward the direction that providesthe steepest loss descent:θt+1 = θt â Î·âÎ¸RËn(f),where η > 0 is the learning rate, which is usually very small, and t denotesthe time (or step) during training. âÎ¸RËn(f) is the gradient of the estimatedrisk on all training data, which can also be written as 1nâni=1âÎ¸L(f(xi), yi).However, the number of training examples n is usually huge, so a vari-ant called stochastic gradient descent (SGD) is mostly applied in practice.Specifically, in SGD, the model updates its parameters using only a smallsubset (or even just one example) of the training data at each step, such asθt+1 = θt â Î·âÎ¸L(f(xi), yi), (xi, yi) â¼ Dtrain. (3.3)203.1. Warm-up with Supervised LearningAlthough each update in SGD is noisy and does not guarantee a decrease inthe total loss, each step is far more computationally efficient and the noisecan even help escape shallow local minima and plateaus in non-convex op-timization landscapes. Over time, averaging over many such noisy updatestypically leads to convergence to a good solution, especially when combinedwith techniques like:â¢ Learning rate schedules (e.g., warm-up or cosine scheduling in Loshchilovand Hutter 2016),â¢ Momentum of historical updates (e.g., Adam in Kingma 2014),â¢ Regularization (e.g., weight decay in Loshchilov and Hutter 2017).SGD is not just a practical algorithm; it also plays a central role in under-standing the learning dynamics of deep model training. Its behavior hasbeen linked to generalization performance (including in Ren et al. 2022 andalso in Chapter 4), implicit regularization (Barrett and Dherin 2020), andeven simplicity bias (in Ren and Sutherland 2024 and also in Chapter 7).3.1.3 Evaluation: Loss, Accuracy, and MoreNow, we can build our model and train it using SGD. At the same time,we want to check whether our model performs well or not, which needsevaluation from different perspectives. Evaluating deep models goes beyondjust checking their accuracy or training loss. Depending on our ultimateaims, i.e., whether we aim for performance, interpretability, robustness, orunderstanding learning dynamics, different evaluation techniques are used.In general, we wish to cover log-probability of the modelâs prediction, ex-pected calibration error, norm or other quantities of hidden embeddings,out-of-distribution probing accuracies, noise sensitivity, 2D-projection of fea-tures, representation similarity, etc. The theme is to provide a more thor-ough understanding of how the modelâs different parts gradually evolve bytracking important quantities during training. Compared with traditionallearning theory such as the PAC learning framework (Valiant 1984), whichusually evaluates the modelâs behavior when fully trained, we focus moreon the dynamical process during training using force analysis. The resultsfrom some classical training dynamics literature, e.g., those on SGD (Mandtet al. 2017; Arora et al. 2018; Xie et al. 2020), those on infinite-width ap-proximation like Neural Tangent Kernel (Jacot et al. 2018), and those onloss landscapes (Foret et al. 2020), are usually considered as the practical213.2. One-Step Influenceassumptions or lemmas during our force analysis.3.2 One-Step Influence3.2.1 The AKG DecompositionRecall that we want to track how the modelâs confidence in an observingexample xo changes after learning an updating example xu. To simplify theanalysis in the CE loss setting, we track the logarithmic probability change:ât(xo) â log pθt+1(y | xo)â log pθt(y | xo).To calculate this, we first approximate log pθt+1(y | xo) using a first-orderTaylor expansion. (We use pt to represent pθt and use p(x) or p(x) inter-changeably to represent p(y | x), for notational conciseness.) This giveslog pt+1(xo) = log pt(xo) + â¨â log pt(xo), θt+1 â Î¸tâ©+O(â¥Î¸t+1 â Î¸tâ¥2).Rearranging the terms and rewriting the inner product leads toât(xo) = âÎ¸ log pt(xo)|Î¸tï¸¸ ︷︷ ï¸¸VÃd(Î¸t+1 â θt)︸ ︷︷ ï¸¸dÃ1+O(â¥Î¸t+1 â Î¸tâ¥2),where d is the number of parameters of the model. If the model updates itsparameters using SGD with a single âupdating example,â we haveθt+1 â θt = âÎ·âÎ¸L(p(xu), yu).To evaluate the leading term, we plug in the definition of SGD and repeat-edly use the chain rule, recalling that p = Ï(z) and z = hÎ¸(x):ât(xo) = âÎ¸ log pt(xo)|Î¸tï¸¸ ︷︷ ï¸¸VÃd(Î¸t+1 â θt)︸ ︷︷ ï¸¸dÃ1= âÎ¸ log pt(xo)|Î¸tï¸¸ ︷︷ ï¸¸VÃd(â Î·âÎ¸L(xu, yu)|Î¸tï¸¸ ︷︷ ï¸¸1Ãd)T=(âz log pt(xo)|ztï¸¸ ︷︷ ï¸¸VÃVâÎ¸zt(xo)|Î¸tï¸¸ ︷︷ ï¸¸VÃd)(â Î·âÎ¸L(xu, yu)|Î¸tï¸¸ ︷︷ ï¸¸1Ãd)T= âz log pt(xo)|ztï¸¸ ︷︷ ï¸¸VÃVâÎ¸zt(xo)|Î¸tï¸¸ ︷︷ ï¸¸VÃd(âÎ·âzL(xu, yu)|ztï¸¸ ︷︷ ï¸¸1ÃVâÎ¸zt(xu)|Î¸tï¸¸ ︷︷ ï¸¸VÃd)T= âÎ·âz log pt(xo)|ztï¸¸ ︷︷ ï¸¸VÃV[âÎ¸zt(xo)|Î¸tï¸¸ ︷︷ ï¸¸VÃd(âÎ¸zt(xu)|Î¸t)Tï¸¸ ︷︷ ï¸¸dÃV] (âzL(xu, yu)|zt)Tï¸¸ ︷︷ ï¸¸VÃ1= âÎ·At(xo)Kt(xo, xu)Gt(xu, yu), (3.4)223.2. One-Step Influencewhich is the core AKG decomposition of our one-step influence function.For the higher-order term, O(â¥Î¸t+1 â Î¸tâ¥2), we know thatθt+1 â θt = âÎ· (âÎ¸zt(xu)|Î¸t)â¤ Gt(xu, yu).The residual term Gt, which is the norm of the gap between two probabili-ties, is obviously bounded. In practice, the gradient of z to θ is also usuallybounded, both because the training process does not diverge, and particu-larly when considering the widely applied practice of gradient clipping. Wehence haveO(â¥Î¸t+1 â Î¸tâ¥2) = O (Î·2â¥âÎ¸zt(xu)â¥2op â¥Gt(xu, yu)â¥2) ,which is typically O(η2) and can be neglected when η is small.The decomposition above holds under mild assumptions. For instance, thefunction f must be differentiable around θ so that the first-order Taylorexpansion exists, and the networkâs gradient should be bounded to ensurethat higher-order terms remain controlled. These are very common assump-tions applied in many existing works, and practitioners also design manymechanisms to stabilize the training, e.g., clipping gradients, smoothing theactivation function (using smoothed variants of ReLU), etc.Another concern is the role played by those higher-order terms in the Taylorseries. While higher-order terms can play an important role over the longrun, as discussed by Damian et al. (2022), we argue that in one-step orshort-horizon analyses, and assuming the learning rate is relatively small,the first-order component dominates the modelâs behavior. A more detailedexploration of higher-order effects is left to future work.3.2.2 Role of ARecalling Equation (3.4), we have At(xo) = âz log pt(xo), which only de-pends on the observing example xo. If we use {p1, . . . , pV } to represent themodelâs prediction on different dimensions, we can write At as:At(xo) = I â 1pâ¤Î¸t =ï£®ï£¯ï£¯ï£¯ï£°1â p1 âp2 · · · âpVâp1 1â p2 · · · âpV. . . . . .. . . . . .âp1 âp2 · · · 1â pVï£¹ï£ºï£ºï£ºï£» .233.2. One-Step InfluenceThis matrix is a centering operator based on pθt . For any length-V vectorb, we have Ab = (I â 1pâ¤Î¸t)b = bâ b¯1, where b¯ = pâ¤Î¸tb is the weighted sumof all elements in b.Imagine, in one time step, that the influence imposed by the term KG is thisvector b. Such an influence will then be normalized by A term dependingon how well this b aligns with current pt(xo): better alignment leads to alarger b¯, which means the knowledge provided by b is not surprising. Hence,the resulting influence is penalized by b¯ for every dimension and vice versa.In other words, A is self-stabilizing the modelâs confidence change based onits current prediction, and vice versa.3.2.3 Role of KWe then check Kt(xo, xu) = (âÎ¸z(xo)|Î¸t)(âÎ¸z(xu)|Î¸t)â¤, which is the productof the modelâs gradients with respect to xo and xu. It is also the empiricalneural tangent kernel (eNTK, Jacot et al. 2018) of the backbone network hθ(recall Figure 3.1, and we assume gÏ to be an identity function here). Mostof the consistent trends observed in this thesis implicitly rely on the followingassumption (although the equations above always hold even without it):During training, the relative influence of learning xuon other different xo is relatively stable (e.g., w.r.t. ranking).This is much less strict than the common âlazy eNTKâ assumption dis-cussed by Arora, Du, Wei Hu, Z. Li, Salakhutdinov, et al. (2019); that is asufficient but not necessary condition for this thesis. We will elaborate onthis assumption later under different problems.Intuitively, we can also interpret Kt as a model-specific similarity measure-ment between different input samples. The relative strength can be approx-imated by the Frobenius norm1 of this matrix: larger â¥Ktâ¥F means that theupdate of xu likely influences the modelâs prediction on xo more.3.2.4 Role of GFinally, we check the role of Gt(xu, yu) = âzL(xu, yu)|zt , which could beconsidered as the âgapâ term between the modelâs current prediction andthe supervisory signal on (xu, yu). Considering a popular cross-entropy loss1â¥Aâ¥F =ââmi=1ânj=1 |aij |2 =âTr(AAâ¤) is similar to the 2-norm of a vector.243.2. One-Step Influencein Equation (3.1), we haveâzL(xu, yu) = ââzeâ¤yu log Ï(z) = ââz log Ïi(z) = â1Ïi(z)âzÏi(z).For each dimension of this gradient, we have the standard result thatâÏiâzk={Ïi(1â Ïk) if i = kÏi(0â Ïk) if i Ì¸= k.Substituting this back leads to the vector form of Gt under CE-loss:Gtce(xu, yu) = pt(xu)â eyu , (3.5)which is a length-V vector. Using a similar method, we can calculate thegap term for the MSE-loss (Lmse = â¥p(xu)â eyuâ¥22) as:Gtmse(xu, yu) = 2câ (pt(xu)â eyu), (3.6)where â is element-wise multiplication and ci = Ïi(1â Ïi).We find that pt(xu) â eyu can be interpreted as the âforceâ imposed bylearning (xu, yu), which starts from the current prediction and points to aone-hot distribution, as demonstrated in Figure 3.2. Note that both thestrength and the direction of this term matter in our analysis.3.2.5 Demonstration using an MNIST ExampleNow we have the formal definition of the AKG decomposition. The MNISTclassification example provided in Figure 3.2 and 3.4 can help us understandthem better. As illustrated in the upper panel in Figure 3.2-(a), our modellearns a number â4â in this update. G is a vector starting from the currentprediction and pointing to the one-hot distribution. The red arrow (on thelabelâs dimension) is the pressure we intentionally imposed, and the orangearrows come from the fact that the probability mass must sum to one.Now, the learning dynamics provide us a way to do âforce analysisâ on thelog-probability of xo. The force originates from the G term and is thenprojected by the K term. Note that K measures the similarity betweenxu and xo from the modelâs gradient perspective. Perhaps surprisingly,we find that this similarity usually aligns with humansâ perception. Forexample, since a number â4â looks more similar to a number â9â than anumber â0â, then learning a number â4â influence p(y | xo = 9) more than253.3. Accumulated InfluenceLearn using SGD(ðð = ,ðð = 4)(a) Adaptation vector created by (ðð, ðð)(b) One-step change with the same ð¢ð¡(ðð, ðð) (large ð)Î log ðð¡(ðð) = âððð¡(ðð) ð¦ð¡(ðð, ðð) ð¢ð¡(ðð, ðð)ð² = ð ð¦|ðð =Identical:ððð¡(ðð)ððð¡+1(ðð)ð² = ð ð|ðð =Similar:ð² = ð ð¦|ðð¨ =Dissimilar:0 ... 4 ... 9ððð¡(ðð)eð¦=4Strong influence, ð ( , ð)( , ð)Weak influenceð¢ð¡(ðð, ðð)Big ð¦ð¡ð¹Small ð¦ð¡ð¹Figure 3.2: The per-step influence in an MNIST experiment. A large learn-ing rate is applied to make the difference more obvious.p(y | xo = 0). Figure 3.2-(b) verifies this relative strength by showingthe probability change of different examples after the same one-step updateof learning a number â4â. First, in the identical case, the 4th dimensionis pulled up, as expected. Then, the learning of this â4â pulls up the 4thdimension rather than the 9th dimension when a number â9â is given, whichintroduces an interesting pairing effect that is hard to interpret by analyzingthe global optimum. Last, since a number â0â is not similar to â4â, thisupdate almost does not influence p(y | xo = 0).3.3 Accumulated Influence3.3.1 Relative Stable K and Pairing EffectThe previous section demonstrates the one-step influence of learning a singleexample. We now turn to discuss the cumulative influence throughout thetraining process. The first thing to check is whether the relatively stableKt assumption holds. Hence in Figure 3.3, we directly visualize the averageâ¥Ktâ¥F throughout the training (we also use Ktuo to represent Kt(xo, xu) to263.3. Accumulated Influence02000400060000123456789yu is 105001000150020000123456789yu is 2050010001500200025000123456789yu is 301000200030000123456789yu is 405001000150020000123456789yu is 50100020000123456789yu is 60100020000123456789yu is 701000200030000123456789yu is 8010002000300040000123456789yu is 90 200 400 600 800 1000Training Epochs10 210 1100Train/test losstrain losstest loss0 200 400 600 800 1000Training Epochs050010001500Mean norm of eNTK0 200 400 600 800 1000Training Epochs0100200300Norm of the eNTK change0.20.40.60.81.0test accuracyFigure 3.3: First three rows: Results showing the relative stability of â¥Ktuoâ¥Ffor learning a specific number (title of each panel) on other numbers (labeledby the colorful digits near the lines). Last row: change of train/test loss,mean of eNTK, and Eu,o[â¥Ktuo âKtâ1uo â¥F ]. (Ktuo is just Kt(xo, xu) for nota-tion conciseness.)make the notation concise). The key findings are:1. The last row roughly indicates different phases throughout the train-ing: the first several epochs (0 â¼ 30) are a bit messy, and the lastseveral epochs (800 â¼ 1000) behave similarly to the finetuning stage;2. Although the norm of eNTK (Eu,o[â¥Ktuoâ¥F ]) and the norm of eNTKâsadaptation (Eu,o[â¥Ktuo âKtâ1uo â¥F ]) changes a lot after 30 epochs, theranking between â¥Ktuoâ¥F on different observing classes are relativelystable, as demonstrated by the upper nine panels;3. The pairing effect between the âsimilarâ inputs is clear, e.g., â4â andâ9â, â5â and â8â, etc;4. The pairing effect between the âdis-similarâ inputs are also clear, e.g.,â6â and â7â, â2â and â5â, etc.5. The pairing effect mentioned previously is not strictly symmetric,273.3. Accumulated Influence(a) Accumulated change of epochs (b) Correlation of the accumulated change4ï 9 7ï 95ï 89ï 45ï 39ï 73ï 5Figure 3.4: Accumulated influence (pairing effect) in an MNIST experiment.which is because of the inconsistent A and G terms.In summary, throughout the majority of the training process, especiallywhen the test accuracy is relatively high, the similarity between differentclasses tends to remain relatively stable. Hence, the pairing effect mentionedin one-step influence can be accumulated. As in Figure 3.4, if we keep train-ing the model, interesting pairing effects emerge in their predictions. Notethat this pattern gradually vanishes as the model approaches convergence.This supports our view that analyzing a modelâs dynamic behavior dur-ing training provides a valuable complement to traditional learning theory,which primarily focuses on asymptotic convergence.3.3.2 Evolution of G and Zig-zag Learning PathIn contrast to Kt, which usually changes slowly with different t, Gt typicallyundergoes significant changes during training. Fortunately, such changesare often predictable owing to their simplicity, i.e., they resemble the formpt(xu)âeyu . As a result, one can perform a straightforward âforce analysisâby examining its norm and direction over the course of training. We thenuse the following toy example to give a better demonstration of this process.Consider a multivariate Gaussian dataset (Toy-Gaussian for short), as il-lustrated in Figure 3.5. The dataset contains N samples, where each sampleis a 3-tuple (x, y,qâ). We can use the following three steps to get one:â¢ Sample label y: we first select the label y â [V ] following a uniform283.3. Accumulated Influenceðâ ð¦ ð¥ = [0.9, 0.1, 0]ðð¦ð = [ð, 0, 0]ðâ ð¦ ð¥ = [0.5, 0.3, 0.2 ]ðð¦ = [ð, 0, 0 ]ðâ ð¦ ð¥ = [0.1, 0.1, 0.8]ðð¦ = [ð, 0, 0 ]Easy:Medium:Hard (Wrong label):A plane.CIFAR 10 - LabelPlane? Ship?Plane???Figure 3.5: Left: a multi-variate Gaussian dataset with V = 3. Right: threetypical sample groups in the Toy-Gaussian and noisy-CIFAR-10 dataset(Krizhevsky, Hinton, et al. 2009).distribution over all V classes.â¢ Sample input x: we sample the input signal x|y=v â¼ N (µv, Ï2I), whereÏ is the noisy level for all the samples. µv is the mean vector for allthe samples in class v. Here, each µv is a 30-dim vector, in whicheach dimension is randomly selected from {âÎ´µ, 0, δµ}. This processis similar to selecting 30 different features for each class.â¢ Calculate ground truth Bayesian probability qâ(y | x): To do this,we use the fact that qâ(y | x) â q(x | y)p(y). As y follows anuniform distribution, we have qâ(y | x) = q(x|y=v)âj Ì¸=v q(x|y=j) . Followingq(x | y = v) â¼ N (µv, Ï2I), we find qâ(y | x) should have a Softmaxform. Specifically, we have:qâ(y = v | x) = esvâj Ì¸=v esj; si = â 12Ï2â¥xâ µiâ¥22.For a finer-grained understanding of the modelâs behavior, we roughly groupall the samples into the following three groups, based on a base difficultydefined as â¥qâ(x)â eyâ¥22 (please also refer to Figure 3.5):â¢ Easy: small base difficulty, where qâ is similar to ey (x locates close293.3. Accumulated Influenceto the cluster center), e.g., an image of a plane that is easy to classify;â¢ Medium: medium base difficulty, where the peak of qâ might stillalign with y (x locates near the boundary of different clusters), but isalready dissimilar to ey, e.g., an image which is hard to tell whetherit is a plane or a ship;â¢ Hard (e.g., wrong labeled): high base difficulty, where the peak of qâmight be different from y (x locates close to another cluster center),e.g., an image of a horse but labeled as a plane.With the preparations above, we can now do a very interesting experiment todemonstrate the evolution of Gt. Specifically, we randomly initialize a 4-layerMLP and train it on the Toy-Gaussian data using SGD (it converges roughlyat the 80th epoch). During the training, we track the modelâs confidencep(xo) on four examples with different difficulties. Since V = 3 and eachp(xo) is a 2-simplex, we can use a barycentric coordinate visualization.2Figure 3.6: Learning path of samples with different base difficulty. Cornerscorrespond to one-hot vectors. Colors represent training time: transparentat initialization, dark blue at the end of training.As demonstrated in Figure 3.6, the two easy samples very quickly move tothe correct location near ey (as indicated by the light color until reaching thecorner). The medium sample takes a less direct route, drifting off slightlytowards qâ, but still directly approaches ey. The hard sample, however, doessomething very different: it first approaches qâ, but then veers off towardsey, giving a âzig-zagâ path. In both the medium and hard cases, there seemsto be some âunknown forceâ dragging the learning path towards qâ.Where does this unknown force come from? Conducting a âforce analysisâfor each epoch helps us to answer this question. Specifically, we can approx-imate the one-step influence defined in Equation (3.4) as the accumulated2In this triangular plane, each corner is a one-hot distribution and the center is [ 13, 13, 13].303.3. Accumulated Influenceinfluence for one epoch, and then decompose it as follows.âep(xo) = âÎ·At(xo)ï£«ï£¬ï£¬ï£­Kt(xo, xo)Gt(xo, yo) + âxu Ì¸=xoxuâDtrainKt(xo, xu)Gt(xu, yu)ï£¶ï£·ï£·ï£¸(This is an approximation in that it neglects the change in A and K duringthe epoch.) In this decomposition, the forces imposed on xo come fromtwo sources: other examples and itself, as demonstrated in Figure 3.7-(a).For the influence from other examples, at any time t, âdissimilarâ sampleswill have small â¥Ktuoâ¥F , and hence only slightly affect p(xo). However,those âneighboringâ samples will have large â¥Ktuoâ¥F , and hence affect itsupdates much more; because qâ is hopefully similar for similar x values, itis reasonable to expect that the mean of ey for data points with similar xwill be close to qâ(xo). Thus, the net effect of updates for xu Ì¸= xo shouldbe to drag p(xo) towards qâ(xo). This is the âunknown forceâ we observed.Another force affecting p(xo) during training, i.e., the update based on(xo, yo), will drive p(xo) towards eyo , even though, in our hard example,it is quite far away from the ground truth qâ(xo).The dynamical interaction among these forces forms the âzig-zagâ learningpath we observed. To better see the interactions of these forces, we showthe learning path of a hard sample during training in Figure 3.7. In thefigure, we use blue lines to demonstrate the changes based on the updates ofother examples, while red lines show the influence of learning its label. Thepurple dot and cross marker show p(xo) at the start and end of the epoch.It is clear that at the beginning of the training, the influence of neighbor-ing examples dominates. That is because near initialization pt(xu) will berelatively flat. Hence, the size of the Gt for âsimilarâ xu and the Gt for xoshould be comparable, as demonstrated in Figure 3.7-(b). That is to say, ifthere are at least a few âsimilarâ training points, pt(xo) will move towardsqâ(xo), as demonstrated in the figure. Throughout training, some of thesesimilar samples will become well-classified, so that their Gt decay very fast,and their updates will no longer exert much force on pt(xo). Thus, the xoupdates begin to dominate, causing the zig-zag pattern as the learning pathturns towards eyo . For easy samples, where qâ and eyo are in the same di-rection, these forces agree and lead to fast convergence. On samples like theâmediumâ example, the two forces broadly agree early on, but the learningpath deviates slightly towards pâ en route to eyo .313.4. Summary: Force Analysis of DNNð¢ð¡ of neighoubrs:ð¢ð¡ of self-update:ð¡ = 0 ð¡ = 10ð¡ = 0ð¡ = 10(b) One-step update at different epoch (c) Detailed observation of ð(ðð) during training(a) Two group of forces in one epochUpdated by gradient of own labelPath on this epochInfluenced by gradient of other horsesðð©ð¥ðð§ð[1, 0, 0]ðð¡ð¨ð«ð¬ð[0, 1, 0]ð©0( )ð©1: [0.3, 0.6, 0.1]Figure 3.7: Demonstration of how the âzig-zagâ learning path emerges in ahard example: evolution of Gt is the key.3.4 Summary: Force Analysis of DNNIn this chapter, we formalize learning dynamics in a basic supervised setting.Leveraging the AKG decomposition, we show how a single update influencesmodel confidence on another example. Through MNIST and Toy-Gaussiancases, we visualize the stability of Kt and the evolution of Gt, which are thekeys to understanding cumulative learning effects. This process parallelsâforce analysisâ in Newtonian mechanics. Specifically, the âforceâ originatesfrom the G term. It is then normalized and projected by the A and K terms,and is ultimately imposed on log p(xo). In the following chapters, we willextend this analytical style to various settings to deepen our understandingof deep neural networksâ behavior.However, there are also important differences between the two settings. Forinstance, our analysis focuses solely on the first-order temporal behavior,without an equivalent notion of inertia or momentum.3 Moreover, the forcein our framework is inherently high-dimensional, in contrast to the three-dimensional space typically considered in classical mechanics.Extending this analogy by introducing additional physical concepts into ourframework, such as higher-order dynamics or energy-based interpretations,could be an interesting direction for future research.3Perhaps the âmomentumâ mechanisms in other optimizers like RMSProp (Hinton,Srivastava, et al. 2012) or Adam (Kingma 2014) are worth incorporating.32Chapter 4Understanding andImproving KnowledgeDistillationChapter 3 provides us with a powerful tool for understanding different phe-nomena during deep modelsâ training. Section 3.3 further demonstrates aninteresting âzig-zagâ learning path in a supervised classification task. Fromthese observations, a natural idea is to stop the training when the modelreaches the inflection point along this path, which is exactly the motivationof a related study of robust classification (S. Liu et al. 2020).4 With thehelp of early-stopping regularization, they design an efficient algorithm toprevent the model from memorizing the wrong labels, and hence behavesmore robustly when the labels are noisy.This chapter extends the problem to a more general setting. We arguethat even in relatively clean-label scenarios, the model is still capable ofautomatically identifying better supervisory signals, which can subsequentlybenefit the training of other models. Specifically, we first show that traininga network with a better supervisory signal, i.e., one closer to the unknownground-truth distribution qâ, leads to a tighter generalization bound underthe cross-entropy loss. Motivated by the analysis of learning dynamics inreal-world image classification tasks, we then propose a method called Filter-KD (Filtered Knowledge Distillation), in which a teacher network, trainedwith early stopping and an exponential moving average (EMA) mechanism,provides improved supervisory signals to a student network. Our empiricalresults not only validate the theoretical predictions of learning dynamics inpractical settings, but also demonstrate that Filter-KD is an effective andmodular tool that can be easily integrated into existing KD frameworks.4We did not find this paper, which is quite similar to our paper Ren et al. (2022), untilcompleting the latter; the paper contains a detailed discussion of the differences.334.1. Background and Related WorkThe discussions and results in this chapter are primarily based on our paperâBetter Supervisory Signals by Observing Learning Pathsâ (Ren et al. 2022).4.1 Background and Related Work4.1.1 One-hot Supervision is not OptimalIn multi-class classification problems, we usually supervise our model withâone-hotâ labels: the one-hot vectors ey which have the y-th dimensionequal to one, and all other dimensions equal zero. Over time, however, ithas gradually become clear that this âdefaultâ setup might not always be thebest choice in practice, in that other schemes can yield better performanceon held-out test sets. One such alternative is to summarize a distributionof human annotations, as Peterson et al. (2019) did for CIFAR10. However,this method requires a lot of human effort, which is hard to scale to otherlarger datasets. Another alternative approach is label smoothing, in whicha mixture of a one-hot vector and the uniform distribution is used to trainthe model (Szegedy et al. 2016). However, this approach manipulates theone-hot labels for all examples (containing both easy and hard ones) in thedataset in the same way, which is also not optimal.4.1.2 Teacher as Better Supervision in KDKnowledge distillation (KD), first training a teacher network on the train-ing set and then a student network on the teacherâs output probabilities wasoriginally proposed for model compression (Hinton et al. 2015). However,we can also understand it as refining the supervision signal: the teacherprovides âsoftâ and input-dependent outputs rather than the hard one-hotlabels to the student. Knowledge distillation is promising because it re-quires no additional annotation effort, and at the same time, can providesample-specific labels based on their difficulties. Perhaps surprisingly, KDcan improve student performance even when the teacher is of exactly thesame form as the student and trained on the same data; this is known asself-distillation (Furlanello et al. 2018; L. Zhang et al. 2019). There havebeen many recent attempts to explain knowledge distillation and specifi-cally self-distillation (e.g. Allen-Zhu and Yuanzhi Li 2020; Tang et al. 2020;Menon et al. 2021), from both optimization and supervision perspectives.We focus on the latter area, where it is usually claimed that the teacherprovides useful âdark knowledgeâ to the student through their labels. Fromthe discussions in Section 3.3, we speculate that such dark knowledge comes344.2. Insights From Learning Dynamicsfrom the mutual influence among training samples, which is well capturedby our learning dynamics analysis framework.4.2 Insights From Learning Dynamics4.2.1 Better Supervisory, Better GeneralizationFrom the student networkâs perspective, the only difference among self-knowledge distillation (self-KD), label smoothing, and standard supervisedlearning lies in the supervisory signal it receives. Consistent with findingsfrom prior work (as well as our own experiments), self-KD typically outper-forms label smoothing, which in turn outperforms standard training. Thisperformance hierarchy suggests that the supervisory signals provided by ateacher network are of higher quality than those offered by label smoothingor one-hot labels. This naturally raises an important question: how can wequantitatively compare the quality of different supervisory signals?To answer this question, we first recap the general form of cross-entropy loss,i.e., âqâ¤ log p, and the most commonly used form restricts q to be a one-hotvector e. If we switch our supervisory signal to another target distributionqtar, the corresponding risk is thenRtar(f,D) â ânâi=1Vâv=11nqtar(yi = v | xi) log p(yi = v | xi)= â 1nnâi=1qTtar(xi) log p(xi) (4.1)With this definition, we propose the following hypothesis as a trend.Hypothesis 1. Suppose we train a model supervised by qtar, that is, weminimize Rtar(f,Dtrain). Then, smaller average L2 distance between qtarand the ground truth qâ on these samples, i.e. small Ex [â¥qtar(x)â qâ(x)â¥2],will in general lead to better generalization performance.This hypothesis is suggested by Proposition 3 of Menon et al. 2021, whichshows that for any predictor f and loss bounded as L(f(x), yË) â¤ â, we haveED[(Rtar(f,D)âR(f))2] â¤ 1nVarx[qTtarL(f(x))]+â2K(Exâ¥qtar(x)â qâ(x)â¥2)2,where L(f(x)) is the loss for each possible label (it is just â log p(x) in ournegative log-likelihood, i.e., NLL, setting) and K is also a constant. When354.2. Insights From Learning DynamicsFigure 4.1: Normalized (divided byâ2) distance between output distribu-tion p and ground truth qâ during the one-hot training in different stages(left to right: initial, early stop, convergence). The bar plots show the den-sity along the corresponding axis. The code for this experiment can be foundin the notebook, and the animation of this figure is in the git repo.n is large, the second term will dominate the right-hand side, implying asmaller average â¥qtarâqââ¥ will lead to Rtar being a better approximation ofthe true risk R; minimizing it should then lead to a better learned model.This suggests that the quality of the supervision signal can be roughly mea-sured by its L2-distance to the ground truth qâ. Appendix A.1 slightlygeneralizes the result of Menon et al. (2021) with bounds based on totalvariation (L1-distance) and KL divergences; we focus on the L2 version herefor simplicity. From Hypothesis 1, we can now treat â¥qtar(x)â qâ(x)â¥2 as aelement-wise measurement of the supervision quality.4.2.2 Good Supervision Emerges during TrainingSince the qâ is usually intractable, we start with the Toy-Gaussian datasetstudied in Section 3.3, since their qâ are well defined and easy to calcu-late. On real datasets, we consider CIFAR-10H (Peterson et al. 2019), inwhich each image is paired with a group of labels provided by humans. Thecorresponding empirical distribution is then a good approximation of qâ(x).Recall the examples demonstrated in Figure 3.6, where the modelâs learningpath on hard examples first moves toward the unknown ground-truth qâ andthen converges to a one-hot distribution. Although the four examples in thisfigure hint that early stopping the modelâs training can help the model finda good prediction, is this still true for most of the examples? (It is possiblethat the good stopping points for different examples differ greatly.) Toverify this, we use the Toy-Gaussian dataset and train a 4-layer MLP to364.2. Insights From Learning Dynamicsconvergence with n = 5000 training samples. In Figure 4.1, we show thescatter plots of each sampleâs base difficulty (how close its qâ is to its ey)versus the modelâs prediction quality (how close the modelâs prediction p isto its qâ), in three different stages during training. At initialization, mostpoints5 have large â¥p(x) â qâ(x)â¥2. By the point of early stopping, mostp values are roughly near qâ. At convergence, however, p(x) â ey, as theclassifier has nearly memorized the training inputs, leading to a diagonalline in the plot. Note that the average label quality (Ex â¥ptar(x)â qâ(x)â¥2)mentioned in Hypothesis 1 is the mean height of all points in the figure, ifwe treat p as ptar. It is clear that the early-stopped model has the potentialto provide better supervisory signals, i.e., better ptar.This trend also persists in more realistic systems, albeit with increased noisein their training paths. Specifically, we randomly initialize a ResNet18 (K.He et al. 2016a) and train it on CIFAR-10 until its convergence. Since nowwe have 10 classes, we first convert a length-10 vector to a length-3 one byretaining the probabilities corresponding to the correct class and the second-highest predicted class, while aggregating the probabilities of the remainingclasses into a single value. The learning path can then be projected onto thesame 2-simplex triangle plane, as demonstrated in Figure 4.2. The panelsin the first column show the learning path of an easy sample.6 We can seethat pt converges quickly towards the left corner, the one-hot distributionfor the correct class, because the color of the scattered points in that figureis quite light. At the early stopping epoch, pt has already converged to ey.However, for a hard sample, it is very difficult to observe any patterns fromthe raw path (blue points): there are points almost everywhere in this plot.This is likely caused by the more complex network and dataset, as well as ahigh learning rate in early training. To find the hidden pattern in this high-variance path, we treat pt as a time series signal with many high-frequencycomponents. Thus, we can collect its low-frequency components via a low-pass filter and then draw that, i.e., taking an exponential moving average(EMA) on pt, using the following equation:pt+1ema = (1â α)ptema + αpt, (4.2)where α controls the strength of the smoothing. In digital signal processing,the equation above is equivalent to a first-order low-pass filter where α5The curve structure is expected: points with qâ â ( 13, 13, 13) are near the middle of thebase difficulty range, and all points are initialized with fairly ambiguous predictions p.6Without knowing qâ, we instead use the âzig-zag scoreâ to roughly measure the diffi-culty. Please also refer to Appendix E of our paper for more details (Ren et al. 2022).374.2. Insights From Learning Dynamicscontrols the cut-off frequency (Smith 2003).Figure 4.2: Learning paths on different problems. In the first two columns,we record pt for each batch, while in the last column, we record it for eachepoch. The first row is training a ResNet18 on CIFAR-10, the second rowis ResNet18 on CIFAR-100, and the last row is LSTM on AGNews.After smoothing the noisy trajectory, we find ptema (the red dots) exhibits asimilar trend: first moving towards the unknown true label before eventuallyturning to memorize the wrong label. Such a trend also exists in many morecomplicated systems, e.g., training a ResNet18 on CIFAR-100 or trainingan LSTM (Hochreiter and Schmidhuber 1997b) on AGNews (Xiang Zhanget al. 2015), as demonstrated in the later rows of this figure.384.3. Proposed Method: Filter-KD4.3 Proposed Method: Filter-KD4.3.1 The AlgorithmInspired by the findings above, we hence propose a knowledge distillation-based method called Filter-KD. The idea is quite simple: since better super-vision leads to better generalization ability (as in Hypothesis 1) and earlystopping the training of one network can provide better ptar, we can firsttrain a teacher network and then use the filtered ptâ²ema as the supervisorysignal for the studentâs training, as demonstrated in Algorithm 1. Specifi-cally, we maintain a look-up table qema â RNÃV to store a moving averageof pt for each training sample. Note that in one epoch, each qema(xn) will beupdated only once.7 We check the early stopping criterion with the help ofa validation set. Afterwards, the teaching supervision qema is ready, and wecan train a student network under its supervision. This corresponds to usinga moving average of the teacher model âin function space,â i.e., averagingthe outputs of the function over time.Note that this framework is easy to incorporate into any other KD methods,as long as the teacher and student have identical output formats. Sucha distillation procedure can also be conducted for multiple generations tofurther improve the modelâs performance, as was done in the âBorn AgainNetworks (BAN)â studied by Furlanello et al. (2018). We analyze a similarsystem in our paper Ren, Lavoie, et al. (2023), which also uses the generalprinciple of learning dynamics discussed in this thesis. For brevity, we onlyfocus on one generation of this process here.4.3.2 Evaluation of Filter-KDIn this part, we consider two main baselines. Other than the standardsupervised learning using one-hot examples (OHT for short in the tables), wefurther compare an important baseline called ESKD (early-stopped KD, i.e.,α = 1 using the same algorithm), which can be considered as an ablation forthe filtering mechanism applied in Filter-KD. Compared to ESKD, Filter-KD can avoid the extremely high variance of pt during training. That isbecause most practical implementations only consider early-stopping at theend of each epoch, which is equivalent to down-sampling the noisy learningpath (as in the last panel of Figure 4.2), further exacerbating the variance of7As suggested by the last column of Figure 4.2, updating the prediction epoch-wise isenough to get a good estimation.394.3. Proposed Method: Filter-KDInput: Dataset {(xn, yn)}Ni=1, two randomly initialized DNNs# Train the teacherInitialize the teacher fteacher, initialize an N Ã V matrix called qemaGo through the entire dataset, calculate qema[n, :] = fteacher(xn)for epoch = 1, 2, ..., T dofor i â {1, 2, . . . , N} in random order dopË = fteacher(xn)qema[n, :] = (1â α) · qema[n, :] + α · pËUpdate parameters of fteacher based on Lce(pË, yn),end forCheck the early stopping criterion; stop training if satisfiedend for# Train the studentFor each input xn, set ptar = qema[n, :]Train the network fstudent under the supervision of ptarAlgorithm 1: Filter-KD. α controls the cut-off frequency of the low-pass filter (0.05 in the experiments here).pt. Thus, ESKD will likely select a bad ptar for many data points. Filter-KD,by contrast, has much more stable predictions.However, such enhanced robustness suffers from higher algorithmic complex-ity: the running time of Filter-KD might be slightly increased. Furthermore,compared to the teaching model in ESKD, the Filter-KD requires a teachingtable qema â RNÃV , which requires substantial memory when the dataset islarge. One alternative avoiding the need for this table would be to insteadtake an average âin parameter space,â like e.g. momentum parameter up-dating as in Szegedy et al. (2015). But we empirically find that, althoughthis helps the model converge faster, it does not lead to a better teachernetwork. Thus, although Filter-KD has clear drawbacks, we hope that ourexplanations here may lead to better practical algorithms in the future.We now compare the performance of Filter-KD and other baselines on a realdataset. To rule out the influence of other hyperparameters, we first focus onself-KD and a fixed temperature Ï = 1, as we want the only difference amongthese methods to be ptar. Thus, we can conclude that the improvement weachieved comes purely from the refined supervision. More results consideringdifferent hyperparameters can be found in our paper (Ren et al. 2022).404.3. Proposed Method: Filter-KDAccuracy â ECE âOHT KD ESKD FilterKD OHT KD ESKD FilterKDCIFAR10 95.34 95.39 95.42 95.63 0.026 0.027 0.027 0.007CIFAR100 78.07 78.40 78.83 80.09 0.053 0.061 0.067 0.029Table 4.1: Quantitative comparison of generalization performance forResNet18 self-distillation (mean value of 5 runs, bold numbers are thebest method). ECE is the expected calibration error (C. Guo et al. 2017),which measures how well the modelâs prediction quality. Please also refer toAppendix B.1 for the formal definition.Table 4.1 clearly shows that Filter-KD performs best in both accuracy andECE. The two ablations (i.e., KD and ESKD) suggest that both the earlystopping and smoothing mechanisms in Filter-KD are important.4.3.3 Auto Noisy-label Fixing AbilityWe then consider the noisy-label classification problem, where we train andvalidate the model on a dataset with some labels randomly flipped. Aftertraining, all the models are evaluated on a clean held-out test set. Theexperiments are conducted on CIFAR (Figure 4.3) and TinyImageNet (Ta-ble 4.2), under different noise ratios Ï: Ï = 0.1 means 10% of the labelsare flipped. In Figure 4.3, an interesting trend can be observed: the en-hancement brought by Filter-KD is not significant when Ï is too small ortoo large. That is reasonable because for small Ï, few samples have badlabels, thus the possible enhancement might not be large. When Ï is toohigh, the labels of similar points become less reliable, and the learning pathwill no longer head as reliably towards qâ. Thus, for very high noise ratios,the performance of Filter-KD decays back towards that of OHT.Noise Ï 0 0.05 0.1 0.2OHT 56.95 53.02 52.02 30.52ESKD 58.61 53.53 52.99 36.55FilterKD 59.32 56.43 55.51 40.81Table 4.2: Noisy-label learning results on TinyImageNet dataset.414.4. Conclusion and Future WorksFigure 4.3: Test accuracy under different noise ratio Ï (this notation is notsoftmax function here). Solid lines are the means, while shaded regions arethe standard errors for 3 runs with different random seeds (shaded range isthe standard error). The last panel compares the influence of different tem-peratures. Each thin rectangle plot represents a different Ï = {0, 0.1, ..., 0.8},in which we plot the results with different Ï = {0.5, 1, 2, 4, 10}.4.4 Conclusion and Future WorksIn this chapter, we first claim that a better supervision signal, i.e., thoseptar that are closer to qâ, leads to better generalization performance; thisis supported by results of Menon et al. (2021) and further empirical resultsgiven here. To further understand how such better supervision emerges, wedirectly observe the behavior of samples with different difficulties by pro-jecting them on a 2D plane. The observed zig-zag pattern is well-explainedby considering the tradeoff between two forces, one pushing the predictionto be near that of similar inputs, the other pushing the prediction towardits training label. Such a phenomenon can be well explained by the learn-ing dynamics framework we discussed in Chapter 3: as long as we use thesample-wise exponential moving average (EMA, which is equivalent to alow-pass filter well studied in signal processing) to extract the dominatingcomponent in their messy learning paths.To apply these findings to real tasks, in which the data and network aremore complex, we propose a practical method called Filter-KD to furtherenhance ptar. Experimental results on various settings (datasets, networkstructures, label noise, and temperatures) not only show the advantage ofthe proposed method as a method for knowledge distillation but also helpverify our learning dynamics framework.However, the proposed Filter-KD method incurs additional computationaland storage costs compared to other baseline approaches. Moreover, the ex-424.4. Conclusion and Future Worksperiments presented in this chapter are limited to the self-distillation setting.Exploring how Filter-KD can be integrated with more advanced knowledgedistillation frameworks remains an interesting direction for future work.Regarding the framework of learning dynamics, most of the theoretical re-sults and experiments in this chapter can be viewed as a validation of theproposed phenomena in a more practical image classification setting. Theprinciples highlighted in our framework are further corroborated by recentindependent research, including Datta et al. (2023) and Ravikumar et al.(2025), which explore similar concepts in diverse fields. This convergence offindings underscores the generalizability of our approach to a wide rangeof related tasks. The next chapter extends this framework to the fine-tuning of large language models (LLMs), where the problem setup is sub-stantially more complex than the standard supervised classification tasksstudied here. Remarkably, under reasonable simplifications and mild as-sumptions, we show that learning dynamics remains a powerful analyticaltool even in such large-scale systems. This finding highlights the potential ofâforce analysisâ as a promising perspective for gaining deeper insights intothe behavior of modern deep learning models.43Chapter 5Understanding LLMFinetuningChapter 3 introduces the analytical framework of learning dynamics, wherewe leverage the AKG decomposition to examine fine-grained model behav-iors at the instance level, clearly identifying and elaborating the roles of itsconstituent terms. Specifically, through the pairing effect illustrated in theMNIST example, we demonstrate that the term Kt(xo, xu), which measuresthe similarity between xo and xu from the modelâs gradient space, remainsrelatively stable throughout training. Using the âzig-zagâ learning path ex-ample, we show how the Gt term, which provides the energy and directionfor the modelâs confidence change, gradually evolves over time. Insightsgained from these âforce analysisâ demonstrations subsequently inspire thedevelopment of the Filter-KD method proposed and examined in Chapter 4.In this chapter, we extend our framework to the finetuning of large languagemodels (LLMs), a rapidly evolving area involving increasingly large andcomplicated deep neural networks. We illustrate how the âforce analysisâperspective offers a unified approach for interpreting model behaviors acrossa diverse array of algorithms and tasks. Specifically, we explore variousfinetuning scenarios for LLMs, covering (though not exhaustively):â¢ Topics: alignment, hallucination, unlearning, self-bias amplification,reasoning, gradient ascent, and token-wise modeling.â¢ Algorithms: Supervised Finetuning (SFT), Direct Preference Opti-mization (DPO, Rafailov et al. 2023) and its variants, and reinforce-ment learning-based methods like Group Relative Policy Optimization(GRPO, D. Guo et al. 2025).We anticipate this list will expand over time, and hope that the analyticaltools inspired by learning dynamics will continue to evolve, addressing ad-ditional key challenges of interest to the broader LLM research community.44Chapter 5. Understanding LLM FinetuningThe structure of this chapter is organized as follows. We first put our anal-ysis into context by highlighting the remarkable success achieved by variousfinetuning methods within the overall pipeline of deploying LLMs. Simul-taneously, we summarize certain unexpected behaviors identified by otherresearchers. We hope to explain these phenomena through the perspectiveof learning dynamics, complementary to existing approaches that primarilyanalyze convergence behavior or global optima.However, extending the fundamental framework proposed in Chapter 3 toLLM finetuning is not straightforward. Depending on the specific problemof interest, several challenges emerge, such as autoregressive modeling, ex-tremely large response spaces, negative gradient occurrences, issues withnon-unique correct labels, and token-wise modeling complexities. Thesechallenges are elaborated upon in subsections beginning with âTheory.âIn subsections beginning with âExplanation,â we demonstrate how our an-alytical framework sheds new light on critical topics that have puzzled theLLM community. Although the frameworkâs application is still in its prelim-inary stage, with experimental validation limited to medium-sized datasets,e.g., Anthropic-HH (Y. Bai et al. 2022), UltraFeedback (G. Cui et al. 2023),reasoning datasets at the MATH level (Hendrycks et al. 2021), and modelsranging from 0.5B to 3B parameters, the observed behaviors consistentlyalign with our theoretical insights. We anticipate that this analysis willprovide valuable perspectives to the community, potentially informing thedesign of more effective and practical algorithms in larger systems.Finally, as an initial exploration of practical implications, we introduce twonovel algorithms addressing tasks related to alignment and mathematicalreasoning, detailed in subsections beginning with âAlgorithm.âThe core content of this chapter primarily draws from the following papers:â¢ âLearning Dynamics of LLM Finetuningâ, our ICLR-2025 paper, pro-vides most of the theoretical analyses and experiments related to SFTand DPO;â¢ âOn the Effect of Negative Gradient in Group Relative Deep Reinforce-ment Optimizationâ, our NeurIPS-2025 paper, analyzes reinforcementlearning-based finetuning approaches for LLMs like GRPO;â¢ âBias Amplification in Language Model Evolution: An Iterated Learn-ing Perspectiveâ, our NeurIPS-2024 paper, discusses the risks associ-ated with self-bias amplification during multi-generation training.455.1. Introduction and Background5.1 Introduction and Background5.1.1 Finetuning LLM Unlocks PotentialThe finetuning of LLMs has become a cornerstone in deploying pretrainedgeneral-purpose language models to specialized tasks, significantly boostingtheir performance and practical applicability (Tie et al. 2025). Initially,standard supervised instruction-finetuning (i.e., SFT) dominated, whereintask-specific labeled datasets were directly used to optimize models throughnegative log-likelihood (NLL) loss. This straightforward method has beenextensively utilized across tasks like sentiment analysis, named entity recog-nition, text classification, question answering, etc.However, the increasing demand for models aligned with human values hasshifted researchersâ attention toward more flexible reinforcement learning-based strategies, notably reinforcement learning from human feedback (RLHF,Y. Bai et al. 2022) using Proximal Policy Optimization (PPO, Schulman etal. 2017). RLHF leverages human preferences as reward signals to iterativelyguide model optimization, hence improving alignment in open-ended taskssuch as dialogue completion, summarization, and code generation.Introduced by OpenAI (Ouyang et al. 2022), current finetuning pipelinestypically consist of three stages: initial SFT to gather instruction-followingability, training a reward model based on human feedback, and subsequentlyapplying RL optimization using PPO. This procedure can be broadly cate-gorized into two phases: instruction tuning and preference tuning.Subsequently, numerous studies have emerged to refine this pipeline. Forinstance, to avoid explicitly training a separate reward model, Rafailovet al. (2023) observed that a finetuned LLM inherently encodes a rewardmodel. Motivated by this insight, they propose Direct Preference Opti-mization (DPO), a streamlined approach where the model directly learnsto distinguish between human-annotated preferred and rejected responses,thus bypassing complexities intrinsic to traditional RL setups. They alsoprovided theoretical justification, demonstrating that under mild assump-tions, DPO is equivalent to PPO. This approach inspired various extensions,including KTO (Ethayarajh et al. 2024), IPO (Azar et al. 2024), SLiC (YaoZhao et al. 2023), SimPO (Meng et al. 2024), etc., also listed in Figure 1.1in Chapter 1. However, similar to the original DPO, these variants relyon pre-collected datasets, leading to off-policy training, thus deviating fromPPOâs original on-policy requirement. Consequently, on-policy variants suchas RLAIF (H. Lee et al. 2023) and OAIF (S. Guo, B. Zhang, et al. 2024)465.1. Introduction and Backgroundhave attracted increasing attention. Nonetheless, these methods introducenew challenges, such as ensuring unbiased real-time evaluations, managingadditional computational costs for verification, and reconsidering whether aseparate reward model might be needed.In response to these issues, reasoning tasks have gained more attentionwithin the research community recently. This has been partly because rea-soning represents a fundamental capability for artificial intelligence, but alsobecause the related tasks typically offer clearly verifiable reward signals (e.g.,correctness of mathematical or programming solutions). This setting, re-ferred to as Reinforcement Learning with Verifiable Rewards (RLVR, YangYue et al. 2025), simplifies reward design and facilitates straightforwardbenchmarking across methods. While traditional PPO naturally fits intoRLVR, the authors of DeepSeek-R1 demonstrated substantial improvementsin reasoning performance using a simple yet effective method, Group Rel-ative Policy Optimization (GRPO, Shao et al. 2024; D. Guo et al. 2025),where the supervisory signal consists solely of correctness feedback on model-generated responses. Subsequently, various GRPO variants have been pro-posed to address limitations of the original approach.Together, these developments have significantly enhanced the flexibility andalignment of LLMs, making them increasingly effective for diverse and so-phisticated applications. In this thesis, we demonstrate that despite theproliferation of algorithms inspired by different theories, our âforce analy-sisâ enabled by learning dynamics can provide a unified framework to under-stand them, as their loss functions all involve gradients with respect to thelog-probability of response sequences. Consequently, we hope our frameworkcan then provide valuable insights (possibly) on why these methods work.5.1.2 Finetuning LLMs Comes with InstabilityWhile LLM finetuning is crucial, it is far from a guaranteed improvement.In fact, when done improperly, finetuning can introduce unexpected behav-iors that degrade performance, reduce robustness, and undermine alignment.One major concern is catastrophic forgetting (McCloskey and Cohen 1989;Ouyang et al. 2022; H. Li et al. 2024; C. Liu et al. 2024), where previouslyacquired general abilities, such as code generation, factual question answer-ing, or basic arithmetic, diminish as the model becomes overspecialized.Such issues are particularly evident in SFT that do not use representativedatasets or that aggressively minimize loss on narrow objectives.475.1. Introduction and BackgroundAnother subtle but damaging consequence is overfitting to spurious patternsin the finetuning dataset. For example, if preference data is biased, say, hu-man annotators consistently rate longer or more polite responses as better,the model may learn to associate verbosity or hedging with correctness, re-gardless of actual answer quality (Saito et al. 2023). This behavior has beenempirically observed in instruction-tuned models like Alpaca (Taori et al.2023) or early RLHF models, where responses became longer but not morehelpful. Relatedly, when reward models are trained on such data, they mayinadvertently reinforce patterns that emphasize fluency or style over sub-stance, which introduces the risk of self-bias amplification discussed by Renet al. (2024). This often leads to hallucination amplification: models con-fidently generate fabricated facts, papers, or statistics with seemingly highplausibility, a behavior worsened by reward hacking in preference optimiza-tion methods like PPO or DPO.Mode collapse is another failure mode common in improperly tuned sys-tems (Holtzman et al. 2020). Here, the modelâs responses become repet-itive, template-like, or even non-grammatical. For instance, models mayrepeatedly respond with boilerplate disclaimers (e.g., âAs an AI developedby OpenAI, ...â) or output nearly degenerate sequences (e.g., âThe capitalof China is the capital of China is the capital ...â). This issue has been docu-mented in several open-source preference optimization methods (e.g., KTO,SimPO), especially when training is too aggressive or data lacks diversity.Furthermore, a growing body of recent work has revealed several counter-intuitive behaviors during LLMsâ finetuning. For instance, studies such asW. Deng, Ren, M. Li, et al. (2025) and Razin et al. (2025) highlight casesof unintentional unalignment arising in both DPO and GRPO frameworks,where the model drifts away from intended human-aligned behavior despitetraining on the correct dataset. In addition, Rafailov et al. (2024) report asurprising phenomenon in DPO: although the model increasingly improvesits ability to distinguish between preferred and rejected responses, the ab-solute quality of both types of responses may degrade during training.While such behaviors may not directly impair the modelâs final performance,they introduce instability and unpredictability in the modelâs outputs, pos-ing risks for real-world deployment. We argue that these seemingly paradox-ical behaviors reflect a lack of control over the underlying learning dynamics,and thus warrant a deeper, more principled explanationâan objective thisthesis aims to address through the lens of our proposed framework.In summary, the examples above highlight a crucial insight: while finetuning485.2. Learning Dynamics of SFTis a powerful tool for LLM adaptation and alignment, we still lack enoughunderstanding of it. Small shifts in data composition, reward shaping, oroptimization objectives can drastically alter behavior in ways that are diffi-cult to anticipate or detect via standard metrics. As a result, different frommost of the current works on explaining or mitigating these phenomena ata dataset-level by analyzing LLMsâ behavior at convergence, we instead fo-cus on a fine-grained observation of the system. Because we believe theunexpected behaviors are sometimes only driven by some key examples orlearning steps, which is exactly what the learning dynamics focuses on. Con-sequently, we hope our framework can then provide valuable insights on why(and when) these methods might fail.5.2 Learning Dynamics of SFTWe start with the most fundamental form of LLM finetuning: SFT. Thisserves as a foundation for our analysis, as we will later demonstrate thatmany more complex finetuning methods can, from a gradient perspective,be reduced to variations of SFT. The primary distinctions among thesemethods often lie in (i) the origin of training samples, (ii) how the algo-rithm dynamically adjusts the equivalent learning rate (eLR) for differentexamples, and (iii) whether the eLRs are allowed to be negative, or (iv)whether eLRs are defined at a token-wise level.Additionally, in this chapter, we adopt a notation change for consistencywith conventions in the reinforcement learning and LLM finetuning litera-ture. Specifically, we denote the modelâs prediction as Ï(y | x) instead ofp(y | x), and use bold symbols x and y to emphasize the multi-token natureof LLM prompts and responses.5.2.1 Theory: Tackling the Auto-regressive ProblemWe now extend our analytical framework to LLMâs finetuning. Recall ourcore AKG decomposition demonstrated in Equation (3.4), which was orig-inally formulated under the single-label prediction setting. However, theLLMâs response usually contains multiple tokens, which resemble a multi-label problem. We hence begin by extending the framework to a multi-labelsetting, for example, predicting both the color and identity of a digit inthe Colored-MNIST dataset. In this setting, the predictions associated withdifferent labels can be modeled as being computed independently, even ifthey share a common backbone network. As a result, we can compute ât495.2. Learning Dynamics of SFTfor each label separately and then stack them together. Specifically, we de-fine ât(xo) â RVÃL, where L is the number of labels. Each column of thismatrix is then calculated as:[ât(xo)ï¸¸ ︷︷ ï¸¸VÃL]l = âÎ·[At(xo)ï¸¸ ︷︷ ï¸¸VÃVÃL]lKt(xo, xu)ï¸¸ ︷︷ ï¸¸VÃV[Gt(xu, yu)ï¸¸ ︷︷ ï¸¸VÃL]l. (5.1)The main difference between this one and Equation (3.4) is the shape ofthese terms; Kt is shared since a shared backbone hθ is assumed.We now examine the loss function. Since the typical loss function usedduring SFT is the negative log-likelihood (NLL) of a given completion yu =[y1, . . . , yL] â VL conditioned on the prompt xu, then we haveLSFT(xu, yu) â âLâl=1log Ï(y = yl | xu, y<l)= âLâl=1eâ¤yl log Ï(y | xu, y<l). (5.2)Note that compared with the multi-label classification problem discussedbefore, where the joint distribution of all labels can be factorized as Ï(y |x) =âl Ï(yl | x), the LLMâs prediction should be Ï(y | x) =âl Ï(yl | x, y<l),which will make the AKG decomposition very complicated.Fortunately, due to the âteacher forcingâ mechanism8 and âcausal maskâused in most off-policy LLM finetuning implementations, we can merge thisfactorization into the definition of the backbone hθ and still keep the formatof Equation (5.1). This is illustrated by Figure 5.1. Using Ï to denote theconcatenation of x and y, the prediction of all tokens of y isÏ (y | Ï) = Softmax column (z) ; z = hθ (Ï) .Here z is a V Ã L matrix where each column contains the logits of theprediction of the l-th token. Our hθ, even though it takes the entire se-quence Ï as its input, will force the model not to refer to the future tokensyâ¥l when making predictions on the l-th token, commonly implementedvia âcausal maskingâ as illustrated in the figure. Then, we can calculate8This implies that in most off-policy finetuning settings, the model generates predic-tions conditioned on y<l as provided in the dataset, rather than on tokens generated bythe model itself, which is different from inference time.505.2. Learning Dynamics of SFTðð1 2 31 2 3 4LLMPredictionsSupervisionNLL-Lossðððððð â (ðð, ðð)Causal mask with prefixLLMMulti-label CE-Loss ðð(ðð|ð, ðð:ðâð) ðð(ðð|ð, ðð:ð³;ð¦ðð¬ð¤)Figure 5.1: The teacher forcing mechanism and the causal mask.(âÎ¸zl(Ïo)|Î¸t)(âÎ¸zl(Ïu)|Î¸t)â¤ on each column of z and stack them to form aV ÃV ÃM ÃL tensor Kt(Ïo,Ïu). The calculation of Gt and At also followsa similar procedure. After some calculations, the resulting decomposition isalmost identical to that in a multi-label classification problem. Assume wehave a response yu of length L associated with xu, stacked into Ïu, and yoof length M associated with xo, stacked into Ïo. The change of the modelâsprediction on the m-th token of yo can be represented as (when gradientsover z have bounded norm):[ât(Ïo)ï¸¸ ︷︷ ï¸¸VÃM]m = âLâl=1Î·[At(Ïo)ï¸¸ ︷︷ ï¸¸VÃVÃM]m[Kt(Ïo,Ïu)ï¸¸ ︷︷ ï¸¸VÃVÃMÃL]m,l[Gt(Ïu)ï¸¸ ︷︷ ï¸¸VÃL]l +O(η2), (5.3)where GtSFT (Ïu) = ÏÎ¸t(y | Ïu) â yu. The calculation of this term can befound in Appendix A.2.Fortunately, our GtSFT is also a tensor starting from the modelâs predictionand pointing to the one-hot supervisions. Then, a similar âforce analysisâapproach can still be applied. Compared with the decomposition in Chap-ter 3, the main difference is that all the input involved here is Ï rather thanmerely x, which allows us to answer questions likeFor a question xu, how does learning the response yuinfluence the modelâs belief about a response yâ²u?515.2. Learning Dynamics of SFTWhat do you mean by cheating?Could you clarify whatyou mean by cheating?Itâs interesting, this isnât a topic that Iâve studied extensively, so Iâll be happy to just think about it together. You said âhowâ instead of âifâ, do you think itâs possible?The purple cat danced under the starry night sky with joyful abandon.Letâs take a look at the next monthâs release schedule. Are there any games youâre particularly excited for?You in cheat poker how do?HAJA de sa @4 ~ åµValid feedback Invalid feedback Ungrammatical SemanticsEdit distanceSimilarity to ðð+ (hypothetical sketch)ðð+What do you mean by playing?ðgpts+ðgptf+ðð¢âððâ ð+ðhumðrnd+Prompt: ððHow do you cheat in poker?1. Chosen response ðð+1.1 GPT rephrase chosen, preserving semantics ðgpts+2. Rejected response ððâ4. Random sentence by GPT ðhum3. Irrelevant from train set ððâ ð+5. Random permuted chosen ð²ð«ð§ð+1.2 GPT rephrase chosen, preserving format ðgptf+2.1 GPT rephrase rejected, preserving semantics ðgptsâ2.2 GPT rephrase rejected, preserving format ðgptfâFigure 5.2: The 2-D plane for different probing responses.5.2.2 Prepare the Probing DatasetAnother major challenge in analyzing SFT lies in the intractably large re-sponse space, i.e., y â Y, which contains V L possible combinations. Unlikeclassification tasks such as MNIST, this huge space makes it infeasible toobserve the modelâs confidence change across all possible responses. As apractical compromise, we focus our analysis on a set of representative re-sponse types anchored around the reference response used during SFT, de-noted by y+u . This notation, commonly adopted in the alignment literature,refers to the preferred response as annotated by humans.To select related responses for analysis, we adopt several heuristic criteria,such as the semantic suitability of a candidate response y to the prompt xuin comparison with y+u , or its structural similarity (e.g., measured via editdistance). Inspired by the structure of standard preference optimizationdatasets like Anthropic-HH (Y. Bai et al. 2022) and UltraFeedback (G.Cui et al. 2023), we partition Y into three subspaces and evaluate a curatedset of response types, as summarized in Figure 5.2. The prompt templatesused to generate these responses and the concrete examples of all 14 responsetypes are provided in Appendix C.1.In the next few subsections, we will do the force analysis and verify ourtheory in practical settings. We first create the training set Dtrain by ran-domly selecting 5000 examples from the training split of the dataset. We525.2. Learning Dynamics of SFTconsider two common datasets, Antropic-HH and UltraFeedback, in all ex-periments. Each example in Dtrain contains three components: the prompt(or question) x, the preferred response y+, and the less preferred responseyâ. SFT finetunes with [x, y+], while DPO uses all three (subscripts ofx and y are removed for conciseness). We repeat the experiments on twoseries of models: pythia-410M/1B/1.4B/2.8B (Biderman et al. 2023) andQwen1.5-0.5B/1.8B (J. Bai et al. 2023).To get a more detailed observation of the learning dynamics, we furthercreate a probing dataset Dprob by randomly selecting 500 examples fromDtrain, and generate several typical responses based on the correspondingx, y+, or yâ, as demonstrated in Figure 5.2. Then for each x in Dprob, wecan observe how log ÏÎ¸t(y | Ï) gradually changes on different types of y. Inshort, Dprob helps us to get a more fine-grained inspection of the learningdynamics, which can not only support our analysis above, but also shedmore light on how the modelâs prediction evolves on the entire Y â RVÃL, avery sparse and huge space. Unless otherwise specified, all curves presentedin the following sections are averaged over 500 probing examples.For brevity, we only showcase some core experiments using one model in thisthesis. Note that we empirically find that the trends on different models arequite consistent in models ranging from 410M to 3B. Please refer to theappendix of our paper for more details.5.2.3 Explanation: Force Analysis on Different ResponsesWe can now conduct a force analysis on LLMâs prediction during SFT. Letâsfirst recap two building blocks of such an analysis:1. Recall the MNIST example, where learning [xu, y+u ] will âpull upâ thosesimilar responses, and the strength is controlled by â¥Kt(Ïo,Ïu)â¥F ;2. Recall the zigzag example, where Gt(Ïu) dynamically changes, shapinghow model confidence shifts in different stages of training.Combining these insights with the heuristics for measuring response simi-larity (as demonstrated in Figure 5.2), we expect that learning from y+u willmost strongly influence valid responses, while exerting little to no effect onungrammatical ones.See Figure 5.3, where we train an LLM using the pair Ïu = [xu, y+u ] for sev-eral epochs. As expected, the modelâs log-probability log Ït(y | Ïu) steadilyincreases for y = y+u throughout training. Interestingly, responses in both535.2. Learning Dynamics of SFTFigure 5.3: Left: the learning curves of y+u and other related responses;right: the force analysis of them.the valid-feedback and invalid-feedback groups exhibit a non-monotonic,quadratic trend: their confidence initially increases in the early stages ofSFT but later decreases as training progresses.This phenomenon can be naturally interpreted through the lens of forceanalysis applied to yâ²u. As illustrated in Figure 5.3-(b), the rephrases ofy+u are subject to two competing forces. On one hand, they experiencea pull-up pressure due to their similarity to y+u , which is similar to theinfluence between â4â and â9â. On the other hand, due to the normalizationconstraint over the response space (i.e., the total probability of each tokenmust sum to one), the increase in confidence for y+u inherently imposes aglobal downward pressure on other responses.As training continues, the modelâs confidence in y+u begins to saturate, caus-ing the âenergyâ supplied by Gt(Ïu) to diminish, similar to how Gt changesin the âzig-zagâ example. When this diminishing pulling force is no longersufficient to counteract the global pushing-down effect, the confidence insimilar but non-identical responses begins to decline.In short, if we conceptualize the response space as a continuous distribution,we might have the upper panel of Figure 5.3-(b), where the nearby responsesare first dragged up and then pushed down.545.2. Learning Dynamics of SFTFigure 5.4: Learning curves and force analysis on ungrammatical responses.To further validate the existence of a global pushing-down pressure, weconduct an ablation study by examining the confidence changes of ungram-matical responses. Comparing the upper panels of Figure 5.4-(b) and Fig-ure 5.3-(b), we observe that responses such as yrnd and y+urnd are locatedfarther from y+u in the response space. This suggests that their correspond-ing similarity terms â¥Ktâ¥F are smaller, indicating weaker coupling to thelearned signal from y+u .In such cases, the pulling-up force becomes negligible, and the pushing-downpressure, arising from the normalization constraint over the response distri-bution, dominates. As a result, the log-probabilities of these ungrammaticalresponses steadily decrease during training, a trend clearly supported by theempirical curves shown in Figure 5.4-(a).The two experiments above highlight the distinct behaviors of grammaticalversus ungrammatical response groups. To further validate the correctnessof the force analysis framework, we present additional, more nuanced ex-periments in Appendix D of Ren and Sutherland (2025). Due to spaceconstraints, we highlight one particularly illustrative example here, whichcompares y+test (a preferred response selected from the test set) with yhum (arandomly generated English sentence from ChatGPT).Intuitively, y+test should be more similar to y+u than yhum, as they are likely555.2. Learning Dynamics of SFTFigure 5.5: Comparison of the inflection point of different response groups(calculated using SciPy on the smoothed curves). Different points in thescatter plot represent the results of different models.drawn from similar distributions (e.g., both are helpful and in a positivetone). Empirically, we also observe that y+test typically consists of responsesto prompts (e.g., answers to questions or replies in dialogue), while yhumtends to include more generic, descriptive statements.The results in Figure 5.5 support this intuition. In the right column of thefigure, we compare the average inflection point9, i.e., the epoch at whichthe log-probability curves begin to decay, for both response types. In thescatter plot, points above the diagonal indicate cases where yhum decaysearlier than y+test. Since the global pushing-down pressure is expected toaffect both groups similarly, this difference in decay timing can be attributed9As demonstrated by the red dotted lines in Figure 5.5, which measures when theconfidence start to decrease consistently.565.2. Learning Dynamics of SFTâ¢ User Prompt:"Write a Python function to calculate the F1 score using scikit-learn.ââ¢ LLMâs hallucinated response:from sklearn.metrics import fscoredef calculate_f1(y_true, y_pred):return fscore(y_true, y_pred)â¢ Where this hallucination comes from:sklearn.metrics.fscore does not exist! But it might have appeared in the training data insidea tutorial or forum post discussing âF1-scoreâ in context, like the examples above!â¢ Where âfscoreâ comes from:... you can calculate fscore easily use sklearn.metrics, ...... To calculate the fscore between two predictions, a straightforward way is to use sklearn or pytorch function ...Figure 5.6: Example of a type-B hallucination in Ravichander et al. 2025.to the pull-up force. Because y+test is more similar to y+u , it receives a strongerpull-up signal, thereby delaying its decline in confidence.5.2.4 Explanation: Why a Specific Hallucination EmergesThe experiments above primarily serve to validate the correctness of ourforce analysis framework in the context of LLM finetuning. Some of theobserved behaviors also help address long-standing concerns, such as âwhySFT alone is insufficient for alignmentâ (Qi et al. 2024), by illustrating howthe model tends to pull up a wide range of similar responses early in training,regardless of their subtle alignment properties.In this subsection, we extend our analysis to offer a hypothetical explanationfor a more widely discussed issue: why SFT may exacerbate hallucination.Notably, the perspective we propose is consistent with findings from recentempirical studies on hallucination, such as Ravichander et al. 2025. Specifi-cally, Ravichander et al. (2025) categorize hallucinations into three distincttypes based on their underlying causes. Type-B hallucinations are whenAn incorrect fact was in the pretraining data, or the fact is taken out of thecontext, i.e., the fact appeared within a specific setting in a document in thetraining data, but when taken in isolation, it loses its original meaning.Figure 5.6 provides a concrete example for this type of hallucination. Intu-itively, this behavior can also be interpreted as the model reusing phrasesor responses that were appropriate in one context but incorrectly applyingthem in a different context. We offer a hypothetical explanation for thisphenomenon by examining combinatory sequences of the form [xu, y+j Ì¸=u],575.2. Learning Dynamics of SFTFigure 5.7: Learning curves of y+u ,y+test and y+j Ì¸=u and their force analysis.We visualize them together because they are all âpreferred response.âwhich can be interpreted as the model using the answer, or specific phrasesfrom the answer, to question B in response to question A. In principle, themodelâs confidence in such a mismatched response should not increase, as itlacks semantic coherence with the given input. However, as shown in Fig-ure 5.7, the confidence in y+j Ì¸=u continues to increase throughout training.Although the rate of increase is slower than that of y+u , it notably does notexhibit the âfirst up, then downâ pattern observed in responses like y+test orother rephrasings.This behavior can be explained by the fact that, unlike y+test, which is influ-enced only by the learning of [xu, y+u ], the confidence in y+j Ì¸=u is affected bytwo upward forces: one is caused by its similarity to [xu, y+u ], and anothercomes from learning its own pairing, i.e., [xj Ì¸=u, y+j Ì¸=u]. The latter effect be-comes even more pronounced when xu and xj Ì¸=u are semantically similar,such as when both prompts pertain to a similar topic: a pattern commonlyobserved in type-B hallucinations (see Figure 5.6).In summary, the experiments presented in this subsection offer a hypothet-ical explanation for one specific type of hallucination. Other forms of hallu-cination may arise from different underlying mechanisms, which are beyondthe scope of this analysis. As for practical strategies to mitigate such issues,we do not offer definitive conclusions at this stage.Nonetheless, our force analysis indicates that the semantic similarity be-585.3. Method Involving Gradient Ascent: DPOtween prompts (i.e., x) plays a significant role in the emergence of hallucina-tions. This suggests that carefully crafting more detailed or explicit promptscould potentially reduce such undesired behaviors. For instance, in the caseof the âfscoreâ example, explicitly distinguishing between executable codeand casual discussion within the prompt may reduce such âspurious corre-lationâ and hence reduce hallucination.5.2.5 Summary of SFT PartIn this section, we begin by extending our AKG decomposition to the SFTof LLMs. Leveraging commonly used mechanisms such as teacher forcingand causal masking, we derive an analogous decomposition, enabling theapplication of a similar force analysis framework.To address the challenge posed by the vast response space, we focus ouranalysis on specific responses selected using heuristic criteria. Based on thissetup, we conduct a series of experiments across different settings to validatethe correctness of our analytical framework. The experimental results notonly align closely with our theoretical predictions but also provide insightsinto widely discussed questions, such as why SFT alone may be insufficientfor alignment and why prolonged SFT can exacerbate hallucination.Finally, returning to the four properties introduced at the beginning of Sec-tion 5.2, and to facilitate comparison between SFT and other finetuningmethods, we summarize the gradient form of SFT as follows:i. Origin of the sample: static, off-policy;ii. Dynamic eLR: No response-level design, only epoch-wise scheduler;iii. Negative eLR: No;iv. Token-level eLR: No.5.3 Method Involving Gradient Ascent: DPOPreference finetuning, which teaches the model to provide responses thatalign better with human preferences, is also an important phase in LLMfinetuning pipelines. Different from the SFT stage above, where the train-ing tells the model âwhat to doâ, many preference finetuning methods alsoteach the model âwhat not to do,â which makes the learning dynamics morecomplex. Building upon our analysis of SFT, we now extend our framework595.3. Method Involving Gradient Ascent: DPOto Direct Preference Optimization (DPO), an effective approach for prefer-ence tuning. As introduced at Section 5.1, DPO was originally proposed asa lightweight alternative to reinforcement learning-based finetuning, aimingto eliminate the need for training a separate reward model.Due to its simplicity and efficiency, the off-policy variant of DPO, wherethe model is finetuned on a pre-collected dataset, has become increasinglypopular in practice. In what follows, we demonstrate how this off-policyDPO setup can be reduced to a form analogous to SFT and how our forceanalysis framework can be naturally applied. The key differences betweenoff-policy DPO and SFT can be summarized along the following dimensions:i. Origin of the sample: static, off-policy (on-policy version exists);ii. Dynamic eLR: response-level, adaptive;iii. Negative eLR: Yes;iv. Token-level eLR: No.5.3.1 Theory: Unified Framework on Different MethodsWe first recap the loss function of DPO, following Rafailov et al. (2023):LDPO(θ) = â E(xu,y+u ,yâu )â¼D[log Ï(Î² logÏÎ¸t(y+u | Ï+u )Ïref(y+u | Ï+u )â β log ÏÎ¸t(yâu | Ïâu )Ïref(yâu | Ïâu ))],(5.4)where Ï(Â·) represent a Sigmoid function.10 β is usually a positive number,controlling the relative scaling of the hidden reward function (Rafailov etal. 2023). y+u and yâu are pre-generated responses, and Ïref is the referencemodel, typically the result of SFT. Ï+u = [xu, y+u ] and Ïâu = [xu, yâu ], whichmeans they share the same x but different y. In most cases, the Ïo discussedin this thesis shares the same input x as the corresponding Ïu.Note that in DPO, both y+u and yâu generate gradients for the modelâsupdate; we hence need to recalculate the AKG decomposition. Specifi-cally, we define ÏÎ¸t(y+u | Ï+u ) = Softmax column(z+) and ÏÎ¸t(yâu | Ïâu ) =Softmax column(zâ), where z+ = hÎ¸(Ï+u ) and zâ = hÎ¸(Ïâu ) respectively.Then, starting from M = L = 1, the decomposition for the DPO loss (simi-10Since Softmax function is inherently a generalization of Sigmoid function, we use thesame notation for them.605.3. Method Involving Gradient Ascent: DPOlar to Equation (5.3) for SFT) could be written as:ât(Ïo)ï¸¸ ︷︷ ï¸¸VÃ1= âÎ¸ log Ït(Ïo)|Î¸tï¸¸ ︷︷ ï¸¸VÃd(Î¸t+1 â θt)︸ ︷︷ ï¸¸dÃ1+O(â¥Î¸t+1 â Î¸tâ¥2)â (âz log Ït(Ïo)|ztï¸¸ ︷︷ ï¸¸VÃVâÎ¸zt(Ïo)|Î¸tï¸¸ ︷︷ ï¸¸VÃd)(â Î·âÎ¸L(xu, y+u , yâu )|Î¸tï¸¸ ︷︷ ï¸¸1Ãd)T= âz log Ït(Ïo)|ztï¸¸ ︷︷ ï¸¸VÃVâÎ¸zt(Ïo)|Î¸tï¸¸ ︷︷ ï¸¸VÃd(âÎ·â[z+;zâ]L|ztï¸¸ ︷︷ ï¸¸1Ã2V[âÎ¸z+(Ï+u );âÎ¸zâ(Ïâu )] |Î¸tï¸¸ ︷︷ ï¸¸2VÃd)T= âÎ·âz log Ït(xo)|ztï¸¸ ︷︷ ï¸¸VÃV[âÎ¸zt(xo)|Î¸tï¸¸ ︷︷ ï¸¸VÃd([âÎ¸z+(Ï+u );âÎ¸zâ(Ïâu )] |Î¸t)Tï¸¸ ︷︷ ï¸¸dÃ2V] (â[z+;zâ]L|zt)Tï¸¸ ︷︷ ï¸¸2VÃ1= âÎ·At(Ïo)[Kt(Ïo,Ï+u );Kt(Ïo,Ïâu )](â[z+;zâ]L|zt)Tâ âÎ·At(Ïo)(Kt(Ïo,Ï+u )GtDPO+(Ï+u )âKt(Ïo,Ïâu )GtDPO-(Ïâu ))(5.5)where [Â·; Â·] are concatenation of two vectors or matrices, GtDPO+(Ï+u ) ââz+LDPO, and GtDPO-(Ïâu ) â âzâLDPO. To calculate the residual terms,we decompose the loss:LDPO(θ) = â log(a)a â Ï(b)b â β(log ÏÎ¸t(y+u | Ï+u )â log ÏÎ¸t(yâu | Ïâu ))â c= âÎ² (LSFT(Ï+u )â LSFT(Ïâu ))â cc â β(log Ïref(y+u | Ï+u )â log Ïref(yâu | Ïâu )), (5.6)where c is not a function of θ. Using the chain rule, the l-th column of theresidual term GtDPO+ can be calculated as (the calculate of GtDPO- is similar)GtDPO+ =âLDPOâaâaâbâz+b|zt= â1aa(1â a)âz+b|z+= Î²(1â a) (ÏÎ¸t(y+u | Ï+u )â y+u ) .By stacking values with different l, we can get the residual term of DPO asGtDPO+ = Î²(1â a)(ÏÎ¸t(y | Ï+u )â y+u)GtDPO- = Î²(1â a)(ÏÎ¸t(y | Ïâu )â yâu)a = Ï(Î² logÏÎ¸t(y+u | Ï+u )ÏÎ¸t(yâu | Ïâu )â β log Ïref(y+u | Ï+u )Ïref(yâu | Ïâu ))(5.7)615.3. Method Involving Gradient Ascent: DPOCombining Equation (5.5) (the AKG decomposition) and Equation (5.7)(the Gt terms), we can get the one-step influence function on the DPO case.We omit the full expression here due to the length of the equation. It is quitesimilar to the SFT case, except that DPO has a A(âK+G+ ââKâGâ)form. Compared with that for SFT, two main differences should be noted,as concluded at the beginning of Section 5.3. First, DPO has a response-level dynamic eLR design. Compare with the SFT case where we haveGtSFT = Ït â y+u , the Gt terms for DPO all contain a scalar in front of thevectors, i.e., β(1 â a), which is what we call response-level dynamic eLR.Specifically, the b term is usually named as the âmarginâ in DPO literature,which represents how well the current policy separates y+u and yâu comparedwith the reference policy. Due to the monotonicity of Ï(Â·), a larger marginleads to larger a = Ï(b), which in turn restrains the strength of GtDPO+/-.In other words, GtDPO+/- automatically provides less energy on the examplesthat are already well separated. We then check the role of β, which controlsthe regularizing effect on the KL distance between ÏÎ¸t and Ïref in the originalRL loss (Rafailov et al. 2023). When the margin is negative, larger β leadsto a smaller a and hence provides stronger GtDPO+/- for the model to âcatchupâ the separating ability of the reference model faster. But when the modelis better and has a positive margin, increasing β will increase a and hencecreate a negative influence on β(1 â a), which makes the model updateless. This aligns well with the claims of Rafailov et al. (2023): the strongerregularizing effect tends to âdrag ÏÎ¸ back towards Ïrefâ when its predictionsdeviate from Ïref too much.Another key distinction is the use of intentional gradient ascent, or the so-called negative gradient. This arises naturally in alignment-focused tasks,where the objective is to train the model to distinguish preferred responsesfrom rejected ones. A straightforward strategy, in this case, is to increasethe modelâs confidence in preferred responses while decreasing it for rejectedones. A visual comparison illustrating the main differences between SFT andDPO in the context of learning dynamics is provided in Figure 5.8.In fact, many DPO variants, such as IPO, KTO, SLiC, and SimPO, can beanalyzed in a similar fashion, as their loss function decompositions closelyresemble that of DPO. For further details, we refer readers to Appendix B.2of our paper (Ren and Sutherland 2025). We will later show that under-standing the role played by the negative gradient is the key to understandingsome unexpected behaviors of these algorithms.625.3. Method Involving Gradient Ascent: DPOâ¢ SFT â¢ Off-policy DPOð²u+ð ð½ð(ð²|ð±ð®)ð ð½ðCaused by squeezing effectð²uâ ð²u+ð²â *ð½(1 â ð)Figure 5.8: Force analysis for SFT and DPO.5.3.2 Theory: Gradient Ascent and Squeezing EffectWe now turn to the learning dynamics of gradient ascent. Negative gradientsare in fact ubiquitous in LLM finetuning, appearing in diverse contexts suchas machine unlearning (Barez et al. 2025), DPO and its variants, and evenin RL-based finetuning methods like GRPO. Actually, whenever a modelis required to penalize, suppress, or forget certain outputs, gradient ascentbecomes a natural, off-the-shelf mechanism.However, unlike gradient descent, which targets the minimization of a well-defined loss function and benefits from established convergence guarantees,many gradient ascent methods involve simply reversing the sign of the equiv-alent learning rate (eLR). Despite its widespread use, this reversal often hasundesirable effects, especially in complex systems like LLMs.In the following subsections, we will highlight several counterintuitive phe-nomena that arise under gradient ascent. These findings help explain certainunexpected behaviors observed during off-policy DPO and offer deeper in-sight into the implications of applying negative gradients in other systems.We start from the one-step influence of gradient ascent, using a commonUnconstrained Feature Model (UFM) applied in many LLM theory papers(e.g., Thrampoulidis 2024; Yize Zhao et al. 2024; Razin et al. 2025). Thismodel simplifies the backbone network hθ(x) to a trainable feature vectorhx, focusing on the dynamics of the last read-out layer w and predictions.Specifically, consider a simple V -class logistic regression problem where eachhigh-dimensional input data x is converted to a length-d feature vector via635.3. Method Involving Gradient Ascent: DPOa deep neural network. In other words, we have hx â RdÃ1. The modeluses a linear read-out layer w â RVÃd to convert the feature vector tologits z = whx and then generate the probability prediction vector p usinga Softmax head Ï(Â·). We consider a common cross-entropy loss function foreach input pair (x, y). In summary, we have:Lce(pt, y) = âeâ¤y log pt; pt = Ï(zt); zt = wthx, (5.8)where t is the index of the step during training and ey is a length-V one-hotvector determined by the ground truth label y. To simplify our analysis, weassume a fixed h and only update the parameters of the read-out layer wusing stochastic gradient descent:wt+1 = wt â Î·âwL = wt â η(pt â ey)hâ¤x , (5.9)where η is the learning rate, which can be negative if we consider a negativegradient during training. With Equation (5.8) and (5.9), we can write downeach dimension of pt and pt+1 after some calculations. To quantitativelyanalyze how the modelâs confidence in each class evolves during training, wedefine the ratio αi â pt+1ipti, which captures the relative change in probabilityfor class i after an update. This allows us to determine whether each piincreases or decreases analytically, and by how much.Due to space constraints, we omit the formal analysis of αi and instead pro-vide high-level insights into its behavior. For detailed calculations, theoret-ical justification, and more experimental supports, please refer to AppendixE of our paper (Ren and Sutherland 2025). In short, considering a simpleL = 1 case, when we impose a negative gradient on label yâu , we can describethe changing of modelâs predictive distribution ÏÎ¸t+1 as follows:â¢ Guarantee: the confidence of yâu , i.e., ÏÎ¸t+1(y = yâu ) will decrease.â¢ Guarantee: the decreased probability mass is largely âsqueezedâ intothe output which was most confident before the update: if yâ =argmaxiâ[V ]{yâu } ÏÎ¸t(y = i), then ÏÎ¸t+1(y = yâ) will increase.â¢ Trend: the rich get richer and the poor get poorer: dimensions withhigh ÏÎ¸t tend to increase, and those with low ÏÎ¸t tend to decrease.â¢ Trend: peakier ÏÎ¸t squeeze more. If the probability mass concentrateson few dimensions in ÏÎ¸t , which is common for a pretrained model, allÏÎ¸t+1(y Ì¸= yâ) decrease (only yâ is ârichâ).645.3. Method Involving Gradient Ascent: DPOâ¢ Demonstration of squeezing effectð ððâ = ð =ðâðððâðð + ððð +â¯â¢ Intuition of squeezing effectâ¢ Verification of squeezing effectð²uâð ð½ðð ð½ðPush down everything but the argmaxProbabilityLog probabilityVocabulary IDVocabulary IDWhat we want What the model doFigure 5.9: Demonstration, intuition, and verification of the squeezing effect.â¢ Trend: smaller ÏÎ¸t(y = yâu ) exacerbate the squeezing effect: if yâu ismore unlikely under ÏÎ¸t , the probability mass of all other ÏÎ¸t+1(y Ì¸= yâ)will be more seriously decreased, and the ÏÎ¸t+1(y = yâ) increases more.The trend described above is referred to as the âsqueezing effectâ in ourpaper. This phenomenon captures how the model redistributes probabilitymass after one update: squeezing it away from less likely (poorer) dimensionsand concentrating it onto more likely (richer) ones. An intuitive illustrationof this effect, along with experimental validation using a logistic regressionexample, is provided in Figure 5.9.Furthermore, the bottom-left panel of the figure gives a helpful intuitionfor why the squeezing effect arises. Suppose the model aims to decreaseÏ(y = 0) during an update. A straightforward approach would be to reducethe corresponding logit z0 = â10, and indeed gradient descent will do so.In addition to that update, however, the model may opt to increase the logitof another dimension, thereby increasing the denominator of the softmax andreducing Ï(y = 0) indirectly. Due to the monotonicity of the exponentialfunctionâs derivative, increasing the logit of the most probable class has thelargest effect on the denominator, making it the most efficient direction forreducing the target probability. This explains the second guarantee above:the model favors reinforcing already strong dimensions, e.g., z1 = 10 here.655.3. Method Involving Gradient Ascent: DPOAs the denominator increases, the probability mass assigned to all otherclasses necessarily decreases, consistent with other claims in the squeezingeffect. This process follows the richer-richer, poorer-poorer principle, whichcan be imagined from the structure of the Gt term. Specifically, by checkingthe ptâey term, we know the gradient contribution is piâ1 for the negativelylabeled (undesired) class yâu and pi for all other classes. This implies thatthe redistributed log-probability mass (prior to normalization) is allocatedto the other dimensions in proportion to their current probabilities pi.Note that the discussions above describe the squeezing effect on the modelâsdistribution given the same input. In other words, the trend above describesât(Ïu) = âÎ·At(Ïu)Kt(Ïu,Ïu)Gt(Ïu),where the observing response is also the updating example. This effect linksthe âforceâ Gt(Ïu) to the âconfidence changeâ ât(Ïu). Then how does thiseffect spread to the modelâs confidence on other responses? We can take thebasic AKG decomposition and substitute the calculated âÎ·Gt(Ïu) to it:ât(Ïo) = âÎ·At(Ïo)Kt(Ïo,Ïu)Gt(Ïu)= At(Ïo)Kt(Ïo,Ïu)ï¸¸ ︷︷ ︸project and normalizeât(Ïu)ï¸¸ ︷︷ ï¸¸squeezed(At(Ïu)Kt(Ïu,Ïu))â ,where â is the pseudo-inverse of a matrix.From the equation above, we observe that a poorly imposed negative gra-dient not only affects the modelâs prediction change on the current trainingpoint ât(Ïu), but could also spread to the confidence of other instances Ïo.This influence depends on the following factors:1. the severity of the squeezing effect on Ïu;2. the similarity between Ïo and Ïu;3. the modelâs current confidence in Ïo.These factors together determine how the negative update propagates be-yond the target instance, potentially leading to unintended shifts in modelbehavior, in one-step learning. Although a more fine-grained analysis is dif-ficult to obtain, we can intuitively expect that if more tokens in DPO areaffected by the squeezing effect, the overall model distribution will tend tobecome peakier, rather than flatter. In the next subsection, we will showanother important mechanism of gradient ascent, which might exacerbatethe squeezing effect further.665.3. Method Involving Gradient Ascent: DPO5.3.3 Theory: Positive Gradients Scatter, Negative UniteThe analysis above considers only the negative gradient in a UFM setting.However, in methods like DPO and GRPO, positive and negative gradientsare typically applied in pairs. Suppose we consider a paired update consist-ing of a negative gradient on yâu and a positive gradient on a different targety+u Ì¸= yâu , conditioned on the same x. The update rule then becomes:wt+1 = wt â ηneg(pt â eyâu )hâ¤x â ηpos(pt â ey+u )hâ¤x .After performing the corresponding calculations, we find that the squeezingeffect cancels out in this case, implying that the overall distribution shouldremain more balanced. However, this conclusion contradicts the empiricalobservations seen in DPO, where the squeezing effect persists.This discrepancy is also noted by Razin et al. (2025), where the authorsargue in their Appendix C that the one-step squeezing analysis is insufficientto capture the full dynamics. Thus, more complex, multi-step interactionsthat accumulate over the course of training should be considered.To do this, we first notice that in DPO and other methods involving pairedpositive and negative gradients, the contexts Ï associated with each re-sponse are typically not perfectly aligned. For example, consider y+u =Ithink ... and yâu =I donât believe .... In this case, only the gradi-ent contributions at the first token can fully cancel out, as both responsesare conditioned on the same input xu. For the remaining tokens, the grow-ing dissimilarity between y+u and yâu weakens the cancellation, allowing thesqueezing effect to accumulate across the sequence.Moreover, due to the next-token prediction nature of LLMs, where multiplecompletions can be semantically valid, the behaviors of positive and negativegradients diverge significantly during training. This contrasts with tradi-tional supervised classification tasks, which typically assume a single ground-truth label. For instance, given a prompt like My favorite video gameis . , plausible completions might include WoW, Star Rail, Trails inthe Sky or Zelda. Positive gradients associated with these responses willspread across nearby tokens, creating a plateau in the modelâs probabilitydistribution, as shown in the upper-left panel of Figure 5.10.In contrast, negative gradients tend to squeeze probability mass into domi-nant directions, even when applied sporadically. These opposing behaviorsgive rise to an important summary of their cumulative influence in trainingwith non-unique labels:675.3. Method Involving Gradient Ascent: DPOð ð½ðð ð½told | Howare | oldyou | are? | youSporadical negative gradientgenerate accumulated squeezing effectâ¢ Accumulation of positive gradientâ¢ Accumulation of negative gradientâ¢ Verification using a 2-gram experimentPositve-onlyBoth, balancedFigure 5.10: Left: accumulated influence on positive and negative gradients.Right: verification on a 2-gram experiment.Positive gradients scatter, negative gradients unite.This principle highlights how asymmetries between positive and negativeupdates can lead to unintended distributional shifts, such as overconfidenceor undesired suppression of valid alternatives.To further validate that paired positive and negative gradients do not cancelout (and thus cannot fully eliminate the squeezing effect) we construct acontrolled experiment under a 2-gram setting. We continue to operate inthe UFM, but now the model learns a 2-gram distribution of the form p(y1 |y0) = Ï(y1 | hy0). Both y0 and y1 are sampled from a vocabulary of sizeV = 16. The feature matrix h â RdÃV is randomly initialized, simulatingthe output of a frozen backbone encoder hθ. For any token y0, its featurevector hy0 corresponds to the y0-th column of h. We construct positive andnegative sequences of equal length L, with all tokens independently drawnfrom a uniform distribution over {0, 1, . . . , 15}. During training, the modelis updated using standard NLL loss applied to each conditional distributionÏ(yi | hyiâ1) along the sequence.To visualize the learning dynamics, we track the modelâs predicted distri-butions over all possible contexts h. This allows us to generate 3D plots,such as those shown in the right column of Figure 5.10. In each plot, the685.3. Method Involving Gradient Ascent: DPOz-axis represents the output probability; each row corresponds to a differentcontext y0, and each line (with 16 dimensions) shows the probability distri-bution over the vocabulary. The gray curves denote the distribution beforetraining, and the blue ones show the result after applying updates from thepaired gradients. Since the vocabulary size is 16, each plot contains 16 rows,illustrating how the output distribution evolves for each context.The left panels of Figure 5.10 clearly illustrate why the accumulated influ-ence of positive and negative gradients is so different. In the positive-onlysetting, we observe that the output distributions become flatter after train-ing, because the pull-up behavior is a âlocalâ behavior: each update onlypulls up a small region surrounding y+u . Such a trend can be well depictedby the positive-only experiment (where we set ηneg = 0).In contrast, when comparable negative gradients exist, although they arealso imposed sporadically, their behavior is global: the probability mass onmost dimensions is squeezed to the same ârichâ one. Hence, as the exper-iment in the bottom-right panel shows, a model trained on paired positiveand negative examples (with equal sequence lengths and ηpos = âÎ·neg),the resulting distributions become noticeably peakier. The probability massunder each context becomes increasingly concentrated around the rich di-mension(s), though not exclusively, as the richest dimension can shift dy-namically during training. This indicates that negative gradients exert aunifying force, drawing the distribution toward dominant directions.These results empirically confirm that the one-step squeezing effect can ac-cumulate over long sequences. While the influence of positive gradientsdisperses across multiple plausible outputs, the negative gradients reinforceeach other, amplifying the squeezing effect, especially when applied repeat-edly over long sequences.5.3.4 Explanation: Unexpected Behaviors of DPOWe now apply the force analysis framework to off-policy DPO. A key dis-tinction between the learning dynamics of SFT and DPO lies in the pres-ence of an intentional negative gradient in the latter, as demonstrated inFigure 5.8. As discussed earlier, the squeezing effect introduces a globalbehavior whereby probability mass is increasingly concentrated on the richdimensions. This effect becomes more pronounced when the negative gradi-ent is applied to dimensions associated with lower-confidence predictions.We begin by examining a widely observed and somewhat counter-intuitive695.3. Method Involving Gradient Ascent: DPOFigure 5.11: Left: training curves of ÏÎ¸t(y+u ) and ÏÎ¸t(yâu ). Right: forceanalysis on y+u in one epochâs training (the dataset contains N problems).phenomenon during DPO training: the decay of both ÏÎ¸t(y+u ) and ÏÎ¸t(yâu )after prolonged training. This behavior, illustrated in Figure 5.11, has beenfrequently reported and discussed in the community, e.g. by Pal et al. (2024)and Rafailov et al. (2024). Some noticed the influence of negative gradients,but didnât study further.To offer a new perspective, we analyze this behavior through the lens offorce analysis. In Figure 5.11-(b), we visualize the dynamics using diagramssimilar to those employed in the SFT section. A key distinction in DPO isthe presence of two explicitly imposed pressures: on both y+u and yâu . Unlikethe SFT case, where a one-step response-level analysis is often sufficient,understanding the dynamics in DPO requires tracking the accumulation ofinfluence both within and across training epochs (and data samples).Focusing on the force exerted on y+u , we identify three major components.First, the self-influence from learning [xu, y+u ] induces a strong upward force,substantially increases the modelâs confidence in y+u . Because this effectis highly localized, we can reasonably ignore the contribution from otherexamples [xi, y+i ] for i Ì¸= u, assuming a diverse enough dataset where thecorresponding kernel norms â¥Ktâ¥F are relatively small.705.3. Method Involving Gradient Ascent: DPOSecond, a downward force arises from learning [xu, yâu ], due to the sharedinput xu. However, this opposing pressure is also localized and its impact istypically moderated by the kernel norm â¥Ktâ¥F between them, which tendsto be smaller than that of the self-induced upward force.Given these dynamics, a natural question arises: why does the confidence iny+u eventually decay with continued DPO training? The key to this seem-ingly paradoxical decay lies in the third group of forces: a global effectinduced by the large negative gradients applied to low-confidence regions,which we refer to as the squeezing effect. Although the individual impactfrom any single negative example is weak, their cumulative influence be-comes substantial when aggregated over the N â 1 negative examples inthe dataset. This global nature allows these aggregated forces to graduallysuppress ÏÎ¸t(y+u ), even though y+u is not directly involved in those updates.Moreover, recall that the squeezing effect intensifies when the target yâu al-ready has a low probability before the update. This condition is exactlywhat we observe in the later stages of DPO training: ÏÎ¸t(yâu ) becomes in-creasingly small during training, thus amplifying the global squeezing effect.Additional evidence for the squeezing effect is provided in the left panel ofFigure 5.12. In this experiment, we first train the model for 10 epochs usingSFT, and then continue train it with DPO. We observe that the modelâsconfidence in most responses decays much more rapidly than in the pureSFT case, although we use the same learning rate. This is especially evidentfor responses such as yrnd and y+urnd, which are never seen during training andbear no resemblance to either y+u or yâu ; thus, they serve as effective proxiesfor evaluating the modelâs distribution on those âfoundationâ dimensions.Their rapid decay suggests the presence of a global pushing-down force,which is a signature of the squeezing effect.The observation above naturally leads to a follow-up question: if all trackedresponses, including y+u , which is subject to direct pull-up pressure, aredecreasing in confidence, where does the probability mass go?The answer again lies in the squeezing effect. As visualized in Figure 5.9, theprobability mass is increasingly concentrated on the rich dimensions duringtraining. To verify this, we monitor the modelâs confidence on a specificconstructed sequence: yâ, in which the l-th token is the argmax predictiongiven the context [xu, y+u,<l]. It is important to note that yâ is not necessarilya fluent or grammatical response; it is not the result of standard greedydecoding, but rather a teacher-forcing token-wise argmax construction.715.3. Method Involving Gradient Ascent: DPOFigure 5.12: More evidence supporting the squeezing effect.These results, also shown in Figure 5.12-(b), make it clear that the modelâspredictive confidence collapses onto the argmax sequence, while confidence inall other responses, including preferred and unseen ones, continues to decay.To assess whether this phenomenon reflects a broader trend in any othercontext, we also evaluate a similar sequence y¯â generated by conditioningon a random prompt [xu, yâ²rnd]. Both sequences exhibit similar behavior:their confidence increases consistently during DPO.In summary, the above analysis and empirical findings provide strong sup-port for our proposed force analysis framework and the associated squeezingeffect. This perspective offers a unified explanation for several nuanced andpreviously unexplained trends in model behavior across diverse responsesunder DPO. While additional empirical illustrations further reinforce theseinsights, we omit them due to space constraints. For more results and dis-cussion, please refer to Appendix D of our paper (Ren and Sutherland 2025).5.3.5 Algorithm: A Simple yet Effective PipelineThe preceding analysis offers a new perspective on why off-policy DPOtraining can introduce problematic behaviors. Although the modelâs abilityto separate preferred from rejected responses continues to improve, as evi-denced by the steadily increasing margin ÏÎ¸t(y+u ) â ÏÎ¸t(yâu ), the significantincrease of ÏÎ¸t(yâ) may lead to undesirable side effects. In particular, de-generation phenomena such as repetitive outputs, often referred to as the725.3. Method Involving Gradient Ascent: DPOPretrainedLLMðð = ððSFT using [ð¥, ð¦+]SFT using [ð¥, ð¦+]AND [ð¥, ð¦â]ððððDPO using [ð¥, ð¦+, ð¦â]DPO using [ð¥, ð¦+, ð¦â]IdenticalDifferentIdenticalððâðððâðððâðððâðððâðððâðDPO 2 epochsDPO 4 epochsDPO 6 epochsDPO 2 epochsDPO 4 epochsDPO 6 epochsð²â onvalleyð²â on plateauFigure 5.13: Illustration of the proposed method and baseline. âEâ is shortfor the âdataset extensionâ.ârepeaterâ effect (Holtzman et al. 2020), can become more severe when themodel assigns excessively high confidence to yâ, as illustrated in Figure 5.15.This is intuitive: as the squeezing effect drives probability mass toward high-frequency tokens, and the model becomes more likely to be âtrappedâ bythose tokens and phrases.Some recent works have recognized this limitation of off-policy DPO andproposed on-policy variants to address it (S. Guo, B. Zhang, et al. 2024).Their experiments demonstrate that on-policy DPO and its variants oftenoutperform their off-policy counterparts. Our framework provides a coherentexplanation for this improvement: under on-policy training, the sampled yâuis more likely to lie within regions of high model confidence, thereby reducingthe influence of the global squeezing effect.However, on-policy methods typically require a strong verifier to providereal-time feedback, effectively reintroducing the need for a reward model,precisely the component that DPO aims to eliminate. This raises a naturalquestion: can we make yâu more âon-policyâ before applying DPO?Motivated by this idea, we propose a simple yet effective pipeline to miti-gate the harmful squeezing effect in off-policy DPO. Our method does notrely on any additional modules; hence, the extra computational overhead isnegligible. Also, it can be seamlessly integrated into existing workflows.Specifically, we can first train the model on both [xu, y+u ] and [xu, yâu ] duringthe SFT stage (i.e., on-policify the rejected responses), and then run theusual DPO, as in Figure 5.13. Following the analysis above, we can expect735.3. Method Involving Gradient Ascent: DPO0 2 4 6 8 10Number of epochs30025020015010050Log probabilityChosen y +ubaselineextend0 2 4 6 8 10Number of epochs30025020015010050yu is the peak nowRejected yu0 2 4 6 8 10Number of epochs500400300200100All rephrasesNon-human sequenceOther responses0 2 4 6 8 10Number of epochs1301201101009080Drop at the start of DPOArgmax response(a) Result on Antropic-HH.0 2 4 6 8 10Number of epochs500450400350300250200Log probabilityChosen y +ubaselineextend0 2 4 6 8 10Number of epochs500450400350300250200yu is high enough nowRejected yu0 2 4 6 8 10Number of epochs500400300200100All rephrasesNon-human sequenceOther responses0 2 4 6 8 10Number of epochs540520500480460440420400Slow at the start of DPOArgmax response150014001300120011001000900800(b) Result on UltraFeedback.Figure 5.14: Learning dynamics of the baseline and the proposed methodwith training data extension.that during this new SFT stage, the region of those responses similar to y+uor yâu will be âpulled upâ simultaneously. This is what we want because inmany cases, both y+u and yâu are reasonably good responses for the questionxu; the new SFT design hence helps to pull up a larger region that containsmore suitable responses compared with the baseline SFT. After that, thelocal push-down pressure imposed during DPO can efficiently decrease themodelâs confidence on yâu and its similar responses. Since yâu is no longer sounlikely before DPO, the squeezing effect is mitigated.We call our training pipeline âextendâ and compare its learning dynamicswith the baseline setting in Figure 5.14. The squeezing effect is mitigatedbecause the confidence of other responses all decays more slowly duringDPO. Furthermore, we also observe a big drop in ÏÎ¸t(yâ) when DPO starts,which supports our analysis more directly.Table 5.1: Win-rate to baseline.DPO Ep. ChatGPT Claude0 0.4729 0.46792 0.6518 0.51514 0.6928 0.60456 0.6667 0.5432To further show that mitigat-ing the squeezing effect indeedbrings benefits, we compare theresponses generated by modelstrained using different methodsby feeding them to ChatGPT andClaude3. Specifically, we firstSFT the model for two epochsusing the two methods discussed745.3. Method Involving Gradient Ascent: DPOFigure 5.15: Prompt used for evaluating modelâs response (from Rafailovet al. 2023), an example feedback from GPT3.5-turbo, and two examplesof the âdegenerateâ effect described in Holtzman et al. 2020. (Sorry for thesmall fonts, since the degenerated messages are usually nonsense. Pleasezoom in if you want the details.)above and call the resulting policy network Ïbase and Ïextend. Then, we con-duct identical DPO training on both Ïbase and Ïextend for several epochs.The win rate of the proposed method against the baseline one is providedin Table 5.1. It is clear that before DPO, Ïbase is better, because Ïextendis explicitly trained on those yâ. However, the Ïextend performs better af-ter DPO several epochs since the squeezing effect is efficiently mitigated.Furthermore, after checking the modelâs responses on some validation prob-lems, we find that although both B2 and E2 inevitably generate degenerateresponses, the proposed method can mitigate this issue to some extent.Despite its simplicity and effectiveness, the proposed method also carriespotential risks. One concern is whether the harmful influence of SFT on yâucan be reliably canceled during the subsequent DPO phase. The methodis generally safe when yâu is slightly inferior to y+u for a given input xu.However, if yâu contains sensitive or inappropriate content, exposing it tothe model, even through negative gradient updates, as in standard DPO,may still pose safety concerns. The high-level goal of our approach is toenhance the on-policy nature of responses subject to negative gradients. To755.4. RL-based Finetuning: GRPOachieve this more safely, one possible strategy is to finetune the model on adiverse set of valid alternative responses before DPO, thereby ensuring thatyâu lies in a flatter (plateau) region of the output distribution rather thana sharp peak. We believe this âon-policifyâ is an interesting and promisingtopic in the future, because there is always more off-policy data than on-policy data on the internet. We can also cut yâu into small chunks or deletethe sensitive information within it before SFT. We believe that exploringthis direction, i.e., designing SFT strategies that shape the initial landscapeto mitigate harmful global effects of the negative gradients, might also bean interesting direction for future work.5.4 RL-based Finetuning: GRPOWe now analyze a more complex and recently popularized finetuning al-gorithm for LLMs: Group Relative Policy Optimization (GRPO)(Shao etal. 2024). GRPO has attracted increasing attention due to its simplicityand effectiveness, particularly on reasoning-heavy tasks. Prominent mod-els such as DeepSeek-R1(D. Guo et al. 2025), DeepSeek-Math (Shao et al.2024), Med-R1 (Lai et al. 2025), and Search-R1 (Jin et al. 2025) have lever-aged GRPO to achieve significant performance gains across domains, in-cluding code generation, mathematical problem solving, medical reasoning,and retrieval-augmented generation. These empirical successes highlightGRPOâs growing importance as a tool for aligning models with task-specificbehaviors using rule-based or heuristic feedback mechanisms.Structurally, GRPO shares several similarities with DPO: both apply re-sponse level updates and introduce negative gradients, with learning ratesdynamically adjusted based on response quality. However, GRPO differs ina critical aspect: rather than relying on a fixed pair of off-policy positiveand negative responses, it samples N on-policy roll-outs per question in eachtraining epoch. These sampled responses (and consequently, the proportionof correct responses) can vary across epochs with the improvement of themodelâs reasoning capability. This on-policy and group-based nature in-troduces additional complexity into the learning dynamics, making it morechallenging to analyze model behavior using static probing methods. In linewith the summaries provided in the SFT and DPO sections, we also providethe characteristics of gradients in GRPO as:i. Origin of the sample: dynamical, on-policy;ii. Dynamic eLR: response-level, token-level;765.4. RL-based Finetuning: GRPOiii. Negative eLR: Yes;iv. Token-level eLR: Yes.To address these complexities, we shift our focus in this section from theprobability evolution of a fixed probing set to a more localized, intra-rolloutanalysis. Specifically, we investigate the influence of individual tokens withindifferent responses answering the same sampled question. This fine-grainedapproach enables us to study how different types of tokens contribute tothe modelâs reasoning improvements under GRPO training. We believesuch a fine-grained analysis is more appropriate for understanding RL-stylealgorithms and can offer insights into questions such as: learning whichtokens impede the modelâs reasoning capabilities most?In this section, we assume the improvement of the modelâs confidence inthose positive responses is a key measurement for its reasoning ability. Wehence investigate why, under GRPO training, the confidence in some pos-itive responses sometimes fails to increase sufficiently, or even decreases, aphenomenon we term the Lazy Likelihood Displacement (LLDisp) problem,originally proposed in our recent work (W. Deng, Ren, M. Li, et al. 2025;W. Deng, Ren, Sutherland, et al. 2025).To address this, we first extend the learning dynamics decomposition toGRPO at the token level, allowing for a finer-grained analysis. We then focusour study within the framework of the Unconstrained Feature Model (UFM),previously introduced in the context of the squeezing effect. LeveragingUFM, we define an influence score that quantifies the directional impact ofindividual tokens. This allows us to identify specific harmful tokens withinnegative responses that contribute to LLDisp.By selectively mitigating the influence of these harmful tokens, we demon-strate that the LLDisp problem can be substantially alleviated. Our ex-perimental results strongly support this claim. Furthermore, the algorithminspired by our analysis leads to notable improvements in the modelâs math-ematical reasoning performance, outperforming several baseline methods.775.4. RL-based Finetuning: GRPO5.4.1 Theory: Decompose the Gradient of GRPOLetâs first recap the loss and gradient of GRPO. Following Shao et al. (2024),the standard GRPOâs loss LËGRPO(Î¸) is:E(x,a)â¼D{yn}Nn=1â¼Ïref(x)ï£®ï£° â1âNn=1 |yn|Nân=1|yn|âl=1min(AnÎ³n,l(Î¸), Anclip (γn,l(θ), εl, Îµh))ï£¹ï£» .(5.10)This loss formulation reveals the underlying mechanism of the algorithm.We begin with a reference model Ïref and a dataset D. In each iteration, aproblem instance x and its corresponding verifier a are sampled from D. Thereference model Ïref is then used to generate N candidate responses {yn}Nn=1by sampling from the distribution Ïref(y | x). The verifier a evaluates eachresponse yn and determines whether it is correct, assigning a binary rewardrn = 1 for correct and rn = 0 for incorrect responses. These rewards arethen standardized to compute the advantage values An for each response:An =rn â EË[{rn}Nn=1]âVËar[{rn}Nn=1].From Equation (5.10), it is clear that the loss is calculated in a token-wisefashion, where the ratio γn,l(θ) depends on both n and l. Specifically, theratio and its gradient are defined as:γn,l(θ) =ÏÎ¸(yn,l | x, yn,<l)Ïref(yn,l | x, yn,<l)âÎ¸Î³n,l =ÏÎ¸(yn,l | x, yn,<l)Ïref(yn,l | x, yn,<l)âÎ¸ log ÏÎ¸(yn,l | x, yn,<l)â cn,lâÎ¸Ln,l, (5.11)where cn,l is a positive constant and Ln,l is the standard cross-entropy loss.In GRPO, the tokens and responses involved in the system become morecomplicated. We will try our best to follow the notations used in this section.First, we still use xo and yo to represent the question and response weare observing. Note that we have N roll-outs for each question x, withN+ of them being correct (r = 1) and Nâ of them being incorrect (r =0). The ratio of the correct and incorrect responses is denoted as Ï+ andÏâ, respectively. Usually, we have Ï+ + Ïâ = 1. Then, we use y+n (or785.4. RL-based Finetuning: GRPOyân ) to represent a correct (or incorrect) response. Note that n â [N ],and we define two index subsets I+ and Iâ to represent the positive andnegative responses, respectively. Hence y+n is a concise expression for yn, n âI+. To make the notations concise, we omit the index of the question.That is because in this part, we would analyze the token-level influence, weuse Ï+n,<l = [xn, y+n,<l], which is the context for the l-th token of the n-thresponse, which happens to be correct.We now check the token-wise learning dynamics in GRPO. Specifically, wewould like to trackât(Ï+o,<m)â log ÏÎ¸t+1(y | xo, y+o,<m)â log ÏÎ¸t (y | xo, y+o,<m) ,i.e., the modelâs confidence change on a correct responseâsm-th token, whichshould be a V Ã 1 vector. Using the first-order Taylor expansion, we have(we abuse the inner product notion here to represent matrix multiplication,which is easier to read, e.g., â¨B,bâ© = Bb):ât(Ï+o,<m)ï¸¸ ︷︷ ï¸¸VÃ1=â©âÎ¸ log ÏÎ¸t(y | xo, y+o,<m)ï¸¸ ︷︷ ï¸¸VÃd, θt+1 â Î¸tï¸¸ ︷︷ ï¸¸dÃ1âª=â©âÎ¸ log ÏÎ¸t(y | xo, y+o,<m),âÎ·âÎ¸LËGRPOâª=â©âÎ¸ log ÏÎ¸t(y | xo, y+o,<m),Î·âNn=1 |yn|Nân=1|yn|âl=1âÎ¸AnÎ³n,l(Î¸)âª= Î·1âNn=1 |yn|ân,lAnâ©âÎ¸ log ÏÎ¸t (y | xo, y+o,<m) , cn,lâÎ¸Ln,lâª ,where cn,lâÎ¸Ln,l is defined in the gradient of γ in Equation (5.11). We alsosimplify LËGRPO by ignoring the clip operation.From the equation above, we can definitely apply a similar decompositionas Equation (5.3) and then conduct force analysis. However, to get moreinsights about the accumulated influence at the token level, we instead focuson the following question:If GRPO wants to improve the confidence of those correct roll-outs,which group of tokens strengthens (or impedes) this?Then, analyzing the whole V dimensions of ât(Ï+o,<m)would be redundant,and also make the decomposition more complicated. As a result, we only795.4. RL-based Finetuning: GRPOfocus on the labelâs dimension of ât, and define our influence function aseâ¤yo,mât(Ï+o,<m)=Î·âNn=1 |yn|ân,lAncn,lâ©eâ¤yo,mâÎ¸ log ÏÎ¸t(y | xo, y+o,<m),âÎ¸Ln,lâª= Î·1âNn=1 |yn|ân,lAncn,l â¨âÎ¸Lo,m,âÎ¸Ln,lâ© (5.12)where the loss function L here is the standard cross-entropy loss. If theright-hand side of this decomposition is positive, then the confidence of them-th token in y+o will increase, which is what we expect. Then, our task isto separate all other tokens (determined by n, l) into two groups (recall thatcn,l > 0, but An can be both positive and negative):â¢ the beneficial ones: n, l where An â¨âÎ¸Lo,m,âÎ¸Ln,lâ© > 0â¢ the harmful ones: n, l where An â¨âÎ¸Lo,m,âÎ¸Ln,lâ© < 0.Obviously, learning the tokens (l-th token in the n-th response) in the ben-eficial group will enhance the modelâs confidence on correct responses, andvice versa. The conditions described above align well with our intuition.We can interpret the inner product between two token-level gradients as ameasure of similarity between y+o,m and yn,lâanalogous to the K term dis-cussed previously, but now we would like to offer more direct insight into itsdirectional effect. When the inner product is large and positive, it indicatesthat the two tokens are highly similar. In such cases, the impact of the tokenyn,l on the update of y+o,m depends on the sign of An: similar tokens frompositively rewarded responses (An > 0) exert a beneficial influence, whilethose from negatively rewarded responses (An < 0) can be detrimental.5.4.2 Theory: Grouping Tokens via Influence ScoresBuilding on the discussion above, we aim to identify the harmful tokens thathinder learning by considering both their advantage values and their similar-ity to a given observing token. Mitigating the influence of such tokens couldaccelerate model training and help resolve the LLDisp issues, i.e., tokens ina positive response exhibit insufficient increase (W. Deng, Ren, M. Li, et al.2025). However, computing the exact inner product between gradients is of-ten intractable in practice, especially within large-scale models. To addressthis, we turn to simplification and approximation.In this subsection, we revisit the Unconstrained Feature Model (UFM) in-troduced in the DPO analysis and propose a tractable approximation to the805.4. RL-based Finetuning: GRPOgradient inner product, which we term the âinfluence scoreâ. This score pro-vides a computationally efficient proxy for measuring the directional impactof one token on another during training.Recall Equation (5.12), our task is to calculate â¨âÎ¸Lo,m,âÎ¸Ln,lâ©. Consid-ering UFM, in which θ â [w;h], where w â RVÃd and h â RdÃ1 are bothtrainable parameters. We use the same subscript system for both h, Ï,and y. So, the feature vector of Ï+o,<m is hence h+o,<m. Then, âÎ¸Lo,m =âÎ¸ log Ï(y | xo, y+o,<m)can be written as â[w;h] log Ï(wh+o,<m), where Ï(Â·) isthe Softmax function. Our inner product then becomes:â©â[w;h] log Ïyo,m(who,m),â[w;h] log Ïyn,l(whn,l)âª . (5.13)Note that calculating this inner product is essentially equivalent to calculat-ing the gradient flow of ddt log Ïtyi , as in Razin et al. (2025) and our W. Deng,Ren, M. Li, et al. (2025). We use a notation system that aligns better withthose in previous chapters here.Then, after several steps (details can be found in Appendix A.3), we haveâ¨âÎ¸Lo,m,âÎ¸Ln,lâ© â α(o,m),(n,l) â (eyo,m â Ïyo,m)â¤(eyn,l â Ïyn,l)hâ¤n,lho,m,(5.14)which is the multiplication of two inner products of the updating and ob-serving tokens (one is on their G-terms and another is on their hidden rep-resentations). In summary, combining with Equation (5.12), we claim ifAnα(o,m),(n,l) > 0, learning the l-th token of the n-th response is beneficial,and vice versa.5.4.3 Explanation: Lazy Likelihood DisplacementRecall that our goal is to use learning dynamics to gain insight into theLLDisp problem in GRPO, wherein some correct (positive) responses fail toreceive sufficient increases in probability during training. The force anal-ysis presented above suggests that tokens from both positive and negativeresponses may contribute to LLDisp. However, suppressing learning fromtokens in positive responses is undesirable, as these tokens also exert bene-ficial âpull-upâ forces that reinforce their own correct outputs.This observation motivates a more targeted strategy: identifying and miti-gating harmful tokens specifically within negative responses. To assess theviability of this approach, we conduct an experiment to evaluate whether815.4. RL-based Finetuning: GRPOattenuating the influence of selected negative responses can effectively alle-viate the LLDisp problem.In particular, we visualize the change in model confidence on positive re-sponses (averaged over N+ responses for each problem) after a single updatestep, using an enlarged learning rate to amplify the observed trends. We be-gin by randomly sampling 40 questions from the MATH dataset (Hendryckset al. 2021), and for each question, we generate N = 8 roll-out responsesusing the initialized model ÏÎ¸0 . Each response is then evaluated by a verifierto determine its correctness, which allows us to compute both its reward rnand advantage An, along with its initial model probability ÏÎ¸0(y+o | x).We then finetune the model on these responses under two different set-tings: (1) standard GRPO, and (2) an ablation variant in which all negativeresponses are ignored, referred to as the Pos Only setting. After train-ing, we re-evaluate the same responses under the updated model to obtainÏÎ¸1(y+o | x). The differenceGap(x) â EoâI+ [ÏÎ¸1(y+o | x)â ÏÎ¸0(y+o | x)]reflects the modelâs improvement in confidence for positive responses onaverage. If Gap(x) is smaller than a threshold (e.g., the average increase onall other questions), it might suffer from the LLDisp problem.As shown in Figure 5.16, the Pos Only setting results in significantly largerconfidence gains for most questions compared to GRPO. This clearly indi-cates that the inclusion of negative responses, while potentially beneficialfor exploration, can hinder the learning of correct predictions.11 Hence, abig gap between GapGPRO(x) and GapPos-Only(x) is also a sign of LLDisp.Then, should we abandon the use of negative responses and rely solely onPos Only training? The answer is no. As demonstrated by prior work likeShao et al. (2024) and D. Guo et al. (2025), negative examples play a criticalrole in penalizing undesirable behaviors, thereby enhancing the robustnessand generalization of the modelâs reasoning capabilities.What is needed, instead, is a more fine-grained approach: a token-levelassessment of influence within negative responses. This is precisely whereour token-wise learning dynamics framework offers valuable guidance: en-abling the identification and mitigation of only the truly harmful componentswithin negative roll-outs, without discarding their overall utility.11The exploration-exploitation trade-off in RL-LLM is discussed in our paper (W. Deng,Ren, Sutherland, et al. 2025).825.4. RL-based Finetuning: GRPOFigure 5.16: Averaged positive response confidence change after one update.This requirement can be readily implemented by computing the averageinfluence score across all positive responses. Based on Equation (5.14), theaverage version is defined as:Î±Ën,l â EoâI+,mâ[|y+o |][Î±(o,m),(n,l)]Î±Ë+n,l â Î±Ën,l,ân â I+; Î±Ëân,l â Î±Ën,l,ân â Iâ (5.15)where we also define the Î±Ë+n,l (and Î±Ëân,l) for the tokens with positive (andnegative) An, respectively. Since we only care about the negative responses,those tokens with Î±Ëân,l > 0 are our targets.Before conducting a token-level experiment, we first check whether our pro-posed influence score is a good approximation of the LLDisp problem at thequestion level. We believe those positive responses suffering from LLDispmight also have a relatively big En,l[Î±Ëân,l]. To validate our hypothesis, weselect 100 questions from the AIME dataset (1983â2023) (Hendrycks et al.2021) and compute two quantities for each question: the observed LLDispmetric Gap(x) and the proposed influence score En,l[Î±Ëân,l]. We then evalu-ate the degree of alignment between these two measures by computing theoverlap among the Top-K ranked problems under each metric.As a baseline, we also calculate the overlap between Gap(x) and a randomlyshuffled ranking. The results, summarized in Table 5.2, show that the Top-K lists obtained from En,l[Î±Ëân,l] have significantly higher overlap with Gap(x)than with the random ranking. This suggests that our proposed influencescore is an effective and computationally efficient proxy for identifying po-tential LLDisp issues before the finetuning starts!Furthermore, we demonstrate some concrete example of what those âharmfultokensâ in negative responses looks like in Figure 5.17. A key insight is835.4. RL-based Finetuning: GRPOFigure 5.17: Highlight the tokens in the harmful group, i.e., those shouldnot be penalized in the negative response.Top-K Qwen-1.5B-deepseek Qwen-1.5B-mathEn,l[Î±Ëân,l] Random En,l[Î±Ëân,l] Random10 50% 17.5% 60% 21.3%15 75% 26.3% 75% 31.9%Table 5.2: Ranking correlation experiment.that many tokens in negative samples may be logically or step-correct, yetGRPO penalizes them indiscriminately using a shared advantage term An.For instance, the red-highlighted tokens in incorrect responses also appear incorrect ones. Consequently, the accumulated gradient force on these tokenswould cancel out in expectation, since En[An] = 0, ân â [N ].5.4.4 Algorithm: Negative Token Hidden RewardBased on the analysis above, the way to refine GRPO is obvious: we canjust penalize those harmful negative tokens by a ratio β < 1. We then setAËn,l = Î²An,ân, l â {n â Iâ : Î±Ëân,l > Ï},and leave the advantage of other tokens unchanged. Since we want to miti-gate the influence of such tokens, we usually have 0 â¤ β â¤ 1. We design Ï as845.4. RL-based Finetuning: GRPOFigure 5.18: Impact of NTHR on Gap(x) in different models.a tunable hyperparameter to control how many tokens are considered harm-ful. Following our paper (W. Deng, Ren, M. Li, et al. 2025), we also callthis value negative token hidden reward (NTHR), which is easy to combinewith GRPO and other variants.We begin by evaluating the effectiveness of NTHR by examining Gap(x)using a bar plot format similar to that in Figure 5.16. In this experiment,we set β = 0, meaning that we entirely exclude the influence of negativetokens whose estimated influence scores Î±Ëân,l exceed a predefined thresholdÏ . We compare NTHR against two baselines. The first is standard GRPO,corresponding to the case where β = 1, in which all negative tokens arefully retained. The second is GRPO+Random, where the same number ofnegative tokens as in the NTHR setting are randomly selected and theirinfluence is removed.As shown in Figure 5.18, NTHR significantly alleviates the LLDisp problem,outperforming both baselines. Moreover, we observe that the extent of im-provement varies across different questions and appears to be correlated withthe modelâs reasoning capability. This observation opens up an interestingdirection for future research: adapting token-level mitigation strategies tothe difficulty of the task and the modelâs current learning state.Having demonstrated the impact of negative gradients on likelihood dynam-ics and the effectiveness of our method in mitigating the LLDisp issue, wenow turn to evaluating the effect of negative gradients on downstream modelperformance. To this end, we finetune models of varying sizes on the MATHdataset (levels 3â5) (Hendrycks et al. 2021), and assess their performanceusing greedy decoding across five math reasoning benchmarks: AIME24,AMC, MATH500, Minerva, and Olympiad. Our results, summarized inTable 5.3, show that GRPO augmented with NTHR consistently improvesperformance across all model sizes. We also verified some claims mentioned855.5. Conclusion and Discussionsin this section, e.g., throwing all the negative tokens might harm the system(as shown by the Pos Only results).In summary, our proposed NTHR method, motivated by a token-level learn-ing dynamics analysis, effectively identifies and attenuates harmful nega-tive updates in standard GRPO. By reducing the effective learning rate(eLR) of the most detrimental tokens, we mitigate the LLDisp phenomenonto some extent. Our experiments on math reasoning tasks across mod-els from 0.5B to 3B demonstrate that NTHR enhances the performance ofGRPO over many datasets, offering a practical and principled improvementto preference-based finetuning.Base model + Method AIME24 AMC MATH500 Minerva Olympiad Avg.Qwen2.5-Math-1.5BBase 3.3 20.0 39.6 7.7 24.9 19.10GRPO 13.3 57.5 71.8 29.0 34.1 41.14Pos Only 10.0 57.5 70.6 30.1 31.0 39.84+NTHR 16.7 57.5 70.8 30.5 34.2 41.94Qwen2.5-0.5B-InsBase 0.0 2.5 33.4 4.4 7.0 9.46GRPO 0.0 7.5 33.8 9.2 8.1 11.72+NTHR 0.0 10.0 36.6 8.1 8.6 12.66Qwen2.5-1.5B-InsBase 0.0 22.5 53.0 19.1 20.7 23.06GRPO 3.3 32.5 57.2 18.8 23.0 26.96+NTHR 6.7 35.0 58.8 21.0 20.9 28.48Qwen2.5-Math-1.5B (deepscaler)Base 3.3 20.0 39.6 7.7 24.9 19.10GRPO 10.0 42.5 72.4 32.4 31.9 37.80+NTHR 16.7 47.5 73.2 29.4 31.4 39.60Qwen2.5-3BBase 10.0 37.5 58.6 26.1 24.6 31.36GRPO 6.7 35.0 66.6 31.2 29.9 33.88+NTHR 10.0 47.5 65.6 31.6 26.8 36.30Table 5.3: Results across selected math benchmarks for different Qwen2.5models and methods. The proposed method consistently provides averageperformance gains on various models.5.5 Conclusion and DiscussionsThis chapter covers a wide range of topics related to LLM finetuning, mak-ing it somewhat dense in content. To aid the reader in recalling the keyideas, we summarize the main takeaways from various perspectives in thisconcluding section. In addition to the core insights, we also highlight some865.5. Conclusion and Discussionsrelated works, outline potential future research directions, and discuss thelimitations of our proposed learning dynamics framework. To facilitate com-parison across different finetuning methods and help readers recall the maincontributions of each section, we provide a summary in Figure 5.20.5.5.1 Summary of each SectionThis chapter begins with an overview of LLM finetuning methods in Sec-tion 5.1. By discussing both the benefits and potential risks introduced byfinetuning, we highlight the need for a fine-grained behavioral analysis ofmodel dynamics. Such an analysis, we argue, could offer new insights intothe advantages and limitations of various LLM finetuning algorithms.We then apply our learning dynamics framework to the most fundamentalfinetuning approach, i.e., supervised finetuning (SFT), in Section 5.2. Withthe help of the teacher-forcing mechanisms and causal mask, we successfullyextend the force analysis and AKG decomposition to a multi-token, auto-regressive setting. We then circumvent the intractability of modeling thefull response space by instead tracking the evolution of model confidenceon selected representative responses. This framework provides novel expla-nations for various empirical observations during SFT training, includingnon-monotonic confidence trajectories and distinct inflection points in thelearning curves of different responses types. Furthermore, we offer a novelhypothesis for why hallucinations are often exacerbated after SFT. Thisexplanation is also supported by multiple recent empirical studies.Building on this foundation, Section 5.3 extends the framework to DirectPreference Optimization (DPO), a more complex off-policy finetuning al-gorithm. Despite its structural differences, DPO also updates parametersvia âÎ¸ log Ï. Hence, we can still derive an AKG-style decomposition forDPO through gradient chain rule expansions. A key distinction is the pres-ence of explicitly imposed negative gradients. Our theoretical and empiricalanalyses reveal that these negative gradients introduce a counter-intuitivesqueezing effect, which causes several undesirable behaviors, such as thedegradation of both log ÏÎ¸t(y+) and log ÏÎ¸t(yâ) over time, and the amplifi-cation of the so-called ârepeater phenomenonâ. Motivated by this analysis,we propose a simple yet effective pipeline to mitigate these issues.In the final section, Section 5.4, we extend our analysis to Group RelativePolicy Optimization (GRPO), an even more complex and dynamic finetun-ing method based on on-policy reinforcement learning. GRPO is commonly875.5. Conclusion and Discussionsapplied in reasoning-intensive tasks. Unlike SFT or DPO, GRPO exhibitsvarying equivalent learning rates (eLRs) at the token level, and its train-ing responses are updated in every iteration. These characteristics makethe response-level force analysis insufficient. To address this, we developa token-level extension of our framework. By analyzing a simplified Un-constrained Feature Model (UFM) (previously used in the analysis of thesqueezing effect), we define a token-wise influence score capable of identify-ing harmful tokens before the parameter updates. Leveraging this insight,we introduce Negative Token Hidden Reward (NTHR), a novel method thatselectively reduces the influence of the most harmful tokens. Our experi-mental results across multiple math reasoning benchmarks demonstrate theconsistent superiority of NTHR over standard GRPO, highlighting both thepractical impact and theoretical grounding of our approach.5.5.2 More on Negative GradientRecently, numerous variants of DPO and GRPO have been proposed toaddress various limitations of the original algorithms. These follow-up worksare often designed to enhance robustness in specific settings, targeting issuessuch as instability, misalignment, or poor generalization. In Figure 5.19, wepresent a force analysis of several representative variants, while the detailedlearning dynamics decompositions of their loss functions are provided inAppendix B of our paper (Ren and Sutherland 2025).Although these methods are motivated by diverse intuitions and theoreticalconsiderations, we find that many of them can be naturally (and partly)interpreted within our learning dynamics framework. Interestingly, despitebeing independently developed, a common theme emerges across these ap-proaches: they often implicitly address similar core aspects of preference op-timization, such as making the training signals more on-policy or restrainingthe influence of negative gradients on low-confidence tokens.Self-play Finetuning (SPIN, proposed by Z. Chen et al. (2024)) serves as arepresentative example of how our proposed framework provides a new per-spective on understanding where its benefits come from. SPIN is motivatedby the observation that responses generated by smaller, pre-finetuned mod-els are generally of lower quality compared to those found in SFT datasets.Leveraging this insight, the authors treat self-generated responses y as neg-ative examples and the ground-truth responses y+u from the SFT dataset aspositive examples. Inspired by the loss structure of Generative Adversar-ial Networks (GANs, Goodfellow et al. 2014), SPIN maximizes the modelâs885.5. Conclusion and Discussionsâ¢ On-policy DPO, IPOð²u+ð²uââ¢ SPINð²u+ð²uââ¢ SPPOð²u+ð²uââ¢ SLiCð²uâ ð²u+ð²refTriggered whengap is smaller than ð¹Figure 5.19: Force analysis for some DPO variants.ability to distinguish between these two types of responses. After a seriesof derivations, the authors arrive at a loss function that is formally equiva-lent to the standard DPO objective, with the key distinction being that allyâu responses are generated on-policy from the current model, while y+u aredrawn from the static dataset. As in Figure 5.19, the on-policy nature ofthe negative responses in SPIN mitigates the squeezing effect efficiently.Furthermore, the SPPO method (a follow-up to SPIN) introduced by YueWu et al. (2024) takes this idea a step further by entirely eliminating theuse of negative gradients. From the perspective of force analysis, the sepa-ration between y+u and yâu can still be achieved by emphasizing their relativestrength, without explicitly penalizing the model through negative updates.Like SPIN, SPPO also relies on on-policy sampling, leading to more efficientand stable learning dynamics.Besides the previous examples where the algorithm alters the negative gra-dient on a global level, here are more examples where a response-level ortoken-level relative strength is controlled. Smaug (Pal et al. 2024), is a goodexample. Theoretically, it assumes that y+u and yâu differ by only a singletoken. Then, the modelâs confidence in y+u after the shared prefix is guar-anteed to decrease during standard DPO training (also an instance of theLLDisp problem). To mitigate this, it dynamically increases the learningrate for the positive gradient by up to 50 times (i.e., their λ = 50) wheneverthe paired yâu is detected to have a low confidence. This strategy alignswell with our force analysis: applying a strong negative gradient in valleyregions is harmful, and thus, amplifying the positive gradient (equivalently,decreasing the relative eLR for negative tokens) can alleviate the influence.Dynamically controlling the relative strength between positive and negativeupdates is also common in reinforcement learning, where it is often imple-mented through reward shaping with a small offset. For instance, a binary895.5. Conclusion and Discussionsreward between (0, 1) can be transformed into (â0.5, 0.5) or (â0.1, 0.9), thelatter of which makes the effective learning rate for positive examples 9times greater than that for negative ones. Notably, the dynamic advantagemechanism in GRPO performs a similar implicit modulation, adapting thegradient magnitude based on relative outcome quality.Dynamically controlling eLR also extends to domains where only negativeupdates are applied, such as machine unlearning. A representative exampleis the Negative Preference Optimization (NPO, Ruiqi Zhang et al. 2024),which modifies the standard DPO loss by removing the positive gradient partentirely. In this setting, the eLR for the negative gradient is dynamicallyadjusted based on the modelâs prediction confidence. This illustrates howgradient magnitude control, informed by prediction certainty, is a generaland effective tool for many tasks, like alignment, reasoning, and unlearning.In summary, our analysis reveals that negative gradients play an increasinglycritical role in LLM finetuning, particularly in the context of reinforcementlearning-based methods. Despite their widespread use, a comprehensiveunderstanding of how these gradients influence model behavior remains lim-ited within the community. We hope that the learning dynamics frameworkand empirical findings presented in this chapter, as well as in our recentwork (W. Deng, Ren, M. Li, et al. 2025; Ren and Sutherland 2025), serveas a foundation for future research. We believe that a deeper explorationof these dynamics holds the potential to yield new insights and guide thedevelopment of more robust and efficient finetuning algorithms.5.5.3 LimitationsWe now briefly discuss the limitations of our analytical framework in thecontext of LLM. The primary issue lies in the gap between changes in modelconfidence and the modelâs actual capabilities. For instance, an increase inconfidence on certain correct responses does not necessarily imply improvedalignment, generalization, or reasoning ability. The answer to such questionsis likely negative in many cases.In traditional supervised learning tasks, monitoring the loss function is ofteneffective for evaluation, since the test data is typically drawn from the samedistribution as the training set. In such cases, the empirical risk closelyapproximates the true risk. However, this assumption does not generallyhold for tasks involving LLMs, which are considerably more complex andharder to build distributional models for. Thus, a systematic evaluation905.5. Conclusion and Discussionsapproach is needed to bridge this conceptual gap.Another limitation is about the assumptions we made in our AKG decompo-sition. Specifically, in most of our analysis of Chapters 4 and 5, we usuallyonly consider â¥Ktâ¥F . But in fact, Kt is a V Ã V matrix which can alterthe direction of the updating force vector provided by Gt. Hence, a moredetailed analysis is required. We believe that theoretical results from theNTK literature may provide useful insights for future work.Furthermore, in the context of LLM finetuning (and more general finetuningmethods), the model often operates in the so-called âkernel regime,â whereK evolves slowly and remains relatively stable, thereby justifying approxi-mations such as those made in the UFM setting (G. Yang 2020). However,in the âfeature learning regime,â where internal representations (and henceK) undergo substantial changes during training, the usefulness of our anal-ysis becomes less certain. We will revisit this question in the next chapterby returning to a simpler supervised learning scenario.The final limitation we want to highlight is the oversimplification of theTransformer architecture in our current framework. As an empirical theoryintended to provide insights into model behavior, incorporating force analy-sis alongside core architectural components, such as the attention mechanismin Transformers, could yield deeper and more informative interpretations.In particular, we believe that integrating our framework with the circuithypothesis (Lynch et al. 2025) presents a promising direction for future ex-ploration.915.5. Conclusion and Discussionsâ¢â¢â¢â¢Static, off-policyEpoch-wise schedulerNoNoStatic, off-policyResponse-level, adaptiveYesNodynamic, on-policyResponse-level, token-levelYesYesMNIST AKG + multi-label + auto-regression SFT AKG + dynamic eLR+ negative gradientSFT AKG + tokenwise modeling+ only observe 1 dim+ consider UFMmore concise AKG+ influence scoreSqueezing effect+ one-stepintroduce UFM+ accumulated:pos scatter v.s neg uniteForce analysis on:+ ââ of rephrases+ inflection pointsimilar ï decay later+ hallucination:increase of ð²ðâ ð¢+Huge Y space+ prob datasetdiff. rephrases Force analysis on:+ ð²ð¢+ and ð²ð¢â â+ ð²â ââ fast+ SFT then DPOmake the trendmore obvious+ extended SFTmitigate this effectForce analysis on:+ LLDisp on some ð²ð¢++ tokens in ð²ð¢â is the key+ harmful tokens are correct tokens with negative eLR(idea: reduce hallucination)Use fine-grained promptto reduce the similaritybetween ð±ð¢+ and ð±ðâ ð¢+Extended SFT pipelineThe win-rate is increasedRepeater is mitigatedNTHREleminate harmful tokensin negative responsesImproved reasoningcapability on math problemsFigure 5.20: A brief summary of this chapter.92Chapter 6Feature Adaptation inTransfer LearningIn previous chapters, the âobjectiveâ in our force analysis has been themodelâs log-probability. In this chapter, we extend this framework by treat-ing the hidden representations as the objective. This shift is motivated bythe belief that internal representations (or even some hidden structures likeattention patterns, circuits, etc.) are critical for understanding how deepneural networks internalize knowledge and generalize to downstream tasks.Compared to the LLM finetuning discussed in Chapter 5, where the modeloften operates in a âNTK regimeâ with almost static features, the trans-fer learning scenarios considered in this chapter involve a nontrivial degreeof feature adaptation. To facilitate a clean and interpretable analysis, weadopt a simplified and controlled setting: selecting a well-pretrained back-bone model, followed by attaching a task head and then performing full-parameter finetuning. This is usually named as a pretraining-finetuningpipeline, which is quite common in different fields. Although our currentfocus is on representation-level adaptation, the techniques presented herecan be extended to more complex and abstract internal mechanisms.Despite this controlled setup, our experiments span a diverse array of tasks,including self-supervised image learning, image classification, molecular prop-erty prediction (both classification and regression), and image segmentation.They also involve architectures ranging from CNNs and GNNs to Transform-ers. The goal is to verify that the principles uncovered via learning dynamicsare both general and model-agnostic. We believe this justifies the potentialapplicability of our analysis to more complex systems such as LLMs.Finally, much of the content in this chapter is based on âHow to PrepareYour Task Head for Finetuningâ (Ren, S. Guo, et al. 2023). While thepaper emphasizes practical finetuning strategies, this chapter reframes theanalysis through the lens of learning dynamics, with a particular focus on936.1. Introductionfeature adaptation. In short, we argue that effective feature adaptationrequires the energy provided by the gradient updates to be both consistentand sufficient. Moreover, this force should not be overly distorted by therandomly initialized task head in the early stages of finetuning.6.1 IntroductionIn the era of deep learning, pretraining a model on a large dataset andadapting it to downstream tasks is a popular workflow. With the help of alarge amount of data and huge computing resources, the pretrained modelcan usually provide beneficial features for the downstream tasks. Such aframework is proven to be efficient and effective in many domains and tasks,e.g., natural language processing, computer vision, graph-based learning,and so on. Although different variants of finetuning methods are widelyapplied, e.g., direct finetuning, finetuning after linear probing (Kumar et al.2022), side-tuning (J. O. Zhang et al. 2020), using different learning rates fordifferent layers (T. Zhang et al. 2021), and more, a detailed understandingof how features are adapted during finetuning remains elusive.Among the various topics within the pretrain-finetune pipeline, feature adap-tation plays a central role in understanding how knowledge embedded in apretrained model is transferred to downstream tasks. It is a common beliefthat most of the knowledge is embedded in the modelâs hidden features, andthe task head plays a role of linear separation on top of them (Goodfellowet al. 2016). This topic is also intimately connected to broader challengesin transfer learning and out-of-distribution (OOD) generalization.Unlike many existing approaches in transfer learning that assume a specificdistribution shift and derive generalization bounds accordingly, this chapterfocuses on the geometric adaptation of hidden representations in deep neuralnetworks. Leveraging the force analysis and learning dynamics frameworkintroduced earlier, we provide a detailed characterization of feature adapta-tion using core notions such as âenergyâ and âdirection.â As in other topicsdiscussed in this thesis, the lens of learning dynamics offers valuable insightinto the process of feature adaptation and further motivates the design ofeffective training strategies in practice.946.2. Learning Dynamics and Feature AdaptationPT-backboneDS-backbonecopyPThead PT-lossDS-head âHP(ð©, ð²)â ð¥DS-backbone DS-headð¥ð¥ âFT(ð©, ð²)BackpropBackpropð¡ð = ðð(ð±ds) ð³ = ðð°(ð¡)â¢ Pretraining (PT) stageâ¢ Head-probing (HP) stageâ¢ Full-tuning (FT) stageðð¡ð­ð±ptð±dsð±dsFigure 6.1: The training pipeline considered in this chapter.6.2 Learning Dynamics and Feature Adaptation6.2.1 Background and MotivationWe start by introducing a popular transfer learning pipeline studied byKumar et al. (2022). As demonstrated in Figure 6.1, it consists of threesequential stages:â¢ Pre-training (PT) stage: This stage involves training a backbone net-work, either through standard supervised learning or self-supervisedlearning methods.â¢ Head-probing (HP) stage: We copy all (or part) of the pretrainedbackbone from the PT stage and attach a randomly initialized task-specific network, referred to as the task head. During this stage, thebackbone parameters are frozen, and only the task head is trained viagradient descent on the objective LHP(p, y).â¢ Full-tuning (FT) stage: Building on the HP stage, we unfreeze thebackbone and jointly train the entire network, including both the back-bone and the task head, using the objective LFT(p, y).956.2. Learning Dynamics and Feature AdaptationFigure 6.2: A transfer learning experiment on ReNet18 (pretrained onImageNet-1K and finetuned on STL10 (Coates et al. 2011).Our primary focus in this chapter is the adaptation of the hidden featurerepresentation, specifically examining how it evolves before and after thefull-tuning stage, as the emoji in Figure 6.1.The motivation comes from rethinking the analysis of Kumar et al. (2022).Specifically, they claim that directly full-tuning the model with a randomlyinitialized task head can distort the well-learned features, and hence harmthe out-of-distribution (OOD) performance significantly. They thus proposeto freeze the backbone and train the task head to its convergence first, andthen finetune the full network together. They theoretically show that afully converged task head will make hidden features almost stable duringfinetuning, and hence preserve the useful features for the downstream tasks.However, intuitively, we sometimes might expect the hidden features toadapt to the downstream tasks to some extent, depending on the similaritybetween the source and target distributions. This idea is supported by theexperiments presented in Figure 6.2. While a fully converged HP stage doesoutperform vanilla finetuning (consistent with the results of Kumar et al.2022), we observe that early-stopped HP, where the HP-accuracy is roughly85%, often achieves even better performance.In these experiments, all FT runs are conducted in identical settings, withthe only difference being the number of epochs spent in the HP phase. Thissuggests that the initialization of the task head at the beginning of FT playsa critical role: it implicitly determines the extent to which the representationis allowed to adapt, thereby influencing the final downstream performance.966.2. Learning Dynamics and Feature Adaptation6.2.2 Formalizing the Feature AdaptationWe now formalize the feature adaptation to describe how it changes. Al-though in practice the backbone model fθ is often highly complex, suchas a stack of convolutional blocks in ResNet or several attention layers inTransformers, for theoretical tractability, we consider a simplified Overpa-rameterized Model (OPM) in this chapter. This model is closely related tothe Unconstrained Feature Model (UFM) introduced in Chapter 5, wherethe task head gw is modeled as a simple linear mapping with w â RVÃh.The main difference between them is that in OPM, we further specify thehidden feature h to an overparameterized linear mapping.In the OPM, the feature representation is defined as h = Bx â RhÃ1, whereB â RhÃd is a large matrix parameterizing the backbone function fB, andx â RdÃ1 is the input data. Importantly, the hidden dimension h is allowedto be significantly larger than the input dimension d, which justifies the termâoverparameterized.âWe assume that B0 is a pretrained backbone obtained from a source task,such that the initial feature h0 = B0x already provides a meaningful repre-sentation. The objectives of the post-training are to leverage this pretrainedbackbone B0, along with an initialized task head w0, to achieve strong per-formance on the target downstream task. Since our target is to observe theadaptation of h and the head-probing stage does not change h, we first focuson the full-tuning stage. Supposing we have N data samples (xn, yn)Nn=1, thecross-entropy training loss function is then written as:LFT = 1NNân=1eâ¤yn log p; p â Ï(z); z â g(h) = wh; h â f(x) = Bx (6.1)where Ï(Â·) is a standard Softmax function and pn = Ï(wBxn).Then, the one-step evolution of the hidden representation ht is12:ht+1o â hto = âÎ·1NNân=1ï£«ï£¬ï£­Kt(xo, xu)ï¸¸ ︷︷ ï¸¸slow-change(âhzt(xu))â¤ï¸¸ ︷︷ ï¸¸direction(pt(xu)â eyn)ï¸¸ ︷︷ ï¸¸energyï£¶ï£·ï£¸+O(Î·2),(6.2)where Kt(xo, xu) is still the eNTK and the Gt term is also pâ e.12The proof is similar to previous chapters; details are given in the paper (Ren, S. Guo,et al. 2023).976.2. Learning Dynamics and Feature AdaptationThe primary differences between the decomposition used here and thosestudied in Chapters 3 to 5 lie in the following two aspects. First, since wenow track the evolution of the hidden representation h rather than the log-probability log p(xo), the A term is no longer present in the decomposition(or, equivalently, it is simply the identity). Second, by explicitly splittingthe network at the hidden embedding h, the K term now corresponds to theeNTK of the backbone network fB. The parameters mapping h to the outputare captured separately as the âdirectionâ term mentioned earlier, specifi-cally âhzt(xu). Under the simplified linear setting assumed in this analysis,this gradient becomes constant across all n examples, i.e., âhzt(xu) = wt.Note that wt plays a very important role in our analysis here: in the forward-pass, wt is operating as a hyper-plane that separates the features of differentclasses; in the backward-pass, it is projecting the energy provided by Gttowards representations. Furthermore, this wt also keeps changing duringthe full-tuning process, which makes the analysis more complicated. We willelaborate on this later in this chapter.We then check the head-probing stage. From the modelâs perspective, all theinfluence imposed via the head-probing stage is through the value of w0, i.e.,the task head before the start of full-tuning. This w0 is also the task headat the end of the HP stage, which is then correlated with the HP-accuracy,as demonstrated by the red curve in Figure 6.2. If we further assume thatthe data used in HP and FT stages are identical, this HP-accuracy is alsothe starting FT-accuracy at t = 0.6.2.3 Factors that Matter: Energy and DirectionNow, we can first describe the evolution of the hidden representation fromt = 1 to t = T using the L2-norm in the following proposition:Proposition 1. Exnâ¥hTn âh0nâ¥2 â¤ c Â·EAIE , where EAIE â Exnâ¥p0nâeynâ¥2 isthe Average Initial Energy (AIE). Here c is a constant, T is the FT epochs,and p0n is the modelâs prediction of sample xn at the beginning of FT (orequivalently, at the end of HP).Although this bound is not so tight13, the underlying phenomenon is quiteinspiring. Specifically, the result states that if we interpret the average L2-distance between the modelâs predictions and their corresponding one-hot13You can find it in Appendix A of our paper (Ren, S. Guo, et al. 2023). Note that thenotations used there differ slightly from those in this thesis.986.2. Learning Dynamics and Feature Adaptation(a) Influence of Ï on energy. (b) Influence of Ï on direction.Figure 6.3: Left: statistics of [p0]y and [ey]y under different HP epochs.Right: approximated change of the âdirectionâ term in Equation (6.2) un-der different Ï . Toy+MLP means pretrain a 4-layer MLP on full MNIST,then transfer to a distorted subset of MNIST. Res50-IN1K(Sup)-STL meanspretrain a ResNet50 on ImageNet-1k using supervised classification, thentransfer to a downstream task on STL10.labels as a form of âenergy,â then the L2-norm of the feature adaptationis upper bounded by this energy. In other words, greater energy inducesgreater feature adaptation, and conversely, lower energy limits the extent ofadaptation. Since the loss function in the HP stage is non-decreasing, thenumber of HP epochs (defined by Ï) is a reasonable approximation of AIE,as illustrated in Figure 6.3-(a). To draw this figure, we collect the value of[p0n]yn , i.e., the predicting probability of the correct label, for all trainingexamples and put them in 10 bins. It is clear that a larger Ï makes theassociated p0 closer to the one-hot distribution.Following Proposition 1, we can link the adaptation of features to Ï via AIE.In the proof, we disentangle the dependence of the direction and energyterms using the Cauchy-Schwarz inequality, i.e., (âhz)â¤(p0âey) â¤ â¥âhzâ¥2 Â·â¥p0 â eyâ¥2. However, the direction term Equation (6.2) indeed plays animportant role, especially at the beginning of finetuning.To verify our hypothesis about the direction term, we depict the change ofthis term during finetuning in the right panel of Figure 6.3. Actually, underour simplified setting, this quantity can be approximated by the norm of thegradients to wt, since â¥âhzt+1 ââhztâ¥2F = â¥wt+1 â wtâ¥22 = Î·2â¥âwLâ¥22. Aswe find â¥wtâ¥2 changes little during the finetuning stage, the large â¥âwLâ¥2is more likely from a big direction change. As illustrated by Figure 6.3-(b),when Ï = 0, âhzt changes a lot at the beginning of FT, which can makeh change in inconsistent directions. When Ï = 1024, the direction termchanges only a little through finetuning. Introducing the directional change996.2. Learning Dynamics and Feature AdaptationFigure 6.4: How hT changes(assume full-tuning T epochs to converge) com-pare with h0 when head-prob Ï epochs.of wt allows for a more comprehensive understanding of feature adaptation.We will elaborate on this in the next section.6.2.4 How Features Evolve under Different SettingsThe findings regarding the direction term inspire us to look deeper at thedifference between a strong adaptation (e.g., Ï = 0) and a mild adaptation(e.g., Ï = 32). To get an overall picture of hâs change, only observing â¥hT âh0â¥2 is not enough. Hence, we analyze the following four quantities related tothe similarity between the features before finetuning and afterwards: â¥hT âh0â¥2, â¥hT â¥2, â¨hT ,h0â©, and cos(hT ,h0). Before delving deep into how weget insights on how these values change, we first describe these interestingtrends using the following proposition.Proposition 2 (Informal). In an overparameterized two-layer linear model,when Ï increases (the AIE decreases), â¥hT â h0â¥22 monotonically decreaseswhile â¥hT â¥22 and â¨hT ,h0â© exhibit a quadratic trend. The trend of cos(hT ,h0)is hard to predict, but there is a phase in which this value increases fast.Figure 6.4 provides a clear demonstration of how these four quantities changewhen head-probing Ï epochs. We also empirically verify these trends in dif-ferent settings, e.g., training a ResNet18 on CIFAR10, training a ResNet50on a subset of IN1K, and training a Graph Convolution Network (GCN,Kipf and Welling 2016) on a Molecular-pcba dataset (a benchmark withmulti-label classification task, Weihua Hu et al. 2020). The results alignwith our proposition quite well.With the help of this proposition, we can conceptualize feature adaptationas a geometric transformation in vector space. In the upper-left panel of1006.2. Learning Dynamics and Feature AdaptationFigure 6.5: Up: A vector-space illustration of how features adapt duringfinetuning. Bottom: Empirical verification of the vector-based intuitionusing a toy example. From left to right: strong, mild, and tiny adaptation.Figure 6.5, we compare various adapted features hT originating from thesame initial feature h0, each corresponding to a different initialization w0obtained from the head-probing stage at different epochs.When Ï is relatively large (implying lower energy), we observe only mildor tiny adaptation. As shown in the curves of Figure 6.4: â¥hT â h0â¥22 issmall; â¥hT â¥2 and â¨hT ,h0â© remains similar to h0; and cos(hT ,h0) changesonly slightly. This indicates that hT has been slightly stretched along thedirection of h0, preserving directional alignment.In contrast, when Ï is small and the adaptation becomes strong, we observea sharp drop in cos(hT ,h0) and a corresponding decline in â¨hT ,h0â©. Thissignifies that hT is no longer aligned with h0; the direction of the featurevector has changed substantially. In other words, the original representationhas been significantly altered and now points to a different direction.The upper-right panel of Figure 6.5 illustrates the three levels of featureadaptation in a 3D vector space. This visualization also provides intuitionfor why mild adaptation is often preferable for downstream tasks. Specif-ically, when adaptation is minimal, the two feature clusters (e.g., the blueand red ellipses) remain largely overlapping, limiting their separability andthereby reducing classification performance. In contrast, a mild adaptation1016.2. Learning Dynamics and Feature Adaptationgently stretches hT , increasing the distance between clusters and enhancingtheir separability, which is a beneficial effect for downstream classification.However, in the strong adaptation case, the features are altered excessively,which can distort the original features and ultimately degrade performance.This trend is further validated using a 3-class toy example shown at thebottom of Figure 6.5, where we visualize the evolution of ht via t-SNE(Van der Maaten and Hinton 2008). The transparency of each point en-codes the time step t during the full-tuning process, and the three panelscorrespond to strong, mild, and tiny adaptation regimes, respectively. Asclearly shown, the level of adaptation correlates with cluster separation andstructure preservation, aligning with our theoretical intuition.6.2.5 Proof Sketch of the TrendsBefore turning to the practical strategies motivated by our analysis of fea-ture adaptation, we first provide a sketch of the proof for Proposition 2 (aregression problem version), whose conclusion may not be immediately in-tuitive. The detailed proof can be found in Appendix D of our paper (Ren,S. Guo, et al. 2023).The proof begins by explicitly writing down the quantities we want to track:â¢ Euclidean distance: d¯euc â Exâ¼D[â¥ht â h0â¥22];â¢ Dot product: d¯dot â Exâ¼D[ht · h0];â¢ Cosine: d¯cos â Exâ¼D[ht·h0â¥htâ¥2ââ¥h0â¥2].Then, under OPM that h = Bx, we can simplify them to:d¯euc =M â[tr(Bt · Bt)â 2tr(Bt · B0) + tr(B0 · B0)]d¯dot =M â tr(Bt · B0)â¥htâ¥22 =M â tr(Bt · Bt); â¥h0â¥22 =M â tr(B0 · B0), (6.3)where M â Exâ¼D[tr(xxâ¤)] is a constant in the following analysis.We then use b â vec(B) â RhâdÃ1 to represent the column-wise concatena-tion of the hà d matrix. Then, all the traces involved in the distance abovecan be simplified using the fact that tr(Bt · B0) = bt · b0:â¢ Euclidean distance: d¯euc â bt · bt â 2bt · b0 + b0 · b0;â¢ Dot product: d¯dot â bt · b0;1026.2. Learning Dynamics and Feature Adaptationâ¢ Cosine: d¯cos â bt·b0âbt·btâb0·b0 .Recall that we want to link the value of Ï to different trends of these dis-tances, and the value of Ï can only influence these distances via the followingpath: Ï â HP-lossâ w0 â Bt â bt â d¯. To get more insights, we approx-imate the behavior of this model in the NTK regime, in which the convergedparameters can be analytically calculated. Specifically, be applying Equa-tion (8) in J. Lee et al. 2019 and assuming tââ, we can have:bt = b0 â (âbq0)â¤Îºâ10 (q0 â Y ), (6.4)where q0 â RNÃ1 is the modelâs prediction on N training samples at thebegining of full-tuning, i.e., t = 0. And κ0 = (âbq0)â¤âbq0 â RNÃN is theeNTK on X, where X â [x1, . . . , xN ] â RNÃd. We use κ here to representthe eNTK on the full dataset, which is calculated slightly differently fromour pair-wise Kt term in other chapters. Specifically, since we assume V = 1in this part, each K(xi, xj) can be considered as an element in κ here.Then, we can write down bt Â·b0 or bt Â·bt by representing bt using b0 followingEquation (6.4). For example, after several steps, tr(Bt · B0) can be writtenas the following quadratic form:tr(Bt · B0) = bt · b0 = b0 · b0 â (âbq0 · b0)â¤Îºâ10 (q0 â Y )= b0 · b0 â qâ¤0 Îºâ10 (q0 â Y ), (6.5)where âbq0 · b0 = q0 is proved in our Lemma 1 in Ren, S. Guo, et al. 2023.Since κ0 = X(B0 · B0 + â¥w0â¥22 · IdÃd)Xâ¤ is usually positive definite, thisquadratic form has a global maximum at qâ0 =12Y .Similarly, we can calculate the critical point of Bt · Bt following a similarprocedure. We omit the details due to the page limits and only summarizethe trend of different distances and quantities in Table 6.1 and Figure 6.6.Remember when q0 = Y , the gradient of B0 is zero, hence ht = h0 (the idealcase in Kumar et al. 2022). When q0 moves linearly from Y to 0, we havethe following three phases as demonstrated in Table 6.1, which is roughlyhow we define âtinyâ, âmildâ, and âstrongâ adaptations.â¢ Tiny, Y â 12Y : ht lengthens its norm with a slight change in thedirection, hence d¯dot also increases;â¢ Mild, 12Y â αY : the norm of ht keeps increasing, but its directionbegins to change, which makes d¯dot decrease;1036.3. Strategies to Control Adaptationâ¢ Strong, αY â 0: the norm of ht begins to decrease and the anglebetween ht and h0 drastically changes, which makes ht changes a lot.q0 = AIE d¯euc d¯dot â¥htâ¥2 d¯cosY â 12Y â â â â ? (â)12Y â αY â â â â ââÎ±Y â 0 â â â â ? (â)Table 6.1: Question marks in the last column mean we cannot accuratelypredict its change. But it usually decreases in experiments.ð¡ð â ð¡ð22YY/2ð¼Y0ð¡ð , ð¡0YY/2ð¼Y0ðð 22 + ð1ðð 22ð¡ð 22YY/2ð¼Y0ðð 22 + ð2ðð 22Figure 6.6: Illustrations of different metrics if they are 1-D quadratic func-tions. The curves in Figure 6.4 are a summarization of them.6.3 Strategies to Control AdaptationWe already have a good understanding of how the features adapt underdifferent settings from the analysis inspired by learning dynamics. Thenext step is then design good HP-FT algorithms to further improve thedownstream performance. The general principle considered here is a âphe-nomenon â hypothesis â solutionâ workflow, in which we use the valida-tion performance for verification. For example, we can have: HP train-accconverge to 100% â not enough energy â use smaller Ï .However, it is hard to know how much energy is beneficial, as the neuralnetwork might not encode the input samples as we humans expect. Hence,we suggest starting from the basic setting, i.e., using a linear head andsweeping Ï to get a high-level understanding of how the energy influencesthe performance. Usually, selecting the optimal Ïâ using validation accuracycan ensure a reasonably good test result. Figure 6.7 provides a good exampleof doing such a sweep under different settings. It is clear that in most cases,1046.3. Strategies to Control AdaptationFigure 6.7: Sweeping Ï from 0 to 200 (in 2n fashion). The valid accuracyof the full-tuning-only setting is the leftmost point in each panel. The first4 columns are on the image classification task, the fifth one is on the graphtask, and the last one is on the image segmentation task.HP to convergence is not the optimal solution, meaning that leaving enoughenergy for feature adaptation would be beneficial.Other than the basic Ï sweeping methods for controlling the adaptationenergy, we also provide some advanced tricks, which are only applicable tospecific scenarios and need to be more careful. For example, if we really wanttiny energy but using the linear head only achieves less than 50% trainingaccuracy after head probing, we can consider an MLP head to increasethe converged training accuracy (hence reduce the energy). If we want amild adaptation, but the training accuracy during HP goes to 100% toofast, using label smoothing during HP can be considered. If we believe thedownstream task only needs low-level features of the backbone, copying theparameters of only one part of the backbone is a good choice. However, theseadvanced tricks also have side effects that are deleterious to the downstreamperformance. Here is a brief introduction to some of them, and the detaileddesign and experimental verifications can be found in our paper.MLP Head: Instead of using a linear head, X. Chen et al. (2020) claimthat using an MLP head can improve the headâs separation ability and henceprovide more tolerance for the featureâs quality. Following the analysis ofthis paper, we can consider this trick when we want small energy, while thelinear head cannot reach a high HP training accuracy even after a long HP.For example, in the first two columns in Table 6.2, the HP-train-acc in thelinear head cases plateaued after reaching 92% or 78%, while a 2-layer MLPcan reach 99% and 95%. As the energy decreases, features adapt less duringfinetuning and the final models generalize better.However, we should be careful when applying this trick if we want to use a1056.3. Strategies to Control Adaptationsmall Ï (i.e., large energy). Because a larger head will make the inconsistentdirection problem of âhzt more serious. Hence, increasing the head capacityin this case would make the head converge more slowly and hence prolongsuch a chaos phase.Sim-DomainA Sup-DomainB Sim-STL Sup-STL Sim-STL Sup-STLHP-train-acc 92.676 78.213 96.875 100 97.607 1001-cos(hT ,h0) 0.1269 0.1422 0.0044 0.017 0.0049 0.0256â¥hT â h0â¥22 5.247 4.752 1.792 6.716 1.731 7.391FT-val-accBaselineLinear-head78.075 61.545BaselineSmall energyηft = 1ηhp = 1 93.914 97.581Small energyηft = 0.9ηhp = 0.9 94.015 97.694HP-train-acc 99.121 94.883 96.875 100 97.982 1001-cos(hT ,h0) 0.1098 0.1104 0.014 0.0549 0.0109 0.0849â¥hT â h0â¥22 4.942 4.206 1.841 9.278 3.041 13.881FT-val-acc2MLP-head78.453 63.773More energyηft = 1ηhp = 0.9 94.304 97.92Opposite Energyηft = 0.9ηhp = 1 93.208 97.039Table 6.2: How lsHP and larger heads influence the feature adaptation.The models are pretrained using SimCLR (Sim, T. Chen et al. 2020) orsupervised (Sup) classification on IN1K. Numbers in blue (red) represent adecrease (increase) compared with their counterparts in the baseline.Label Smoothing HP: Recall that the energy is upper bounded by â¥ey âp0â¥22, where p0 is the modelâs prediction after HP for Ï epochs. p0 convergesto the labels used in HP when Ï ââ. Hence, instead of changing Ï , we canalso manipulate the labels in HP to achieve a similar goal. One simple yeteffective way is label smoothing (Mu¨ller et al. 2019). By setting the labelsduring HP as ηhpey +1âÎ·hpV â u, where u is a uniform V -class categoricaldistribution, the HP stage can always reserve at least (1â ηhp) â â¥ey â uâ¥2energy for the following feature adaptation, even Ï ââ. Such a trick (lsHPfor short) is quite helpful when the HP-train-acc converges to 90%+ veryfast, yet we still want a mild adaptation, like the example shown in thesecond two columns in Table 6.2. With lsHP, we see that the features adaptmore, even though the HP-train-accs are unchanged.To verify that the aforementioned improvement comes from the reservedenergy during lsHP, we further try using smoothed labels during full-tuning(e.g., ηft = 0.9). The results match our analysis: when ηhp = ηft = 0.9,the reserved energy disappears, as the labels of the two phases are the sameagain. Hence, all the numbers under this condition are quite similar to thebaseline case (ηhp = ηft = 1). For the âopposite energyâ case, we observe alarger adaptation but a worse test performance. That is because the reservedenergy makes the features adapt in opposite directions. These results remindus that if we decide to use smooth label in both FT and HP (e.g., we assumemost of the samples in the downstream dataset are ambiguous), we need toset ηhp â¤ ηft to ensure a correct force direction.In summary, the lsHP trick is suitable for scenarios where the pretrained1066.4. Conclusion and Future Worksfeatures are pretty well and the standard HP converges to 90%+ very fast.When the HP-train-acc is too low, the assumption used in HP, i.e., p0 con-verges to the labels, no longer holds. Then, introducing the lsHP would makethe modelâs behavior more unpredictable, and should hence be avoided.6.4 Conclusion and Future WorksThis chapter extends our learning dynamics framework from tracking log-probabilities to analyzing the adaptation of hidden feature representations.Although the resulting decomposition differs slightly from earlier cases, thecore concepts of âenergyâ and âdirectionâ remain applicable.Motivated by this new decomposition, we find that the standard Head-Probefollowed by Full-Tune (HPâFT) pipeline proposed by Kumar et al. (2022)may underestimate the importance of feature adaptation, particularly whenthe source and target tasks differ substantially. To better characterize thisprocess, we analyze the learning dynamics of the hidden feature h underthe overparameterized model (OPM) assumption. Beyond the conventionalL2-distance, we introduce four additional quantities to capture the natureof feature adaptation: â¥hT â¥2, â¨hT ,h0â©, â¥hT â h0â¥2, and cos(hT ,h0).The non-trivial trends uncovered by these metrics, illustrated in Figure 6.4and validated through both theoretical analysis and empirical experiments,provide insights into how features evolve under varying energy conditions.Specifically, when the energy is low (i.e., in the tiny or mild adaptationregimes), the final feature hT is gently stretched along the direction of thepretrained feature h0. In contrast, under high-energy conditions, hT tendsto be severely distorted, as visually illustrated in Figure 6.5.Based on these observations, we speculate that a suitable adaptation energyof hT is the key when the pretrained features are not perfect for the down-stream tasks. Under different experimental settings, we illustrate how toachieve better downstream performance by controlling feature adaptations.Finally, there are still many open questions not addressed here. For exam-ple, methods to quantitatively or even adaptively analyze the discrepancybetween the pretrain and downstream tasks would be very useful in knowingwhat kind of head probing to perform in advance. The learning dynamicsframework could also be further extended to more practical settings, ratherthan OPM. We left these interesting directions for our future work.107Chapter 7Simplicity Bias andCompositionalityThis chapter explores a fundamental topic in representation learning: sim-plicity bias and compositional generalization. Unlike previous chapters,which begin by introducing a specific system we want to analyze using learn-ing dynamics, we start here by discussing three closely related concepts:â¢ The Platonic Representation Hypothesis (Huh et al. 2024);â¢ Kolmogorov complexity (Kolmogorov 1963) and compression for AGI(Rae 2023);â¢ Learning speed advantage and simplicity bias (Valle-Perez et al. 2018).Upon examining their assumptions and implications, we find that they allconverge on a common principle: learning representations that adhere to Oc-camâs Razor, i.e., favoring simpler explanations that can lead to better gen-eralization. Interestingly, they observe that many large, well-trained modelsacross different modalities naturally exhibit this preference for simplicity.In other words, simplicity bias appears to be an inherent and ubiquitousproperty of modern deep learning systems.To investigate the origin of this bias, we study a compositional generalizationproblem through the lens of learning dynamics. Our empirical analysis showsthat simpler mappings are learned more quickly, a phenomenon that canbe well-explained using force analysis on model predictions, similar to thetechniques employed in earlier chapters. Motivated by this, we implement amulti-generation self-distillation approach, also known as iterated learning,to further amplify the simplicity bias. Experimental results across varioustasks and modalities strongly support our theoretical insights.While our method improves compositional generalization in practice, ourprimary goal in this chapter is to highlight how learning dynamics offer a1087.1. Papers Covered in this Chapternew explanatory framework for understanding simplicity bias. This micro-level perspective provides valuable insights into a foundational aspect ofgeneralization and may guide the development of more efficient algorithmsfor learning structured and robust representations.7.1 Papers Covered in this Chapter7.1.1 The Platonic Representation HypothesisThe Platonic Representation Hypothesis, introduced by Huh et al. (2024),proposes that high-performing deep learning models across different modal-ities and training paradigms tend to converge toward a shared internal rep-resentation space. Drawing inspiration from Platoâs theory of ideal forms,the authors suggest that this convergence reflects a common underlying sta-tistical structure of the world. The models in different modalities couldindependently discover this structure through their learning.The central point of the above discussion is the assumption that there ex-ists a unique (in terms of topology) and consistent generative mechanismunderlying the data we observe from the world, which is also one of our coreassumptions in this chapter, as in Figure 7.1. This assumption is implicitlyrelated to the âlow-dimensional manifoldâ hypothesis commonly applied inmany machine learning textbooks. In essence, it posits that the primarygoal of representation learning is to discover a mapping from input datato representations that faithfully capture the topological structure of thisunderlying generative mechanism. Once a model successfully learns such amapping, systematic generalization would, in principle, be achieved.Another important conclusion from their observations is that many high-performing models across different modalities tend to spontaneously con-verge to such ground-truth-aligned representations. In other words, mostwell-trained models tend to naturally follow Occamâs Razor, favoring sim-pler and more structured solutions. A particularly striking phenomenon,often summarized as âall good models are good in similar ways, while allbad models fail differently,â has been empirically demonstrated across arange of vision and language models. Huh et al. (2024) provide several hy-pothetical explanations, and we will focus on the most related one, i.e., thesimplicity bias, in this chapter.1097.1. Papers Covered in this Chapter(a) The Platonic Representation Hypothesis[ð, ð]ð¿ð¨ð¿ð©ðð¨ ðð©GenX(ð, ð)ViT on IN1K GPT on languageðð ðâð(ð±)ðð(ð³) ð²â ð¦, ð¦ð:ð â ð©ð ððGenX(ð,ð) GenY(ð, ð)(b) System model studied in this chapterFigure 7.1: The data-generating assumption, details in Section 7.2.7.1.2 Compression for AGIThe idea of compression for AGI was originally proposed in the early 1960sthrough Solomonoff induction, and it has recently regained attention dueto the success of applying scaling laws in pretraining the LLMs. The talkâCompression for AGI â (Rae 2023) provides a clear and compelling sum-mary of this hypothesis. In it, Rae argues that the ability to compress infor-mation, i.e., using the simplest representation to record all the observations,is not merely correlated with intelligence, but is in fact a synonym with it.This perspective revitalizes a long-standing belief in machine learning: thatbetter compression implies deeper understanding and reasoning.Rae supports this claim by examining how modern LLMs operate as ad-vanced lossless compressors. Trained via next-token prediction, these mod-els implicitly learn to compress vast corpora of human-generated text bycapturing patterns, structures, and semantics. As evidence, he shows thatLLMs outperform traditional compression algorithms, such as gzip and evendomain-specific compressors designed for the Hutter Prize benchmark.This notion of compression is also closely tied to the âMinimum DescriptionLengthâ principle, which serves as a practical approximation of Kolmogorovcomplexity. If we assume that all observed data are generated by a consis-tent underlying mechanism, as postulated by the Platonic RepresentationHypothesis, then achieving higher compression implies that the model haslearned a representation closer to the ground-truth generative process, and1107.1. Papers Covered in this Chaptervice versa. Following Raeâs line of reasoning, developing more capable mod-els is equivalent to building models that achieve better compression, whichin turn means they implement mappings with lower Kolmogorov complexity.However, a key challenge remains: how can we measure compression rateor Kolmogorov complexity in practice? Rae proposes an elegant proxy: theintegral under the training loss curve during next-token prediction. Thiscorresponds to the relative learning speed of the model. In other words, rel-evant learning speed, a measurable and practical quantity in deep learning,becomes a bridge connecting abstract theoretical concepts such as OccamâsRazor, compression, and Kolmogorov complexity.14By combining the insights from the Platonic Representation Hypothesis andCompression for AGI, we observe that many modern pretrained models ap-pear to be naturally following a trajectory toward better compression, evenwithout being explicitly optimized for it. This chapter seeks to answer acritical question:Why do models spontaneously adhere to this principle of simplicity bias?To do so, we start by analyzing this phenomenon in a highly simplifiedsetting, using learning dynamics as our primary analytical tool.7.1.3 Related Discussions in Our WorksThe materials presented in this chapter are drawn from several of our previ-ous works. Since the topics discussed above are very general, which is a bithard to formalize in detail, we hence narrow them down to compositionalgeneralization as a starting point. The initial idea of connecting composi-tionality with learning speed advantage originates from our paper, âCompo-sitional Languages Emerge in a Neural Iterated Learning Modelâ (Ren et al.2020). In that work, we studied a simple Lewis referential game (Lewis 2008)within the emergent communication framework. We argued that, much likethe human cognitive system, deep neural networks exhibit a preference forcompositional mappings. This bias manifests as a learning speed advantagefor simpler structures and can be progressively amplified through an iter-ated learning setup. To understand the roots of this phenomenon, we begananalyzing the mutual influence between examples during training, which isone of the themes of learning dynamics.14It is also important to note, however, that directly optimizing for learning speed canbe risky due to Goodhartâs Law (Strathern 1997).1117.2. Compositionality and CompressibilityBuilding on this foundation, our paper, âImproving Compositional General-ization Using Iterated Learning and Simplicial Embeddingsâ (Ren, Lavoie,et al. 2023), extended our analysis and algorithm to more general repre-sentation learning problems. The goal was to enhance systematic general-ization across different tasks and modalities. As a theoretical justification,we first propose a compositionality ladder and claim that relying on merelythe mutual information cannot separate compositional mappings from othergeneral bijective mappings. We then employed a group-theoretic tool andshowed that the Kolmogorov complexity can distinguish those compositionalmappings due to their structural compactness.However, since Kolmogorov complexity is not computable, we must findsome other mechanisms to understand this problem. Subsequently, we moveour focus back to the learning speed advantage discussed in Ren et al. (2020).By applying force analysis to the components of different mappings with dis-tinct simplicity, we offer a novel interpretation of why simpler mappings tendto be learned faster in gradient descent. These findings implicitly alignedwith the âcompression rateâ discussed by Rae (2023). In short, becausemodels cannot absorb all information in a single update, they must acquireknowledge sequentially. When the learning of one example increases themodelâs confidence on other related examples, the update is constructive.This leads to a simplicity bias: examples that form aligned forces duringtraining are more likely to be learned more efficiently, whereas examples thatexert contradictory forces are harder to learn. This interpretation is devel-oped further in our paper, âUnderstanding Simplicity Bias Toward Compo-sitional Mappings via Learning Dynamicsâ (Ren and Sutherland 2024).7.2 Compositionality and Compressibility7.2.1 The Data Generating AssumptionWe start by introducing the data-generating mechanism and its relationshipto representation learning in this section. As illustrated in Figure 7.1-(b), thesemantic generating factors, also known as latent variables, are divided intotwo groups: the task-relevant factors (or semantic generating factors) G =[G1, ..., Gm], and task-irrelevant (or noise) factors O. This division dependson our understanding of the task; for example, if we only want to predict thedigit identity of an image in the color-MNIST dataset (Arjovsky et al. 2019),then m = 1 and G1 represents the digit identity. All the other generatingfactors, such as color, stroke, angle, and possible noise, are merged into O.1127.2. Compositionality and CompressibilityIf we want to predict a function that depends on both identity and color,e.g., identifying blue even numbers, we could have G = [G1, G2] with G1 theidentity and G2 the color.Each input sample x â X is determined by a deterministic function GenX(G,O).The task label(s) y â Y only depend on the factors G and possible inde-pendent noise ϵ, according to the deterministic function GenY(G, ϵ). Note(x,O) â¥ (y, ϵ) | G, and that O, G, and ϵ are independent. The data-generating distribution P (x, y) is determined by the latent distributionsP (G) and P (O), along with the GenX and GenY. We assume GenX andGenY are fixed across environments (the ârules of productionâ are consis-tent), while P (G) and P (O) might change between training and test.For compositional generalization, we wish to model the problem of general-izing to new combinations of previously seen attributes: understanding âredcircleâ based on having seen âred squareâ and âblue circle.â Thus, we mayassume that the supports of P (G) are non-overlapping between train andtest. (If this assumption is not true, it only makes the problem easier.) Insummary, our goal is to find an algorithm A such that, when trained on adataset Dtrain â¼ Pntrain, A achieves small test risk RPtest(A(Dtrain)). HerePtrain and Ptest should satisfy these conditions:â¢ Ptrain and Ptest have G, O, ϵ jointly independent, and x = GenX(G,O),y = GenY(G, Ïµ).â¢ GenX and GenY are the same for Ptrain and Ptest.â¢ In challenging cases, we may have supp[Ptrain(G)]â© supp[Ptest(G)] = â .7.2.2 The Ladder of CompositionalityFor compositional generalization, we expect the model to extract atomicsemantic features from the training data, and systematically re-combinethem in a procedure akin to how the data is generated. We thus considera typical representation learning framework, which resembles the inverse ofthe data generation process in Figure 7.1-(b). We use a backbone hθ : X âZ to convert the input signal x into a representation z, and a task headgÏ : Z â Y to solve the given task based on that representation. Theprediction of the model is yË = (g â¦ h)(x).Intuitively, we would like our learned z to uncover the hidden G, and gÏ(z) torecover GenY(G, ϵ). We thus analyze how the relationship between z and Ginfluences the modelâs generalization capability, building off principles such1137.2. Compositionality and CompressibilityStage IVStage IIIStage IIStage I â¢ ð» ð ð³ > 0ð³ contains insufficient info. about ðâ¢ ð» ð ð³ = 0; ð» ð³ ð > 0ð³ contain irrelevent non-ð info.â¢ ð» ð ð³ = 0; ð» ð³ ð = 0ð³ only has info. in ð (bijection)but their structures can be differentâ¢ ð³ â ðisomorphic bijectionð³maintains the structure of ðTrain longerBigger modelData augmentationSelf-supervised LearningCompressibility andexpressivity pressuresFigure 7.2: The ladder of compositionality stating the requirements of zusing the entropy-related measurement. H(Â·) is the Shannon Entropy of adistribution. Details in Appendix A of our paper (Ren, Lavoie, et al. 2023).as information bottleneck (Tishby et al. 1999). Inspired by the âladder ofcausationâ (Pearl and Mackenzie 2018), we propose a âladder of compo-sitionalityâ in Figure 7.2, which outlining a series of conditions on z andG. We hypothesize that the compositional generalization roughly requiresreaching the highest rung of that ladder:Hypothesis 2. To generalize compositionally, the learned z should captureexactly the information in G and nothing more (G to z should be a bijection),and moreover it should preserve the âstructureâ of G (i.e. the mapping fromG to z should be an isomorphism).More on this hypothesis, the ladder, and relationship to models of disentan-glement (Higgins et al. 2018) are discussed in Appendix A of our paper (Ren,Lavoie, et al. 2023). In short, we find that a model trained using commonlearning methods relying on mutual information between input x and super-vision y cannot reliably reach the final stage of the ladder: it is necessary toseek other inductive biases in order to generalize compositionally.In the remainder of this chapter, we will demonstrate that the topologicalsimilarity between z and G, the Kolmogorov Complexity of the correspond-ing mapping M : Z â G, and the learning speed advantage can separate1147.2. Compositionality and Compressibilitythose high-compressed compositional mappings from other bijections (atleast in the scope of compositional generalization problem).7.2.3 Kolmogorov Complexity of Different BijectionsTo build a link between compositionality and Kolmogorov complexity, wecan first describe different bijections between z and G using group theory,and then apply the description length to compare the complexity of a typicalelement. Specifically, assuming z â Z,G â G and |Z| = |G|, the space ofall bijections between z and G is an isomorphism of a symmetric group S|G|.If G = [G1, ..., Gm] and each Gm has v different possible values, |G| = vm.For clarity in the analysis, we assume z also has the same shape. Then, anybijection between z and G can be represented by an element in Svm .The space of compositional mapping, which is a subset of all bijections, hasmore constraints. Recall how a compositional mapping is generated: wefirst select zi for each Gj in a non-overlapping way. Such a process canbe represented by an element in Sm. After that, we will assign differentâwordsâ for each zi, which can be represented by an element in Sv. As wehave m different zi, this procedure will be repeated m times. In summary,any compositional mapping can be represented by an element in the groupSmv â Sv â Svm , where â is the semidirect product in group theory. Thecardinality of Svm is significantly larger than Smv â Sv, and so a randomlyselected bijection is unlikely to be compositional. We thus have:Proposition 3 (Informal). For m, v â¥ 2, among all bijections, any compo-sitional mapping has much lower upper bounds of its Kolmogorov complexitythan a typical non-compositional mapping.We can prove this by constructing descriptive protocols for each bijection.As a compositional mapping has more reused rules, its description lengthcan be smaller (see Appendix A.4 for more discussions).We now link the compositionality to a formal description of the simplicity,i.e., Kolmogorov complexity. In the next section, we will showcase that thecompositional mappings are, in general, learned faster than other bijections,and hence link the learning speed advantage to simplicity bias.1157.3. Simplicity Bias and Learning Dynamics00 0110 1100 0110 1100 0110 11â Holistic â¡ Compositionalâ¢ Other â£ Degenerate00 0110 11â£â â¡â¢â¢(a) Four groups of mappings (b) Roles of them in hypothesis spaceFigure 7.3: Left: four groups of mappings in our Toy256 example. Right:their roles, reproducing Figure 7 of Huh et al. (2024).7.3 Simplicity Bias and Learning DynamicsThe discussions above link the concepts of compositionally, simplicity, andKolmogorov complexity under an idealized setting, which also aligns wellwith many related works. This section further links these concepts to thelearning speed advantage from a learning dynamics perspective. Specifically,we would first demonstrate that a neural network naturally favors simplermappings using experiments under manual settings, and then provide a de-tailed explanation of why such a tendency exists using force analysis.7.3.1 Toy256 Dataset and Different MappingsTo verify our claim, we consider a multi-label classification problem and cre-ate 256 different datasets (each only contains 4 examples) for each mappingM in our Toy256 setting. For example, the dataset for the compositionalmapping in Figure 7.3-(a) should be {(blue box, 00), (blue circle, 01),(red box, 10), (red circle, 11)}, where the label â01â means y¯1 = 0 andy¯2 = 1. We then randomly initialize a neural network as our hθ and con-catenate two Sigmoid functions as our gÏ. With the same initialization andall hyperparameters, we train the network to converge for each dataset.To facilitate the discussion, we categorize all 256 possible mappings into fournon-overlapping groups:1167.3. Simplicity Bias and Learning Dynamicsâ¢ Compositional mappings: These are isomorphisms of the ground-truthgenerating factors. Each dimension of the representation encodes a dis-tinct feature using a stable, interpretable vocabulary. These mappingsare highly compressed and reside on the fourth rung of the composi-tionality ladder. There are exactly 2!Ã 22 = 8 such mappings.â¢ Holistic mappings: These are non-compositional bijections. Whilethey preserve one-to-one correspondence between inputs and outputs,they lack the structured alignment seen in compositional mappings.As a result, they occupy the third rung of the ladder. Excluding the8 compositional cases, there are 4!â 8 = 16 holistic mappings.â¢ Degenerate mappings: These map all objects to the same output.While they are minimal in complexity, their high ambiguity rendersthem generally ineffective for downstream tasks. There are 4 such puredegenerate mappings.â¢ Other mappings: This group includes all remaining mappings, whichmay exhibit partial degeneracy or non-bijective behavior. These map-pings fall somewhere between the above categories in terms of structureand utility. We donât further separate them in this thesis.The roles these mappings play within the hypothesis space are illustratedin Figure 7.3. In essence, our objective is to identify mappings that areboth unambiguous and simple, ideally, those that align with a compositionalgenerating mechanism, and hence have a shorter description length.7.3.2 The Experimental SetupsTo verify the generality of our claim, we consider several different experi-mental settings. For the input signals, we first consider two types of one-hot concatenation vectors. One is the concatenation of two 2-dim vectors(OHT2 for short). For example, blue box and red circle are encoded as[0101] Â·W4Ãd and [1010] Â·W4Ãd, respectively, where W4Ãd is a randomlyinitialized matrix fixed for all 256 mappings. Another setting is OHT3, whichconsiders a redundant dimension for each attribute, where blue box andred circle are encoded as [010010] Â·W6Ãd and [100100] Â·W6Ãd, respec-tively. We also consider the vision input, where the image is sampled andcolored from the dSprite dataset (Matthey et al. 2017).We use different network structures for different input modalities. For theone-hot input, we use an MLP with three hidden layers with a width of1177.3. Simplicity Bias and Learning Dynamics(a). Learning curves of 256 mappingsâ ððâðð¥ð¨ð ð·ð(ðð|ð<ð)(b). Learning speed and coding length (c). Compression rate in Raeâs talkFigure 7.4: Left: The learning curves of all 256 mappings. Middle: Thecorrelation between the learning speed and the minimum description lengthof all mappings. Right: the compression rate in LLMâs training. The figurecomes from Touvron et al. 2023 and is discussed in Raeâs talk (Rae 2023).128. For the image input, we consider both a 3-layer MLP and a ResNet9(K. He et al. 2016b) with narrower hidden layers. The task heads for all thenetworks are identical: we add two separate linear projection layers with sizeh Ã 2 on the output of the backbone. After that, we take Softmax on eachof these outputs to generate probabilistic predictions. When calculatingthe loss function, we consider both cross-entropy (CE) loss and a meansquare error (MSE) loss, where the latter is calculated between the predictedprobability vector and a one-hot distribution of the ground truth labels.When optimizing the network, we consider both SGD and Adam. Unlessotherwise stated, the learning rate is set to be 10â3, weight decay is 5â10â4,and all other parameters are set to be the default values. Note that allhyperparameters (including the initialization of the network) are shared forall 256 experiments in each group.7.3.3 Simpler Mappings are Learned FasterSee Figure 7.4-(a) where we show the training curves of 256 runs in ourdefault setting. Since the only difference among these runs is the datasetgenerated by different mappings, it is safe to conclude that the differencein their learning speed is caused by the inherent bias of the modelâs learn-ing behavior on this problem. From this figure, we see that compositionalmappings are learned faster than holistic ones. However, some mappings arelearned even faster, which makes sense because those non-bijection mappings1187.3. Simplicity Bias and Learning Dynamicscontain degenerate components, i.e., two or more objects are mapped to thesame z, which means they are simpler. That also explains why the fourdegenerate mappings, which map all four objects to the same z, are learnedfastest among all 256 mappings. Nevertheless, as shown in Figure 7.3-(b),only those mappings that contain no degenerate component can perfectlyexplain all the training observations unambiguously. In other words, beforea battle between compositional mappings and holistic mappings, those non-bijections are already ruled out by the training accuracy. In other words,they cannot even achieve the third level of the compositionality ladder.To further verify the existence of the learning speed advantage, we quantifythe learning speed using the concept of âconvergence time,â i.e., the areaunder the learning curve. A smaller convergence time means the mappingis learned faster. This metric is similar to the C-score (Z. Jiang et al. 2021),which describes a training exampleâs difficulty. Also, if the model is trainedwith cross-entropy loss and all examples only appear once, this metric isthe compression rate for the entire dataset (Rae 2023), as shown in Fig-ure 7.4-(c). These works also hint that learning speed is deeply related tocompression, simplicity, and generalization ability.Furthermore, we also want to calculate explicitly the upper bounds on eachmappingâs Kolmogorov complexity. Since the mapping space studied in ourToy256 setting is simple enough, we can first create the CFGs for eachmapping and then convert them to a piece of description sequence using themethod provided by Kirby et al. (2015). After that, we can use Huffmancoding (Huffman 2007) and calculate the coding length in bits for eachmapping. Due to the page limits, we omit the calculation of the codinglength here; please refer to Appendix B of our paper for more details (Renand Sutherland 2024). In short, a smaller coding length means the mappingis simpler. It is clear in Figure 7.4-(b) that compositional mappings arethe simplest bijections. Note that all the non-bijection mappings containdegenerate components, hence are simpler than bijections. The figure alsoclearly demonstrates that the learning speed is strongly correlated to thecoding length (with Ï > 0.65 and p < 10â30), matching our hypothesis well.Lastly, we can also track a metric called âKernel Alignmentâ studied in thePlatonic Representation Hypothesis (Huh et al. 2024). Different from thecoding length discussed above, which depends on how we design the codingmechanism, the kernel alignment-based approach is model-agnostic and canmathematically describe the representationâs similarity to the ground truthfactors. Specifically, they define this concept from the following three levels:1197.3. Simplicity Bias and Learning Dynamics1. A representation, which maps the input signal to a dense representa-tion, is a function f : X â Rn. Our hθ(x) plays a similar role;2. A kernel, κ : X Ã X â R, characterizes the similarity between twoelements in X . In a dense representation case, the inner product isusually applied, i.e., κ(xi, xj) = â¨f(xi), f(xj)â©. Our paper considerHamming distance, because our G and z are all categorical variables;3. A kernel-alignment metric m : K Ã K â R, measures the similaritybetween two kernels.For the third-level measurement, Huh et al. (2024) use Centered Kernel Dis-tance, a kernel-alignment metric throughout their paper. Actually, many re-lated works in compositional generalization also have similar measurements,e.g., the topological similarity proposed by Brighton and Kirby (2006):Topsim(G, z) â Corr(dG(G(i),G(j)), dz(z(i), z(j))), (7.1)This definition also follows three steps: z are representations generated byfeeding x to hθ; dG and dz are distance measurements (or kernel in thesecond step above) for spaces G and Z; Corr(Â·, Â·) is Spearmanâs correlation,measuring the relationship between the output of two functions (kernels).In short, higher topological similarity means similar objects in G are mappedto similar positions in Z. If we consider G as another modality of theprojected ground truth, the topological similarity is just a special kernel-alignment metric used in the Platonic Representation Hypothesis (Huh etal. 2024). Also, some other measurements of compositionality like TRE(Tree Reconstruction Error, Andreas 2019), representation disentanglement(Higgins et al. 2018), etc., follow this general principle.Then, using this kernel alignment metric (i.e., topological similarity), wefurther examine their correlation to the learning speed in Table 7.1. It isclear that in most cases, the two quantitative metrics above align well. Bothof them support the claim that simpler mappings are indeed learned fasterunder different settings. One exceptional case is training a ResNet withimage input using the Adam optimizer. The simplicity bias is even reversedcompared with the results using SGD. This phenomenon hints that the biasin DNNsâ learning is also influenced by the inherent bias of specific networkstructures and optimizers. We left this for our future work.1207.3. Simplicity Bias and Learning DynamicsTable 7.1: The statistical correlation between learning speed and simplicity.CL-Conv.Time Topsim-Conv.Time CL-Conv.Time Topsim-Conv.TimeInput Optim. LossÏ p Ï pInput Optim. LossÏ p Ï pCE 0.6475 8.1*1e-32 -0.7101 1.4*1e-40 CE 0.6866 5.0*1e-37 -0.5911 1.6*1e-25SGDL2 0.5793 2.4*1e-24 -0.7817 5.3*1e-54SGDL2 0.5932 1.1*1e-25 -0.6057 5.1*1e-27CE 0.6598 2.3*1e-33 -0.5731 9.4*1e-24 CE 0.5403 8.4*1e-21 -0.6720 5.5*1e-35OHT2MLPAdamL2 0.5378 1.4*1e-20 -0.7223 1.2*1e-42ImageMLPAdamL2 0.4433 9.5*1e-14 -0.6585 3.4*1e-33CE 0.5976 3.5*1e-26 -0.7963 2.2*1e-57 CE 0.6711 7.4*1e-35 -0.2619 2.2*1e-5SGDL2 0.6386 9.8*1e-31 -0.7311 4.5*1e-44SGDL2 -0.015 0.8159 -0.0297 0.6358CE 0.5672 3.4*1e-23 -0.6418 4.1*1e-31 CE -0.5115 1.8*1e-18 0.0423 0.4999OHT3MLPAdamL2 0.5582 2.3*1e-22 -0.7026 2.1*1e-39ImageResNetAdamL2 -0.2876 2.8*1e-06 0.0197 0.75387.3.4 Force Analysis of Different MappingsThe results above bridge the simplicity bias to the modelâs learning behavior,where the latter can be further explained using our force analysis framework.Remember our model generates probabilistic predictions on both y1 and y2using Sigmoid functions. Then, we can directly write down the predictedprobability of each mapping as a product of eight terms, as in the top lineof Figure 7.5. In this figure, we demonstrate how the modelâs confidencein different M is updated when learning specific training samples (the left-most column). For example, in the first row of the compositional mappingin the figure, the model learns (blue box, 00). Then the correspondingP (y1 = 0 | blue box) and P (y2 = 0 | blue box) are significantly improved,since they are directly updated by learning this example. Furthermore, re-call how learning a number â4â influences the number â9â in Chapter 3, themodelâs predictions on those âsimilarâ (measured using Hamming distance)input examples would also be indirectly updated (represented by the smallarrows in the figure). As a result, the modelâs confidence on blue circleand red box, which share one attribute with the learned blue box, are in-fluenced more by this update. Furthermore, since a compositional mappingalways utilizes consistent values to represent the same attribute (e.g., z1 = 0always encodes the blue color), all the direct and indirect updates align wellwith the compositional mapping. That is why the training loss of such map-pings decreases faster. On the contrary, for a holistic mapping, we observeseveral contradictions between the direct and indirect updates: learning re-quires the model to use more updates to counteract those negative influences.That partially explains why compositional mappings are learned faster thanholistic ones when training a neural network using gradient descent.1217.4. Conclusion and Discussions, 00, 01, 10, 11P y1 = 0 â P y2 = 0 â P y1 = 0 â P y2 = 1 â P y1 = 1 â P y2 = 0 â P y1 = 1 â P y2 = 1P M =, 00, 11, 10, 01P y1 = 0 â P y2 = 0 â P y1 = 1 â P y2 = 1 â P y1 = 1 â P y2 = 0 â P y1 = 0 â P y2 = 1P M =CompositionalHolisticFigure 7.5: Force analysis of two types of mappings.7.4 Conclusion and DiscussionsThis chapter began by introducing two influential concepts that are closelytied to the success of modern large-scale models: the Platonic Representa-tion Hypothesis (Huh et al. 2024) and Compression for AGI (Rae 2023).Both ideas are grounded in Occamâs Razor, emphasizing the role of sim-plicity of the learned representations in effective generalization. They arguethat the scaling laws observed in contemporary models can be attributed toa pursuit of more efficient compression of observations.In particular, Huh et al. (2024) posit the existence of unique and stableground-truth data-generating mechanisms and empirically demonstrate thatwell-pretrained models, across various modalities, tend to converge to similartopological structures in their representation spaces. They suggest thatsuch convergence reflects an implicit simplicity bias that adheres to OccamâsRazor and may underlie the success of these models.Such a tendency is also observed by many researchers during the pretrain-ing of LLMs (Rae 2023). In the talk given by Rae, he first introduces manyrelated concepts, e.g., Solomonoff induction, minimum description length,Kolmogorov complexity, and generalization, and then links them to the com-pression rate that a model can achieve. He argues that a model with a highcompression rate must have a better understanding of the world. Such ameasurement can then be estimated by the integral between the modelâstraining curve and the x-axis in a next-token prediction task. Then, thosemodels that learn more quickly are inherently more intelligent, and thisprinciple forms a cornerstone of the scaling laws in LLMsâ pretraining.1227.4. Conclusion and DiscussionsBuilding on these insights, this chapter offers a novel mechanistic explana-tion for why gradient-based training tends to favor simpler, more structuredsolutions. We focus on a toy but illustrative task: learning compositionalrepresentations under a data-generating assumption akin to that in Huh etal. 2024. We argue that among all possible mappings from inputs to hiddenrepresentations, those that reflect the compositional structure of the datacan generalize perfectly and (crucially) have significantly lower Kolmogorovcomplexity, as shown via group-theoretic arguments.We then argue that the simplicity bias mentioned above should be embod-ied in the learning speed advantage in deep learning systems. To verify thisclaim, we conduct several experiments in different settings on our Toy256dataset. The results show that these compositional mappings are consis-tently learned faster. Through force analysis of the modelâs confidence up-dates across different data samples, we find that compositional mappingsbenefit from aligned âunified forcesâ across examples, while unstructuredmappings experience contradictory gradient directions. This provides a con-crete explanation for the emergence of simplicity bias during training.Although our experimental setting is deliberately minimal, similar findingshave emerged across diverse practical contexts. For instance, Le Priol et al.(2021) and Bengio et al. (2019) observe that models learn causal directionsmore easily than anti-causal ones. Likewise, Chiyuan Zhang et al. (2016)show that label permutation, analogous to creating holistic mappings fromcompositional ones, dramatically slows down learning and degrades general-ization, despite perfect memorization being achievable. Our framework ex-plains this through disrupted mutual influence: shuffled labels cause learningforces to interfere destructively rather than constructively.In conclusion, we believe that the inherently sequential nature of learn-ing in deep models makes the mutual influence between examples critical.Although only verified on a very simple manual setting, the force analysisshowcases some important principles. If each exampleâs gradient contributescoherently (or at least orthogonally when learning new knowledge) to oth-ers, training becomes significantly more efficient. According to Rae (2023),this leads to higher compression rates and higher intelligence. We thus be-lieve that analyzing learning speed and learning dynamics is a promisingdirection for future research, with the potential to illuminate fundamentalaspects of deep learning systems. We would consider more practical settingsand further extend our analytical framework in our future work.123Chapter 8Conclusion and DiscussionAfter a long journey through the study of learning dynamics and deep learn-ing, this thesis comes to a close. Given the wide range of topics covered, weprovide a summary diagram in Figure 8.1 to help readers recall the maincontent of the thesis. Key concepts and representative figures are attachednear each sectionâs box to highlight the core ideas of individual chapters.We began in Chapter 1 and Chapter 2 with a discussion of contemporarydeep learning theory, contrasting math-like and physics-like aspects. Weargued that the rapidly evolving nature of modern deep learning systemscalls for a greater emphasis on local intuitions and empirical trends. Thismotivates our proposal to study model behavior from a local, microscopicperspective. In particular, we introduced the idea of learning dynamics,which aims to analyze the âforcesâ exerted on model predictions when asingle example is learned via gradient descent.In Chapter 3, we formalized this framework using influence functions andproposed the AKG decomposition. In this formulation, the force originatesfrom G, is projected and normalized by K and A, and is ultimately imposedon the log-probability of the observing example xo. We interpreted K asthe similarity between xu and xo, which is usually relatively stable duringtraining. This is illustrated by the mutual influence between digits â4â andâ9â in our MNIST example. G captures the source of the force and evolvesdynamically during training. Our âzig-zag learning pathâ example vividlydemonstrates how these forces interact at different stages during training.Motivated by this zig-zag pattern, we explored a natural application: au-tomatic noisy label correction. As discussed in Chapter 4, early-stoppedmodels often learn to identify the correct labels by themselves. We studiedthis in the context of knowledge distillation and introduced Filter-KD, apipeline that leverages better supervisory signals from a teacher model toimprove generalization. We theoretically show that supervision aligning bet-ter with the ground-truth distribution qâ leads to stronger generalization,124Chapter 8. Conclusion and Discussionespecially on ambiguous or âhard-to-tellâ samples. Experiments support theeffectiveness of this method well.Next, in Chapter 5, we extended our force analysis to the finetuning ofLLMs, which is a highlight of this thesis. To make the next-token predic-tion task analytically tractable, we introduced simplifications and a probingdataset that reveals how different supervision signals affect the output dis-tribution over the huge Y space. We analyzed finetuning methods such asSFT, DPO, and GRPO, offering a unified perspective on their learning be-havior. Our framework uncovered the important, often overlooked, role ofnegative gradients. We propose the âsqueezing effectâ, and further explore itboth theoretically and empirically. Extending this framework to other LLMadaptation methods presents a promising direction for future research.In Chapter 6, we explored how our framework could be applied beyond out-put dynamics, e.g., to the adaptation of hidden features and other internalrepresentations. We analytically modeled how intermediate features evolveunder different levels of adaptation energy during finetuning, by observingfour distance-related quantities of the hidden features.Finally, in Chapter 7, we returned to the foundational question that mo-tivated our study: Where does the simplicity bias come from? Using acarefully designed toy task, we connected our findings to several influentialideas in deep learning theory, including the Platonic Representation Hy-pothesis, Occamâs Razor, and compression for AGI. Although the setup isdeliberately constrained, we believe the theoretical insights can shed morelight on this deep and fundamental question.To conclude, we acknowledge that the learning dynamics framework is still inits early stages. For instance, we have yet to fully explore the role of A or thedirectional effects embedded in K. The interaction of our framework withspecific architectural components (e.g., attention) or advanced optimizers(e.g., Adam) also remains underexplored. Future work may also extendour analysis from the âkernelâ regime to the more general and commonfeature learning regime. Given the connections between NTK and GaussianProcesses, combining them offers another compelling direction.Ultimately, I hope this thesis is not the end but another beginning of abroader investigation into force analysis and learning dynamics in deep learn-ing. The directions outlined here are open invitations for future exploration.125Chapter 8. Conclusion and DiscussionChapt. 1Introduction Chapt. 3Fundamentals of Learning DynamicsChapt. 5Case 2: UnderstandingLLMâs finetuningChapt. 6Case 3: Feature Adaptationin Transfer LearningChapt. 7Case 4: Simplicity Biasand CompositionalityChapt. 2OverviewChapt. 4Case 1: BetterKnowledge DistillationGoal:Solution:Figure 8.1: Summary and highlights of each chapter.126BibliographyZeyuan Allen-Zhu and Yuanzhi Li (2020). âTowards understanding ensem-ble, knowledge distillation and self-distillation in deep learning.â arXivpreprint arXiv:2012.09816.Jacob Andreas (2019). âMeasuring Compositionality in Representation Learn-ing.â International Conference on Learning Representations.Martin Arjovsky, Le´on Bottou, Ishaan Gulrajani, and David Lopez-Paz(2019). âInvariant risk minimization.â arXiv preprint arXiv:1907.02893.Sanjeev Arora, Nadav Cohen, and Elad Hazan (2018). âOn the optimizationof deep networks: Implicit acceleration by overparameterization.â Inter-national conference on machine learning. PMLR, pages 244â253.Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov,and Ruosong Wang (2019). âOn Exact Computation with an InfinitelyWide Neural Net.â NeurIPS. arXiv: 1904.11955.Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang(2019). âFine-grained analysis of optimization and generalization for over-parameterized two-layer neural networks.â International conference onmachine learning. PMLR, pages 322â332.Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos,Mark Rowland, Michal Valko, and Daniele Calandriello (2024). âA generaltheoretical paradigm to understand learning from human preferences.âInternational Conference on Artificial Intelligence and Statistics. PMLR,pages 4447â4455.Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng,Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li,Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, KemingLu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan,Sinan Tan, Jianhong Tu, PengWang, Shijie Wang, Wei Wang, ShengguangWu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang,Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingx-uan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou,127BIBLIOGRAPHYXiaohuan Zhou, and Tianhang Zhu (2023). âQwen Technical Report.âarXiv: 2309.16609.Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, NovaDasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, etal. (2022). âTraining a helpful and harmless assistant with reinforcementlearning from human feedback.â arXiv: 2204.05862.Fazl Barez, Tingchen Fu, Ameya Prabhu, Stephen Casper, Amartya Sanyal,Adel Bibi, Aidan OâGara, Robert Kirk, Ben Bucknall, Tim Fist, et al.(2025). âOpen problems in machine unlearning for ai safety.â arXiv preprintarXiv:2501.04952.David GT Barrett and Benoit Dherin (2020). âImplicit gradient regulariza-tion.â arXiv preprint arXiv:2009.11162.Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky (2017). âSpectrally-normalized margin bounds for neural networks.â Advances in neural in-formation processing systems 30.Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Se´bastienLachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal (2019).âA meta-transfer objective for learning to disentangle causal mechanisms.âarXiv preprint arXiv:1901.10912.Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley,Kyle OâBrien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Puro-hit, USVSN Sai Prashanth, Edward Raff, et al. (2023). âPythia: A suitefor analyzing large language models across training and scaling.â ICML.PMLR, pages 2397â2430.Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K War-muth (1989). âLearnability and the Vapnik-Chervonenkis dimension.âJournal of the ACM (JACM) 36.4, pages 929â965.Henry Brighton and Simon Kirby (2006). âUnderstanding linguistic evolu-tion by visualizing the emergence of topographic mappings.â Artificial life12.2, pages 229â242.Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton(2020). âA simple framework for contrastive learning of visual representa-tions.â International conference on machine learning. PmLR, pages 1597â1607.Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He (2020). âImprovedbaselines with momentum contrastive learning.â arXiv preprint.Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu(2024). âSelf-play fine-tuning converts weak language models to stronglanguage models.â arXiv preprint arXiv:2401.01335.128BIBLIOGRAPHYXiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang(2025). âGpg: A simple and strong reinforcement learning baseline formodel reasoning.â arXiv preprint arXiv:2504.02546.Adam Coates, Andrew Ng, and Honglak Lee (2011). âAn analysis of single-layer networks in unsupervised feature learning.â Proceedings of the four-teenth international conference on artificial intelligence and statistics.JMLR Workshop and Conference Proceedings, pages 215â223.Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni,Guotong Xie, Zhiyuan Liu, and Maosong Sun (2023). âUltraFeedback:Boosting Language Models with High-quality Feedback.â arXiv: 2310.01377 [cs.CL].Alex Damian, Eshaan Nichani, and Jason D Lee (2022). âSelf-stabilization:The implicit bias of gradient descent at the edge of stability.â arXivpreprint arXiv:2209.15594.Arghya Datta, Subhrangshu Nandi, Jingcheng Xu, Greg Ver Steeg, He Xie,Anoop Kumar, and Aram Galstyan (2023). âMeasuring and mitigating lo-cal instability in deep neural networks.â arXiv preprint arXiv:2305.10625.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei (2009).âImagenet: A large-scale hierarchical image database.â 2009 IEEE con-ference on computer vision and pattern recognition. Ieee, pages 248â255.Wenlong Deng, Yi Ren, Muchen Li, Danica J Sutherland, Xiaoxiao Li,and Christos Thrampoulidis (2025). âOn the Effect of Negative Gradi-ent in Group Relative Deep Reinforcement Optimization.â arXiv preprintarXiv:2505.18830.Wenlong Deng, Yi Ren, Danica J Sutherland, Christos Thrampoulidis, andXiaoxiao Li (2025). âToken Hidden Reward: Steering Exploration-Exploitationin GRPO Training.â 2nd AI for Math Workshop @ ICML 2025. url:https://openreview.net/forum?id=eyPPunAzSr.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova (2019).âBert: Pre-training of deep bidirectional transformers for language un-derstanding.â Proceedings of the 2019 conference of the North Americanchapter of the association for computational linguistics: human languagetechnologies, volume 1 (long and short papers), pages 4171â4186.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min-derer, Georg Heigold, Sylvain Gelly, et al. (2021). âAn Image is Worth16x16 Words: Transformers for Image Recognition at Scale.â ICLR 2021.Simon S. Du, Kangcheng Hou, Russ R Salakhutdinov, Barnabas Poczos,Ruosong Wang, and Keyulu Xu (2019). âGraph neural tangent kernel:129BIBLIOGRAPHYFusing graph neural networks with graph kernels.â Advances in neuralinformation processing systems 32.Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and DouweKiela (2024). âKto: Model alignment as prospect theoretic optimization.âarXiv preprint arXiv:2402.01306.Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur (2020).âSharpness-aware minimization for efficiently improving generalization.âarXiv preprint arXiv:2010.01412.Kunihiko Fukushima (1980). âNeocognitron: A self-organizing neural net-work model for a mechanism of pattern recognition unaffected by shift inposition.â Biological cybernetics 36.4, pages 193â202.Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, andAnima Anandkumar (2018). âBorn again neural networks.â Proceedings ofInternational Conference on Machine Learning. PMLR, pages 1607â1616.Ian J Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio(2016). Deep learning. Volume 1. 2. MIT press Cambridge.Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, DavidWarde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio (2014). âGener-ative adversarial nets.â Advances in neural information processing systems27.Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger (2017). âOncalibration of modern neural networks.â International Conference on Ma-chine Learning. PMLR, pages 1321â1330.Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, RunxinXu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. (2025). âDeepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.âarXiv preprint arXiv:2501.12948.Shangmin Guo, Yi Ren, Stefano V Albrecht, and Kenny Smith (2024). âlp-NTK: Better Generalisation with Less Data via Sample Interaction DuringLearning.â The Twelfth International Conference on Learning Represen-tations.Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, FelipeLlinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al.(2024). âDirect language model alignment from online ai feedback.â arXivpreprint arXiv:2402.04792.Hangfeng He and Weijie Su (2020). âThe Local Elasticity of Neural Net-works.â International Conference on Learning Representations.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun (2016a). âDeepresidual learning for image recognition.â Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition, pages 770â778.130BIBLIOGRAPHYKaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun (2016b). âDeepresidual learning for image recognition.â Proceedings of the IEEE confer-ence on computer vision and pattern recognition, pages 770â778.Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart,Eric Tang, Dawn Song, and Jacob Steinhardt (2021). âMeasuring mathe-matical problem solving with the math dataset.â arXiv preprint arXiv:2103.03874.Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey,Danilo Rezende, and Alexander Lerchner (2018). âTowards a definition ofdisentangled representations.â arXiv: 1812.02230.Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mo-hamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen,Tara N Sainath, et al. (2012). âDeep neural networks for acoustic mod-eling in speech recognition: The shared views of four research groups.âIEEE Signal processing magazine 29.6, pages 82â97.Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky (2012). âNeural net-works for machine learning lecture 6a overview of mini-batch gradientdescent.â Cited on 14.8, page 2.Geoffrey Hinton, Oriol Vinyals, and Jeff Dean (2015). âDistilling the knowl-edge in a neural network.â arXiv preprint arXiv:1503.02531.Sepp Hochreiter and Ju¨rgen Schmidhuber (1997a). âLong short-term mem-ory.â Neural computation 9.8, pages 1735â1780.â (1997b). âLong short-term memory.â Neural computation 9.8, pages 1735â1780.Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi (2020).âThe Curious Case of Neural Text Degeneration.â ICLR.Jiwoo Hong, Noah Lee, and James Thorne (2024). âOrpo: Monolithic prefer-ence optimization without reference model.â arXiv preprint arXiv:2403.07691.Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, andHeung-Yeung Shum (2025). âOpen-Reasoner-Zero: An Open Source Ap-proach to Scaling Up Reinforcement Learning on the Base Model.â arXiv:2503.24290 [cs.LG]. url: https://arxiv.org/abs/2503.24290.Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren,Bowen Liu, Michele Catasta, and Jure Leskovec (2020). âOpen GraphBenchmark: Datasets for Machine Learning on Graphs.â arXiv preprintarXiv:2005.00687.David A Huffman (2007). âA method for the construction of minimum-redundancy codes.â Proceedings of the IRE 40.9, pages 1098â1101.Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola (2024).âPosition: The platonic representation hypothesis.â Forty-first Interna-tional Conference on Machine Learning.131BIBLIOGRAPHYL Hui (2021). âEvaluation of neural architectures trained with square lossvs cross-entropy in classification tasks.â ICLR 2021.Arthur Jacot, Franck Gabriel, and Cle´ment Hongler (2018). âNeural tangentkernel: Convergence and generalization in neural networks.â NeurIPS.Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer (2021).âCharacterizing Structural Regularities of Labeled Data in Overparame-terized Models.â International Conference on Machine Learning. PMLR,pages 5034â5044.Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, DongWang, Hamed Zamani, and Jiawei Han (2025). âSearch-r1: Training llmsto reason and leverage search engines with reinforcement learning.â arXivpreprint arXiv:2503.09516.Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyan-skiy, and Ping Tak Peter Tang (2016). âOn large-batch training for deeplearning: Generalization gap and sharp minima.â arXiv preprint arXiv:1609.04836.Diederik P Kingma (2014). âAdam: A method for stochastic optimization.âarXiv preprint arXiv:1412.6980.Thomas N Kipf and Max Welling (2016). âSemi-supervised classificationwith graph convolutional networks.â arXiv preprint arXiv:1609.02907.Simon Kirby, Monica Tamariz, Hannah Cornish, and Kenny Smith (2015).âCompression and communication in the cultural evolution of linguisticstructure.â Cognition 141, pages 87â102.Pang Wei Koh and Percy Liang (2017). âUnderstanding black-box predic-tions via influence functions.â International conference on machine learn-ing. PMLR, pages 1885â1894.Andrei N Kolmogorov (1963). âOn tables of random numbers.â Sankhya¯:The Indian Journal of Statistics, Series A, pages 369â376.Alex Krizhevsky, Geoffrey Hinton, et al. (2009). âLearning multiple layersof features from tiny images.âAlex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton (2012). âImagenetclassification with deep convolutional neural networks.â Advances in neu-ral information processing systems 25.Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and PercyLiang (2022). âFine-tuning can distort pretrained features and underper-form out-of-distribution.â International Conference on Learning Repre-sentations.Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xiaofeng Yang (2025).âMed-r1: Reinforcement learning for generalizable medical reasoning invision-language models.â arXiv preprint arXiv:2503.13939.132BIBLIOGRAPHYRe´mi Le Priol, Reza Babanezhad, Yoshua Bengio, and Simon Lacoste-Julien(2021). âAn analysis of the adaptation speed of causal models.â Interna-tional Conference on Artificial Intelligence and Statistics. PMLR, pages 775â783.Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, RichardHoward, Wayne Hubbard, and Lawrence Jackel (1989). âHandwrittendigit recognition with a back-propagation network.â Advances in neuralinformation processing systems 2.Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, ThomasMesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, andAbhinav Rastogi (2023). âRlaif: Scaling reinforcement learning from hu-man feedback with ai feedback.âJaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman No-vak, Jascha Sohl-Dickstein, and Jeffrey Pennington (2019). âWide neuralnetworks of any depth evolve as linear models under gradient descent.âAdvances in neural information processing systems.David Lewis (2008). Convention: A philosophical study. John Wiley & Sons.Hongyu Li, Liang Ding, Meng Fang, and Dacheng Tao (2024). âRevisitingcatastrophic forgetting in large language model tuning.â arXiv preprintarXiv:2406.04836.Chengyuan Liu, Yangyang Kang, Shihang Wang, Lizhi Qing, Fubang Zhao,Changlong Sun, Kun Kuang, and Fei Wu (2024). âMore than catastrophicforgetting: Integrating general capabilities for domain-specific llms.â arXivpreprint arXiv:2405.17830.Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda (2020). âEarly-learning regularization prevents memorization ofnoisy labels.â Advances in Neural Information Processing Systems.Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du,Wee Sun Lee, and Min Lin (2025). âUnderstanding r1-zero-like training:A critical perspective.â arXiv preprint arXiv:2503.20783.Ilya Loshchilov and Frank Hutter (2016). âSgdr: Stochastic gradient descentwith warm restarts.â arXiv preprint arXiv:1608.03983.â (2017). âDecoupled weight decay regularization.â arXiv preprint arXiv:1711.05101.Aengus Lynch, Benjamin Wright, Caleb Larson, Kevin K. Troy, Stuart J.Ritchie, So¨ren Mindermann, Ethan Perez, and Evan Hubinger (2025).âAgentic Misalignment: How LLMs Could be an Insider Threat.â An-thropic Research. https://www.anthropic.com/research/agentic-misalignment.Stephan Mandt, Matthew D Hoffman, and David M Blei (2017). âStochasticgradient descent as approximate bayesian inference.â Journal of MachineLearning Research 18.134, pages 1â35.133BIBLIOGRAPHYLoic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner (2017).âdSprites: Disentanglement testing Sprites dataset.âMichael McCloskey and Neal J Cohen (1989). âCatastrophic interference inconnectionist networks: The sequential learning problem.â Psychology oflearning and motivation. Volume 24. Elsevier, pages 109â165.Warren S McCulloch andWalter Pitts (1943). âA logical calculus of the ideasimmanent in nervous activity.â The bulletin of mathematical biophysics5, pages 115â133.Yu Meng, Mengzhou Xia, and Danqi Chen (2024). âSimpo: Simple pref-erence optimization with a reference-free reward.â Advances in NeuralInformation Processing Systems 37, pages 124198â124235.Aditya Krishna Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim,and Sanjiv Kumar (2021). âA statistical perspective on distillation.â In-ternational Conference on Machine Learning. PMLR, pages 7632â7642.Marvin Minsky and Seymour A Papert (1969). Perceptrons: An Introductionto Computational Geometry. MIT Press.Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, JoelVeness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas KFidjeland, Georg Ostrovski, et al. (2015). âHuman-level control throughdeep reinforcement learning.â nature 518.7540, pages 529â533.Rafael Mu¨ller, Simon Kornblith, and Geoffrey E Hinton (2019). âWhen doeslabel smoothing help?â Advances in neural information processing sys-tems.Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Sre-bro (2017). âExploring generalization in deep learning.â Advances in neu-ral information processing systems 30.Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova Das-Sarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, AnnaChen, et al. (2022). âIn-context learning and induction heads.â arXivpreprint arXiv:2209.11895.Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright,Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, AlexRay, et al. (2022). âTraining language models to follow instructions withhuman feedback.â NeurIPS, pages 27730â27744.Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu,and Colin White (2024). âSmaug: Fixing failure modes of preference op-timisation with dpo-positive.â arXiv preprint arXiv:2402.13228.Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn (2024). âDis-entangling length from quality in direct preference optimization.â arXivpreprint arXiv:2403.19159.134BIBLIOGRAPHYJudea Pearl and Dana Mackenzie (2018). The Book of Why: The New Sci-ence of Cause and Effect. Basic Books.Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and OlgaRussakovsky (2019). âHuman uncertainty makes classification more ro-bust.â Proceedings of the IEEE/CVF International Conference on Com-puter Vision, pages 9617â9626.Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ah-mad Beirami, Prateek Mittal, and Peter Henderson (2024). âSafety align-ment should be made more than just a few tokens deep.â arXiv preprintarXiv:2406.05946.Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.(2018). âImproving language understanding by generative pre-training.âJack Rae (2023). âCompression for AGI.â url: https://www.youtube.com/watch?v=dO4TPJkeaaU.Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn (2024). âFromr to qâ: Your language model is secretly a q-function.â arXiv preprintarXiv:2404.12358.Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Ste-fano Ermon, and Chelsea Finn (2023). âDirect preference optimization:Your language model is secretly a reward model.â NeurIPS.Abhilasha Ravichander, Shrusti Ghela, DavidWadden, and Yejin Choi (2025).âHALoGEN: Fantastic LLM Hallucinations and Where to Find Them.âarXiv preprint arXiv:2501.08292.Deepak Ravikumar, Efstathia Soufleri, Abolfazl Hashemi, and Kaushik Roy(2025). âTowards memorization estimation: Fast, formal and free.â Forty-second International Conference on Machine Learning.Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora,and Boris Hanin (2025). âUnintentional Unalignment: Likelihood Dis-placement in Direct Preference Optimization.â The Thirteenth Interna-tional Conference on Learning Representations.Yi Ren, Shangmin Guo, Wonho Bae, and Danica J Sutherland (2023). âHowto Prepare Your Task Head For Finetuning.â ICLR 2023.Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B Cohen, and Simon Kirby(2020). âCompositional languages emerge in a neural iterated learningmodel.â ICLR 2020.Yi Ren, Shangmin Guo, Linlu Qiu, Bailin Wang, and Danica J Sutherland(2024). âBias Amplification in Language Model Evolution: An IteratedLearning Perspective.â NeurIPS.Yi Ren, Shangmin Guo, and Danica J Sutherland (2022). âBetter Supervi-sory Signals by Observing Learning Paths.â ICLR 2022.135BIBLIOGRAPHYYi Ren, Samuel Lavoie, Michael Galkin, Danica J Sutherland, and AaronC Courville (2023). âImproving compositional generalization using iter-ated learning and simplicial embeddings.â Advances in Neural Informa-tion Processing Systems 36, pages 60547â60572.Yi Ren and Danica J Sutherland (2024). âUnderstanding Simplicity Biastowards Compositional Mappings via Learning Dynamics.â arXiv preprintarXiv:2409.09626.â (2025). âLearning Dynamics of LLM Finetuning.â ICLR-2025.Frank Rosenblatt (1957). The Perceptronâa perceiving and recognizing au-tomaton. Technical report. Cornell Aeronautical Laboratory.Sebastian Ruder (2016). âAn overview of gradient descent optimization al-gorithms.â arXiv preprint arXiv:1609.04747.David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams (1986).âLearning representations by back-propagating errors.â nature 323.6088,pages 533â536.Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto (2023).âVerbosity bias in preference labeling by large language models.â arXivpreprint arXiv:2310.10076.John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and PhilippMoritz (2015). âTrust region policy optimization.â International confer-ence on machine learning. PMLR, pages 1889â1897.John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and OlegKlimov (2017). âProximal policy optimization algorithms.â arXiv preprintarXiv:1707.06347.Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi,Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. (2024). âDeepseek-math: Pushing the limits of mathematical reasoning in open languagemodels.â arXiv preprint arXiv:2402.03300.David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre,George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, VedaPanneershelvam, Marc Lanctot, et al. (2016). âMastering the game of Gowith deep neural networks and tree search.â nature 529.7587, pages 484â489.Steven Smith (2003).Digital signal processing: a practical guide for engineersand scientists. Newnes.Marilyn Strathern (1997). ââImproving ratingsâ: audit in the British Univer-sity system.â European review 5.3, pages 305â321.Ilya Sutskever, Oriol Vinyals, and Quoc V Le (2014). âSequence to sequencelearning with neural networks.â Advances in neural information processingsystems 27.136BIBLIOGRAPHYChristian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed,Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and AndrewRabinovich (2015). âGoing deeper with convolutions.â Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 1â9.Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbig-niew Wojna (2016). âRethinking the Inception architecture for computervision.â Proceedings of the IEEE Conference on Computer Vision andPattern Recognition, pages 2818â2826.Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schnei-der, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar(2024). âPreference Fine-Tuning of LLMs Should Leverage Suboptimal,On-Policy Data.â arXiv: 2404.14367 [cs.LG].Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed HChi, and Sagar Jain (2020). âUnderstanding and improving knowledgedistillation.â arXiv preprint arXiv:2002.03532.Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li,Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto (2023). âAl-paca: A strong, replicable instruction-following model.â Stanford Centerfor Research on Foundation Models. 3.6, page 7.Christos Thrampoulidis (2024). âImplicit optimization bias of next-tokenprediction in linear models.â arXiv preprint arXiv:2402.18551.Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai,Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, et al. (2025). âA surveyon post-training of large language models.â arXiv e-prints, arXivâ2503.Naftali Tishby, Fernando C Pereira, and William Bialek (1999). âThe in-formation bottleneck method.â Allerton Conference on Communication,Control, and Computing. arXiv: physics/0004057.Naftali Tishby and Noga Zaslavsky (2015). âDeep learning and the informa-tion bottleneck principle.â 2015 ieee information theory workshop (itw).Ieee, pages 1â5.Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothe´e Lacroix, Baptiste Rozie`re, Naman Goyal, EricHambro, Faisal Azhar, et al. (2023). âLlama: Open and efficient founda-tion language models.â arXiv preprint arXiv:2302.13971.Leslie G Valiant (1984). âA theory of the learnable.â Communications of theACM 27.11, pages 1134â1142.Guillermo Valle-Perez, Chico Q Camargo, and Ard A Louis (2018). âDeeplearning generalizes because the parameter-function map is biased towardssimple functions.â arXiv preprint arXiv:1805.08522.137BIBLIOGRAPHYLaurens Van der Maaten and Geoffrey Hinton (2008). âVisualizing datausing t-SNE.â Journal of machine learning research 9.11.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin (2017). âAttentionis all you need.â Advances in neural information processing systems 30.Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, SebastianBorgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Met-zler, et al. (2022). âEmergent abilities of large language models.â arXivpreprint arXiv:2206.07682.Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quan-quan Gu (2024). âSelf-play preference optimization for language modelalignment.â arXiv preprint arXiv:2405.00675.Zeke Xie, Issei Sato, and Masashi Sugiyama (2020). âA diffusion theory fordeep learning dynamics: Stochastic gradient descent exponentially favorsflat minima.â arXiv preprint arXiv:2002.03495.Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Ben-jamin Van Durme, Kenton Murray, and Young Jin Kim (2024). âCon-trastive preference optimization: Pushing the boundaries of llm perfor-mance in machine translation.â arXiv preprint arXiv:2401.08417.Greg Yang (2020). âTensor programs ii: Neural tangent kernel for any ar-chitecture.â arXiv preprint arXiv:2006.14548.Zhaorui Yang, Qian Liu, Tianyu Pang, Han Wang, Haozhe Feng, MinfengZhu, and Wei Chen (2024). âSelf-Distillation Bridges Distribution Gap inLanguage Model Fine-Tuning.â arXiv: 2402.13669.Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, YuYue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. (2025).âDapo: An open-source llm reinforcement learning system at scale.â arXivpreprint arXiv:2503.14476.Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, andGao Huang (2025). âDoes reinforcement learning really incentivize reason-ing capacity in llms beyond the base model?â arXiv preprint arXiv:2504.13837.Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and OriolVinyals (2016). âUnderstanding deep learning requires rethinking gener-alization.â arXiv preprint arXiv:1611.03530.Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and JitendraMalik (2020). âSide-tuning: a baseline for network adaptation via addi-tive side networks.â European Conference on Computer Vision. Springer,pages 698â714.138Jixiao Zhang and Chunsheng Zuo (2025). âGrpo-lead: A difficulty-awarereinforcement learning approach for concise mathematical reasoning inlanguage models.â arXiv preprint arXiv:2504.09696.Kaichen Zhang, Yuzhong Hong, Junwei Bao, Hongfei Jiang, Yang Song,Dingqian Hong, and Hui Xiong (2025). âGVPO: Group variance pol-icy optimization for large language model post-training.â arXiv preprintarXiv:2504.19599.Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, andKaisheng Ma (2019). âBe Your Own Teacher: Improve the Performance ofConvolutional Neural Networks via Self-Distillation.â ICCV. arXiv: 1905.08094.Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei (2024). âNegative preferenceoptimization: From catastrophic collapse to effective unlearning.â arXivpreprint arXiv:2404.05868.Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and YoavArtzi (2021). âRevisiting Few-sample {BERT} Fine-tuning.â InternationalConference on Learning Representations. url: https://openreview.net/forum?id=cO1IH43yUF.Xiang Zhang, Junbo Zhao, and Yann LeCun (2015). âCharacter-level convo-lutional networks for text classification.â Advances in neural informationprocessing systems 28.Xiaojiang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang, Zheng Lin,Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, etal. (2025). âSrpo: A cross-domain implementation of large-scale reinforce-ment learning on llm.â arXiv preprint arXiv:2504.14286.Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh,and Peter J Liu (2023). âSlic-hf: Sequence likelihood calibration with hu-man feedback.â arXiv preprint arXiv:2305.10425.Yize Zhao, Tina Behnia, Vala Vakilian, and Christos Thrampoulidis (2024).âImplicit geometry of next-token prediction: From language sparsity pat-terns to model representations.â arXiv preprint arXiv:2408.15417.Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu (2020). âGradi-ent descent optimizes over-parameterized deep ReLU networks.â Machinelearning 109.3, pages 467â492.139Appendix AProofs and CalculationsA.1 Risk Estimate Variance Bound in Section 4.2The following result is a generalization of Proposition 3 of Menon et al. 2021,whose proof we replicate and extend here:Proposition 4. Let L be any bounded loss, L(f(x), y) â¤ â < â, and con-sider Rtar of Equation (4.1). For any predictor f : X â âK , we haveED[(Rtar(f,D)âR(f))2] â¤ 1nVarx[qTtarL(f(x))]+ ξ,where ξ can be any of the following seven quantities:â2K(Exâ¥qtar(x)â qâ(x)â¥2)2â2(Exâ¥qtar(x)â qâ(x)â¥1)22â2(ExâKL(qtar(x) â¥qâ(x)))22â2 ExKL(qtar(x) â¥qâ(x))2â2(ExâKL(qâ(x) â¥qtar(x)))22â2 ExKL(qtar(x) â¥qâ(x))â2 Ex[KL(qtar(x) â¥qâ(x)) + KL(qâ(x) â¥qtar(x))].Proof. To begin,ED[(Rtar(f,D)âR(f))2]= VarD[Rtar(f,D)âR(f)]+(ED[Rtar(f,D)âR(f)])2.For the variance term, since R(f) is a constant and Rtar is an average of ni.i.d. terms, we getVarD[Rtar(f,D)âR(f)] = 1nVarx[qtar(x)TL(f(x))].140A.2. Calculation of the GtSFT(Ïu) in Equation (5.3)The other term, asRtar is an average of i.i.d. terms andR(f) = Ex[qâ(x)â¤L(f(x))],is (ED[Rtar(f,D)âR(f)])2=(Ex[(qtar(x)â qâ(x))â¤L(f(x)])2.For the first bound, we apply the Cauchy-Schwarz inequality,(qtar(x)â qâ(x))â¤L(f(x)) â¤ â¥qtar(x)â qâ(x)â¥2â¥L(f(x))â¥2;since the elements of L(f(x)) are each at most â, the term â¥L(f(x))â¥2 is atmostâKâ.For the other bounds, we instead apply Ho¨lderâs inequality, yielding(qtar(x)âqâ(x))â¤L(f(x)) â¤ â¥qtar(x)âqâ(x)â¥1â¥L(f(x))â¥â â¤ ââ¥qtar(x)âqâ(x)â¥1.The KL bounds follow by Pinskerâs inequality and then Jensenâs inequality.The last bound, for the Jeffreys divergence, combines the two KL bounds.A.2 Calculation of the GtSFT(Ïu) in Equation (5.3)We here showcase a standard GtSFT(Ïu) = âzL(Ïu) |zt considering a stan-dard NLL loss. As the auto-regression nature of the SFT loss is alreadyencoded in the causal mask used in hθ, the columns in Gt(Ïu) are indepen-dent of each other, which can be separately calculated. Plus, the summationover l can also be achieved by left-multiplying a length-L all-one vector 1.Specifically, the SFT loss for each l is:[LSFT(Ïu)]l = â log Ï(yl = yu | Ïu)= âeâ¤yu log Ï(yl | Ïu)= âeâ¤yu log (Ï(zl)) ,where yu is for the l-th dimension of yu. The gradient of L on z can be thencalculated as:[GtSFT(Ïu)]l = âzl [LSFT(Ïu)]lï¸¸ ︷︷ ï¸¸1ÃV=ï£«ï£­âÏ[LSFT(Ïu)]lï¸¸ ︷︷ ï¸¸VÃ1ï£¶ï£¸â¤âzlÏï¸¸ ︷︷ ï¸¸VÃV= â (eyu â Ï)â¤âzlÏ = Ï(yl | Ïu)â eyu , (A.1)141A.3. The Inner Product Calculation in GRPO (Proof of Equation (5.14)where â is element-wise division.To calculate the equation above, we first recall the NLL loss of the l-th tokenis [LSFT]l = âeâ¤y+l log Ï, where Ï = Ï(z). Then, âzLï¸¸ï¸·ï¸·ï¸¸1ÃV= âÏLï¸¸ï¸·ï¸·ï¸¸1ÃVâzÏï¸¸ï¸·ï¸·ï¸¸VÃV. For eachdimension of âzLl, we have âLÏi = 0 if Ïi Ì¸= yl and âLÏi = â 1Ïi if Ïi = yl. Bywriting it in vector form, we have âzL = â(eyl â Ï)â¤âzÏ. For âzÏ, wehave:âzÏ =ï£®ï£¯ï£¯ï£¯ï£°Ï1(1â Ï1) âÏ2Ï1 · · · âÏV Ï1âÏ1Ï2 1â Ï2Ï2 · · · âÏV Ï2. . . . . .. . . . . .âÏ1ÏV âÏ2ÏV · · · 1â ÏV ÏVï£¹ï£ºï£ºï£ºï£» .Combining this matrix and the 1ÃV vector (eylâÏ)â¤, where the only non-zero term is 1Ïk at the k = yl position. So, left multiplying by this vector isactually first selecting the k-th row of âzÏ, and then multiplying 1Ïk to it.In summary, we have:âzL = â 1Ïk[âÏkÏ1,âÏkÏ2, . . . ,âÏk(1â Ïk), . . . ,âÏkÏV ]â¤= [Ï1, Ï2, . . . , Ïk â 1, . . . , ÏV ]â¤= Ï â ekBy stacking the terms with different l â [L], we can getGtSFT(Ïu) = âzLSFT(Ïu)|zt = ÏÎ¸t(y | Ïu)â eyu (A.2)A.3 The Inner Product Calculation in GRPO(Proof of Equation (5.14)This appendix provides detailed steps for getting Equation (5.14), i.e.,â¨âÎ¸Lo,m,âÎ¸Ln,lâ© â α(o,m),(n,l) â (eyo,m â Ïyo,m)â¤(eyn,l â Ïyn,l)hâ¤n,lho,m.Recall Equation (5.12) and UFM, in which θ â [w;h], where w â RVÃd andh â RdÃ1 are both trainable parameters. We use the same subscript systemfor both h, Ï, and y. So, the feature vector of Ï+o,<m is hence h+o,<m. Then,âÎ¸Lo,m = âÎ¸ log Ï(y | xo, y+o,<m)can be written as â[w;h] log Ï(wh+o,<m),where Ï(Â·) is the Softmax function. Our inner product then becomes:142A.3. The Inner Product Calculation in GRPO (Proof of Equation (5.14)â©â[w;h] log Ïyo,m(who,m),â[w;h] log Ïyn,l(whn,l)âª . (A.3)To make the derivation easy to follow, we suggest keeping the shape ofthese matrices in mind during the derivation. First, â[w;h] log Ïyo,m(who,m)is taking the gradient of one scalar (i.e., the yo,m-th token of the log Ï) toall the parameters, so it should be a very long (V â d+ d)Ã 1 vector:â[w;h] log Ïi â[â log ÏiâwË1, . . . ,â log ÏiâwËV âd,â log Ïiâh1, . . . ,â log Ïiâhd]= [âwË log Ïi;âh log Ïi]where wË â RV âdÃ1 is the column-wise flatten version of w. Using thisdecomposition, the inner product in Equation (A.3) can be computed byfirst evaluating the inner products between the gradient of each componentseparately, and then summing them together, i.e.,â©â[w;h] log Ïi,â[w;h] log Ïjâª = â¨âwË log Ïi,âwË log Ïjâ©+ â¨âh log Ïi,âh log Ïjâ© .(A.4)Letâs start with âw log Ïyo,m(who,m). Note that taking the gradient of w isnot a typo here, since âwË is just a column wise rearrange of âw, and wealready have the matrix representation of âw log in our previous sections.Specifically, we haveâw log Ïyo,m(who,m)ï¸¸ ︷︷ ï¸¸VÃd= (eyo,m â Ïo,mï¸¸ ︷︷ ï¸¸VÃ1)hâ¤o,mï¸¸ï¸·ï¸·ï¸¸1Ãd,Each column of the above matrix is (eyo â Ïo)hom,i, where hom,i is the i-thelement of ho,m. The inner product of two gradients can be calculated as:â©âwË log Ïyo,m ,âwË log Ïyn,lâª = dâi=1(eyo,m â Ïyo,m)â¤(eyn,l â Ïyn,l)hom,ihnl,i= (eyo,m â Ïyo,m)â¤(eyn,l â Ïyn,l)dâi=1hom,ihnl,i= (eyo,m â Ïyo,m)â¤ï¸¸ ︷︷ ï¸¸1ÃV(eyn,l â Ïyn,l)ï¸¸ ︷︷ ï¸¸VÃ1hâ¤o,mhn,lï¸¸ ︷︷ ï¸¸1Ã1(A.5)Similarly, the gradient with respect to h isâh log Ïyo,m(who,m)ï¸¸ ︷︷ ï¸¸dÃ1= wâ¤ï¸¸ï¸·ï¸·ï¸¸dÃV(eyo,m â Ïyo,mï¸¸ ︷︷ ï¸¸VÃ1).143A.4. Proof of Kolmogorov Complexity Bound in Section 7.2The inner product is then calculated as:â©âh log Ïyo,m ,âh log Ïyn,lâª = (wâ¤(eyo,m â Ïyo,m))â¤ (wâ¤(eyn,l â Ïyn,l))= (eyo,m â Ïyo,m)â¤ï¸¸ ︷︷ ï¸¸1ÃVwwâ¤ï¸¸ ︷︷ ï¸¸VÃV(eyn,l â Ïyn,l)ï¸¸ ︷︷ ï¸¸VÃ1. (A.6)Now, substituting Equation (A.6) and Equation (A.5) back to Equation (A.4),we can get the decomposition of the learning dynamics between two tokens:â¨âÎ¸Lo,m,âÎ¸Ln,lâ© =(eyo,m â Ïyo,m)â¤(hâ¤n,lho,m · IV +wwâ¤)(eyn,l â Ïyn,l)=(eyo,m â Ïyo,m)â¤(eyn,l â Ïyn,l)hâ¤n,lho,m+ (eyo,m â Ïyo,m)â¤wwâ¤(eyn,l â Ïyn,l), (A.7)where IV is a V Ã V identity matrix. This decomposition also resemblesa AKG style: the difference is that eyo,m â Ïyo,m is only one column of A.The G term is still starting from Ï and pointing to e (note that we still haveâÎ· outside the inner product). The good news is that our K term is moreinterpretable, which sheds more light on the token-level mutual influence.Ideally, we would compute every term in Equation (A.7) during training inorder to accurately evaluate the inner product. However, in practice, calcu-lating the term involving wwâ¤ is significantly more computationally inten-sive than computing hâ¤h, due to the dimensionality differences: w â RVÃdwhereas h â RdÃ1. Moreover, we hypothesize that the hidden represen-tations h, which tend to be relatively stable throughout training, alreadycapture sufficient information about token-level similarity. Therefore, wepropose a simplified approximation by retaining only the first term on theright-hand side of Equation (A.7). This approximation forms the basis ofour influence score, denoted by α, which serves as a practical and efficientsurrogate for measuring token-level influence during GRPO training:â¨âÎ¸Lo,m,âÎ¸Ln,lâ© â α(o,m),(n,l) â (eyo,m â Ïyo,m)â¤(eyn,l â Ïyn,l)hâ¤n,lho,m.(A.8)A.4 Proof of Kolmogorov Complexity Bound inSection 7.2Proposition 3 (Informal). For m, v â¥ 2, among all bijections, any compo-sitional mapping has much lower upper bounds of its Kolmogorov complexitythan a typical non-compositional mapping.144A.4. Proof of Kolmogorov Complexity Bound in Section 7.2Proof. Recall the fact that any bijection from z to G can be represented byan element in the symmetry group Svm . From the definition of the symmetrygroup, we know each element in Svm can be represented by a permutationmatrix of size vm. As there is only one 1 in each row and column of apermutation matrix, any permutation matrix can be uniquely represented bya permuted sequence of length vm. Specifically, assume we have a sequence ofnatural numbers {1, 2, ..., vm}, each permuted sequence Perm({1, 2, ..., vm})represents a distinct permutation matrix, and hence represents a distinctbijection from z to G. In other words, we can encode one bijection from zto G using a sequence of length vm, i.e., Perm({1, 2, ..., vm}), and bound thecorresponding Kolmogorov complexity (in bits) asKC(bijection) â¤ vm · log2 vm = vm Â·m · log2 v, (A.9)As an arbitrary bijection from z to G doesnât have any extra information toimprove the coding efficiency, Equation (A.9) provides an upper bound ofthe minimal Kolmogorov complexity.On the contrary, as each compositional mapping can be represented by an el-ement in Smv âSm, we can encode the mappings more efficiently. Specifically,we need to first usem sequences with length v, i.e., Perm({1, 2, ..., v}), to rep-resent the assignment of âwordsâ for each zi. After that, we need a sequenceof lengthm, i.e., Perm({1, 2, ...,m}) to encode the assignment between zi andGj . The corresponding Kolmogorov complexity is then bounded asKC(comp) â¤ v · log2 v +m · log2m, (A.10)Although this is only an upper bound, by a counting argument most suchmappings must have a complexity no less than, say, a constant multiple ofthat bound.To compare their Kolmogorov complexity, we can define a ratio as γ âKC(bijection)KC(comp) . Obviously, when m â¤ v, γ â¥ mâvmâ12 , which is larger than 1 aslong as m, v â¥ 2. When m > v, γ â¥ vm log2 v2 log2m , which is also larger than 1when m, v â¥ 2.Actually, there might be some mappings that are not purely compositionalor holistic. For example, we can have a mapping with ziâ¤10 sharing thereused rules while other zi>10 doesnât. Then this type of mapping can berepresented by an element in S10v âS10âSvmâ10 . As a mapping in this subsetshares 10 common rules, its Kolmogorov complexity is between KC(bijection)and KC(comp). Intuitively, for all bijections, smaller KC(Â·) means highercompressibility and higher compositionality.145Appendix BOther Metrics Used in theThesisB.1 Calculation of Expected Calibration ErrorExpected calibration error is a measurement about how well the predictedconfidence represents the true correctness likelihood (C. Guo et al. 2017).For example, if our model gives 100 predictions, each with confidence, say,p(y = v | x) â [0.7, 0.8]). Then we might expect there are 70 â¼ 80 correctpredictions among those 100 ones. To calculate this, we first uniformlydivide [0, 1] into M bins, with each bin represented by Im â(mâ1M ,mM]and Bm is all the x samples whose confidence falls into Im. Then, ECE iscalculated as:ECE =Mâm=1|Bm|n|acc(Bm)â conf(Bm)| , (B.1)where acc(Bm) =1|Bm|âiâBm 1(yËi = vi), conf(Bm) =1|Bm|âiâBm p(y =vi | xi), vi is the true label and yËi = argmax p(y | xi) is the modelâs pre-diction. All the ECEs mentioned in this thesis are calculated by settingM = 10.146Appendix CMore Details about theExperimentsC.1 The Probing Dataset in Chapter 5Besides the 9 types of responses demonstrated in Figure 5.2 in the main con-text, we further extend our probing responses to 14 types, as follows. Fur-thermore, we also create another probing dataset (named Dprobtest) whereall x comes from the test set. Compared with Dprobtrain that we used in themain context, all the prompts and responses in Dprobtest are never exposedto the model during finetuning. By comparing the learning curves of thesetwo probing datasets, we can figure out the difference between the modelâsprediction of those directly influenced responses (y appears during training)and the indirectly influenced ones (y that the model never sees during train-ing). Finally, we believe the level of the âon-policyâ property (which is veryimportant for the preference finetuning, as discussed in Tajwar et al. 2024)could also be introduced as the second axis in our 2-D plane. We left theexploration of this interesting direction in our future work.â¢ YVF: valid feedback following the instruction xu:0. yÏ0 , response generated by feeding xu to LLM before SFT1. y+u , the chosen (i.e., the preferred) response to xu1.1 y+selfr, rephrase y+u using Ï0, algorithm from Zhaorui Yang etal. 2024;1.2 y+gpts, rephrase y+u using ChatGPT, keep the semantics whilechanging the format;1.3 y+gptf, rephrase y+u using ChatGPT, keep the format while chang-ing the semantics;147C.1. The Probing Dataset in Chapter 52. yâu , the rejected (i.e., the less preferred) response to xu2.1 yâselfr, rephrase yâu using Ï0, algorithm from Zhaorui Yang etal. 2024;2.2 yâgpts, rephrase yâu using ChatGPT, keep the semantics whilechanging the format;2.3 yâgptf, rephrase yâu using ChatGPT, keep the format while chang-ing the semantics;â¢ YIF: irrelevant feedback to xu that are still recognizably human lan-guage (in these datasets, roughly âinternet-standardâ English):3. y+j Ì¸=u, chosen response for a different question xj Ì¸=u selected fromthe training set;4. y+test, chosen response of a question xtest selected from test set;5. yhum, a ârandomâ English sentence generated by ChatGPT withas many words as y+u ;â¢ YUn-gram: ungrammatical sequences:6. y+rnd, a random permutation of the words of y+u , or a randomlygenerated sequence;7. yâ²rnd, a random permutation of the words of a generated sentenceas in yhum.148C.1. The Probing Dataset in Chapter 5Below is an instruction that describes a task along with a reference answer. Using the reference answer as a guide, write your own response.### Instruction: {ð±}### Reference Answers: {ð²+}### Response:â¢ Prompt for self-rephrase: ð²ð¬ðð¥ðð«+â¢ Prompt for GPT-rephrase semantics keeping ð²ð ð©ð­ð¬+Given the reference sentence, please generate an output sentence. Please use different words as much as possible while keeping the meaning of the reference sentence unchanged. Please only return the output sentence.### Reference sentence: {ð²+}### Response: â¢ Prompt for GPT-rephrase format keeping ð²ð ð©ð­ð+Given the reference sentence, please generate an output sentence. Please change the meaning of the reference sentence as muchas possible while keeping the format of it. Please only return the output sentence.### Reference sentence: {ð²+}### Response: Figure C.1: The prompts used to generate y+selfr, y+gpts, and y+gptf. Therephrases of rejected samples are generated similarly. The self-rephrase tem-plate comes from Zhaorui Yang et al. 2024.Human: What will remove cigarette odor from a room?Assistant: I believe there are products that will work at removing odors. Can you tell me whatâs going on?Human: What do you mean?Assistant:Prompt: ð±0. Initial responseIâm not a chemist, but Iâm guessing that most of the cigarette odor comes from the burning tobacco. I know we can buy products that reduce the smell of smoke, and that is what most manufacturers claim. Would that also work to remove cigarette odor in your apartment?1. Chosen responseYou mentioned something about a cigarette smell, can you tell me what happened?1.1 Self-rephraseI believe there are products that will work at removing cigarette odor.1.2 GPT-rephrase, semantics keepingYou brought up the issue of a cigarette odor, could you provide me with more details?1.3 GPT-rephrase, format keepingCan you provide details on the odor you noticed earlier?2. Rejected responseIf you tell me about your situation, I can try to give you more help.2.1 Self-rephrase2.2 GPT-rephrase, semantics keepingShould you share details of your circumstances with me, I will endeavor to provide further assistance.2.3 GPT-rephrase, format keepingIf you share your circumstances with me, I'll do my best to provide you with additional assistance.3. Irrelavent from train setLetâs take a look at the next monthâs release schedule. Are there any games youâre particularly excited for?4. Irrelavent from test setYouâve searched through quite a few results and havenât come across a recipe you like yet.5. Random sentenceThe purple cat danced under the starry night sky with joyful abandon.6. Random permuted ð²ð®+me about mentioned can smell, tell happened? You cigarette something you a what7. Pure random tokensyou a through few Youâve recipe yet. and across quite a searched come havenât results likeIâm not sure what you mean. Iâm not sure what youâre asking.Figure C.2: Example of all possible responses for one x in our probingdataset. Note that the pure random token is generated by first creating arandom sentence, then randomly permuting its tokens.149
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Sutherland, Danica J.
Learning dynamics of deep learning -- force analysis of deep neural networks
Text