Spectral Newton
No mentions found
This entity hasn't been tracked yet, or Iris is still building its knowledge base.
Related Articles from SNS
Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning
Announce Type: replace Abstract: We investigate why deep neural networks suffer from loss of plasticity in continual learning, and thus fail to learn new tasks without reinitializing parameters. We show that this failure is preceded by Hessian spectral collapse at new-task initialization, where meaningful curvature directions vanish and gradient descent becomes ineffective. Analyzing a linearized ReLU network, we derive explicit $\epsilon$-rank conditions for successful training and prove...
Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent
arXiv:2606.02596v1 Announce Type: new Abstract: The curvature exponent $\alpha$ in $h_k \propto \sigma_k^\alpha$ -- governing how Hessian eigenvalues scale with gradient singular values -- varies systematically across layer types ($\alpha \approx 2$ for convolutions, $\approx 1$ for transformer attention, $< 1$ for MLP up-projections). We prove the Spectral Alignment Decomposition: $\alpha = 2 + d\log\Phi_k / d\log\sigma_k$, where $\Phi_k$ measures alignment between Kronecker factor...
Comparison of the potential energy for different equilibrium configurations of symmetric and asymmetric floating drops
arXiv:2602.10120v2 Announce Type: replace Abstract: We provide a numerical method for computing solutions to a free boundary problem arising from the equilibrium state of a floating drop. This numerical method is based on a Newton's method for the underlying nonlinear boundary value problems, and at each iterative step a Chebyshev spectral collocation method is employed. The problems considered here are those that can be described by using generating curves, and include problems in...
Comparison of the potential energy for different equilibrium configurations of symmetric and asymmetric floating drops
arXiv:2602.10120v2 Announce Type: replace-cross Abstract: We provide a numerical method for computing solutions to a free boundary problem arising from the equilibrium state of a floating drop. This numerical method is based on a Newton's method for the underlying nonlinear boundary value problems, and at each iterative step a Chebyshev spectral collocation method is employed. The problems considered here are those that can be described by using generating curves, and include problems in...
A Note on Stability for Orthogonalized Matrix Momentum with Client Sampling
Announce Type: new Abstract: We study finite-sample generalization for a client-sampled distributed optimization scheme with matrix-valued parameters and orthogonalized momentum updates. The central quantity is the gap between the population and empirical objectives at the returned model when only a subset of clients participates in each round. Under independent heterogeneous client data, unequal local sample counts, and fixed aggregation weights, we derive a finite-round upper-tail...
Spectral Scaling Laws of Muon
arXiv:2606.04058v2 Announce Type: replace Abstract: Orthonormalized update rules have rapidly become a leading choice of optimizer for training large language models, with recent open-source state-of-the-art models adopting Muon. To keep these updates tractable, Muon performs the orthonormalization with the Newton--Schulz (NS) iteration. Since NS is only approximate, directions with small singular values fail to be orthonormalized.
Spectral Scaling Laws of Muon
arXiv:2606.04058v1 Announce Type: new Abstract: Orthonormalized update rules have rapidly become a leading choice of optimizer for training large language models, with recent open-source state-of-the-art models adopting Muon. To keep these updates tractable, Muon performs the orthonormalization with the Newton--Schulz (NS) iteration.