Arxiv Daily Digest

We analyze a simple one-hidden-layer neural network with ReLU activation functions and fixed biases, with one-dimensional input and output. We study both continuous and discrete versions of the model, and we rigorously prove the convergence of the learning process with the L^2 squared loss function and the gradient descent procedure. We also prove the spectral bias property for this learning process. Several conclusions of this analysis are discussed; in particular, regarding the structure and properties that activation functions should possess, as well as the relationships between the spectrum of certain operators and the learning process. Based on this, we also propose an alternative activation function, the full-wave rectified exponential function (FReX), and we discuss the convergence of the gradient descent with this alternative activation function.

The self-attention mechanism is central to the success of Transformer architectures. However, standard row-stochastic attention has been shown to suffer from significant signal degradation across layers. In particular, it can induce rank collapse, resulting in increasingly uniform token representations, as well as entropy collapse. Recent work has highlighted the benefits of doubly stochastic attention as a form of entropy regularization. In this paper, we study rank collapse across network depth and show that doubly stochastic attention matrices normalized with Sinkhorn algorithm preserve rank more effectively than standard Softmax row-stochastic ones. We derive a theoretical bound for the pure self-attention rank decay when using Sinkhorn normalization and find that rank decays to one doubly exponentially with depth.

The Polyak-Łojasiewicz (PŁ) condition is often invoked in nonconvex optimization because it allows fast convergence of algorithms beyond strong convexity. A function f on a Riemannian manifold M is globally PŁ if ||∇f(x)||² ≥ 2μ(f(x) − f*) for all x. We show that if f is also smooth and M is contractible, then the PŁ condition imposes a firm global structure: such a function is necessarily of the form f(x) = f* + ||φ(x)||² (a nonlinear sum of squares) where φ is a submersion. The proof hinges on showing that the end-point map of negative gradient flow on f is a trivial smooth fiber bundle over S. This leads to a striking dichotomy: either the minimizer set is diffeomorphic to Euclidean space (and f becomes a convex quadratic under smooth coordinates), or S must display genuinely exotic geometry.

Decentralized stochastic optimization has emerged as a fundamental paradigm for large-scale machine learning. However, practical implementations often rely on biased gradient estimators arising from communication compression or inexact local oracles, which severely degrade convergence. We propose Decentralized Momentum Tracking with Biased Gradients (Biased-DMT). We establish a comprehensive convergence theory in nonconvex settings and show it achieves linear speedup in the number of agents. The analysis decouples network topology from data heterogeneity. When the gradient oracle introduces only absolute bias, the method eliminates structural heterogeneity error and converges to the exact physical error floor.

Decentralized optimization over directed networks is frequently challenged by asymmetric communication and the inherent high variance of stochastic gradients. We propose the Stochastic Momentum Tracking Push-Pull (SMTPP) algorithm, which tracks the momentum term rather than raw stochastic gradients within the Push-Pull architecture. This design decouples the variance reduction capacity from the algebraic connectivity of the graph. SMTPP rigorously compresses the unavoidable steady-state error floor into a minimal neighborhood determined by network connectivity and gradient variance.

Monotone variational inequalities provide a unifying framework for convex minimization, equilibrium computation, and convex-concave saddle-point problems. We develop parameter-free extragradient methods with non-asymptotic last-iterate guarantees. For globally Lipschitz operators, our algorithm achieves an o(1/√T) last-iterate rate. We then extend the framework to locally Lipschitz operators via backtracking line search.