← back to courses

No Gradients, No Problem?

A Comparative Study of Zeroth-Order Optimizers for Memory-Efficient LLM Fine-Tuning

Maria Pilligua, Nil Biescas, Chengheng Li Chen

EPFL CS-439 Optimization for Machine Learning · Spring 2026 · Final grade 6/6

 Report  Code
Training objectives comparison
Best logit accuracy on MultiRC (Qwen3.5) when each method is trained on cross-entropy, token accuracy, or logit accuracy directly. Cross-entropy wins for almost every method; only Sparse-MeZO holds up across all three.

Fine-tuning LLMs with Adam is memory-hungry because backpropagation stores activations and a gradient for every parameter. Zeroth-order (ZO) optimizers avoid that by estimating the gradient from forward passes alone, keeping memory close to inference. Many ZO variants have appeared recently, but each paper reports on a different model, dataset, and budget, which makes them hard to compare.

We put ten ZO optimizers and an AdamW baseline behind a single training harness, fine-tuning Qwen2.5-0.5B and Qwen3.5-0.8B on two SuperGLUE tasks. Only the optimizer changed between runs. Sparse-MeZO came out on top, five points below AdamW at about a third of the memory, and it was the only ZO method that clearly improved when we moved to the larger backbone. Most of the recent variants did not beat plain MeZO in our setup. We also tested a property unique to ZO, training on non-differentiable accuracy directly instead of on cross-entropy, and found the cross-entropy surrogate still wins for almost every method.