The PEFT & Quantization Field Guide: LoRA, QLoRA, DoRA, AWQ, and FP8

1. Executive Summary#

Fine-tuning and deploying large models within realistic GPU budgets requires mastering two orthogonal techniques: Parameter-Efficient Fine-Tuning (PEFT) for training, and Post-Training Quantization (PTQ) for inference. Treating these as black-box config flags leads to degraded accuracy and suboptimal throughput.

2. PEFT Internals: LoRA, QLoRA, and DoRA#

A. LoRA (Low-Rank Adaptation, Hu et al. 2021)#

Based on the hypothesis that weight updates during adaptation have a low "intrinsic rank," LoRA freezes the pre-trained weight matrix W_0 (dimension d x k) and injects two small trainable matrices A (r x k) and B (d x r), where rank r << min(d, k):

W = W_0 + (alpha / r) * (B * A)

• Zero Inference Overhead: Once trained, B * A is multiplied out and added directly into W_0, adding zero latency at serving time.
• Best Practice: Always apply LoRA to all linear layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), not just attention Q and V. Rank r = 16 to 64 with alpha = 2 * r is the empirical sweet spot.

B. QLoRA (Dettmers et al., 2023)#

Reduces fine-tuning memory by 4x via three innovations:

1. NF4 (4-bit NormalFloat): Pre-trained neural network weights follow a zero-centered normal distribution. NF4 uses information-theoretically optimal quantiles spaced unevenly so each 4-bit bin holds an equal number of weights.
2. Double Quantization: Quantizes the quantization scaling constants themselves from FP32 to FP8, saving ~0.37 bits per parameter.
3. Paged Optimizers: Uses NVIDIA unified memory to automatically page AdamW optimizer states to CPU RAM during sequence-length memory spikes.

C. DoRA (Weight-Decomposed Low-Rank Adaptation, Liu et al. 2024)#

Decomposes pre-trained weights into magnitude (m, a 1 x k vector) and direction (V / ||V||_c), applying LoRA strictly to the directional component while training magnitude directly. DoRA closely mirrors full fine-tuning learning dynamics and consistently beats standard LoRA at lower ranks.

3. Inference Quantization: AWQ vs. GPTQ vs. FP8#

Senior Takeaway: On Hopper (H100) and Ada (L40S) GPUs, W8A8 FP8 is often faster under high batch concurrency than 4-bit W4A16 (AWQ/GPTQ). Why? Because W4A16 must dequantize 4-bit weights back to FP16 in registers before executing FP16 matrix math, making it compute-bound at large batch sizes, whereas W8A8 executes natively inside FP8 Tensor Cores at 2x the FLOP rate.