Does Lower Precision Always Save Energy? Benchmarking Energy–Performance Trade-Offs in Weight-Quantized Large Language Model Inference on a GPU
Artykuł w czasopiśmie
MNiSW
140
Lista 2024
| Status: | |
| Autorzy: | Laskowski Jan, Tomiło Paweł, Kozłowski Edward, Laskowska Agnieszka, Piątek Radosław |
| Dyscypliny: | |
| Aby zobaczyć szczegóły należy się zalogować. | |
| Rok wydania: | 2026 |
| Wersja dokumentu: | Elektroniczna |
| Język: | angielski |
| Numer czasopisma: | 20 |
| Wolumen/Tom: | 19 |
| Numer artykułu: | 4893 |
| Strony: | 1 - 21 |
| Impact Factor: | 3,9 |
| Scopus® Cytowania: | 0 |
| Bazy: | Scopus |
| Efekt badań statutowych | NIE |
| Materiał konferencyjny: | NIE |
| Publikacja OA: | TAK |
| Licencja: | |
| Sposób udostępnienia: | Witryna wydawcy |
| Wersja tekstu: | Ostateczna wersja opublikowana |
| Czas opublikowania: | W momencie opublikowania |
| Data opublikowania w OA: | 16 października 2026 |
| Abstrakty: | angielski |
| Large language model (LLM) inference is an increasing component of AI energy demand, while the energy effects of weight quantization depend on workload phase and implementation. We benchmarked four LLM families—Gemma 4 E4B IT, Qwen3.5 4B, Nemotron 3 Nano 4B, and Ministral 3 3B Instruct—across Q4_K_M, Q6_K, Q8_0, and BF16 weight representations on a single NVIDIA GeForce RTX 4090 using llama.cpp/CUDA. Five work- loads separated prefill from autoregressive generation, and GPU energy was estimated from NVML telemetry. In PP-512, no statistically significant differences were detected under the employed measurement protocol; in PP-1024, precision affected total energy and mean GPU power. In all generation scenarios, model family, precision, and their interaction significantly affected the primary endpoints. Relative to BF16, Q4_K_M reduced dynamic energy per token by 63.6–85.7% and increased generation throughput by 85.2–160.2% across all four model families. Ministral 3 3B Instruct–Q4_K_M ranked first in TOPSIS, whereas Nemotron 3 Nano 4B–Q4_K_M required less GPU memory. Findings are limited to single-request execution on one RTX 4090/llama.cpp-CUDA stack; output quality was not assessed, and 50 ms NVML sampling limits interpretation of short prefill measurements |
