Efficient Lightweight Vision–Language Model for Thermal Image Captioning and Question Answering on Mobile Devices
Artykuł w czasopiśmie
MNiSW
100
Lista 2024
| Status: | |
| Autorzy: | Baran Katarzyna |
| Dyscypliny: | |
| Aby zobaczyć szczegóły należy się zalogować. | |
| Rok wydania: | 2026 |
| Wersja dokumentu: | Drukowana | Elektroniczna |
| Język: | angielski |
| Wolumen/Tom: | 14 |
| Strony: | 166754 - 166764 |
| Impact Factor: | 4,2 |
| Scopus® Cytowania: | 0 |
| Bazy: | Scopus |
| Efekt badań statutowych | NIE |
| Materiał konferencyjny: | NIE |
| Publikacja OA: | TAK |
| Licencja: | |
| Sposób udostępnienia: | Witryna wydawcy |
| Wersja tekstu: | Ostateczna wersja opublikowana |
| Czas opublikowania: | W momencie opublikowania |
| Data opublikowania w OA: | 27 października 2026 |
| Abstrakty: | angielski |
| Thermal imaging provides robust visual information in low-light, privacy-preserving scenarios, but integrating it into multimodal language models for mobile deployment is challenging due to computational limits. This paper proposes ThermalMobileVLM, a lightweight vision-language model that replaces the standard vision encoder with a custom efficient CNN optimized for thermal images. Fine-tuned on thermal face emotion datasets and outdoor scenes, the model supports high-quality captioning and visual question answering. Through structured pruning and mixed-precision quantization, it achieves real-time performance on smartphones, reducing model size by 85% compared to MobileVLM-1.7B (from 3,450 MB to 510 MB after INT4/INT8 quantization) while retaining 98.7% of the uncompressed model’s VQA accuracy. Through comprehensive qualitative analysis, human evaluation on a human-written caption subset, feature-inversion privacy analysis, and comparisons with state-of-the-art models including GPT-4V and Qwen-VL, we show that thermal-specific architectural design provides clear and measurable benefits in both accuracy and efficiency compared to generic alternatives, even after fine-tuning. Experiments show its advantages in privacy-sensitive and low-visibility applications, including affective computing and night-time perception. |
