Analyzing Quantized Small Language Models for Efficient Edge Deployment

Ⅰ. Introduction

The emergence of Transformer-based architectures^{[ 1]} has transformed the field of natural language processing (NLP), giving rise to large language models (LLMs) that exhibit exceptional performance in tasks such as language comprehension and generation. Models like OpenAI o1, GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet have showcased the ability to produce human-like text, sparking widespread academic interest and commercial applications. Simultaneously, the availability of open-source LLMs such as Meta’s Llama has democratized access to advanced language models, fostering a growing ecosystem for their distribution, deployment, and utilization in various services and platforms, including Ollama, LangChain, vLLM, and Hugging Face.

The utilization of LLMs is hindered by substantial computational and memory requirements, which poses significant challenges for deployment on resource- constrained edge devices. To mitigate these challenges, the concept of small language models (SLMs) has emerged as a promising alternative for edge deployment. Based on the literature, SLMs are often defined as having fewer than one billion parameters^{[ 2]} or less than 10 billion parameters^[3]. These scaled-down models are more suitable for deployment on resource-constrained edge devices, as they exhibit reduced computational and memory footprints.

While existing research has examined various characteristics of LLMs, such as inference latency, memory usage, and accuracy^[4], these studies primarily focus on high-performance server hardware, neglecting the distinct constraints of edge devices. Assessments of small language models specifically designed for edge deployment remain limited. This gap is crucial, as the successful integration of SLMs could unlock a diverse range of NLP applications on edge platforms, including ondevice assistants, context-aware IoT systems, and realtime translation services while maintaining acceptable performance levels.

Quantization methods^[5,6] have demonstrated the potential to substantially reduce the memory requirements and computational complexity of language models, thereby facilitating the deployment of large-scale models on resource-constrained edge platforms like the NVIDIA Jetson devices. Previous studies have investigated the application of quantization techniques in general neural networks^[7] and diverse NLP applications^[8,9]. However, the intricate relationship between model scale, quantization approach, and performance on edge hardware remains an area that requires further exploration.

Building upon prior research on quantization, model evaluation, and SLMs, this study extends those insights to edge computing. Specifically, we provide a comprehensive assessment of quantized SLMs on resource- constrained hardware. We evaluate the performance and efficiency-measured in terms of inference time, resource utilization, and power consumption- of 72 quantized models. These models, which range from 1 billion to 3 billion parameters and incorporate various quantization configurations, are designed for deployment on devices such as the NVIDIA Jetson Orin Nano. Our contributions are threefold: (1) we assess the viability of deploying quantized SLMs on edge devices; (2) we investigate the trade-offs between model accuracy and efficiency on resource-constrained hardware; and (3) we provide practical insights for selecting models and quantization configurations to support efficient NLP service deployments on edge devices.

Ⅱ. Related Work

Small Language Models: Wang et al.^[10] have provided a detailed survey of small language models, clarifying their definitions and exploring their use cases in resource-constrained environments. Their work underscores the potential of SLMs to deliver competitive performance with significantly reduced parameter counts, laying the groundwork for further exploration of these models in edge hardware applications. Our study builds upon these insights by specifically evaluating quantized SLMs on edge devices, extending the analysis to scenarios with tight resource budgets.

Quantization Techniques: Jacob et al.^[7] demonstrated the feasibility of integer-arithmetic-only inference through quantization, enabling significant model size reduction while maintaining acceptable accuracy levels. This pioneering research has influenced subsequent advancements in quantization techniques. Building upon this foundation, Gholami et al.^[5] provided a comprehensive survey of various quantization methods, exploring their applications, challenges, and effectiveness across a range of scenarios. Lee et al.^[9] conducted an extensive evaluation of quantized, instruction- tuned LLMs, analyzing models with up to 405 billion parameters. Their research highlights the trade-offs between quantization methods and model performance at scale. However, their focus on servergrade hardware limits the applicability of their findings to edge deployments, where computational resources and memory are constrained.

Model Storage Formats: The development of efficient model storage formats, such as GGUF (GPT-Generated Unified Format)^[11], has been instrumental in enabling the deployment of quantized models on devices with limited resources. GGUF supports diverse quantization configurations while offering enhanced metadata storage capabilities. This extensibility ensures that models can be seamlessly loaded and executed across multiple platforms, facilitating systematic experimentation and reproducibility. Our work leverages the GGUF format to manage a variety of quantization configurations in a unified manner.

Ⅲ. Methodology

3.1 Hardware and Software Setup

Hardware: All experiments were conducted on an NVIDIA Jetson Orin Nano, which features a 1024-core NVIDIA Ampere GPU with 32 Tensor Cores, a 6-core Arm Cortex-A78A CPU, and 8GB of LPDDR5 memory.

Oper ating System: Ubuntu 22.04 LTS (64-bit).

LLM Serving Fr amework: We used the Ollama framework to serve the quantized LLMs due to its support for various quantization methods and compatibility with edge devices. The Ollama provides prequantized model variants, ensuring efficient execution on the Jetson device.

3.2 Model and Configuration

The models selected for evaluation comprise Meta’s Llama 3.2 (1B and 3B) and Alibaba’ s Qwen 2.5(1.5B and 3B). These represent two modern families of open-source SLMs. We applied 13 distinct quantization configurations using the GGUF storage format to each model to reduce both memory and computational overhead on edge devices. The configurations, denoted as ‘q2_K’, ‘q3_K_L’, ‘q3_K_M’, ‘q3_K_S’, ‘q4_0’, ‘q4_1’, ‘q4_K_M’, ‘q4_K_S’, ‘q5_0’, ‘q5_1’, ‘q5_K_M’, ‘q5_K_S’, and ‘q6_K’, are explained as follows:

· 'q2' to 'q6' (Quantization Bit-Width): The numeral denotes the bit-width used to represent each weight. For instance, ‘q2’ indicates a 2-bit precision, while ‘q6’ signifies a 6-bit precision. Generally, a higher bit-width allows for a more nuanced representation of the model’s weights, which can better preserve accuracy but at the cost of increased resource usage.

· ‘0’ and ‘1’ (Quantization Types): The suffixes ‘0’ and ‘1’ indicate the quantization type as defined in frameworks like llama.cpp. In ‘type-0’ quantization, each weight is reconstructed as w = dq, where d is the block scale, and q is the quantized value. In ‘type-1’ quantization, the reconstruction uses w = dq + m, where m represents the block minimum. These types differ in whether an offset (minimum) is used for each quantization block.

· ‘K’ (Quantization Method): The presence of a K indicates the use of K-Quantization, a technique that divides the model’s weights into large blocks, which are further split into smaller subblocks. Each sub-block is assigned its scale and minimum value, which are then quantized to a limited number of bits based on the target bitwidth. This approach helps the model maintain its performance despite the reduced precision by capturing distribution differences within large weight blocks.

· ‘L’, ‘M’, and ‘S’ (Quantization Var iants): These letters denote variant schemes to balance accuracy and efficiency by adjusting the bit-widths used for different parts of the model. In the ‘L’ (Large) variant, critical tensors are quantized with a higher bit-width to preserve precision, while all other tensors use the target bit-width. The ‘M’ (Medium) variant employs an intermediate bit-width for the critical tensors- lower than that of the ‘L’ variant but higher than the base target-thus compromising performance and compression. The ‘S’ (Small) variant uniformly applies the base target bit-width across all tensors, maximizing quantization efficiency but potentially sacrificing some accuracy.

For clarity, Table 1 provides a summary of the quantization configuration nomenclature and their key characteristics.

We chose the above models and quantization settings (as provided by the Ollama framework) following an initial screening based on prior works^[2,3] and pilot experiments on the Jetson Orin Nano (8GB memory). The selection of Llama 3.2 and Qwen 2.5 was motivated by their complementary architectural and training characteristics, ensuring our findings’ generalizability and robustness. Llama 3.2, introduced by Meta in September 2024, includes lightweight models with 1B and 3B parameters and multimodal variants with 11B and 90B parameters. This study specifically utilizes the lightweight versions optimized for efficient deployment on edge devices. The architecture features enhancements such as Grouped-Query Attention (GQA), significantly improving inference scalability. Furthermore, instruction-tuned versions of Llama 3.2 have been aligned with human preferences using supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF)^[12]. Qwen 2.5, developed by Alibaba and also released in September 2024, complements Llama 3.2 by providing an alternative architecture pre-trained on an extensive corpus comprising 18 trillion tokens. Qwen 2.5 excels particularly in knowledge-intensive and reasoning tasks. Its variants, fine-tuned for instruction following and structured outputs, utilize RLHF extensively to enhance alignment and model responsiveness^[13]. Including these two distinct model families-differentiated by Western and Chinese origins, distinct pre-training datasets, and varying tokenization strategies- strengthens the general applicability of our study. The comprehensive combination of models and quantization strategies (2 model families × 2 model sizes × 13 quantization configurations = 72 variants) facilitated an in-depth evaluation of how quantization affects key performance metrics, including storage efficiency, inference speed, and output accuracy.

3.3 Benchmark Dataset

We used a subset of the MMLU-Pro benchmark^[14] to evaluate model accuracy, ensuring consistent and challenging test conditions across different models and quantization configurations. Specifically, we selected three knowledge domains from MMLU-Pro: computer science (410 questions), engineering (969 questions), and mathematics (1351 questions). MMLU-Pro is a more challenging extension of the original MMLU(Massive Multitask Language Understanding) benchmark^[15], featuring complex reasoning questions and an increased number of answer choices (up to 10) to reduce the chance of correct guesses by random chance. Each question in these subsets is multiple-choice with one correct answer and 9 distractors, making naive guessing only 10% likely to succeed.

3.4 Evaluation Protocol

For each model variant, we measured several metrics: accuracy, inference time, resource utilization, and power consumption. Accuracy on MMLU-Pro was computed as the fraction of questions answered correctly. We report accuracy under two conditions: including random guesses and excluding random guesses. This distinction is made because the evaluation script is designed such that if a model’ s generated response does not contain a recognizable answer option, a random answer is selected on its behalf (simulating a guess). Given the benchmark’ s format of up to 10 choices, a random guess has a 10% chance of being correct. Including these guesses in accuracy can inflate the score of very poorly performing models. Therefore, we also calculate accuracy, excluding those cases where we remove all questions where the model failed to produce an answer and a guess was inserted. The “excluding” accuracy reflects the model’ s performance on the subset of questions it meaningfully attempted. By comparing both, we can gauge how much a model relies on guesswork. We also recorded the inference time per question (averaged over all queries in a domain), using Ollama’ s timing logs on the Jetson. CPU and GPU utilization were monitored via system tools (with GPU utilization focusing on the Jetson’ s integrated GPU), and power consumption was measured using the Jetson’ s onboard power meter, capturing average power draw (in milliwatts, mW) during inference on each domain. These measurements allow us to evaluate efficiency in terms of both computational load and energy usage. All experiments were run in a controlled setting on the Jetson (performance mode enabled, consistent ambient conditions) to ensure fair comparison across models and configurations.

Ⅳ. Experimental Results

This section presents an in-depth analysis of the experimental results for the quantized Llama 3.2 and Qwen 2.5 models. We first discuss accuracy outcomes (with and without random guess adjustments) across the different configurations, then examine inference latency and subsequently analyze the relationship between model size and performance. Finally, we consider resource utilization patterns and provide qualitative insights into the practical usability of these models.

4.1 Accuracy Analysis

Tables 2 and 3 summarize the accuracy of Llama 3.2 and Qwen 2.5 models on the MMLU-Pro benchmark, both including and excluding random guesses. The highest accuracy in each category is shown in bold. Each value represents accuracy, while the numbers in parentheses indicate the number of questions answered correctly over the total number of questions attempted. For instance, in Table 1, Llama 3.2 1B q2_K achieves 0.115 (47/410) when including random guesses, indicating 47 out of 410 questions were correct (i.e., 11.5% accuracy). Excluding random guesses, the model’ s accuracy is 0.125 (43/345), meaning 43 out of 345 questions were correct (12.5% accuracy). From these figures, we can infer that 65 questions were randomly guessed, of which 4 were answered correctly. The MMLU-Pro benchmark evaluates language models on multiple-choice questions, with an average of 9.188, 9.583, and 9.771 answer choices for computer science, engineering, and math, respectively. This corresponds to probabilities of 0.109, 0.104, and 0.102, respectively, for randomly selecting the correct answer in each domain. Conventional MMLU-Pro evaluations include random guesses when a model’ s response is unextractable, but this paper examines results both with and without random guesses to offer deeper insights into model performance. Excluding random guesses provides a clearer picture of a model’ s predictive capability, as random responses can skew accuracy metrics. Notably, even small quantized models achieve higher accuracy when random guesses are excluded, underscoring their non-trivial predictive abilities.

Tables 2 and 3 reveal significant performance differences among categories. Generally, the math category exhibits the best performance, followed by computer science and engineering. Moreover, the optimal quantization technique varies by category. For example, Qwen 2.5 3B q4_1 performs best in math and computer science, yet in engineering it is surpassed by q5_K_S, which shows 5.9% higher accuracy. Similarly, for Llama 3.2 3B, the best-performing quantizer differs across categories. These variations indicate that even within MMLU-Pro results, it is essential to evaluate model performance by the required category.

When comparing Tables 2 and 3, Qwen 2.5 outperforms Llama 3.2 according to the highest accuracy criteria. In the math category, Qwen 2.5 3B q4_1 achieves 51.4%, notably higher than Llama 3.2 3B q5_K_S at 40.2%. However, Qwen 2.5 exhibits considerable performance variability across quantization configurations, particularly at low bit configurations (‘q2_K’, ‘q3_K_S’, ‘q3_K_M’, ‘q3_K_L’). Notably, Qwen 2.5 3B q2_K failed to extract any answers in engineering and math, and only three were extractable in computer science-all of which were incorrect, as shown in Table 2.

4.2 Inference Time Analysis

Tables 4 to 7 present the inference times for Llama 3.2 1B, Llama 3.2 3B, Qwen 2.5 1.5B, and Qwen 2.5 3B models across various quantization configurations. Each reported value is the mean, with parentheses showing the standard deviation. The inference time varies substantially by MMLU-Pro category, typically increasing from computer science to engineering and then math. Many outliers emerge at lower bit configurations: in Qwen models, q2_K on both 1.5B and 3B is particularly affected, while in Llama models, q3_K_S, q3_K_M, and q3_K_L on the 3B model exhibit notable outliers.

Figures 1 and 2 provide a more granular view of inference times per question for selected quantization settings (q2_K, q3_K_S, q4_K_M). Fig. 1 displays Llama 3.2 1B and 3B results for computer science, while Fig. 2 shows Qwen 2.5 1.5B and 3B in the same domain. Each dot represents the inference time for a single question, and the red dashed line indicates the average. For Qwen 2.5 1.5B and 3B, q2_K incurs very high inference times (23.366 and 137.915, respectively), attributable to numerous values exceeding 100. As quantization configurations increase, inference times become more consistent.

4.3 Relationship Between Accuracy, Inference Time, and Model Size

Figure 3 illustrates the relationship between accuracy and inference time, while Figure 4 shows how accuracy varies with model size. Figure 5 depicts the relationship between inference time and model size. In this context, "accuracy" refers to the average performance across computer science, engineering, and math domains, including random guesses. In Figures 3 and 4, Qwen 2.5 3B q4_1 achieves the highest overall accuracy at 40%, surpassing Qwen 2.5 3B q5 and q6. For Llama 3.2 3B model, the q6_K configuration yields the highest accuracy at 33.04%. For Llama 3.2 3B, q3_K_S, q3_K_M, and q3_K_L configurations result in higher inference times while maintaining fair accuracy. In contrast, the same configurations applied to Qwen 2.5 3B lead to moderate inference times but lower accuracy. Additionally, Qwen 2.5 3B models using the q4_K_S, q4_0, and q4_1 configurations exhibit higher accuracy with longer inference times, whereas q2_K configuration produces very low accuracy with longer inference times. Comparing Qwen 2.5 1.5B and Llama 3.2 1B models, Qwen 2.5 1.5B generally shows lower inference times and higher accuracy across most quantization configurations-even though it has more parameters than Llama 3.2 1B. Similarly, Qwen 2.5 3B often outperforms Llama 3.2 3B in terms of both accuracy and inference speed.

We also revealed distinct clustering patterns as presented in Figures 4 and 5: Llama models are relatively well-clustered, whereas Qwen models exhibit great dispersion. This variability suggests that selecting the optimal quantization configuration for Qwen models is necessary and requires extensive experimentation. For example, even though Qwen 2.5 1.5B q6_K and Qwen 2.5 3B q2_K configurations have similar sizes, Qwen 2.5 1.5B q6_K significantly outperforms Qwen 2.5 3B q2_K in both accuracy and inference time.

4.4 Resource Utilization and Power Consumption Analysis

Deploying models on edge devices requires raw inference time and understanding of how they utilize hardware resources (CPU/GPU) and how much power they draw. During our tests, we monitored CPU load, GPU load, and power usage on the Jetson Orin Nano. Tables 8 to 11 present the resource utilization and power consumption for the Llama 3.2 and Qwen 2.5 models under each quantization configuration.

· Llama 3.2 1B Models: CPU utilization ranges from 1.726% to 3.587%, while GPU utilization peaks at 90.094%. Power consumption varies between 9825.426 mW and 11311.145 mW. Highbit configurations exhibit elevated power consumption despite reduced CPU load, likely due to increased GPU activity.

· Llama 3.2 3B Models: CPU utilization remains low (1.201%-2.398%), but GPU utilization fluctuates significantly, from 62.607% to 85.782%. Power consumption spans 9547.107 mW to 11458.084 mW. The q4_K_S configuration demonstrates the highest power draw, while q3_K_S balances lower GPU utilization with reduced energy consumption.

· Qwen 2.5 1.5B Models: CPU utilization ranges from 2.295% to 4.288%, with GPU utilization between 69.554% and 84.696%. Power consumption peaks at 10515.761 mW and drops to 9447.782 mW, indicating that lower-bit quantizations enhance energy efficiency.

· Qwen 2.5 3B Models: CPU utilization is modest (1.372%-3.383%), but GPU utilization varies widely, from 63.123% to 84.773%. Power consumption ranges from 9194.816 mW to 10667.693 mW. The q4_K_S configuration incurs the highest power cost, while q3_K_S achieves the lowest.

These findings underscore that one must align the quantization configuration with deployment priorities when optimizing for edge deployment. If the priority is energy efficiency, one might avoid configurations that cause long tail latencies or GPU thrashing (even if they save memory), because they can consume more power. If the priority is maximum accuracy, one might choose a higher bit-width and accept a moderate increase in power. Our analysis provides concrete data for these trade-offs. Such data can guide decisions depending on whether an application (say, a battery- powered IoT device vs. a plugged-in smart kiosk) values efficiency over accuracy or vice versa.

4.5 Qualitative Evaluation and Use Cases

While the above quantitative results shed light on performance metrics, it is equally important to consider qualitative aspects and the usability of these quantized models in real application scenarios. Ultimately, an edge-deployed language model must deliver useful responses to end-users within acceptable time frames. Here, we discuss some qualitative observations and the practical implications of our findings.

Firstly, the user-perceived quality of answers can degrade in non-obvious ways when models are heavily quantized. For configurations like q2_K or q3_K_S that showed very low accuracy, the content of their outputs often reflected confusion or failure. For example, in our trials, Qwen 2.5 3B with q2_K frequently produced irrelevant or malformed answers, or no answer at all, for many questions. This aligns with the quantitative result that it answered zero questions correctly. This model would be unusable in a practical setting such as an on-device engineering Q&A assistant a user would receive either incorrect answers or an “I don’ t know” far too often. In contrast, a model with slightly higher precision (say Qwen 2.5 3B q4_1) might produce mostly correct and fluent answers in the same scenario, resulting in a dramatically better user experience. This highlights that beyond accuracy percentages, there is a threshold of quality below which a model’ s outputs may be nonsensical or unhelpful to a human. Our results suggest that quantization beyond 4-bit for complex QA tasks crosses that threshold for these models.

Secondly, consider application scenarios like ondevice virtual assistants, real-time translators, or voicecontrolled IoT devices (as mentioned in the Introduction). Such applications require accuracy, real- time interaction constraints, and user satisfaction considerations. For instance, a real-time translator deployed on a mobile device must generate translations quickly and accurately. Suppose we were to deploy Llama 3.2 1B q3_K_S (3-bit, small variant) to such an application. In that case, a user might experience very fast responses (due to the small model and aggressive quantization), but the translations could be garbled or incorrect, rendering the service ineffective. On the other hand, using Llama 3.2 3B q5_0 (5-bit, type-0) might yield slower responses but much more reliable translations. User feedback in such a case would likely favor the slightly slower but more accurate model. This trade-off between speed and output quality must be tuned to the use case: for a casual chatbot that doesn’t require high factual accuracy, a highly quantized small model might suffice, and the occasional nonsensical answer might be acceptable or even humorous. However, for a critical task (say, a medical information assistant on an edge device), one would choose the configuration that maximizes answer correctness, even if it means using more power or time.

We did not perform a formal user study, but qualitatively, we found that excluding random guesses from the evaluation (as discussed earlier) is a proxy for removing bogus answers. Users in a real scenario effectively “exclude” random guesses themselves by disregarding nonsensical responses. Our observation that excluding guesses modestly boosts measured accuracy suggests that some model outputs -especially from low-bit models-are essentially random. Thus, from a usability standpoint, those configurations have a high rate of unacceptable outputs. For practical deployment, one might implement safeguards such as confidencebased answer rejection. Indeed, one could design the system to detect when the model is likely guessing (e.g., low likelihood scores) and either prompt the user again or offload the query to a larger model in the cloud if possible. This difference could affect user trust; users might prefer a system that occasionally says “I’m not sure” (an understandable response) over one that returns gibberish. In summary, our practical evaluation suggests that there is a floor on quantization for real-world use beyond which the user experience degrades sharply. In our experiments, 4-bit quantization (especially with type-0/ 1 or K with some higher-precision variant) tended to be the lowest precision that still maintained reasonably coherent outputs across all domains. Configurations at 2-bit or 3-bit often led to outputs that would not meet user expectations in many applications. Therefore, developers targeting edge deployment should consider the qualitative behavior of models. Additionally, one can ensure better user satisfaction by aligning model choice with the application. These qualitative insights complement the quantitative metrics, reinforcing that the “best” configuration depends on the context of use, not just on raw performance numbers.

Ⅴ. Conclusions

This study presents the first systematic evaluation of quantized SLMs for deployment on resourceconstrained edge devices, focusing on the NVIDIA Jetson Orin Nano. By analyzing 72 quantized models of Llama 3.2 (1B and 3B parameters) and Qwen 2.5 (1.5B and 3B parameters) across 13 quantization configurations (q2_K to q6_K), we demonstrate the potential and limitations of these models for edge deployment. Our findings indicate that Qwen 2.5 models generally achieve higher accuracy and lower latency than Llama 3.2 models of similar size. Still, Qwen models are also more sensitive to aggressive quantization (suffering steep accuracy drops at low bit-widths). Excluding random guesses from evaluation improves the measured accuracy, while low-bit configurations (e.g., Qwen 2.5 3B q2_K) rely heavily on guessing due to the failure of answer extraction. We also observed that accuracy and inference time vary significantly across domains (computer science, engineering, and math), highlighting the importance of domain-specific model selection and quantization tuning. Resource utilization analysis shows that CPU utilization remains low (1-4%), while GPU was the main workhorse, reaching up to 90% utilization. Power consumption ranged from 9.2 W to 11.5 W under load. These results underscore the need to balance computational efficiency and energy consumption when deploying quantized models on edge devices.

Although our experiments were conducted on a specific hardware platform (Jetson Orin Nano), the insights are also relevant to other edge platforms. The qualitative trends we observed are likely to generalize across different hardware architectures. For edge devices that lack a GPU and rely on CPUs, quantization would remain crucial for fitting models in memory, though such devices may experience overall slower inference; the relative performance differences between, say, 4-bit and 6-bit models might be similar, but absolute latencies would increase. One could potentially deploy slightly larger models or use higher precision without exceeding resource limits on more capable edge accelerators or devices with greater memory (for example, a Jetson AGX Orin). Still, the same trade-off between efficiency and accuracy would apply. We expect that the optimal quantization strategy (e.g., choosing 4-bit vs. 8-bit) might shift with hardware – some processors are optimized for certain bit-widths- yet the core finding stands: moderate quantization offers big wins with tolerable impact on accuracy, whereas extreme quantization risks making the model ineffective. Our discussion on hardware generalizability is necessarily qualitative; verifying these hypotheses on diverse edge hardware (GPUs, CPUs, NPUs, DSPs across different vendors) is an important avenue for future work.

In addition to extending across hardware, future work should examine mixed-precision quantization strategies, where different layers of the model are quantized to different bit levels based on their sensitivity. Hardware-specific optimizations could further improve performance. Moreover, expanding evaluations to include multimodal and multilingual tasks will be valuable, especially since models like Llama 3.2 and Qwen 2.5 are inherently multilingual and some have vision or audio capabilities. We also suggest that future research incorporate user-centric evaluations (e.g., human feed-back on output quality, usability studies in actual edge applications) to complement quantitative metrics - addressing the gap between benchmark performance and real-world user satisfaction. By tackling these areas, the community can build on the findings of this work to enable more robust and energy-efficient NLP services on the next generation of resource-constrained edge devices.