Want to deploy a large model locally to save money and protect data privacy? That's a great idea!

But diving into the world of models, you're immediately bombarded with various parameters and versions: 7B, 14B, 32B, 70B... So many parameters for the same model, which one should you choose?

What level is my computer at, and which model can it actually run?

Don't panic! This article will help you sort things out, telling you in the simplest way how to choose hardware for local deployment of large models! Guaranteed to clear up your confusion after reading!

There is a Hardware Configuration and Model Size Reference Table at the bottom of this article.

Understanding Large Model Parameters: What Do 7B, 14B, 32B Represent?

• Meaning of Parameters: Numbers like 7B, 14B, 32B represent the number of parameters in a Large Language Model (LLM), where "B" stands for Billion. Parameters can be thought of as the "weights" the model learns during training; they store the model's understanding of language, knowledge, and patterns.
• Parameter Count vs. Model Capability: Generally, the more parameters a model has, the more complex it is. Theoretically, it can learn and store richer information, capturing more complex language patterns, and thus perform more powerfully in understanding and generating text.
• Resource Consumption vs. Model Size: Models with more parameters also require more computational resources (GPU computing power), more memory (VRAM and system RAM), and more data for training and running.
• Small Models vs. Large Models:
  • Large Models (e.g., 32B, 65B, or larger): Can handle more complex tasks, generate more coherent and nuanced text, and may perform better in areas like knowledge Q&A and creative writing. However, they have high hardware requirements and run relatively slower.
  • Small Models (e.g., 7B, 13B): Consume fewer resources, run faster, and are more suitable for running on resource-limited devices or in latency-sensitive applications. Small models can also perform well on some simple tasks.
• The Trade-off in Choice: Choosing model size involves a trade-off between model capability and hardware resources. More parameters don't always mean "better"; you need to choose the most suitable model based on your actual application scenario and hardware conditions.

What Kind of Hardware Do I Need to Run a Local Model?

• Core Requirement: Video RAM (VRAM)

  • Importance of VRAM: When running a large model, the model's parameters and intermediate calculation results need to be loaded into VRAM. Therefore, the size of VRAM is the most critical hardware metric for running local large models. Insufficient VRAM will prevent the model from loading, force you to use only very small models, or severely degrade performance.
  • Bigger is Better: Ideally, having a GPU with as much VRAM as possible is best, allowing you to run larger models and get better performance.

• Secondary Importance: System Memory (RAM)

  • Role of RAM: System RAM is used to load the operating system, run programs, and serve as a supplement to VRAM. When VRAM is insufficient, system RAM can act as "spillover" space, but it's much slower (because RAM is much slower than VRAM) and will significantly reduce model running efficiency.
  • Sufficient RAM is Also Important: It's recommended to have at least 16GB or even 32GB+ of system RAM, especially if your GPU VRAM is limited. More RAM can help alleviate VRAM pressure.

• Processor (CPU)

  • Role of CPU: The CPU is primarily responsible for data preprocessing, model loading, and some model computation tasks (especially when using CPU offloading). A better-performing CPU can improve model loading speed and assist the GPU with computation to some extent.
  • NPU (Neural Processing Unit): Some laptops are equipped with an NPU, a hardware unit specifically designed to accelerate AI computations. An NPU can accelerate specific types of AI operations, including inference for certain large models, thereby improving efficiency and reducing power consumption. If your laptop has an NPU, it's a plus, but the GPU remains the core for running local large models. NPU support and effectiveness depend on the specific model and software.

• Storage (Hard Drive/SSD)

  • Role of Storage: You need enough disk space to store model files. Large model files are typically huge; for example, a quantized 7B model might still be 4-5GB, while larger models can require tens or even hundreds of GBs.
  • SSD is Better than HDD: Using a Solid State Drive (SSD) instead of a Hard Disk Drive (HDD) can significantly speed up model loading.

Hardware Priority

1. Video RAM (VRAM) (Most Important)
2. System Memory (RAM) (Important)
3. GPU Performance (Computing Power) (Important)
4. CPU Performance (Auxiliary Role)
5. Storage Speed (SSD is better than HDD)

What If I Don't Have a Dedicated GPU?

• Integrated Graphics and CPU Running: If you don't have a dedicated GPU, you can still run models using integrated graphics (like Intel Iris Xe) or rely entirely on the CPU. However, performance will be significantly limited. It's recommended to focus on running 7B or even smaller, highly optimized models and use techniques like quantization to reduce resource demands.
• Cloud Services: If you need to run large models but lack local hardware, consider using cloud GPU services like Google Colab, AWS SageMaker, RunPod, etc.

How to Run a Local Model?

For beginners, it's recommended to use some user-friendly tools that simplify the process of running local models:

• Ollama: Operated via command line, but installation and use are very simple, focusing on running models quickly.
• LM Studio: Features a clean and intuitive interface, supporting model download, model management, and one-click running.

Hardware Configuration and Model Size Reference Table

Swipe left/right to see all

X86 Laptops
Laptop with Integrated Graphics (e.g., Intel Iris Xe)	Shared System Memory (8GB+ RAM)	8-bit, even 4-bit quantization	≤ 7B (Heavily Quantized)	* Very basic local running experience, suitable for learning and light experimentation.* Performance is limited, inference speed is slow.* It's recommended to use 4-bit or lower precision quantized models to minimize VRAM usage as much as possible.* Suitable for running small models like TinyLlama.
Entry-level Gaming/Thin-and-Light Laptop with Dedicated GPU (e.g., RTX 3050/4050)	4-8 GB VRAM + 16GB+ RAM	4-bit - 8-bit quantization	7B - 13B (Quantized)	* Can run 7B models relatively smoothly; some 13B models can also run with quantization and optimization.* Suitable for experimenting with some mainstream small to medium-sized models.* Note that VRAM is still limited; running large models will be challenging.
Mid-to-High-End Gaming Laptop/Mobile Workstation (e.g., RTX 3060/3070/4060)	8-16 GB VRAM + 16GB+ RAM	4-bit - 16-bit (Flexible Choice)	7B - 30B (Quantized)	* Can run 7B and 13B models more comfortably and has the potential to try models around 30B (requires good quantization and optimization).* Can choose different quantization precisions based on needs to balance performance and model quality.* Suitable for exploring a wider variety of medium to large models.

---

ARM (Apple M Series)
Raspberry Pi 4/5	4-8 GB RAM	4-bit quantization (or lower)	≤ 7B (Heavily Quantized)	* Limited by memory and computing power, mainly used for running very small models or as an experimental platform.* Suitable for researching model quantization and optimization techniques.
Apple M1/M2/M3 (Unified Memory)	8GB - 64GB Unified Memory	4-bit - 16-bit (Flexible Choice)	7B - 30B+ (Quantized)	* The unified memory architecture makes memory usage more efficient; even an 8GB M-series Mac can run models of a certain size.* Higher memory versions (16GB+) can run larger models and even try models above 30B.* Apple chips have advantages in energy efficiency.

---

NVIDIA GPU Computers
Entry-level Dedicated GPU (e.g., RTX 4060/4060Ti)	8-16 GB VRAM	4-bit - 16-bit (Flexible Choice)	7B - 30B (Quantized)	* Performance is similar to mid-to-high-end gaming laptops, but desktops have better cooling and can run stably for long periods.* Good cost-performance, suitable for entry-level local LLM enthusiasts.
Mid-range Dedicated GPU (e.g., RTX 4070/4070Ti/4080)	12-16 GB VRAM	4-bit - 16-bit (Flexible Choice)	7B - 30B+ (Quantized)	* Can run medium to large models more smoothly and has the potential to try models with even more parameters.* Suitable for users with higher requirements for local LLM experience.
High-end Dedicated GPU (e.g., RTX 3090/4090, RTX 6000 Ada)	24-48 GB VRAM	8-bit - 32-bit (or higher)	7B - 70B+ (Quantized/Native)	* Can run the vast majority of open-source LLMs, including large models (e.g., 65B, 70B).* Can try higher bit precisions (e.g., 16-bit, 32-bit) for optimal model quality, or use quantization to run even larger models.* Suitable for professional developers, researchers, and heavy LLM users.
Server-grade GPU (e.g., A100, H100, A800, H800)	40GB - 80GB+ VRAM	16-bit - 32-bit (Native Precision)	30B - 175B+ (Native/Quantized)	* Designed specifically for AI computing, featuring massive VRAM and extremely strong computing power.* Can run very large models and even perform model training and fine-tuning.* Suitable for enterprise applications, large-scale model deployment, and research institutions.

Table Additional Notes

• Quantization: Refers to reducing the numerical precision of model parameters, for example, from 16-bit floating-point (float16) to 8-bit integer (int8) or 4-bit integer (int4). Quantization can significantly reduce model size and VRAM usage and accelerate inference speed, but may slightly reduce model accuracy.
• Heavy Quantization: Refers to using very low bit-precision quantization, such as 3-bit or 2-bit. Can further reduce resource requirements, but model quality degradation may be more noticeable.
• Native: Refers to running the model at its original precision, such as float16 or bfloat16. Provides the best model quality but has the highest resource requirements.
• Quantized Parameter Range: The "Recommended LLM Parameter Range (After Quantization)" in the table refers to the approximate range of model parameters that the hardware can run smoothly under reasonable quantization. The actual model size and performance that can be run also depend on factors like specific model architecture, degree of quantization, software optimization, etc. The parameter ranges given here are for reference only.
• Unified Memory: A characteristic of Apple Silicon chips where the CPU and GPU share the same physical memory, resulting in higher data exchange efficiency.