AI models for Private chatbot / enterprise assistant
A private chatbot answers staff questions in your own words, from your own material — policies, manuals, past tickets, product data — and runs entirely inside your network, so no question and no document is ever sent to an outside provider.
The engine is a large language model. Its size, measured in billions of parameters (written 8B, 70B and so on), sets both the quality of the answers and the hardware needed to serve them. A smaller model answers straightforward questions quickly on modest hardware; a larger model reasons over longer material and follows complicated instructions, at a higher cost per answer. Company knowledge is supplied by retrieval — the relevant passages are found and handed to the model with each question — rather than by retraining it.
Models in this group take text as input: the user’s question plus passages retrieved from your own documents. Each specification table gives three hardware tiers — Minimum, the smallest setup on which the model runs correctly; Medium, the usual production configuration; and High, a configuration sized for peak volume. The rate is the number of sample inputs processed per hour on that hardware. Use these rates for initial sizing. Before production, benchmark your own data to validate accuracy, latency, throughput and cost. The sample input here is one question answered in about 300 words, with roughly 4,000 words of retrieved context supplied to the model.
Llama 3.3 70B
Vendor: Meta
What it does: a general-purpose assistant model that follows instructions well and writes clear prose in a dozen major languages. The usual first choice for a company-wide assistant where answer quality matters more than cost per answer.
Requirement
Minimum
Medium
High
GPU type
2× RTX 4090 (reduced precision)
1× H100 80 GB
4× H100 80 GB
VRAM
48 GB combined
80 GB
320 GB combined
vCPUs
16
24
64
RAM
64 GB
128 GB
512 GB
Server
2× RTX 4090 24 GB
1× H100 SXM 80 GB
4× H100 SXM 80 GB
Rate (answers/hour)
≈ 120
≈ 600
≈ 3,000
Llama 3.1 8B
Vendor: Meta
What it does: a small assistant model that answers short factual questions and summarises retrieved passages. Fast and cheap enough to put in front of a whole workforce, but it loses accuracy on long chains of reasoning.
Requirement
Minimum
Medium
High
GPU type
RTX 3090
RTX 4090
1× H100 80 GB
VRAM
16 GB
24 GB
80 GB
vCPUs
8
12
24
RAM
32 GB
48 GB
128 GB
Server
1× RTX 3090 24 GB
1× RTX 4090 24 GB
1× H100 SXM 80 GB
Rate (answers/hour)
≈ 500
≈ 1,400
≈ 6,000
Qwen2.5 72B
Vendor: Alibaba Cloud
What it does: an assistant model with unusually broad language coverage, including Chinese, Japanese, Korean, Arabic and most European languages. Chosen when the same assistant must serve offices in several regions.
Requirement
Minimum
Medium
High
GPU type
2× RTX 4090 (reduced precision)
1× H100 80 GB
4× H100 80 GB
VRAM
48 GB combined
80 GB
320 GB combined
vCPUs
16
24
64
RAM
64 GB
128 GB
512 GB
Server
2× RTX 4090 24 GB
1× H100 SXM 80 GB
4× H100 SXM 80 GB
Rate (answers/hour)
≈ 110
≈ 560
≈ 2,800
Mistral Small 3
Vendor: Mistral AI
What it does: a mid-sized model that fits on a single professional card while still handling multi-step instructions. A good balance when one server must serve a department without a queue forming.
Requirement
Minimum
Medium
High
GPU type
RTX 4090
1× L40S 48 GB
1× H100 80 GB
VRAM
24 GB
48 GB
80 GB
vCPUs
12
16
32
RAM
48 GB
64 GB
128 GB
Server
1× RTX 4090 24 GB
1× L40S 48 GB
1× H100 SXM 80 GB
Rate (answers/hour)
≈ 260
≈ 700
≈ 2,200
Gemma 2 27B
Vendor: Google
What it does: a compact model tuned for safe, on-topic replies, which makes it suitable for assistants that face customers as well as staff. It refuses out-of-scope requests more readily than most models its size.
Requirement
Minimum
Medium
High
GPU type
RTX 4090 (reduced precision)
1× L40S 48 GB
2× H100 80 GB
VRAM
20 GB
48 GB
160 GB combined
vCPUs
12
16
48
RAM
48 GB
64 GB
256 GB
Server
1× RTX 4090 24 GB
1× L40S 48 GB
2× H100 SXM 80 GB
Rate (answers/hour)
≈ 200
≈ 620
≈ 2,400
Phi-4 14B
Vendor: Microsoft
What it does: a small model trained heavily on reasoning tasks, so it handles arithmetic and step-by-step logic better than its size suggests. Suited to assistants that answer questions about figures and rules.
Requirement
Minimum
Medium
High
GPU type
RTX 3090
RTX 4090
1× L40S 48 GB
VRAM
16 GB
24 GB
48 GB
vCPUs
8
12
24
RAM
32 GB
48 GB
96 GB
Server
1× RTX 3090 24 GB
1× RTX 4090 24 GB
1× L40S 48 GB
Rate (answers/hour)
≈ 380
≈ 1,100
≈ 3,200
DeepSeek-V3
Vendor: DeepSeek
What it does: a very large model that activates only part of itself for each question, so it reasons at close to frontier quality while costing less per answer than its total size implies. Reserved for assistants doing genuinely hard analytical work.
Requirement
Minimum
Medium
High
GPU type
Not practical below 8 cards
8× H100 80 GB
16× H100 80 GB
VRAM
—
640 GB combined
1,280 GB combined
vCPUs
—
96
192
RAM
—
1 TB
2 TB
Server
—
8× H100 SXM 80 GB
2× 8× H100 SXM 80 GB
Rate (answers/hour)
—
≈ 900
≈ 2,000
Choosing between them
The right size depends on how hard the questions are and how many people ask them at once. Our consultants review the material the assistant must cover, the reasoning the answers require and your expected number of daily users, then recommend a model and a serving configuration to match.
At the start of a project we may run a short proof of concept on a sample of your own data, measuring the accuracy and the throughput the model actually achieves on your material. That replaces the estimates on this page with real figures, so the cost and the schedule for the full engagement are known before it is committed.
Share a representative sample, expected volume, latency target and deployment location for Private chatbot / enterprise assistant. We benchmark the shortlisted models, recommend the lowest-cost GPU configuration that meets the target, and scale it from pilot capacity to a dedicated production cluster.