Model reference — Documents
Duplicate detection finds text that says the same thing twice — the same ticket raised through two channels, the same contract stored in two systems, the same paragraph copied between reports — including where the wording differs.
Exact duplicates are trivial to find by comparing checksums. The useful work is near-duplicates: a document reformatted, a paragraph lightly edited, a claim submitted twice with a different date. That requires comparing meaning, which is done by fingerprinting every text and looking for fingerprints that sit close together. Because comparing everything against everything is expensive, the practical pipeline is a cheap filter first, then a careful comparison of the pairs that survive.
Models in this group take text or whole documents as input: plain text, PDFs, scanned pages and office files. Each specification table gives three hardware tiers — Minimum, the smallest setup on which the model runs correctly; Medium, the usual production configuration; and High, a configuration sized for peak volume. The rate is the number of sample inputs processed per hour on that hardware. Use these rates for initial sizing. Before production, benchmark your own data to validate accuracy, latency, throughput and cost. The sample input here is one document of about 500 words fingerprinted for comparison.
Vendor: Beijing Academy of Artificial Intelligence
What it does: fingerprints each document so near-duplicates sit close together even when reworded or translated. The strongest general choice, and it handles long documents without splitting.
| Requirement | Minimum | Medium | High |
|---|---|---|---|
| GPU type | RTX 3060 | RTX 4090 | A100 80 GB |
| VRAM | 8 GB | 24 GB | 80 GB |
| vCPUs | 6 | 12 | 24 |
| RAM | 16 GB | 32 GB | 64 GB |
| Server | 1× RTX 3060 12 GB | 1× RTX 4090 24 GB | 1× A100 SXM 80 GB |
| Rate (documents/hour) | ≈ 6,000 | ≈ 30,000 | ≈ 90,000 |
Vendor: Microsoft
What it does: a smaller, quicker fingerprinting model, well suited to a large collection where a slightly higher miss rate is acceptable in exchange for cost.
| Requirement | Minimum | Medium | High |
|---|---|---|---|
| GPU type | RTX 3060 | RTX 4090 | A100 80 GB |
| VRAM | 6 GB | 24 GB | 80 GB |
| vCPUs | 4 | 8 | 24 |
| RAM | 16 GB | 32 GB | 64 GB |
| Server | 1× RTX 3060 12 GB | 1× RTX 4090 24 GB | 1× A100 SXM 80 GB |
| Rate (documents/hour) | ≈ 9,000 | ≈ 45,000 | ≈ 140,000 |
Vendor: Microsoft (Sentence-Transformers)
What it does: a very small fingerprinting model used as the cheap first filter over millions of documents, narrowing the field before a stronger model compares the survivors.
| Requirement | Minimum | Medium | High |
|---|---|---|---|
| GPU type | RTX 3060 | RTX 4090 | A100 80 GB |
| VRAM | 4 GB | 24 GB | 80 GB |
| vCPUs | 4 | 8 | 24 |
| RAM | 8 GB | 32 GB | 64 GB |
| Server | 1× RTX 3060 12 GB | 1× RTX 4090 24 GB | 1× A100 SXM 80 GB |
| Rate (documents/hour) | ≈ 30,000 | ≈ 120,000 | ≈ 400,000 |
Vendor: Princeton NLP
What it does: tuned for sentence-level similarity, which is what finds a copied paragraph inside two otherwise different documents.
| Requirement | Minimum | Medium | High |
|---|---|---|---|
| GPU type | RTX 3060 | RTX 4090 | A100 80 GB |
| VRAM | 6 GB | 24 GB | 80 GB |
| vCPUs | 4 | 8 | 24 |
| RAM | 16 GB | 32 GB | 64 GB |
| Server | 1× RTX 3060 12 GB | 1× RTX 4090 24 GB | 1× A100 SXM 80 GB |
| Rate (documents/hour) | ≈ 20,000 | ≈ 80,000 | ≈ 240,000 |
Vendor: Microsoft
What it does: reads two documents together and judges whether they are duplicates, which is far more accurate than comparing fingerprints and is therefore used to confirm the pairs the filter proposed.
| Requirement | Minimum | Medium | High |
|---|---|---|---|
| GPU type | RTX 3060 | RTX 4090 | A100 80 GB |
| VRAM | 6 GB | 24 GB | 80 GB |
| vCPUs | 4 | 8 | 24 |
| RAM | 16 GB | 32 GB | 64 GB |
| Server | 1× RTX 3060 12 GB | 1× RTX 4090 24 GB | 1× A100 SXM 80 GB |
| Rate (document pairs/hour) | ≈ 12,000 | ≈ 50,000 | ≈ 140,000 |
The right pipeline depends on the size of your collection and what counts as a duplicate in your business — an identical claim and a similar one need different treatment. Our consultants review your data and your definition, then recommend the filter, the comparison model and the similarity threshold.
At the start of a project we may run a short proof of concept on a sample of your own data, measuring the accuracy and the throughput the model actually achieves on your material. That replaces the estimates on this page with real figures, so the cost and the schedule for the full engagement are known before it is committed.
Share a representative sample, expected volume, latency target and deployment location for Duplicate / near-duplicate text detection. We benchmark the shortlisted models, recommend the lowest-cost GPU configuration that meets the target, and scale it from pilot capacity to a dedicated production cluster.