Content Deduplication & Similarity Detection

Detect near-duplicate and semantically similar content using Brinicle vector search

Overview

Content deduplication and similarity detection are critical for data quality: removing duplicate entries, finding plagiarized content, grouping related articles, and merging near-identical records. Brinicle's VectorEngine enables fast approximate nearest neighbor search, making it efficient to find similar content even across millions of records.

How It Works

Embed each document or record as a dense vector using a sentence embedding model
Index all vectors in Brinicle
For each document, search for its nearest neighbors
Neighbors within a distance threshold are considered duplicates or near-duplicates

Implementation

Indexing Content

import numpy as np
import brinicle

EMBED_DIM = 384

engine = brinicle.VectorEngine(
    "dedup_index",
    dim=EMBED_DIM,
    M=48,
    ef_construction=1024,
    ef_search=512,
)

engine.init(mode="build")

# Example content items
content_items = [
    {"id": "c1", "text": "Machine learning is a subset of artificial intelligence."},
    {"id": "c2", "text": "Machine learning is a branch of artificial intelligence."},  # near-duplicate
    {"id": "c3", "text": "Deep learning uses neural networks with multiple layers."},
    {"id": "c4", "text": "Neural networks with many layers are used in deep learning."},  # near-duplicate
    {"id": "c5", "text": "Python is a popular programming language for data science."},
]

for item in content_items:
    vector = embed(item["text"])  # your embedding function
    engine.ingest(item["id"], vector)

engine.finalize()

Finding Duplicates

DUPLICATE_THRESHOLD = 0.15  # adjust based on your embedding model

def find_duplicates(items, k=5):
    duplicates = []

    for item in items:
        query_vector = embed(item["text"])
        results = engine.search_with_distance(query_vector, k=k)

        for doc_id, distance in results:
            if doc_id != item["id"] and distance < DUPLICATE_THRESHOLD:
                duplicates.append({
                    "original": item["id"],
                    "duplicate": doc_id,
                    "distance": distance,
                })

    return duplicates

dupes = find_duplicates(content_items)
for d in dupes:
    print(f"{d['original']} ≈ {d['duplicate']} (distance: {d['distance']:.4f})")

Batch Similarity Detection

For large datasets, use batch search for better throughput:

def batch_similarity_check(items, k=5, n_jobs=4):
    vectors = np.array([embed(item["text"]) for item in items]).astype(np.float32)
    results = engine.search_batch(vectors, k=k, n_jobs=n_jobs)

    similar_pairs = []
    for i, item in enumerate(items):
        for doc_id in results[i]:
            if doc_id != item["id"]:
                similar_pairs.append((item["id"], doc_id))

    return similar_pairs

Clustering Similar Content

Beyond deduplication, you can use Brinicle for content clustering:

def build_similarity_clusters(items, k=10):
    clusters = {}
    visited = set()

    for item in items:
        if item["id"] in visited:
            continue

        query_vector = embed(item["text"])
        results = engine.search_with_distance(query_vector, k=k)

        cluster = [item["id"]]
        visited.add(item["id"])

        for doc_id, distance in results:
            if doc_id not in visited and distance < 0.3:
                cluster.append(doc_id)
                visited.add(doc_id)

        clusters[item["id"]] = cluster

    return clusters

Scaling to Large Datasets

Brinicle's disk-first design means memory usage stays low even with millions of documents:

On SIFT 1M vectors (128 dimensions), Brinicle uses similar memory to FAISS while maintaining a disk-backed index
Streaming ingest allows you to build indexes without loading the entire dataset into RAM
Batch search with n_jobs enables parallel similarity checks

# Streaming ingest for large datasets
engine.init(mode="build")

for item in stream_content():  # your streaming function
    vector = embed(item["text"])
    engine.ingest(item["id"], vector)

engine.finalize()

Choosing the Distance Threshold

The right threshold depends on your embedding model and use case:

Use Case	Recommended Threshold
Exact deduplication	0.05 - 0.10
Near-duplicates	0.10 - 0.20
Similar content	0.20 - 0.35
Related topics	0.35 - 0.50

Start with a conservative threshold and adjust based on manual review of results.

Content Deduplication & Similarity Detection

On this page