Manticore Search Just Made Embeddings 14× Faster — Here's How They Did It

Manticore Search 27.1.5 shipped this week with a rebuilt ONNX Runtime backend that makes auto-embeddings ~14× faster on average. Same hardware, same model, same weights. The old Candle/SentenceTransformers path crawled at 5–11 documents per second. The new path hits 70–230. At peak, with the right batch size, they're seeing 233 docs/sec at 4.3ms per document.

If you're building any system where embedding speed matters — RAG pipelines, agent memory stores, real-time semantic search — this is the most interesting database optimization story I've seen this quarter. Because it's not just a number. The engineering decisions behind that number are directly applicable to how you think about embedding inference in production.

Why Embedding Speed Is a Hard Constraint

Here's the thing about auto-embeddings in a database: the database runs the model on every INSERT. Your ingest throughput is whatever the embedding step can sustain. If you're building an agent that needs to embed thousands of documents at startup, or a RAG pipeline that re-indexes nightly, your bottleneck isn't the database engine — it's the model inference.

The old Manticore path (Candle, a Rust ML framework, plus SentenceTransformers for tokenization) ran into a wall that looked familiar to anyone who's tried to scale embedding inference: top showed the box well under full load, but throughput was flat at 5–11 docs/sec regardless of concurrency or batch size. CPU was busy, but no work was getting done.

That's the classic symptom of lock contention and poor parallelism. Manticore's team decided to throw out the inference stack and rebuild on ONNX Runtime.

What ONNX Actually Changes

ONNX Runtime (ORT) is Microsoft's cross-platform inference engine for models in the Open Neural Network Exchange format. It's been around for years, but for embeddings specifically, it solves a specific problem: you can convert any SentenceTransformer model to ONNX once, and then ORT runs it with graph optimizations, operator fusion, and thread-safe concurrent access.

The key insight the Manticore team landed on is something most Rust ONNX wrappers get wrong: on Linux and macOS, ORT's C Run() API is thread-safe. You can share a single Session across many concurrent callers without any locking.

let session = ort::session::Session::builder()?
    .with_optimization_level(GraphOptimizationLevel::Level3)?
    .with_intra_threads(0)?          // let ORT pick (= all cores)
    .with_intra_op_spinning(false)?  // do NOT busy-wait between calls
    .with_flush_to_zero()?           // kill denormals on attention softmax
    .with_approximate_gelu()?        // ~10% faster activation, no quality loss
    .commit_from_file(&onnx_path)?;

That one line — intra_op_spinning(false) — was arguably the single biggest win. Without it, ORT's intra-op thread pool defaults to spinning between dispatches. Every core at 100%, throughput lower than spinning off. Flipping it immediately raised throughput and dropped CPU usage at the same time. If you're using ONNX for anything, check your spinning config right now.

The Thread-Safety Trick That Unlocked Everything

This is the part that matters if you're building your own embedding pipeline. The conventional wisdom is: put the session behind a Mutex, or build a session pool. Both are wrong when ORT's Run() is thread-safe.

Manticore's solution is a deliberate unsafe wrapper to bypass Rust's borrow checker:

#[cfg(not(target_os = "windows"))]
struct SessionWrapper {
    inner: std::cell::UnsafeCell<ort::session::Session>,
}
#[cfg(not(target_os = "windows"))]
unsafe impl Sync for SessionWrapper {}
#[cfg(not(target_os = "windows"))]
unsafe impl Send for SessionWrapper {}

impl SessionWrapper {
    fn with_session<R>(&self, f: impl FnOnce(&mut Session) -> R) -> R {
        f(unsafe { &mut *self.inner.get() })
    }
}

On Windows, where ORT's threading isn't guaranteed safe, they fall back to a Mutex. But on Linux and macOS, this single shared session pattern means N concurrent workers all hit the same session with zero locking overhead. No pool management, no mutex contention, no session initialization cost per worker.

Why Batching Is the Wrong Answer (And the Right One)

Here's where Manticore's team went against the textbook. Most embedding libraries batch inputs — pack N documents together, tokenize them, run inference once on the batch. The theory is better GPU utilization. The reality, for CPU-based embedding, is the padding tax.

Every document in a batch gets padded to max_len. Real inputs are highly variable in length — a 50-token document gets padded to 256 if that's the batch's max. The model spends most cycles on padding tokens. Single-doc batches only do work proportional to actual token count.

Manticore's final design is the opposite of the textbook recipe:

fn predict_pipelined(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>, _> {
    let bs = batch_size();
    // Small input — single tokenize + infer, no thread overhead.
    if texts.len() <= bs {
        return Self::tokenize_and_infer(&self.session, &self.tokenizer, texts, ...);
    }
    // Large input — split across workers, each running 1-doc-at-a-time
    // through the SHARED session.
    let num_workers = (texts.len() / bs).min(available_cpus()).max(1);
    let docs_per_worker = texts.len().div_ceil(num_workers);
    std::thread::scope(|s| {
        for worker_texts in texts.chunks(docs_per_worker) {
            s.spawn(move || {
                for text in worker_texts {
                    Self::tokenize_and_infer(&session, &tokenizer,
                        std::slice::from_ref(text), ...)?;
                }
                Ok(())
            });
        }
    });
}

Two paths:

• Single document path: zero thread spawning, zero coordination overhead, 4.3ms latency.
• Bulk path: workers process sequentially internally, sharing the lock-free session. No padding tax because each worker handles one doc at a time.

The result: throughput increases as you reduce batch size, which is the opposite of what most embedding libraries do. Batch size 1 is their fastest configuration at 143 docs/sec (8 threads). Batch size 128? Also 147 docs/sec. The flat scaling profile means you can tune for latency without sacrificing throughput.

The Performance Numbers That Matter

All benchmarks on a 16-core / 32-thread server with all-MiniLM-L12-v2-onnx:

Compare to the old Candle path which sat flat at 10 docs/sec regardless of settings. That's not a typo.

The old path was hard-locked by lock contention on the tokenizer and poor parallelism in Candle's CPU backend. The new path hits 143 docs/sec at batch size 1 — a 14× improvement — and stays there across batch sizes because the session is lock-free and the tokenizer is fast enough to keep up.

What This Means for Agent and RAG Systems

If you're using Manticore as your vector store — and if you're building agents that need local, fast, semantic search, you should be paying attention — this change means your ingest bottleneck just moved. The 14× speedup on embeddings translates directly to:

• Faster index builds. If you're re-indexing a 100K document corpus at 10 docs/sec, that's ~2.8 hours. At 143 docs/sec, it's ~12 minutes.
• Lower latency inserts. Real-time ingestion pipelines (document feeds, chat logs, event streams) can now keep up with write-heavy workloads without backpressure.
• Cheaper hardware. Get the same throughput on fewer cores. The 14× improvement is on the same hardware — you could scale down and still beat the old performance.

But the bigger story is about ONNX as a standard. Manticore's result is reproducible: the same ORT patterns apply whether you're running embeddings in PostgreSQL with pgvector, in a custom Rust pipeline, or in a Python service with onnxruntime. The three lessons — disable intra-op spinning, share sessions across threads, avoid padding tax — are portable to any ONNX-based embedding deployment.

The Bottom Line

Manticore Search 27.1.5 is out now. If you're already running Manticore with auto-embeddings, the ONNX path auto-enables when your model file has the .onnx extension — drop in the converted weights and you get the speedup with zero code changes. Most popular SentenceTransformer models are available pre-converted on HuggingFace under the Xenova/ namespace.

For everyone else: Manticore's engineering team just published a masterclass in production embedding optimization. Read the full article for the deeper technical details, including their approach to ONNX model conversion and thread configuration. The code-level choices they made — thread-safe session sharing, disabling intra-op spinning, single-doc inference — are the kind of practical engineering that actually moves the needle in production systems. Apply them wherever you run embeddings.