Picture an operations lead at a distributor with four years of quotes, delivery notes and supplier emails. The plan is a retrieval system: chunk the documents, embed them, store the vectors, and let an assistant answer "what did we quote this customer for this part last spring". Four store names come back, pgvector, Pinecone, Qdrant and Weaviate, each with a benchmark page where it wins. Has anyone run all four on the same machine with the same data? Not in anything we could find. One independent benchmark leaves Pinecone out and measures a narrow thing carefully; three vendor benchmarks each measure what favours the vendor. No scandal in that, but it sets how the numbers have to be read.
What the public benchmarks measure.
ANN-Benchmarks is the reference most vector-store pages point at. The paper describes "a tool for evaluating the performance of in-memory approximate nearest neighbor algorithms".[1] In-memory: the whole index sits in RAM. Approximate: the index may miss some true neighbours, and the benchmark measures how many it misses against how fast it answers.
The quality measure is recall, "the ratio of the number of points in the result tuple that are true nearest neighbors"; the performance measure is "the number of queries it can answer per second (its QPS)".[1] Every chart on the site plots the two against each other for 10 nearest neighbours. The point is the curve, not any single dot: a store can always trade recall for speed by turning one parameter.
The setup is deliberately narrow: "The experiment loop is, by default, run on a single CPU in a single thread", and "query points are sent one by one to the algorithm instance".[1] The README repeats it: "All benchmarks are single-CPU."[2] That is a fair way to compare algorithms. A retrieval system behind an assistant is used differently: queries arrive together, over a network, filtered by customer and date, while new documents are being written.
Three of the four stores are on the site: pgvector, Qdrant and Weaviate, with results "as of April 2025". Pinecone is not listed and the README mentions no managed service.[2] The README also says the project "is no longer actively maintained". The paper's findings are about algorithms, not products: "graph-based algorithms provide by far the best performance on most of the datasets", and "HNSW is often the fastest algorithm, but PANNG is more robust if there is no global structure in the dataset".[1] The abstract's own summary: "very different approaches to [...] search yield comparable quality-performance trade-offs".
ANN-Benchmarks does not measure filters, inserts and deletes, concurrent load, network or cost. The NeurIPS 2023 Big ANN competition covered part of that gap, "filtered search, out-of-distribution data, sparse and streaming variants".[3]
Its filter track used 10 million 192-dimension image vectors; "the 100,000 queries consisted of one image embedding and one or two tags". Its streaming track ran "a sequence of batches of insertion operations, deletion operations, and search commands in a ratio of roughly 4:4:1". Entries on the filter track were scored on the highest throughput that still reached 90% recall at 10. The winner, ParlayANN, reached about 37,900 queries per second, more than 11 times the organisers' Faiss baseline: filtered search is its own engineering problem, and the gap between a purpose-built filtered index and a standard one is an order of magnitude.
On the streaming track the first declared winner scored 0.985 recall; a caching error was found, and after rescoring a different entry won at about 0.887.[3] A benchmark with updates in it is hard enough to score that the organisers got it wrong once.
| benchmark | who ran it | machine | stores in it | filters | updates |
|---|---|---|---|---|---|
| ANN-Benchmarks[2] | independent | r6i.16xlarge, one CPU core, one query at a time, in-memory | pgvector, Qdrant, Weaviate; not Pinecone | none | none |
| Big ANN NeurIPS'23[3] | independent | Azure D8lds_v5, 8 vCPU, 16 GB | research entries, not products | 10M vectors, 1–2 tags per query | insert, delete, search 4:4:1 |
| Tiger Data[5] | vendor (pgvectorscale) | EC2 r6id.4xlarge, 16 vCPU, 128 GB | Postgres vs Pinecone s1 and p2 pods; serverless excluded | none reported | none reported |
| Qdrant[6] | vendor | Azure D8s v3, 8 vCPU, 25 GB memory cap | Qdrant vs Elasticsearch, Milvus, Redis, Weaviate; not pgvector, not Pinecone | separate run, last updated Feb 2023 | none reported |
| Weaviate[7] | vendor | GCP n4-highmem-16, 16 vCPU, 128 GB | Weaviate only | unfiltered by design | none reported |
Three vendor benchmarks, read as claims.
Tiger Data (pgvector plus pgvectorscale, versus Pinecone). By Avthar Sewrathan, John Pruitt and Matvey Arye, June 2024, updated February 2026. Timescale, now Tiger Data, makes pgvectorscale, the extension under test: the vendor measuring itself.[5] On 50 million Cohere embeddings of 768 dimensions, against a Pinecone s1 pod configuration at 99% recall, the authors report "28x lower p95 latency", "16x higher query throughput" and "75 % lower monthly cost when self-hosted on AWS EC2"; against a p2 configuration at 90% recall, "1.4x lower p95 latency and 1.5x higher query throughput" and "79 % lower monthly cost". The Postgres side was one EC2 r6id.4xlarge (16 vCPUs, 128 GB) at $835 a month running pgvector 0.7.0; the s1 setup was priced at $3,241 a month, the p2 at $3,889.
Read the authors' own caveats. They used "a modified fork of ANN-Benchmarks", so their numbers are not comparable to the site's. They excluded Pinecone serverless, calling it rate-limited and suited to under 5 queries per second: their characterisation, not Pinecone's. They "did not explore every combination possible", and pgvectorscale's index ran with non-default settings tuned by its makers.[5] The $835 also buys a server nobody is yet operating, backing up or patching.
Qdrant (versus four open-source rivals). Qdrant compares itself with Elasticsearch, Milvus, Redis and Weaviate on four datasets of 1 to 10 million vectors, including 1 million 1,536-dimension OpenAI embeddings, on 8-vCPU Azure machines: "we ran all the engines in docker and limited their memory to 25GB". The single-node data is dated June 2024; the filtered run, February 2023.[6] The headline: "Qdrant achieves highest RPS and lowest latencies in almost all the scenarios." Asked whether the results are biased, the page answers "Probably, yes": its team knows its own product best and may have missed tuning options on the others. Closed-source services were excluded because they could not run on the same hardware, so Pinecone is absent; pgvector is absent too. Its design rule: "the speed of the vector search engines should to be compared only if they achieve the same precision" (the stray "to" is in the original).
Weaviate (versus itself). Weaviate's ANN benchmark compares it with nobody. It runs 10,000 requests per configuration on a GCP n4-highmem-16 (16 vCPUs, 128 GB) and reports recall, throughput and latency on four datasets of 1 to 10 million objects. On the 1-million, 1,536-dimension OpenAI set its recommended configuration reports recall at 10 of 97.24%, 5,639 queries per second and a p99 of 4.43 ms.[7] Each request is "an unfiltered vector search across the entire dataset for the given test". Two lines worth keeping: "HNSW is known to perform considerably worse with random vectors than with real-world datasets", and "This benchmark is open source, so you can reproduce the results yourself." Neither the version nor the run date is stated.
How to read a vendor benchmark.
None of the three vendor pages is dishonest: each states its hardware, datasets and most exclusions, and on each the vendor's product wins. Readers skim to the chart. Five questions turn the chart back into a sentence you can act on.
1. Is the recall stated, and is every bar at the same recall? Qdrant's rule, compare speed only at equal precision, is the whole game.[6] Tiger Data ran 99% and 90%, and the throughput gap between Postgres and Pinecone shrinks from 16x to 1.5x between them.[5] A chart with no recall on it tells you nothing.
2. Who is missing? Pinecone is absent from ANN-Benchmarks and Qdrant's page; pgvector from Qdrant's; Pinecone serverless, the product a small business would actually buy, from Tiger Data's. Weaviate compares with nobody. Whatever the reason, the page is silent on the one you are considering.
3. Was the rival tuned by its own people? Qdrant says it may have missed tuning options.[6] Tiger Data says the Pinecone configurations may not be optimal and its own index used tuned settings.[5] When the home team tunes both sides, the gap is an upper bound.
4. Does the dataset look like yours? Benchmark sets are public embeddings of Wikipedia, images and search passages. Your corpus is a few hundred thousand chunks of quotes and emails in French and English, and Weaviate's own page warns that HNSW behaves differently on different data.[7]
5. Were filters and updates in the run? Almost never (fig 1). The one independent benchmark that included them found an 11x gap between a purpose-built filtered index and the baseline, and found scoring updates error-prone.[3] If your queries carry "this customer, last 18 months" and documents arrive daily, the unfiltered static chart is a different workload.
What you pay for, in the vendors' units.
Benchmarks talk in queries per second. Invoices talk in something else, and the units differ by store. For a business handling thousands of queries a day rather than thousands a second, the units matter more than the throughput.
Pinecone serverless meters read units, write units, storage and egress.[8] The read rule is the one to understand: "A query uses 1 RU for every 1 GB of namespace size, with a minimum of 0.25 RUs per query." Results requested and metadata returned do not change it: "Only the size of the namespace does." Storage is billed per gigabyte of index size; the docs' own example, 500,000 records of 768 dimensions with 500 bytes of metadata and 10 sparse values, comes to 1.83 GB. The pricing page lists read units at $16 to $18 per million on Standard ("varies by cloud and region"), storage at $0.33 per GB per month, a $50 monthly minimum on Standard and $500 on Enterprise. The free Starter plan allows 2 GB and 1 million read units a month on AWS us-east-1 only; a $20 Builder plan adds region choice.[8]
Put the docs' example through the docs' rule (our arithmetic, from their formula and list prices). A 1.83 GB namespace costs roughly 2 read units per query; a million queries a month is about 2 million read units, $32 to $36 at Standard prices, plus about $0.60 of storage, inside the $50 minimum. Pinecone may round the gigabytes; the docs do not say. The shape is what matters: query cost grows with how much you store, not how many results you ask for, so the cheapest Pinecone design keeps each namespace small.
pgvector has no price of its own: you pay for the Postgres server it runs in, which many businesses already have. The README states the trade-off: HNSW "has better query performance than IVFFlat (in terms of speed-recall tradeoff)" but "has slower build times and uses more memory"; IVFFlat is the reverse.[4] The hard limit is stated too: an index on the standard vector type holds up to 2,000 dimensions, the half-precision type 4,000. HNSW arrived in 0.5.0 (August 2023), iterative index scans for filtering in 0.8.0 (October 2024); the latest release is 0.8.7, dated 2026-10-01.[4] Qdrant and Weaviate are open source, hosted or self-run; fig 3 carries their published figures, and their cloud prices were not in the pages we read.
| store | published figure or limit | what it tells an operator |
|---|---|---|
| pgvector[4] | Index up to 2,000 dims (vector) or 4,000 (halfvec). HNSW defaults m 16, ef_construction 64, ef_search 40. Filtering "applied after the index is scanned"; iterative scans since 0.8.0. Latest release 0.8.7, 2026-10-01. | Check your embedding model's dimensions against 2,000 before anything else. Cost is the Postgres box you already run. Filter-heavy queries need the 0.8.0 settings and an indexed filter column. |
| Pinecone serverless[8] | 1 RU per GB of namespace per query (min 0.25). $16–$18 per million RU, $4–$4.50 per million WU, $0.33/GB/mo (Standard). $50/mo minimum. Starter: 2 GB, 1M RU/mo, us-east-1 only. Max 20,000 dims, 40 KB filterable metadata, top_k 10,000. Namespaces per index: 100 (Starter) to 1,000,000 (Enterprise). | Query cost scales with namespace size, so partition small. Free tier is a single US region. Region choice starts at the $20 Builder plan. No server to run. |
| Qdrant[6] | Vendor benchmark on an 8-vCPU server with a 25 GB memory cap, 1M to 10M vectors, including 1M at 1,536 dims. Separate filtered-search run (Feb 2023). Open source; cloud or self-run. | 8 vCPUs and under 25 GB held 1M OpenAI-size vectors in the vendor's own run: a sizing floor for a self-run node. Price not in the pages read. |
| Weaviate[7] | Vendor benchmark on 16 vCPU, 128 GB. On 1M vectors at 1,536 dims: recall@10 97.24%, 5,639 QPS, p99 4.43 ms (vendor-reported, unfiltered, same machine for client and server). Open source; cloud or self-run. | Thousands of queries per second on one large machine, unfiltered: far more throughput than a small business needs, so the choice turns on operations. Price not in the pages read. |
The decision is operational.
Weaviate's own unfiltered figure on a 1-million-vector set is 5,639 queries per second.[7] An assistant handling a few thousand questions a day needs a fraction of one percent of that; every throughput chart says "enough", and five other questions decide.
Who runs it? pgvector, Qdrant and Weaviate can be self-run; Pinecone serverless cannot. Tiger Data's $835 machine[5] needs someone to patch it, back it up and rebuild its index after a bad deploy. If you already pay someone to run Postgres, pgvector adds an extension to an existing job; if nobody on the payroll runs a database, a hosted store is the cost of not hiring that person. A managed Postgres service with the pgvector extension is the third option: someone else patches the server, pgvector still has no price of its own, and the region question in the next section applies to it too. We did not read those services' pages for this note.
Where do the vectors live? The chunk stored beside each vector often carries personal information: a contact name, a direct email, a phone number in a quote or a delivery note. Quebec's private-sector privacy act, as amended by Law 25, requires a privacy impact assessment before personal information is communicated outside Quebec or entrusted to a provider outside Quebec to hold it, weighing sensitivity, purposes, protective measures and the destination's legal framework, with a written contract; BLG's May 2024 summary says this covers entrusting a third party outside Quebec with "keeping personal information on its behalf".[9] Doable, but work, and the reason region matters. Pinecone's free plan runs in AWS us-east-1 only; region choice begins on the paid Builder plan.[8] We did not verify any vendor's Canadian region list for this note. A self-run store on a Canadian machine avoids the question; a hosted store answers it with a region list and a contract (our note on running AI on your own hardware has the general version).
How many vectors, and how wide? Multiply your chunk count by your embedding model's dimensions. pgvector will not index past 2,000 dimensions on the standard type (4,000 on half precision)[4]; Pinecone accepts up to 20,000[8]. On count, Qdrant's own run held 1 million 1,536-dimension vectors on 8 vCPUs under 25 GB[6] and Weaviate's used a 16 vCPU, 128 GB machine for 1 to 10 million[7]; below a million vectors the hardware question is small for a self-run store, and Pinecone serverless sizes itself.
How filter-heavy are the queries? "Show me what we quoted this customer" is a filter first and a similarity search second. pgvector's README is direct: "With approximate indexes, filtering is applied after the index is scanned", and with the default ef_search of 40 and a filter matching 10% of rows, "only 4 rows will match on average"; the 0.8.0 iterative scans and an index on the filter column are the fixes.[4] The vendor benchmarks leave filters mostly out (fig 1), and the Big ANN filter track showed how far a filtered index can diverge from a standard one[3]. If most of your queries carry a filter, test that query, not the unfiltered one.
How often does the data change? A corpus that grows daily and is corrected weekly is the streaming track, not the static one. pgvector's README says HNSW "has slower build times" and that IVFFlat has a training step, so an IVFFlat index built on an early sample drifts as data changes.[4] Pinecone charges writes per kilobyte with a 5-unit minimum per request[8], so many tiny updates are the expensive pattern and batched writes the cheap one.
What to do on Monday.
Write the five answers down. Who will run it, where the data may live, how many chunks at how many dimensions, what share of queries carry a filter, how often documents change. One page; the evaluation is checking two finalists against it.
Check the hard limits before the soft numbers. Dimensions against pgvector's 2,000[4]; namespace size against Pinecone's read-unit rule and metadata against its 40 KB filterable limit[8]; region against your Law 25 assessment[9]. A hard limit removes a candidate in a minute; a throughput chart never does.
Run the smallest honest test on your own data. Take 50,000 real chunks and 200 real questions, embed them once, load the two finalists, and compute recall at 10 against an exact search, as ANN-Benchmarks does[1]. Then run the same questions with your real filter on. At 50,000 chunks this fits on a laptop in an afternoon. If the finalists land within a few points of recall at the latency you need, stop measuring and decide on operations.
Price the people, not the pods. Put the hosted store's per-unit estimate next to the hours of whoever would run the self-hosted one, incidents included. Tiger Data's chart compares a machine with a service[5]; yours should add the salary line to the machine.
Keep the retrieval layer swappable. Store the raw chunks and their metadata in a plain table you control and treat the vector index as a derived artifact you can rebuild. The embedding model will change before the store does, and the index must be rebuilt then anyway; it also keeps the Law 25 answer reversible.
What this does not claim.
It does not rank the four stores. No source we read puts all four on one machine, and we did not run one. The numbers in figs 2 and 3 are each vendor's own and are not comparable across rows.
The independent evidence is narrow and ageing. ANN-Benchmarks measures one core, one query at a time, in memory, no filters or updates; its last results are from April 2025 and it is no longer maintained.[2] The Big ANN results compare research entries, not products.[3]
Prices move, and the Law 25 point is not legal advice. Pinecone's figures are from its pricing page and docs as read on 2026-10-21 and "vary by cloud and region".[8] The Law 25 point rests on a law firm's summary of section 17[9]; whether a given region and contract satisfy the assessment is a question for your privacy officer or counsel.
If you are choosing a vector store for a retrieval system and want a second read on the five answers, describe the corpus and workload (no credentials or customer data) through the contact form. We will write back, free, with the hard limits that apply and the smallest honest test for your data. Retrieval systems that stay swappable are part of our workflow automation services.