Vector Stores

A vector store stores numeric vectors and retrieves the items whose vectors are nearest to a query vector. In AI applications, those vectors are commonly embeddings: compact numeric representations of text, images, audio, users, products, or other objects.

Vector stores make questions such as these practical:

• Which documentation passages are semantically related to this question?
• Which products resemble the one a customer is viewing?
• Have we processed a support ticket with the same meaning before?
• Which tools are relevant to the user’s current goal?
• Which saved memories may matter in this conversation?

A vector store is not a language model, and it does not generate an answer. It is a retrieval component. It finds candidate data; application code or a model then decides how to use that data.

This article explains why vector stores exist, their roles in AI systems, popular implementations, and how to use one from Java.

From Content to Vectors #

An embedding model maps an input into a fixed-length array of numbers:

"How can I change my password?"
        |
        v
  embedding model
        |
        v
[0.018, -0.214, 0.092, ..., 0.037]

The individual numbers generally have no useful human-readable meaning. Their geometry matters: inputs with related meaning tend to produce vectors that are near each other in the embedding space.

Suppose an application has already embedded these passages:

A: "Reset your password from Account Settings."
B: "Invoices are available on the Billing page."
C: "Use a security key for passwordless sign-in."

It embeds the query “I forgot my login password” with the same embedding model, then searches for the nearest stored vectors. Passage A should rank highly even though the query and passage do not share exactly the same words.

Each stored record usually contains more than a vector:

id:          handbook:authentication:17
vector:      [0.018, -0.214, ...]
content:     Reset your password from Account Settings.
metadata:
  tenant_id: acme
  document_id: handbook:authentication
  version: 8
  language: en
  section: Password Recovery
  visibility: employee

The vector enables similarity search. The ID, original content, and metadata let the application interpret, authorize, cite, update, and delete the result.

Why Ordinary Databases Are Not Enough #

Relational and document databases are excellent at exact conditions:

WHERE tenant_id = 'acme'
  AND status = 'PUBLISHED'
  AND created_at >= DATE '2026-01-01'

Full-text indexes are excellent at lexical search: tokenization, stemming, phrases, spelling, field weighting, and exact terms. They can find passages containing password reset.

Neither operation alone answers “Which items are closest in meaning to this new input?” An embedding with 1,536 dimensions is a point in a 1,536-dimensional space. Finding the nearest points requires vector distance operations and specialized indexes.

This does not mean every AI application needs a vector store.

• A model that calls a deterministic weather API does not need one.
• A small, fixed prompt may fit directly in the model’s context.
• Exact identifiers, names, error codes, and legal phrases may be better served by full-text or relational search.
• A few hundred vectors can be scanned in memory without a separate database.

A vector store becomes useful when the application repeatedly searches a growing collection by semantic similarity and needs filtering, persistence, concurrency, updates, or low latency.

Distance and Similarity #

A search ranks vectors using a metric compatible with the embedding model and the way its vectors are normalized.

Cosine similarity measures the angle between vectors:

It emphasizes direction rather than magnitude and is common for text embeddings. A larger cosine similarity means greater similarity.

Dot product, or inner product, includes vector magnitude. It is often fast and is appropriate when the embedding model was trained for it. For unit-normalized vectors, dot product and cosine ranking are equivalent.

Euclidean distance, or L2 distance, measures straight-line distance. A smaller distance means the vectors are closer.

Do not choose a metric by name alone. Follow the embedding model’s guidance and configure the index and query with the same metric. A database may return a distance, a similarity, or a transformed score; score thresholds are therefore not automatically portable between stores or embedding models.

Exact and Approximate Search #

An exact nearest-neighbor search compares the query with every eligible vector. It gives exact results but its work grows with the collection. This can be perfectly adequate for a small corpus or an offline evaluation baseline.

Large collections normally use approximate nearest-neighbor search, or ANN. ANN indexes inspect only promising candidates, trading some recall for lower latency.

HNSW #

Hierarchical Navigable Small World, or HNSW, builds a multilayer graph whose edges connect nearby vectors. Search traverses the graph toward promising neighbors.

HNSW commonly offers a strong latency-recall tradeoff, but its graph consumes memory and can be expensive to build. Parameters controlling graph density, construction effort, and search effort affect memory, ingestion speed, latency, and recall.

IVF #

Inverted-file indexes partition vectors into clusters or lists. At query time, the store searches only the most relevant partitions.

Searching more partitions improves recall but costs more time. Some IVF variants require training on representative data, so the index should be rebuilt or retuned when the data distribution changes substantially.

Quantization #

Quantization stores a compressed representation of vectors. Product quantization, scalar quantization, and binary quantization can reduce memory and accelerate candidate search at the cost of precision.

A common design retrieves candidates using compressed vectors and reranks those candidates with full-precision vectors or a separate reranker.

ANN Is a Tunable System #

topK = 10 does not guarantee that an approximate index found the true ten nearest items. Measure recall@K against exact search on representative queries, then tune the index for the required balance of:

• recall;
• p50, p95, and p99 latency;
• memory;
• index build time;
• update throughput; and
• cost.

Vendor benchmarks using different data, filters, hardware, and recall targets are not directly comparable.

Roles in AI Applications #

Retrieval-Augmented Generation #

Retrieval-augmented generation, or RAG, is the most familiar use:

source documents
  -> parse
  -> split into chunks
  -> embed
  -> vector store

user question
  -> embed
  -> retrieve relevant chunks
  -> optionally rerank
  -> add selected evidence to the prompt
  -> language model
  -> answer with citations

RAG gives a model current or private information without training the model. The vector store is only the retrieval stage. It does not guarantee that the retrieved passages are correct, that all relevant passages were found, or that the model will faithfully use them.

Semantic Search #

An application can return vector results directly instead of passing them to a language model. This is useful for documentation search, support-ticket deduplication, media discovery, and related-item features.

Semantic search is good at paraphrases but can be weak for exact identifiers. A query for NullPointerException at PaymentService.java:417 may need exact token matching more than semantic proximity.

Hybrid Search #

Hybrid search combines vector and lexical retrieval:

vector candidates ----+
                      +--> rank fusion or reranker --> final results
keyword candidates ---+

This often works better than choosing one search method. Semantic retrieval handles paraphrases; lexical retrieval preserves product codes, names, quoted phrases, and rare terms. Results can be combined with Reciprocal Rank Fusion, weighted scores, or a learned reranker.

Scores from unrelated retrieval algorithms should not be added without normalization. Their numeric ranges have different meanings.

Recommendations and Similar Items #

Products, documents, users, and behavior sequences can be embedded. Nearest neighbors then provide candidate recommendations. Business rules still need to remove unavailable, unsafe, already-consumed, or unauthorized items before ranking and presentation.

Agent Memory #

An agent can embed selected facts, summaries, or past events and retrieve memories relevant to the current task. A vector store is not memory by itself: the application must decide what is worth remembering, when it expires, how it is corrected, and who may access it.

Saving every conversation message indefinitely creates noise and privacy obligations. Retrieval similarity also does not imply that a memory is true or still current.

Semantic Caching #

A semantic cache reuses a prior result when a new request is sufficiently similar. This can reduce model cost and latency, but it is risky when small differences matter:

"Can user Alice read account 42?"
"Can user Bob read account 42?"

The questions are semantically close but may require opposite answers. Cache keys and filters must include tenant, authorization scope, model and prompt version, tool state, locale, and other correctness dimensions. Do not use semantic similarity to bypass a fresh authorization or financial decision.

Tool and Example Selection #

Applications with hundreds of tools can embed tool descriptions and retrieve a small candidate set for the model. The same technique selects few-shot examples, policies, workflows, or prompt fragments related to the current request.

Retrieval narrows the model’s choices; it must not grant permissions. The application still enforces which retrieved tools the user may invoke.

Ingestion Is Half the System #

Search quality depends on what was indexed.

Parse and Normalize #

Extract meaningful text from HTML, PDF, office documents, source repositories, or application records. Remove repeated navigation and boilerplate without discarding headings, tables, code boundaries, or other useful structure.

Optical character recognition, table extraction, and malformed PDFs can introduce errors before the embedding model sees the content.

Chunk Deliberately #

Embedding an entire manual as one vector produces an average representation that may match no specific question. Split content into independently useful chunks.

Chunks that are too large mix topics and consume model context. Chunks that are too small lose definitions and relationships. Preserve headings and parent document identifiers, and use overlap only when it measurably helps.

There is no universally correct token count. Evaluate chunking against real questions and source formats.

Add Operational Metadata #

Useful metadata includes:

• tenant and access classification;
• source and parent document IDs;
• source URI and title;
• language;
• version or content hash;
• chunk position;
• creation and expiration time; and
• embedding model and indexing generation.

Metadata is essential for access control and lifecycle management, not merely display.

Make Indexing Repeatable #

An ingestion pipeline should support:

• stable IDs or deterministic upserts;
• retries without duplicate chunks;
• replacement of all chunks from an updated source;
• deletion when a source is deleted;
• checkpoints and dead-letter handling;
• rate-limit-aware embedding batches; and
• an atomic switch from an old index generation to a new one.

Changing the embedding model or dimensions normally requires re-embedding the whole corpus. Build a new collection or table generation, validate it, then switch traffic. Do not mix incompatible embeddings in one searchable vector field.

Retrieval Is More Than topK #

A robust retrieval pipeline often performs:

1. query normalization or rewriting;
2. query embedding;
3. mandatory metadata filtering;
4. vector and possibly lexical candidate retrieval;
5. deduplication or diversity selection;
6. reranking;
7. a relevance or evidence-quality decision; and
8. context assembly within a token budget.

topK is the candidate count, not a quality guarantee. A fixed similarity threshold is also fragile across query types and model migrations.

Filter Before Disclosure #

Authorization must be applied by the store or retrieval layer before content reaches the model:

tenant_id = authenticated tenant
AND classification IN caller's allowed classifications
AND active = true

Do not retrieve across all tenants and ask the model to ignore forbidden documents. Post-filtering can leak content and can also return too few results.

ANN indexes and selective filters interact in implementation-specific ways. Some stores prefilter efficiently; others retrieve candidates and then filter, or require tuning to continue searching after many candidates are rejected. Test with the same filter selectivity used in production.

Rerank Candidates #

Embedding similarity is useful for inexpensive candidate generation. A cross-encoder or model-based reranker can examine the query and each candidate together and produce a more precise final order.

For example, retrieve 50 candidates, rerank them, and give the best 5 to the chat model. This improves precision but adds latency and cost.

Popular Implementations #

“Vector database” is often used broadly. Some products are dedicated vector databases; others add vector fields and ANN indexes to an existing relational, search, document, graph, or cache system.

Other valid choices include Cassandra-compatible systems, Neo4j, cloud data warehouses, embedded libraries, and managed database services with vector extensions. FAISS and Apache Lucene provide vector indexing primitives but are libraries rather than complete remote databases with application-level multi-tenancy, backups, and service operations.

How to Choose #

Start with requirements, not benchmark headlines:

• corpus size and vector dimensions;
• query and update throughput;
• p95 and p99 latency at a stated recall target;
• metadata filter complexity and selectivity;
• lexical and hybrid-search requirements;
• consistency, update, and deletion semantics;
• backup, replication, regional, and disaster-recovery needs;
• self-hosted versus managed operation;
• tenant isolation and encryption;
• supported Java client and framework integration;
• observability and index-tuning controls; and
• total cost, including vector memory and replicas.

If PostgreSQL already stores the source data and the expected corpus is moderate, pgvector is often the simplest first implementation. If search is a core product capability, Elasticsearch or OpenSearch may avoid building a second lexical system. A dedicated or managed vector database becomes attractive when vector workload, scale, isolation, or operational ownership deserves its own service.

The best choice can be the existing system that meets measured requirements. Adding a new distributed database solely because it is marketed for AI creates another source of truth, network hop, backup plan, and on-call burden.

Java Integration Options #

A Java application can integrate at three levels.

Framework abstraction

Spring AI’s VectorStore provides a common document, metadata, ingestion, and similarity-search API. This is convenient for RAG and permits substantial application code reuse between supported stores.

Vendor SDK

Dedicated Java clients expose provider-specific features. Qdrant, Milvus, Pinecone, Weaviate, Elasticsearch, OpenSearch, Redis, and other systems provide Java SDKs or supported Java clients. Use them when the application needs native hybrid queries, index controls, collection management, asynchronous bulk operations, or features absent from a framework abstraction.

Native database protocol

For pgvector, ordinary PostgreSQL JDBC and SQL are sufficient. This gives full control and integrates naturally with transactions and relational joins.

An abstraction is not a portability guarantee. Dimensions, score meanings, filter syntax, index creation, hybrid search, and consistency remain store-specific. Keep provider-specific configuration behind an application retrieval interface rather than spreading it through controllers and prompts.

Integrating pgvector with Spring AI #

The following example uses Spring AI 2.0, PostgreSQL, and pgvector.

Add Dependencies #

Import the Spring AI BOM, add one embedding-model starter, and add the pgvector store starter:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.ai</groupId>
      <artifactId>spring-ai-bom</artifactId>
      <version>2.0.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-vector-store-pgvector</artifactId>
  </dependency>
  <dependency>
    <groupId>org.postgresql</groupId>
    <artifactId>postgresql</artifactId>
    <scope>runtime</scope>
  </dependency>
</dependencies>

The vector store needs an EmbeddingModel. It does not use the chat model to create embeddings.

Configure the Store #

spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/knowledge
    username: knowledge_app
    password: ${KNOWLEDGE_DB_PASSWORD}
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      embedding:
        options:
          model: text-embedding-3-small
    vectorstore:
      pgvector:
        initialize-schema: false
        index-type: HNSW
        distance-type: COSINE_DISTANCE
        schema-name: ai
        table-name: knowledge_chunk
        schema-validation: true

Spring AI can initialize the schema when spring.ai.vectorstore.pgvector.initialize-schema=true, but the default is false. Production systems should normally manage the extension, table, indexes, ownership, and migrations through Flyway, Liquibase, or the existing database-change process rather than granting schema privileges to the application.

If dimensions are not configured, Spring AI can obtain them from the embedding model when creating the table. Once the vector column exists, changing to a model with different dimensions requires a new compatible schema and re-indexing.

Ingest Documents #

import java.util.List;
import java.util.Map;
import org.springframework.ai.document.Document;
import org.springframework.ai.transformer.splitter.TokenTextSplitter;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.stereotype.Service;

@Service
public class KnowledgeIndexer {

    private final VectorStore vectorStore;
    private final SourceDocumentRepository sources;
    private final TokenTextSplitter splitter;

    public KnowledgeIndexer(
            VectorStore vectorStore,
            SourceDocumentRepository sources) {
        this.vectorStore = vectorStore;
        this.sources = sources;
        this.splitter = new TokenTextSplitter();
    }

    public void index(String tenantId, String sourceId) {
        SourceDocument source = sources.require(tenantId, sourceId);

        Document document = new Document(
                source.text(),
                Map.of(
                        "tenant_id", tenantId,
                        "source_id", source.id(),
                        "title", source.title(),
                        "version", source.version(),
                        "active", true));

        List<Document> chunks = splitter.apply(List.of(document));
        vectorStore.add(chunks);
    }
}

This illustrates the data path, not a complete synchronization design. A production indexer needs deterministic chunk IDs, replacement or tombstoning of the prior source version, retries, batching, and a record of which embedding generation produced each vector.

Spring AI batches embedding requests to stay within embedding-model token limits. Batching does not replace chunking: a single oversized document still needs to be split deliberately.

Search with an Authorization Filter #

import java.util.List;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder;
import org.springframework.stereotype.Service;

@Service
public class KnowledgeSearch {

    private final VectorStore vectorStore;

    public KnowledgeSearch(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
    }

    public List<Document> search(
            String tenantId,
            String query,
            int limit) {
        var filters = new FilterExpressionBuilder();
        var authorizationFilter = filters.and(
                filters.eq("tenant_id", tenantId),
                filters.eq("active", true))
                .build();

        SearchRequest request = SearchRequest.builder()
                .query(query)
                .topK(limit)
                .similarityThreshold(0.72)
                .filterExpression(authorizationFilter)
                .build();

        return vectorStore.similaritySearch(request);
    }
}

Spring AI’s portable filter expression is converted to the native store filter. The structured builder keeps values separate from the filter language and avoids constructing an expression by concatenating user-controlled text.

The tenant ID must come from authenticated application context, not from a model-generated argument. Tune the similarity threshold with evaluation data. If weak retrieval should result in “I do not know,” make that an explicit application decision rather than assuming the store always returns useful evidence.

Use Results in RAG #

For a straightforward question-answering flow, Spring AI can connect the store to ChatClient through QuestionAnswerAdvisor:

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder;

public final class KnowledgeAssistant {

    private final ChatClient chatClient;
    private final VectorStore vectorStore;

    public KnowledgeAssistant(
            ChatClient.Builder builder,
            VectorStore vectorStore) {
        this.chatClient = builder.build();
        this.vectorStore = vectorStore;
    }

    public String answer(String tenantId, String question) {
        var filter = new FilterExpressionBuilder()
                .eq("tenant_id", tenantId)
                .build();

        var search = SearchRequest.builder()
                .topK(8)
                .similarityThreshold(0.72)
                .filterExpression(filter)
                .build();

        var advisor = QuestionAnswerAdvisor.builder(vectorStore)
                .searchRequest(search)
                .build();

        return chatClient.prompt()
                .user(question)
                .advisors(advisor)
                .call()
                .content();
    }
}

The example creates a filtered Advisor for each request instead of registering one global, unfiltered Advisor. tenantId must come from trusted authentication context. More elaborate RAG flows can use RetrievalAugmentationAdvisor to customize query rewriting, retrieval, document joining, reranking, and prompt augmentation.

Preserve source IDs and titles so the response can cite evidence. A citation proves which chunk was supplied; it does not prove that the chunk is true or that the answer accurately reflects it.

Using pgvector Directly with JDBC #

Applications that do not use Spring AI can call an embedding API, store the result, and query pgvector with JDBC.

Add the pgvector JDBC type:

<dependency>
  <groupId>com.pgvector</groupId>
  <artifactId>pgvector</artifactId>
  <version>0.1.6</version>
</dependency>

Create the extension and table through a database migration:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE knowledge_chunk (
    id text PRIMARY KEY,
    tenant_id text NOT NULL,
    source_id text NOT NULL,
    content text NOT NULL,
    metadata jsonb NOT NULL,
    embedding vector(1536) NOT NULL
);

CREATE INDEX knowledge_chunk_tenant_idx
    ON knowledge_chunk (tenant_id);

CREATE INDEX knowledge_chunk_embedding_hnsw_idx
    ON knowledge_chunk
    USING hnsw (embedding vector_cosine_ops);

The dimension 1536 must match the selected embedding model. The operator class must match the query metric.

With the pgvector Java library registered on the PostgreSQL connection, a query can bind a PGvector value:

import com.pgvector.PGvector;
import java.sql.Connection;
import java.util.ArrayList;
import java.util.List;
import javax.sql.DataSource;

public final class PgVectorKnowledgeSearch {

    private static final String SQL = """
            SELECT id,
                   source_id,
                   content,
                   1 - (embedding <=> ?) AS similarity
              FROM knowledge_chunk
             WHERE tenant_id = ?
             ORDER BY embedding <=> ?
             LIMIT ?
            """;

    private final DataSource dataSource;
    private final EmbeddingClient embeddings;

    public PgVectorKnowledgeSearch(
            DataSource dataSource,
            EmbeddingClient embeddings) {
        this.dataSource = dataSource;
        this.embeddings = embeddings;
    }

    public List<SearchHit> search(
            String tenantId,
            String text,
            int limit) throws Exception {
        float[] query = embeddings.embed(text);
        PGvector vector = new PGvector(query);

        try (Connection connection = dataSource.getConnection()) {
            PGvector.registerTypes(connection);

            try (var statement = connection.prepareStatement(SQL)) {
                statement.setObject(1, vector);
                statement.setString(2, tenantId);
                statement.setObject(3, vector);
                statement.setInt(4, limit);

                try (var rows = statement.executeQuery()) {
                    List<SearchHit> hits = new ArrayList<>();
                    while (rows.next()) {
                        hits.add(new SearchHit(
                                rows.getString("id"),
                                rows.getString("source_id"),
                                rows.getString("content"),
                                rows.getDouble("similarity")));
                    }
                    return hits;
                }
            }
        }
    }
}

EmbeddingClient is an application interface around the chosen embedding provider. The query embeds text before opening the database connection so a slow remote model call does not occupy a JDBC connection.

The SQL uses <=>, pgvector’s cosine-distance operator. Subtracting the distance from 1 produces cosine similarity. Both ORDER BY and LIMIT are important for index use. Bind vectors and filters as parameters; do not insert them into SQL strings.

Direct SQL is useful when vector search must join transactional data:

SELECT c.id, c.content
FROM knowledge_chunk c
JOIN document_acl a ON a.source_id = c.source_id
WHERE c.tenant_id = ?
  AND a.principal_id = ?
ORDER BY c.embedding <=> ?
LIMIT 10;

However, verify query plans and ANN recall with realistic filters. A relational join does not automatically guarantee that the query planner will combine the ANN index and authorization predicates efficiently.

Operating a Vector Store #

Treat a vector store as production data infrastructure.

Monitor:

• ingestion lag and failures;
• embedding API latency, rate limits, and cost;
• vector count by tenant and generation;
• deleted or orphaned chunks;
• query latency and timeout rate;
• candidates examined and filter selectivity where available;
• index build and memory usage;
• exact-versus-ANN recall on a benchmark set; and
• retrieval quality on real application questions.

Back up source content, metadata, and synchronization state. Embeddings can be recomputed, but doing so may be slow and expensive. Know whether the chosen store’s backup includes index structures or rebuilds them during restore.

Deletion deserves special attention. Removing a source from the primary application must eventually remove all of its chunks, vectors, lexical-index entries, caches, and replicas. Privacy deletion is not complete while an old embedding generation still contains the content.

Common Mistakes #

Using vectors for exact lookup

Fetch an order by order ID with a database key, not semantic similarity.

Embedding whole documents

One vector for a long, multi-topic document gives poor passage-level retrieval.

Mixing embedding models

Vectors from different models or dimensions generally do not inhabit a compatible space.

Treating similarity as truth

A close passage is related, not necessarily correct, current, or authoritative.

Ignoring lexical search

Vector search often misses exact codes, names, and rare strings that a text index handles well.

Filtering after retrieval

This can disclose unauthorized data and destroy recall for allowed data.

Using a universal threshold

Similarity scores change with models, metrics, stores, corpora, and query types.

Skipping evaluation

A fluent generated answer can hide retrieval failure. Measure retrieval recall, ranking, citation support, and end-to-end answer quality separately.

Adding a database before measuring

An in-memory exact scan or existing PostgreSQL/search platform may satisfy the initial workload with much less operational complexity.

Further Reading #

• Spring AI Vector Databases
• Spring AI pgvector Integration
• pgvector
• Qdrant Documentation
• Milvus Documentation
• Weaviate Documentation
• Pinecone Documentation
• Spring AI