Skip to content

Architecture & Methodology

VectorZip is a unified, easy-to-use embedding compression library designed to bridge the gap between high-capacity contextual models (like BGE-M3 or Qwen2.5) and the strict resource limitations of modern Vector Databases in Retrieval-Augmented Generation (RAG) applications.

This document outlines the theoretical foundation and the mathematical pipelines of the core algorithms integrated into the VectorZip ecosystem, with a primary focus on our signature proprietary Discrete Cosine Transform (DCT) compression method.

1. The Capacity Bottleneck of Small Models

In modern RAG architectures, reducing embedding dimensionality is highly desirable to save RAM and decrease vector distance computation latency. The industry standard approach is to train "Small" native models (e.g., BGE-Small-en-v1.5, yielding 384 dimensions).

However, native low-dimensional models suffer from a fundamental parameter deficiency. A 33-Million parameter model physically cannot store the vocabulary, multilinguality, and structural understanding required for complex enterprise RAG tasks.

VectorZip operates under the Same-Family Parametric Transfer Hypothesis:

By applying post-hoc compression to the outputs of a high-capacity model (e.g., 1024 dimensions), the resulting compressed vector (e.g., 384 dimensions) inherits the advanced linguistic representations and parametric knowledge of the large backbone, far surpassing the capabilities of a native small model.

2. The Nature of NLP Vectors & The Spectral Challenge

Why can't we just truncate vectors or apply simple PCA?

Unlike audio waveforms or image matrices, dense NLP embeddings have no inherent spatial or temporal ordering. The sequence of dimensions \([d_1, d_2, ..., d_n]\) in an embedding vector is an arbitrary byproduct of the neural network's internal weight initialization.

Because the dimensions are unordered, the vector exhibits extreme high-frequency "noise" when viewed as a signal. Standard compression algorithms (like Fourier transforms or Discrete Cosine Transforms) fail entirely on high-frequency noise because they are designed to compress smooth continuous signals.

Our signature DCT compression method solves this through a novel two-stage algorithm (TSP reordering + DCT-II projection) developed specifically for VectorZip.

3. The DCT Pipeline: Stage 1 - TSP Dimensional Reordering

To enable spectral compression, we must first "smooth" the signal. We achieve this by reordering the dimensions such that highly correlated features are placed adjacently.

We formulate this as a Traveling Salesperson Problem (TSP) over the dimensions.

  1. Covariance Matrix: We compute the empirical covariance matrix \(\Sigma\) over a representative calibration corpus.
  2. Distance Metric: We define the "distance" between any two dimensions \(i\) and \(j\) as: [ D_{i,j} = 1 - \frac{|\Sigma_{i,j}|}{\sqrt{\Sigma_{i,i} \cdot \Sigma_{j,j}}} ]
  3. TSP Optimization: We solve the TSP using a Nearest-Neighbor heuristic followed by a full 2-Opt local search to find the optimal permutation \(P\) that minimizes the total path distance.

The result is a static permutation that groups related neural features together. When applied to any vector, it artificially creates a smooth, low-frequency signal without altering its mathematical content.

4. The DCT Pipeline: Stage 2 - Orthonormal Spectral Projection (DCT-II)

Once the vector is transformed into a "smooth" pseudo-signal by the TSP permutation, it becomes highly compressible.

We project the permuted vector using the Type-II Discrete Cosine Transform (DCT-II):

\[ X_k = \sum_{n=0}^{N-1} x_n \cos \left[ \frac{\pi}{N} \left( n + \frac{1}{2} \right) k \right] \quad k = 0, \dots, N-1 \]

Energy Compaction

The DCT-II exhibits a property known as energy compaction. For smooth signals, almost all the variance (or semantic "energy") is concentrated in the first few low-frequency coefficients (\(X_0, X_1, \dots, X_{K-1}\)).

We truncate the high-frequency tail of the transformed vector, keeping only the first \(K\) dimensions (where \(K\) is the target dimensionality, e.g., 384).

Cosine Similarity Invariance

Because the DCT-II is an orthonormal transformation, it preserves dot products and vector norms exactly. As long as the truncated high-frequency components contain near-zero energy, the angular cosine similarity between any two compressed vectors remains virtually identical to their high-dimensional counterparts.

Conclusion

VectorZip does not "learn" new representations; it provides a unified, highly optimized library to compress and decompress embeddings easily. While VectorZip includes classic methods like PCA, Product Quantization, and Matryoshka slicing, our custom-built TSP-smoothed DCT algorithm stands out by mathematically extracting the maximum density of information from existing embeddings, providing developers with high-fidelity, out-of-distribution robust vector compression.