Skip to content

RaBitQ Library 0.8.0

This release gives the Python indexes a Faiss-style train → add → search workflow, adds optional k-means sampling, tightens input and state validation, and renames three C++ query-factor accessors.

Python changes

  • IvfIndex, HnswIndex and SymqgIndex follow train → add → search. IVF-RaBitQ and HNSW-RaBitQ learn their own centroids in train, configured by index.cp, or accept external centroids through set_centroids. Storage grows on add, so constructors no longer take max_elements. SymphonyQG builds its graph in add, takes nbits (4, 8, or 32 for raw vectors) instead of quantization_bits, and its search replaces search_batch.
  • 0.7 constructors, build, is_built, positional add arguments and the SymphonyQG build, search_batch and quantization_bits still work, but raise a DeprecationWarning and will be removed in 0.9. The migration guide maps each old call to its 0.8 form.
  • Behavior changes that apply without a warning, detailed in the migration guide:
    • train and add default to every available CPU, including 0.7-style add(x) calls; search and the deprecated build keep one thread.
    • Multi-threaded HNSW-RaBitQ add is not reproducible; pass num_threads=1 for the same graph on every run.
    • HNSW-RaBitQ capacity grows on add, and search limits k to the stored point count.
    • SymphonyQG search accepts an omitted, zero or small ef, as HNSW-RaBitQ does, and one seed now drives both rotation and graph construction.
  • RaBitQKMeans and QGKMeans accept max_points_per_centroid to train on a seeded sample, and gain assign(x) for new vectors. Sampling is off by default for these classes; index train samples at most 256 vectors per centroid. See training-set sampling.
  • Indexes reject NaN, infinite and oversized coordinates in data, centroids and queries with ValueError. See input requirements.
  • HnswIndex.search clamps ef to the stored point count instead of failing on a very large value, with unchanged results.

C++ changes

  • Breaking: query-factor accessors take a g_ prefix, without aliases:

    Old New Classes
    k1xsumq() g_k1xsumq() SplitSingleQuery, BatchQuery, SplitBatchQuery, symqg::QuantizedQuery
    kbxsumq() g_kbxsumq() SplitSingleQuery, SplitBatchQuery
    kbase_sumq() g_kbase_sumq() SplitSingleQuery

    The optional k1xsumq pointer parameter of BatchQuery and the SymphonyQG search entry points is now named g_k1xsumq; its position and type are unchanged. - Breaking: saving an index that stores no points throws std::logic_error. Earlier releases wrote such an HNSW index as an empty file; loading old files is unchanged. - Breaking: the installed header rabitqlib/utils/array.hpp and its rabitqlib::Array class are removed; use standard containers instead. - IVF indexes can be created without a point count, trained or given centroids, then filled with add. Pass the rotation seed to this constructor: four arguments such as IVF(dim, num_clusters, 4, METRIC_IP) select the point-count constructor instead. HNSW indexes keep their capacity argument and can be trained or given centroids before the first add. Existing constructors and construct are unchanged. See IVF-RaBitQ and HNSW-RaBitQ. - On MSVC, libc++ and libstdc++ before GCC 11, a seed now gives a different FHT/Kac rotation, and therefore different RaBitQKMeans results, because the signs no longer depend on the standard library. faster_config reads a frozen scale table, so its t_const can shift by a few percent on those toolchains. GCC 11+ Linux results are unchanged. A seeded MatrixRotator is now reproducible on the same toolchain. Saved indexes store their rotation and are unaffected. - Index entry points reject non-finite and oversized coordinates with std::invalid_argument. HNSW cluster_id_of validates its argument, IVF loading rejects trailing bytes, SymphonyQG rejects ef_construction of zero, and the advanced encoders check their preconditions.

Fixes

  • Code compiled for AVX2 or AVX-512 no longer reaches generic callers through shared inline functions and templates, which could fail on CPUs without those instructions. CI now checks ISA-specific builds for such functions.
  • All three indexes save a regular file by writing a temporary file beside it and renaming it into place, so a failed save leaves the existing file unchanged. The replaced file keeps its permission bits, but not its owner or other hard links. Symbolic links and other non-regular destinations are still written in place.
  • HNSW-RaBitQ rejects a zero capacity with ValueError (C++: std::invalid_argument) and reports allocation failure as MemoryError (C++: std::bad_alloc) in construction, add, resize and load alike.
  • IVF initializers clamp nprobe to the cluster count and size their output, instead of reading and writing out of range when called directly.
  • AVX2 and AVX-512 FHT rotation accepts overlapping input, as the other backends do.
  • A failed copy assignment leaves the target object in its previous state, and HNSW search no longer prefetches addresses outside its allocation.

Internal changes

Index, SIMD and binding internals share more helpers, and identifier naming is enforced by clang-tidy. Several pull requests optimize query preparation, LUT quantization and graph search; their measurements are recorded in those pull requests and do not establish a general result. No performance measurement was made for k-means sampling or for the 2,000-cluster threshold at which index train switches to QGKMeans. The published benchmarks retain their recorded 0.7.1 provenance.

The package declares its license as an SPDX expression with expanded PyPI metadata, and the repository adds a security policy and a software citation.

Upgrading

Python code written for 0.7 runs unchanged apart from the behavior changes above; follow the deprecation warnings and the migration guide before 0.9. Index files from 0.7 load as trained indexes, and no conversion is required.

C++ consumers must rename the three query-factor accessors, replace rabitqlib::Array, and must not save empty indexes. For example, zvec, which uses the core and FastScan components, built and passed its RaBitQ tests after renaming one kbxsumq() call in a test. The stable core API v1 and component targets are unchanged. Rebuild C++ consumers with matching headers and libraries; no cross-version C++ binary ABI is promised.

Use the installation guide for supported platforms and the quick start for a complete Python example.