OpenData Vector: MIT-licensed Vector Search on Object Storage | OpenData

OpenData Vector fills the gap between running pgvector and paying a vendor many multiples of hardware costs to operate a search database for you. MIT-licensed and built on SlateDB, OpenData Vector is a stateless, durable, and highly-available search engine that runs anywhere with access to Object Storage. It is designed to be simple enough to operate yourself, and efficient enough to serve 100M vectors for roughly $350/mo. There’s growing consensus in the database community that object storage is a “good thing”. The 99.999999999% durability SLAs, the cost efficiency (1/4 storage cost, $0 cross-AZ networking), and the strong consistency are all distributed system nightmares that are solved by object storage. Because of this, over the last decade online systems have steadily increased their dependency on object storage. The first generation systems were tiered, the second were fully disaggregated and the third (and current) generation are stateless. ┏━Gen 1: Tiered Storage━━━━━━━━━━┓ ┏━Gen 2: Disaggregated━━━━━━━━━━━┓ ┃ ┃█ ┃ ┃█ ┃ ╭──────────────────────╮ ┃█ ┃ ╭──────────────────────╮ ┃█ ┃ │ Query │ ┃█ ┃ │ Query │ ┃█ ┃ ╰───────────┬──────────╯ ┃█ ┃ ╰───────────┬──────────╯ ┃█ ┃ │ ┃█ ┃ │ ┃█ ┃ ┌─────────┘ ┃█ ┃ ┌──────────┘ ┃█ ┃ │ ┃█ ┃ │ ╭─────────╮ ┃█ ┃ ╔══▼═╗ ╔════╗ ╔════╗ ┃█ ┃ ┌──▼─┐ │ Cluster │ ┌────┐┃█ ┃ ║ A ├────▶ B ├────▶ C ║ ┃█ ┃ │ A ├───▶ Manager ◀────┤ B │┃█ ┃ ╚══┬═╝ ╚══┬═╝ ╚══┬═╝ ┃█ ┃ └──┬─┘ ╰─────────╯ └──┬─┘┃█ ┃ └─────────┼─────────┘ ┃█ ┃ └──────────┬────────────┘ ┃█ ┃ │ ┃█ ┃ │ ┃█ ┃ ┌───────────▼──────────┐ ┃█ ┃ ╔═══════════▼══════════╗ ┃█ ┃ │ S3 (cold) │ ┃█ ┃ ║ S3 (durable state) ║ ┃█ ┃ └──────────────────────┘ ┃█ ┃ ╚══════════════════════╝ ┃█ ┃ ┃█ ┃ ┃█ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛█ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛█ ██████████████████████████████████ ██████████████████████████████████ ┏━Gen 3: Stateless━━━━━━━━━━━━━━━┓ ┃ ┃█ ┃ ╭──────────────────────╮ ┃█ ┃ │ Query │ ┃█ ┃ ╰───────────┬──────────╯ ┃█ ┃ │ ┃█ ┃ ┌─────────┼─────────┐ ┃█ ┃ │ │ │ ┃█ ┃ ┌──▼─┐ ┌──▼─┐ ┌──▼─┐ ┃█ ┃ │ A │ │ B │ │ C │ ┃█ ┃ └──┬─┘ └──┬─┘ └──┬─┘ ┃█ ┃ └─────────┼─────────┘ ┃█ ┃ │ ┃█ ┃ ╔═══════════▼══════════╗ ┃█ ┃ ║ S3 (only truth) ║ ┃█ ┃ ╚══════════════════════╝ ┃█ ┃ ┃█ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛█ ██████████████████████████████████ The second generation systems (such as Chroma & Milvus) improved on the first in that durability and replication were delegated to object…