AI pipelines for Azure HorizonDB is now
available in public preview. AI pipelines let you describe an AI data ingestion
workflow, chunking, embedding, extraction, generation, and ranking,
declaratively in SQL, and run it as a fault-tolerant pipeline inside the
database.
Execution is durable: pipelines survive crashes, retry failed steps
automatically, checkpoint incremental work, and resume long-running jobs from
the last completed step.
This removes the boilerplate ingestion service that many GenAI
applications rebuild today. Pipelines run on change, so embeddings stay in sync
as source rows are inserted or updated, and only new or changed rows need to be
re-embedded.
When the embedding model,
dimensions, or chunk size change, backfill reprocesses every row as a single
durable instance that resumes from the last checkpoint instead of restarting
from zero. AI pipelines are part of the azure_ai extension, built on pg_durable, and work with AI model management, pgvector,
DiskANN vector search, and full-text search to provide an end-to-end retrieval
path inside one database.
Learn
more.