docs(geneva): document Blob v1 and Blob v2 UDF workflows by justinrmiller · Pull Request #316 · lancedb/docs

justinrmiller · 2026-07-20T18:29:23Z

Summary

Reworks the Geneva blobs page (docs/geneva/udfs/blobs.mdx) to cover both Lance blob encodings end-to-end:

Comparison table: v1 (large_binary + lance-encoding:blob metadata, storage version ≥ 2.0) vs v2 (lance.blob_field / lance.blob.v2 extension, storage version ≥ 2.2, descriptor-based checkpoints, pylance>=9.0.0b24).
Writing from UDFs: v1 scalar + batched examples (refreshed); new v2 section with the struct-nested lance.blob_field pattern (scalar dict returns and batched lance.blob_array / StructArray construction), plus the required new_table_data_storage_version table options.
Reading in UDFs: v1 BlobFile input (existing, cleaned up — resolves the old TODO); new v2 section showing payloads arriving as bytes and dotted-path input_columns.
Reading back: new section for Table.take_blobs (both encodings, incl. dotted nested paths) and bulk dataset.read_blobs by row address.
Covers both the simple top-level BlobType() output form (supported as of lancedb/geneva#1059, which fixed the extension annotation being dropped on commit) and the struct-nested production pattern.

Companion to the Geneva core PR adding Blob v2 output support (lancedb/geneva, branch gen-378-blob-v2-core).

Linear: GEN-378

Test plan

Every code pattern was executed against a local Geneva backfill (v1 scalar/batched writes, take_blobs reads, BlobFile input, v2 struct scalar/batched writes, nested take_blobs, read_blobs by address, v2 bytes input with dotted input_columns).
mint broken-links passes.

🤖 Generated with Claude Code

Rework the blobs page to cover both Lance blob encodings end-to-end: declaring v1 (large_binary + lance-encoding:blob) and v2 (lance.blob_field struct) output columns, storage-version requirements (2.0 vs 2.2), reading blob columns as UDF inputs (BlobFile vs bytes), and reading payloads back via take_blobs / read_blobs. All snippets verified against a local Geneva backfill run. Linear: GEN-378 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mintlify · 2026-07-20T18:30:32Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
lancedb-bcbb4faf	🟢 Ready	View Preview	Jul 20, 2026, 6:30 PM

Geneva now preserves the extension annotation on top-level computed blob v2 columns (lancedb/geneva#1059), so document the scalar BlobType() form as the simplest pattern and drop the warning. Linear: GEN-378 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mintlify · 2026-07-20T18:48:27Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
lancedb-bcbb4faf	🟡 Building	–	Jul 20, 2026, 6:29 PM

prrao87

Minor nits, I recommend fixing (or leaving comments in the source) and we can get this merged.

prrao87 · 2026-07-22T00:38:52Z

+Geneva UDFs can consume and produce [Lance Blob](https://docs.lancedb.com/tables/multimodal) columns — large binary objects (roughly 1 MB to 100s of MB per value) such as images, audio clips, or documents. Blob columns keep large payloads out of the regular column data path and provide lazy, file-like access on read.

-## Reading Blobs
+Lance has two blob encodings, and Geneva supports both:


We could phrase it this way as long as we update it when we deprecate blob v1 and only support blob v2. AFAIK, lance-encoding:blob will be deprecated in LanceDB once blob v2 support lands in OSS.

prrao87 · 2026-07-22T00:39:55Z

 ---

-Geneva supports UDFs that take [Lance Blobs](https://docs.lancedb.com/tables/multimodal) (large binary objects) as input and has the ability to write out columns with binaries encoded as Lance Blobs.  Lance blobs are an optimization intended for large objects (1's MBs -> 100MB's) and provide a file-like object that lazily reads large binary objects.
+Geneva UDFs can consume and produce [Lance Blob](https://docs.lancedb.com/tables/multimodal) columns — large binary objects (roughly 1 MB to 100s of MB per value) such as images, audio clips, or documents. Blob columns keep large payloads out of the regular column data path and provide lazy, file-like access on read.


I know we link at the bottom of this page, but for this particular definition of a blob column, I think the Lance blob API docs are just more apt to point to the actual definition? WDYT?

prrao87 · 2026-07-22T00:40:50Z

+| **Use when** | the table already exists on an older storage version | creating new tables — cheaper checkpoints and bulk writes at scale |

-For scalar UDFs, blob columns are expected to be of type `BlobFile`
+For Blob v2 output columns on data storage version 2.2+, Geneva checkpoints compact **descriptors** instead of the payload bytes and writes payloads through PyLance's bulk blob writer, which significantly reduces checkpoint I/O for large backfills. This requires `pylance>=9.0.0b24`.


We're linking to a beta release of pylance, which isn't ideal. Worth leaving a comment in the source to change this to an official release once it's available.

dantasse

lgtm, thanks for updating!

one question I have: I thought that keeping multimodal data inline was a key advantage of LanceDB. what are those benefits, and does blob v2 mean we lose those benefits?

answer to myself after thinking a little more: no, I think it's fine. keeping data inline is mostly about ease of use - you don't have to make sure the destination files don't move, no need to load your images after you load your data, etc. as well as fast retrieval. so if blob v2 is still handling all that for the user, then the user gets the benefit whether it's physically stored inline or not.

I am not sure if it's worth including a bit about this at the start. what do you think?

justinrmiller · 2026-07-22T17:03:03Z

lgtm, thanks for updating!

one question I have: I thought that keeping multimodal data inline was a key advantage of LanceDB. what are those benefits, and does blob v2 mean we lose those benefits?

answer to myself after thinking a little more: no, I think it's fine. keeping data inline is mostly about ease of use - you don't have to make sure the destination files don't move, no need to load your images after you load your data, etc. as well as fast retrieval. so if blob v2 is still handling all that for the user, then the user gets the benefit whether it's physically stored inline or not.

I am not sure if it's worth including a bit about this at the start. what do you think?

Blob V2 supports both. Some customers with TBs of data don't want the data to be copied twice.

tsmithv11

LGTM. Probably worth waiting for blob v2 to merge this.

tsmithv11 · 2026-07-22T17:34:24Z

+| **Declaration** | `pa.large_binary()` + `lance-encoding:blob` field metadata | `lance.blob_field(...)` (the `lance.blob.v2` extension type) |
+| **Table requirement** | data storage version ≥ 2.0 | data storage version ≥ 2.2 |
+| **Storage** | payload bytes inline in the column | payloads packed into sidecar blob files, referenced by descriptors |
+| **Use when** | the table already exists on an older storage version | creating new tables — cheaper checkpoints and bulk writes at scale |


Based on your conversation above - could be worth adding a caveat that, depending on the payload size and table configuration, values may be stored inline, packed into shared sidecar files, or written to dedicated sidecar files.

mintlify Bot deployed to staging - docs July 20, 2026 18:30 View deployment

mintlify Bot deployed to staging - docs July 20, 2026 18:42 View deployment

justinrmiller requested review from brendanclement, dantasse, prrao87 and tsmithv11 and removed request for tsmithv11 July 20, 2026 20:35

prrao87 approved these changes Jul 22, 2026

View reviewed changes

dantasse approved these changes Jul 22, 2026

View reviewed changes

tsmithv11 approved these changes Jul 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

docs(geneva): document Blob v1 and Blob v2 UDF workflows#316

docs(geneva): document Blob v1 and Blob v2 UDF workflows#316
justinrmiller wants to merge 2 commits into
mainfrom
gen-378-blob-v1-v2-docs

justinrmiller commented Jul 20, 2026 •

edited

Loading

Uh oh!

mintlify Bot commented Jul 20, 2026

Uh oh!

mintlify Bot commented Jul 20, 2026

Uh oh!

prrao87 left a comment

Uh oh!

prrao87 Jul 22, 2026

Uh oh!

prrao87 Jul 22, 2026

Uh oh!

prrao87 Jul 22, 2026

Uh oh!

dantasse left a comment

Uh oh!

justinrmiller commented Jul 22, 2026

Uh oh!

tsmithv11 left a comment

Uh oh!

tsmithv11 Jul 22, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

justinrmiller commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

mintlify Bot commented Jul 20, 2026

Uh oh!

mintlify Bot commented Jul 20, 2026

Uh oh!

prrao87 left a comment

Choose a reason for hiding this comment

Uh oh!

prrao87 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

prrao87 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

prrao87 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

dantasse left a comment

Choose a reason for hiding this comment

Uh oh!

justinrmiller commented Jul 22, 2026

Uh oh!

tsmithv11 left a comment

Choose a reason for hiding this comment

Uh oh!

tsmithv11 Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

justinrmiller commented Jul 20, 2026 •

edited

Loading

tsmithv11 Jul 22, 2026 •

edited

Loading