Clarify the journey between the training data-loading pages
After the changes in #294, training/index.mdx now provides a fairly complete introduction to loading training data with LanceDB. It covers direct table access, StreamingDataset, filtering, shuffling, transformations, distributed splits, and resumability. It ends by briefly introducing Permutation as the lower-level alternative.
The next sidebar page is currently called PyTorch integration. That title is broad, and the page initially appears to restart the data-loading discussion with Table, Permutation, and DataLoader. For someone reading the pages in sequence, it is not immediately clear why they have moved to a new page or how its purpose differs from the one they just finished.
The underlying distinction is useful and should remain:
Table is the simplest direct-access option.StreamingDataset is the complete streaming loader, with prefetching, transforms, distributed splitting, elastic determinism, and resumability.Permutation is the lower-level, map-style option for users who want explicit control over row selection and ordering while managing batching through PyTorch.
The documentation should make that choice clear. The first page should remain focused on the overall training-data workflow and the recommended StreamingDataset path. Its final section should explain when someone would choose Permutation instead and then point naturally to the PyTorch page.
The PyTorch page should be framed specifically around using Table and Permutation with torch.utils.data.DataLoader. Its unique value is the practical integration detail: collation, output formats, column selection, worker processes, remote connections, and connection factories. It should not read like a second general introduction to data loading.
A title such as PyTorch DataLoader would make this narrower purpose clearer than PyTorch integration. The page should also contain an explicit section such as Use a permutation with DataLoader, giving the preceding page a meaningful destination to link to.
This issue is about titles, scope, and narrative continuity between the two existing pages. It is not intended to move StreamingDataset into a separate document or substantially rewrite the feature documentation from #294.
Clarify the journey between the training data-loading pages
After the changes in #294,
training/index.mdxnow provides a fairly complete introduction to loading training data with LanceDB. It covers direct table access,StreamingDataset, filtering, shuffling, transformations, distributed splits, and resumability. It ends by briefly introducingPermutationas the lower-level alternative.The next sidebar page is currently called PyTorch integration. That title is broad, and the page initially appears to restart the data-loading discussion with
Table,Permutation, andDataLoader. For someone reading the pages in sequence, it is not immediately clear why they have moved to a new page or how its purpose differs from the one they just finished.The underlying distinction is useful and should remain:
Tableis the simplest direct-access option.StreamingDatasetis the complete streaming loader, with prefetching, transforms, distributed splitting, elastic determinism, and resumability.Permutationis the lower-level, map-style option for users who want explicit control over row selection and ordering while managing batching through PyTorch.The documentation should make that choice clear. The first page should remain focused on the overall training-data workflow and the recommended
StreamingDatasetpath. Its final section should explain when someone would choosePermutationinstead and then point naturally to the PyTorch page.The PyTorch page should be framed specifically around using
TableandPermutationwithtorch.utils.data.DataLoader. Its unique value is the practical integration detail: collation, output formats, column selection, worker processes, remote connections, and connection factories. It should not read like a second general introduction to data loading.A title such as PyTorch DataLoader would make this narrower purpose clearer than PyTorch integration. The page should also contain an explicit section such as Use a permutation with DataLoader, giving the preceding page a meaningful destination to link to.
This issue is about titles, scope, and narrative continuity between the two existing pages. It is not intended to move
StreamingDatasetinto a separate document or substantially rewrite the feature documentation from #294.