mirror of
https://github.com/ZoneTree/ZoneTree.git
synced 2026-10-28 20:57:50 +08:00
Improve Storage docs.
This commit is contained in:
Ahmed Yasin Koculu
@@ -8,6 +8,7 @@
|
||||
"concepts/": "iconoir:book",
|
||||
"concepts/index.mdx": "iconoir:book",
|
||||
"storage/": "iconoir:database",
|
||||
"storage/index.mdx": "iconoir:database",
|
||||
"benchmark/": "iconoir:stats-report",
|
||||
"durability/": "iconoir:shield-check",
|
||||
"tuning/": "iconoir:settings",
|
||||
@@ -37,10 +38,11 @@
|
||||
"concepts/op-index.mdx",
|
||||
"concepts/value-mutability.mdx",
|
||||
"storage/",
|
||||
"storage/disk-segments.md",
|
||||
"storage/file-stream-providers.md",
|
||||
"storage/memory-usage.md",
|
||||
"storage/compression.md",
|
||||
"storage/index.mdx",
|
||||
"storage/disk-segments.mdx",
|
||||
"storage/file-stream-providers.mdx",
|
||||
"storage/memory-usage.mdx",
|
||||
"storage/compression.mdx",
|
||||
"benchmark/",
|
||||
"benchmark/benchmark.mdx",
|
||||
"durability/",
|
||||
|
||||
@@ -362,7 +362,7 @@ Treat callbacks as notifications and keep them short; enqueue expensive transfer
|
||||
|
||||
Some maintenance record counts include multiple physical versions of a key across segments. Use them as storage-pressure measurements rather than unique-key counts. Coordinate aggregate memory and I/O budgets across hosted partitions, and release iterators and backup activity before removing a replica's files.
|
||||
|
||||
See [maintenance](../usage/maintenance.mdx), [file stream providers](../storage/file-stream-providers.md), and the [API overview](../reference/api-overview.md) for the individual controls.
|
||||
See [maintenance](../usage/maintenance.mdx), [file stream providers](../storage/file-stream-providers.mdx), and the [API overview](../reference/api-overview.md) for the individual controls.
|
||||
|
||||
## Implementation references
|
||||
|
||||
|
||||
@@ -132,7 +132,7 @@ Bottom merging has its own explicit entry point, `maintainer.StartBottomSegments
|
||||
|
||||
Watch read-only segment accumulation alongside merge duration and write rate. A growing queue means frozen memory is arriving faster than maintenance is clearing it. Budget compaction resources and record limits together; the mutable-segment limit alone does not cap the tree's total memory use.
|
||||
|
||||
See [maintenance](../usage/maintenance.mdx) and [memory usage](../storage/memory-usage.md) for operational controls.
|
||||
See [maintenance](../usage/maintenance.mdx) and [memory usage](../storage/memory-usage.mdx) for operational controls.
|
||||
|
||||
## The controls behind the shape
|
||||
|
||||
|
||||
@@ -66,7 +66,7 @@ Single-part disk storage has four variations selected from the runtime key and v
|
||||
|
||||
Strings and reference-containing values need a variable-size representation. A custom serializer returning the same number of bytes for every value does not by itself turn a managed value type into an unmanaged one.
|
||||
|
||||
File names and counts depend on the selected layout and compression options. Use [disk segments](../storage/disk-segments.md) for file-level details rather than treating one sample file listing as universal.
|
||||
File names and counts depend on the selected layout and compression options. Use [disk segments](../storage/disk-segments.mdx) for file-level details rather than treating one sample file listing as universal.
|
||||
|
||||
## Interpret the empty disk slot
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ ZoneTree protects recent writes with a write-ahead log according to the configur
|
||||
|
||||
The default mode is `AsyncCompressed`. It is the normal starting point for most applications: WAL protection stays enabled and writes can remain very fast because WAL work is handled through a background path.
|
||||
|
||||
Compression method, compression level, and block-size details are covered in [compression](../storage/compression.md).
|
||||
Compression method, compression level, and block-size details are covered in [compression](../storage/compression.mdx).
|
||||
|
||||
## Choosing A Mode
|
||||
|
||||
|
||||
@@ -130,7 +130,7 @@ First actions:
|
||||
* dispose iterators promptly,
|
||||
* review value shape and value size.
|
||||
|
||||
See [memory usage](../storage/memory-usage.md).
|
||||
See [memory usage](../storage/memory-usage.mdx).
|
||||
|
||||
## Disk Usage Grows
|
||||
|
||||
|
||||
@@ -1,147 +0,0 @@
|
||||
# Compression
|
||||
|
||||
ZoneTree supports compression in two different storage paths:
|
||||
|
||||
* WAL compression for recent log records,
|
||||
* disk segment compression for persistent random-access sorted data.
|
||||
|
||||
Both use the same compression method/level model, but they serve different workloads and have different default block sizes.
|
||||
|
||||
## Compression Methods
|
||||
|
||||
ZoneTree supports:
|
||||
|
||||
* `LZ4`
|
||||
* `Zstd`
|
||||
* `Brotli`
|
||||
* `Gzip`
|
||||
* `None`
|
||||
|
||||
Compression levels are validated against the selected method. The default profile uses Zstd compression because it is a good general-purpose balance for storage-engine workloads.
|
||||
|
||||
## WAL Compression
|
||||
|
||||
The default WAL mode is `AsyncCompressed`.
|
||||
|
||||
Default WAL compression:
|
||||
|
||||
```text
|
||||
method: Zstd
|
||||
level: Zstd0
|
||||
block: 256 KB
|
||||
```
|
||||
|
||||
Compressed WALs reduce log size and can improve write-path IO when data compresses well. They also add compression work. The async compressed mode is the default because it keeps WAL protection enabled while preserving high write throughput for most applications.
|
||||
|
||||
Sync compressed WAL uses the same compressed-block WAL file shape and tail behavior. Use Sync WAL when the application specifically needs the plain synchronous WAL path; use sync-compressed WAL when compressed WAL storage or throughput matters and its tail durability tradeoff is acceptable.
|
||||
|
||||
## Disk Segment Compression
|
||||
|
||||
Disk segment compression is block-based random-access compression.
|
||||
|
||||
Default disk compression:
|
||||
|
||||
```text
|
||||
method: Zstd
|
||||
level: Zstd0
|
||||
block: 4 MB
|
||||
```
|
||||
|
||||
Compressed disk segment files store sorted data in compressed blocks. ZoneTree can seek to a block, read it, decompress it, and then serve keys or values from the decompressed block.
|
||||
|
||||
This is different from compressing a whole file as one stream. ZoneTree keeps random access.
|
||||
|
||||
## Block Cache Interaction
|
||||
|
||||
Compressed disk reads use a decompressed block cache. When a block is read, ZoneTree can keep the decompressed block in memory so nearby or repeated reads do not have to read and decompress the same block again.
|
||||
|
||||
The important relationship is:
|
||||
|
||||
```text
|
||||
disk compression block size -> decompressed block cache unit size
|
||||
```
|
||||
|
||||
Larger disk compression blocks can improve compression ratio and sequential read behavior, but each cached block keeps more decompressed data in memory. Smaller blocks can make random reads cheaper, but may reduce compression density and increase metadata overhead.
|
||||
|
||||
Block cache cleanup is controlled by the maintainer:
|
||||
|
||||
* `BlockCacheLifeTime`
|
||||
* `InactiveBlockCacheCleanupInterval`
|
||||
|
||||
See [read path caching](../tuning/read-path-caching.md).
|
||||
|
||||
## Why WAL And Disk Defaults Differ
|
||||
|
||||
ZoneTree intentionally uses smaller blocks for WAL and larger blocks for disk segments.
|
||||
|
||||
WAL blocks serve recent writes and recovery. Smaller blocks keep log recovery and tail handling more responsive.
|
||||
|
||||
Disk segment blocks serve persistent sorted data. Larger blocks can improve compression density and are often a better fit for range reads, sparse lookups, and block cache reuse.
|
||||
|
||||
## When Compression Helps
|
||||
|
||||
Compression can help when:
|
||||
|
||||
* values are text, JSON, repeated structures, or otherwise compressible,
|
||||
* storage bandwidth is more expensive than CPU,
|
||||
* disk space matters,
|
||||
* read workloads repeatedly touch nearby compressed blocks,
|
||||
* large disk segments benefit from smaller physical files.
|
||||
|
||||
## When Compression Costs More
|
||||
|
||||
Compression may cost more than it saves when:
|
||||
|
||||
* payloads are already compressed,
|
||||
* CPU is the bottleneck,
|
||||
* values are tiny and mostly random,
|
||||
* random reads constantly touch cold blocks,
|
||||
* compression block size is too large for the read pattern.
|
||||
|
||||
For large values, compression should be tested with real payloads. A large JSON value and a large already-compressed image have very different storage behavior.
|
||||
|
||||
## Configuration
|
||||
|
||||
Configure WAL compression through `WriteAheadLogOptions`:
|
||||
|
||||
```csharp
|
||||
using ZoneTree.Options;
|
||||
|
||||
using var zoneTree = new ZoneTreeFactory<int, string>()
|
||||
.SetDataDirectory("data/app")
|
||||
.ConfigureWriteAheadLogOptions(options =>
|
||||
{
|
||||
options.CompressionMethod = CompressionMethod.Zstd;
|
||||
options.CompressionLevel = CompressionLevels.Zstd0;
|
||||
options.CompressionBlockSize = 256 * 1024;
|
||||
})
|
||||
.OpenOrCreate();
|
||||
```
|
||||
|
||||
Configure disk segment compression through `DiskSegmentOptions`:
|
||||
|
||||
```csharp
|
||||
using ZoneTree.Options;
|
||||
|
||||
using var zoneTree = new ZoneTreeFactory<int, string>()
|
||||
.SetDataDirectory("data/app")
|
||||
.ConfigureDiskSegmentOptions(options =>
|
||||
{
|
||||
options.CompressionMethod = CompressionMethod.Zstd;
|
||||
options.CompressionLevel = CompressionLevels.Zstd0;
|
||||
options.CompressionBlockSize = 4 * 1024 * 1024;
|
||||
})
|
||||
.OpenOrCreate();
|
||||
```
|
||||
|
||||
`SetDiskSegmentCompressionBlockSize` is also available as a convenience method on `ZoneTreeFactory`.
|
||||
|
||||
## Practical Model
|
||||
|
||||
Think about compression in ZoneTree this way:
|
||||
|
||||
* WAL compression optimizes recent durable log records.
|
||||
* Disk compression optimizes persistent sorted segment files.
|
||||
* Disk compression block size also defines the main read-cache unit.
|
||||
* The maintainer controls inactive decompressed block cleanup.
|
||||
* Real payloads matter more than generic compression claims.
|
||||
@@ -0,0 +1,133 @@
|
||||
---
|
||||
title: Compression
|
||||
description: Configure WAL and disk compression independently, understand block cache costs and persisted settings, and compare profiles with representative workloads.
|
||||
---
|
||||
|
||||
<SectionHeader class="zt-guide-intro" eyebrow="STORAGE / COMPRESSION" title="Trade CPU, I/O, and cache memory deliberately" titleTag="h1" titleId="compression-title" subtitle="WAL compression handles recent log records. Disk compression handles sorted random-access blocks. Configure and measure these two paths independently."/>
|
||||
|
||||
<Grid columns="1" columnsMd="2" class="gap-3 zt-guide-outcomes">
|
||||
<Panel tone="accent" variant="surface" class="p-0" bodyClass="p-4"><strong>WAL / recent writes</strong><p>The default is AsyncCompressed with Zstd level 0 and 256 KiB blocks. The WAL mode also determines logging behavior.</p></Panel>
|
||||
<Panel class="p-0" bodyClass="p-4"><strong>Disk / sorted records</strong><p>The default is Zstd level 0 and 4 MiB blocks. Reads decompress the blocks containing the requested data.</p></Panel>
|
||||
</Grid>
|
||||
|
||||
## Separate the two paths
|
||||
|
||||
Both paths expose a compression method, method-specific level, and block size, but through different option objects:
|
||||
|
||||
| Control | WAL | Disk segments |
|
||||
| --- | --- | --- |
|
||||
| Options | `WriteAheadLogOptions` | `DiskSegmentOptions` |
|
||||
| Default method / level | `Zstd` / `CompressionLevels.Zstd0` | `Zstd` / `CompressionLevels.Zstd0` |
|
||||
| Default block size | `256 * 1024` bytes, or 256 KiB. | `4 * 1024 * 1024` bytes, or 4 MiB. |
|
||||
| Valid block-size range | 256 KiB through 16 MiB. | 1 MiB through 64 MiB. |
|
||||
| Main work | Append logging and recovery replay. | Writing sorted output and serving block-addressable reads. |
|
||||
| Scope of settings | Newly created WALs. Existing WALs retain their own recorded options. | Newly written disk files. Existing files retain their recorded block size and method. |
|
||||
|
||||
Changing compression options does not rewrite existing files. A merge may write new files with the new profile while reusing eligible parts with the earlier profile. Multipart descriptors have their own metadata encoding; disk data-block settings do not configure that encoding.
|
||||
|
||||
<AlertBox class="zt-guide-callout" tone="info" title="Choose recovery guarantees through the WAL mode" icon="tabler:shield-check">
|
||||
<p>A compression method or level does not define when a write is durable. <code>AsyncCompressed</code>, <code>SyncCompressed</code>, plain <code>Sync</code>, and <code>None</code> have different logging and failure behavior. Choose the mode with the <a href="/docs/durability/wal-modes/">WAL modes guide</a>, then tune compression within that choice.</p>
|
||||
</AlertBox>
|
||||
|
||||
## Select a method and its own level
|
||||
|
||||
`CompressionMethod` supports `LZ4`, `Zstd`, `Brotli`, `Gzip`, and `None`. Use the matching constants from `CompressionLevels`; a numeric level is not a comparable quality setting across codecs.
|
||||
|
||||
| Method | Example level constants | Interpretation |
|
||||
| --- | --- | --- |
|
||||
| `Zstd` | `Zstd0`, `Zstd1`, `Zstd3` | Codec-specific numeric levels. `Zstd0` is ZoneTree's default. |
|
||||
| `LZ4` | `LZ4Fastest`, `LZ4HighCompression9` | Fast mode or a supported high-compression level. |
|
||||
| `Brotli` | `BrotliFastest`, `BrotliOptimal`, `BrotliSmallestSize` | Named .NET compression-level choices. |
|
||||
| `Gzip` | `GzipFastest`, `GzipOptimal`, `GzipSmallestSize` | Named .NET compression-level choices. |
|
||||
| `None` | Level is unused. | Stores uncompressed payload bytes within the selected storage path. |
|
||||
|
||||
ZoneTree validates the level against the method. Do not carry a level constant from one codec to another just because both constants are integers. Use `CompressionLevels.IsValid(method, level)` when accepting an application-supplied profile.
|
||||
|
||||
## Configure WAL and disk together
|
||||
|
||||
This complete example explicitly selects a WAL mode, uses LZ4 for its log blocks, and keeps Zstd for disk records. It finishes a controlled merge and verifies the value remains readable:
|
||||
|
||||
```csharp
|
||||
using ZoneTree;
|
||||
using ZoneTree.Options;
|
||||
|
||||
using var tree = new ZoneTreeFactory<int, string>()
|
||||
.SetDataDirectory("data/compression-profile")
|
||||
.ConfigureWriteAheadLogOptions(options =>
|
||||
{
|
||||
options.WriteAheadLogMode = WriteAheadLogMode.AsyncCompressed;
|
||||
options.CompressionMethod = CompressionMethod.LZ4;
|
||||
options.CompressionLevel = CompressionLevels.LZ4Fastest;
|
||||
options.CompressionBlockSize = 256 * 1024;
|
||||
})
|
||||
.ConfigureDiskSegmentOptions(options =>
|
||||
{
|
||||
options.CompressionMethod = CompressionMethod.Zstd;
|
||||
options.CompressionLevel = CompressionLevels.Zstd0;
|
||||
options.CompressionBlockSize = 4 * 1024 * 1024;
|
||||
})
|
||||
.OpenOrCreate();
|
||||
using var maintainer = tree.CreateMaintainer();
|
||||
|
||||
tree.Upsert(42, new string('x', 1000));
|
||||
|
||||
// No concurrent writers in this example.
|
||||
maintainer.EvictToDisk();
|
||||
await maintainer.WaitForBackgroundThreadsAsync();
|
||||
|
||||
if (!tree.TryGet(42, out var value) || value.Length != 1000)
|
||||
throw new InvalidOperationException("Expected the stored payload.");
|
||||
|
||||
Console.WriteLine($"Read {value.Length} characters after merging.");
|
||||
```
|
||||
|
||||
The two methods are intentionally independent; this is an example profile, not a universal performance recommendation. `SetDiskSegmentCompressionBlockSize()` is a factory shorthand for the disk block-size option.
|
||||
|
||||
## Understand the unit of a disk read
|
||||
|
||||
Disk files contain individually addressable blocks. A cold read locates and reads the relevant compressed block, decompresses it, and serves the requested bytes. Subsequent reads can reuse the decompressed block while it remains cached.
|
||||
|
||||
```text
|
||||
record lookup
|
||||
→ locate data/header block
|
||||
→ use cached bytes, or read and decompress the block
|
||||
→ decode the requested key/value
|
||||
→ retain reusable bytes and applicable materialized entries
|
||||
```
|
||||
|
||||
This preserves random access without decompressing the whole file. One record can still touch multiple blocks or devices, particularly with variable-size values and separate headers.
|
||||
|
||||
| Block-size change | Potential benefit | Cost to measure |
|
||||
| --- | --- | --- |
|
||||
| Larger disk blocks | More neighboring data per decompression; potentially better compression ratio. | More bytes read and retained for isolated cold accesses. |
|
||||
| Smaller disk blocks | Less unrelated block data for scattered reads. | More block boundaries and metadata; potentially worse compression ratio. |
|
||||
|
||||
Decompressed cache memory follows the uncompressed block size, not the compressed file size. Materialized key/value objects and other caches add memory beyond those bytes. Choosing `CompressionMethod.None` keeps the block container and its `.z` filenames; it does not disable the read caches.
|
||||
|
||||
## Coordinate block retention with the maintainer
|
||||
|
||||
The standard maintainer releases inactive read buffers according to `BlockCacheLifeTime` and its cleanup interval. Shorter retention can reduce inactive memory while causing more reads and decompression when data becomes hot again.
|
||||
|
||||
The cleanup timer captures its interval when started. Use the restart sequence in [memory usage](memory-usage.mdx#configure-retention-and-observe-the-write-set) when changing that interval, and include materialized objects when measuring the cache footprint.
|
||||
|
||||
## Compare profiles with your data
|
||||
|
||||
Use the same records, key order, concurrency, and read/write mix for each candidate profile. Separate cold reads from warmed-cache reads, and measure merge work alongside request latency.
|
||||
|
||||
| Measure | Why it changes the decision |
|
||||
| --- | --- |
|
||||
| Compressed bytes / original bytes | Reveals whether actual payloads benefit. Text and already-compressed images can behave very differently. |
|
||||
| CPU and allocations | Captures codec work and temporary memory costs. |
|
||||
| Write and merge throughput | Shows whether compression helps storage bandwidth or becomes a bottleneck. |
|
||||
| Cold point-read latency | Exposes decompression work for isolated block accesses. |
|
||||
| Warm reads and range scans | Measures block reuse and sequential locality. |
|
||||
| Retained memory and merge peaks | Includes decompressed blocks, materialized records, and overlapping buffers. |
|
||||
| Recovery time | Measures the selected WAL profile with representative log volume. |
|
||||
|
||||
Compare newly written datasets when you want a clean profile comparison. Reopening an old tree with new options can leave a mixture of old and new files, making a filename or factory configuration alone misleading evidence of the active codecs.
|
||||
|
||||
<Panel class="p-0 zt-guide-next-panel" bodyClass="p-5">
|
||||
<strong>Evaluate compression as part of the storage budget</strong>
|
||||
<p>Pair physical file measurements with cache ownership and read patterns before changing the defaults.</p>
|
||||
<Flex wrap="true" class="gap-3"><BtnLink href="/docs/storage/memory-usage/" tone="accent" icon="tabler:arrow-right" iconPosition="end">Memory usage</BtnLink><BtnLink href="/docs/tuning/large-values/" tone="neutral" variant="outline">Large values</BtnLink><BtnLink href="/docs/storage/disk-segments/" tone="neutral" variant="outline">Disk segment files</BtnLink></Flex>
|
||||
</Panel>
|
||||
@@ -1,81 +0,0 @@
|
||||
# Disk Segment Files
|
||||
|
||||
Disk segments are immutable persistent record sets. This page describes their physical file shape. For terminology, see [segments](../concepts/segments.mdx). For sizing and merge tuning, see [disk segment tuning](../tuning/disk-segments.md).
|
||||
|
||||
## File Categories
|
||||
|
||||
ZoneTree stores disk segment data through random-access devices. The common file categories are:
|
||||
|
||||
| Category | Purpose |
|
||||
| --- | --- |
|
||||
| `.data` | serialized key/value bytes or fixed-size record data |
|
||||
| `.head` | headers and offsets for layouts that need them |
|
||||
| `.sparse` | optional sparse index entries |
|
||||
| `.multi` | multipart disk segment descriptor |
|
||||
|
||||
Compression usually adds the `.z` suffix to the physical file.
|
||||
|
||||
```text
|
||||
single-part segment 10
|
||||
10.data.z
|
||||
10.head.z
|
||||
10.sparse.z
|
||||
|
||||
multipart segment 20
|
||||
20.multi
|
||||
part files for segment 11
|
||||
part files for segment 12
|
||||
part files for segment 13
|
||||
```
|
||||
|
||||
Fixed-size key/value layouts may not need a `.head` file. Variable-size layouts use headers and offsets so ZoneTree can keep random access while supporting strings, arrays, and other variable-size serialized values.
|
||||
|
||||
## Single-Part Layouts
|
||||
|
||||
Single-part disk segments choose one of four layouts from the key and value shape:
|
||||
|
||||
| Layout | Physical idea |
|
||||
| --- | --- |
|
||||
| fixed key + fixed value | records can be addressed directly in `.data` |
|
||||
| fixed key + variable value | fixed keys and value offsets are stored in `.head`; value bytes live in `.data` |
|
||||
| variable key + fixed value | key offsets and fixed values are stored in `.head`; key bytes live in `.data` |
|
||||
| variable key + variable value | key/value offsets are stored in `.head`; key/value bytes live in `.data` |
|
||||
|
||||
This is a storage optimization. The logical API stays the same for all layouts.
|
||||
|
||||
## Multipart Descriptor
|
||||
|
||||
A multipart disk segment is one logical disk segment made from ordered parts. Its `.multi` descriptor stores the part segment ids and boundary key/value information needed to route reads to the correct part.
|
||||
|
||||
```text
|
||||
20.multi
|
||||
part ids: 11, 12, 13
|
||||
first/last keys for each part
|
||||
first/last values for each part
|
||||
```
|
||||
|
||||
The parts are regular `IDiskSegment` instances. A part can be carried forward into a later multipart segment when its key range is unchanged by a merge.
|
||||
|
||||
## Sparse Index File
|
||||
|
||||
The `.sparse` file stores selected key/value/index entries. It helps ZoneTree position disk searches without loading every key into memory.
|
||||
|
||||
`DefaultSparseArrayStepSize` controls sparse index density:
|
||||
|
||||
| Value | Effect |
|
||||
| --- | --- |
|
||||
| smaller step | more sparse entries, faster positioning, more memory |
|
||||
| larger step | fewer sparse entries, less memory, more local search |
|
||||
| `0` | disables default sparse array creation/loading |
|
||||
|
||||
## Immutability
|
||||
|
||||
Disk segment files are immutable after creation. Merge operations create new disk segment files and then drop old files when they are no longer needed.
|
||||
|
||||
Iterators and live backup can pin disk segments while they read or copy files. A pinned segment is not rewritten; its deletion is delayed until the pin is released.
|
||||
|
||||
## Restore And Backup
|
||||
|
||||
`IDiskSegment.GetFiles()` returns the physical files that belong to a disk segment. For multipart segments, this includes the `.multi` descriptor and every part file.
|
||||
|
||||
Live backup stores these immutable files and streams in-memory records separately. Restore uses the recorded file order to rebuild the `DiskSegment` and bottom segment layout.
|
||||
@@ -0,0 +1,126 @@
|
||||
---
|
||||
title: Disk Segment Files
|
||||
description: Read ZoneTree's disk file layouts, multipart descriptors, and sparse indexes, and understand how merges and reader leases affect file lifetime.
|
||||
---
|
||||
|
||||
<SectionHeader class="zt-guide-intro" eyebrow="STORAGE / PHYSICAL LAYOUT" title="Read the files behind a disk segment" titleTag="h1" titleId="disk-segments-title" subtitle="A logical disk segment can own several files and reuse parts from earlier merges. Understand that structure before interpreting a directory listing or planning a backup."/>
|
||||
|
||||
<Grid columns="1" columnsMd="3" class="gap-3 zt-guide-outcomes">
|
||||
<Panel class="p-0" bodyClass="p-4"><strong>Sorted records</strong><p>Data and optional headers provide random access to serialized keys and values.</p></Panel>
|
||||
<Panel tone="accent" variant="surface" class="p-0" bodyClass="p-4"><strong>Reusable ranges</strong><p>A multipart descriptor connects ordered parts into one logical segment.</p></Panel>
|
||||
<Panel class="p-0" bodyClass="p-4"><strong>Reader leases</strong><p>Retired files remain available while an iterator or backup still needs them.</p></Panel>
|
||||
</Grid>
|
||||
|
||||
## Recognize the file categories
|
||||
|
||||
The default random-access device manager names files with a numeric segment ID followed by a category. Current writers use the block-container format for data, headers, and sparse indexes, adding `.z` to those filenames.
|
||||
|
||||
| File | What it contains | When to expect it |
|
||||
| --- | --- | --- |
|
||||
| `<id>.data.z` | Serialized record bytes. | A single-part segment, including each physical part of a multipart segment. |
|
||||
| `<id>.head.z` | Keys or values with fixed-size headers containing offsets and lengths. | Layouts where the key, value, or both have variable size. |
|
||||
| `<id>.sparse.z` | A persisted sample of key/value/index entries. | When a default sparse index is enabled and saved. |
|
||||
| `<id>.multi` | Part IDs and boundary keys and values. | A multipart logical segment. Its metadata has its own encoding. |
|
||||
|
||||
The suffix describes the container. Selecting `CompressionMethod.None` still produces `.z` data files with uncompressed block payloads; it does not switch to a different record layout.
|
||||
|
||||
An illustrative multipart segment with two variable-value parts looks like this:
|
||||
|
||||
```text
|
||||
20.multi logical segment descriptor
|
||||
├─ part 11
|
||||
│ 11.head.z keys + value locations
|
||||
│ 11.data.z value bytes
|
||||
│ 11.sparse.z sampled records
|
||||
└─ part 12
|
||||
12.head.z
|
||||
12.data.z
|
||||
12.sparse.z
|
||||
```
|
||||
|
||||
These are segment files. The directory also contains metadata and WAL files that describe other parts of the database. Start with the [Storage overview](index.mdx) for the full set of concerns.
|
||||
|
||||
## Understand the four record layouts
|
||||
|
||||
Each physical single-part segment selects a layout from the runtime key and value types. Types with no managed references use the fixed-size path; strings, arrays, and types containing references use the variable-size path.
|
||||
|
||||
| Key / value shape | Header file | Data file |
|
||||
| --- | --- | --- |
|
||||
| Fixed / fixed | Not needed. | Consecutive serialized key/value pairs. |
|
||||
| Fixed / variable | Fixed keys with value offsets and lengths. | Serialized values. |
|
||||
| Variable / fixed | Key offsets and lengths with fixed values. | Serialized keys. |
|
||||
| Variable / variable | Offsets and lengths for both fields. | Serialized keys and values. |
|
||||
|
||||
The public read/write API stays the same. A custom serializer that happens to emit a constant number of bytes does not by itself move a reference type onto the fixed-size path. Fixed-size serializers must preserve the size contract expected by that path; see [serializers and comparers](../concepts/serializers-and-comparers.mdx).
|
||||
|
||||
## Follow multipart ownership
|
||||
|
||||
The default disk segment mode is `MultiPartDiskSegment`. Its descriptor stores the IDs and first/last key and value information for its ordered parts. Reads use those boundaries to select the relevant part.
|
||||
|
||||
A merge can reuse an eligible part whose key range is unaffected instead of rewriting its records. The new logical segment then references that part's existing files. As a result:
|
||||
|
||||
- A logical segment ID need not match the IDs of its data files.
|
||||
- A new descriptor does not imply every part was newly written.
|
||||
- Part sizes and reuse eligibility affect merge work and write amplification.
|
||||
|
||||
The [segments guide](../concepts/segments.mdx) explains the logical units and reuse rules. [Disk segment tuning](../tuning/disk-segments.md) covers size controls.
|
||||
|
||||
## Budget for the sparse index
|
||||
|
||||
Sparse entries retain a sampled key, value, and record index in memory. They narrow a disk search to a smaller interval without retaining every record. Sampled values can make the index significant when values are large.
|
||||
|
||||
`DiskSegmentOptions.DefaultSparseArrayStepSize` defaults to `1024`: roughly one sampled record per 1,024 records, plus boundary handling.
|
||||
|
||||
| Setting | Effect |
|
||||
| --- | --- |
|
||||
| Smaller positive step | More sampled records and retained objects; narrower search intervals. |
|
||||
| Larger positive step | Fewer samples; wider intervals to search on disk. |
|
||||
| `0` | Disables default sparse-array creation and loading. |
|
||||
|
||||
The index is an optional search accelerator. Disabling it does not remove records or change their order. The byte cost depends on the sampled keys and values, not just the entry count.
|
||||
|
||||
## Inspect files through the API
|
||||
|
||||
`IDiskSegment.GetFiles()` enumerates a segment's physical files. For a multipart segment it includes the descriptor and the files of every part. This complete example writes records, finishes a controlled merge with no concurrent writers, and lists the current disk and bottom segments:
|
||||
|
||||
```csharp
|
||||
using ZoneTree;
|
||||
|
||||
using var tree = new ZoneTreeFactory<int, string>()
|
||||
.SetDataDirectory("data/storage-files")
|
||||
.OpenOrCreate();
|
||||
using var maintainer = tree.CreateMaintainer();
|
||||
|
||||
for (var key = 0; key < 10; ++key)
|
||||
tree.Upsert(key, $"Record {key}");
|
||||
|
||||
maintainer.EvictToDisk();
|
||||
await maintainer.WaitForBackgroundThreadsAsync();
|
||||
|
||||
var segments = tree.Maintenance.BottomSegments
|
||||
.Prepend(tree.Maintenance.DiskSegment);
|
||||
|
||||
foreach (var segment in segments)
|
||||
{
|
||||
foreach (var file in segment.GetFiles())
|
||||
Console.WriteLine($"{file.SegmentId}: {file.FileName}");
|
||||
}
|
||||
```
|
||||
|
||||
File IDs depend on the tree's history. An empty current disk slot may enumerate no files even when bottom segments contain records.
|
||||
|
||||
<AlertBox class="zt-guide-callout" tone="info" title="A file listing is not a live backup" icon="tabler:shield-check">
|
||||
<p><code>GetFiles()</code> describes a segment; it does not freeze the tree or acquire a backup lease. During concurrent maintenance, use the live-backup API to coordinate segment lifetimes and capture in-memory records and restore metadata.</p>
|
||||
</AlertBox>
|
||||
|
||||
## Understand when files disappear
|
||||
|
||||
Merges publish replacement segments rather than rewriting the record data of an active segment. Once a segment is retired, its files can be reclaimed when no reader or backup still leases them. Sparse-index files are auxiliary persisted indexes, separate from that immutable record data.
|
||||
|
||||
Dispose iterators and backup handles promptly. Extra files after a merge can be legitimate while those readers are alive; deleting files based on their age or filename can break an active tree. Use [diagnostics](../operations/diagnostics.md) to investigate failed cleanup.
|
||||
|
||||
<Panel class="p-0 zt-guide-next-panel" bodyClass="p-5">
|
||||
<strong>Connect file layout to resource use</strong>
|
||||
<p>Compression controls the block payloads. Memory usage depends on the indexes and read caches built around them.</p>
|
||||
<Flex wrap="true" class="gap-3"><BtnLink href="/docs/storage/compression/" tone="accent" icon="tabler:arrow-right" iconPosition="end">Compression</BtnLink><BtnLink href="/docs/storage/memory-usage/" tone="neutral" variant="outline">Memory usage</BtnLink><BtnLink href="/docs/durability/backups/" tone="neutral" variant="outline">Live backups</BtnLink></Flex>
|
||||
</Panel>
|
||||
@@ -1,171 +0,0 @@
|
||||
# File Stream Providers
|
||||
|
||||
ZoneTree writes physical storage through `IFileStreamProvider`.
|
||||
|
||||
This abstraction lets the default ZoneTree storage pipeline open, read, write, replace, and delete files without being tied to one backing store.
|
||||
|
||||
The default is `LocalFileStreamProvider`. It stores data on the local file system and is the right choice for normal persistent databases.
|
||||
|
||||
## Where It Is Used
|
||||
|
||||
The file stream provider is used by the default implementations for:
|
||||
|
||||
* write-ahead log files,
|
||||
* metadata files,
|
||||
* disk segment files,
|
||||
* temporary files used during replacement and compaction.
|
||||
|
||||
When you create a factory without passing a provider, ZoneTree uses `LocalFileStreamProvider`.
|
||||
|
||||
```csharp
|
||||
using ZoneTree;
|
||||
|
||||
using var zoneTree = new ZoneTreeFactory<int, string>()
|
||||
.SetDataDirectory("data/app")
|
||||
.OpenOrCreate();
|
||||
```
|
||||
|
||||
To use another provider, pass it to the factory constructor.
|
||||
|
||||
```csharp
|
||||
using ZoneTree;
|
||||
using ZoneTree.AbstractFileStream;
|
||||
|
||||
IFileStreamProvider fileStreamProvider = new LocalFileStreamProvider();
|
||||
|
||||
using var zoneTree = new ZoneTreeFactory<int, string>(fileStreamProvider)
|
||||
.SetDataDirectory("data/app")
|
||||
.OpenOrCreate();
|
||||
```
|
||||
|
||||
The factory uses that provider when it creates the default `WriteAheadLogProvider` and `RandomAccessDeviceManager`.
|
||||
|
||||
Advanced applications can replace those higher-level components directly with `SetWriteAheadLogProvider` or `SetRandomAccessDeviceManager`.
|
||||
|
||||
## Local File Provider
|
||||
|
||||
`LocalFileStreamProvider` is the default provider.
|
||||
|
||||
Use it for:
|
||||
|
||||
* persistent databases,
|
||||
* crash recovery,
|
||||
* write-ahead log durability,
|
||||
* normal production storage,
|
||||
* live backup and restore.
|
||||
|
||||
It delegates to the local file system, so data can survive process exit, machine restart, and application redeploys as long as the underlying storage is available.
|
||||
|
||||
## In-Memory File Provider
|
||||
|
||||
`InMemoryFileStreamProvider` stores ZoneTree's physical files in process memory.
|
||||
|
||||
It still runs ZoneTree's normal stream-based storage flow: metadata, WAL files, disk segment files, temporary files, and merge output are represented as files, but those files live inside the provider instance instead of on disk.
|
||||
|
||||
Use it when you want ZoneTree's ordered storage engine behavior and do not need durability.
|
||||
|
||||
Good use cases:
|
||||
|
||||
* fast tests that should not touch the local file system,
|
||||
* high-RAM machines running durability-optional jobs,
|
||||
* temporary indexing, sorting, staging, or deduplication,
|
||||
* data transfer and transformation pipelines where source data can be replayed,
|
||||
* cache-like or rebuildable datasets.
|
||||
|
||||
```csharp
|
||||
using ZoneTree;
|
||||
using ZoneTree.AbstractFileStream;
|
||||
using ZoneTree.Options;
|
||||
|
||||
var provider = new InMemoryFileStreamProvider();
|
||||
|
||||
using var zoneTree = new ZoneTreeFactory<long, string>(provider)
|
||||
.SetDataDirectory("transfer-job")
|
||||
.ConfigureWriteAheadLogOptions(options =>
|
||||
{
|
||||
options.WriteAheadLogMode = WriteAheadLogMode.None;
|
||||
})
|
||||
.OpenOrCreate();
|
||||
```
|
||||
|
||||
For normal in-memory-provider workloads, prefer `WriteAheadLogMode.None`.
|
||||
|
||||
With `InMemoryFileStreamProvider`, WAL files are also stored in process memory. They do not protect against process termination, machine restart, power loss, or losing the provider instance. Keeping WAL enabled can still be useful in tests that specifically exercise WAL behavior, replay logic, compaction, or recovery code while the provider instance is alive.
|
||||
|
||||
Use a durable provider, such as `LocalFileStreamProvider`, when WAL durability matters.
|
||||
|
||||
## In-Memory Semantics
|
||||
|
||||
`InMemoryFileStreamProvider` is optimized for ZoneTree's own stream usage, not exact file-system emulation.
|
||||
|
||||
Important behavior:
|
||||
|
||||
* data is scoped to the provider instance,
|
||||
* all data is lost when that instance is lost,
|
||||
* writes, truncates, and replacements are visible immediately,
|
||||
* large files use chunked backing storage,
|
||||
* small files avoid large chunk allocation,
|
||||
* `ReadAllBytes` is a small-file convenience API and may throw for very large files,
|
||||
* `Replace` uses fast in-memory object handoff rather than copying large buffers,
|
||||
* `FileShare` is intentionally ignored,
|
||||
* open streams can keep referencing an old in-memory file object after delete or replace,
|
||||
* per-stream `Position` is not thread-safe.
|
||||
|
||||
Use this provider only when those boundaries are acceptable.
|
||||
|
||||
## Custom Providers
|
||||
|
||||
Implement `IFileStreamProvider` when ZoneTree files need to live somewhere other than the local file system or the built-in in-memory provider.
|
||||
|
||||
Examples include:
|
||||
|
||||
* encrypted local storage,
|
||||
* test harnesses with fault injection,
|
||||
* remote object stores with local buffering,
|
||||
* custom storage services,
|
||||
* embedded environments with their own file abstraction.
|
||||
|
||||
A custom provider must return `IFileStream` instances and implement file operations such as existence checks, directory creation, deletion, replacement, and small metadata reads.
|
||||
|
||||
```csharp
|
||||
using ZoneTree.AbstractFileStream;
|
||||
|
||||
public sealed class MyFileStreamProvider : IFileStreamProvider
|
||||
{
|
||||
public IFileStream CreateFileStream(
|
||||
string path,
|
||||
FileMode mode,
|
||||
FileAccess access,
|
||||
FileShare share,
|
||||
int bufferSize = 4096,
|
||||
FileOptions options = FileOptions.None)
|
||||
{
|
||||
// Return an IFileStream backed by your storage layer.
|
||||
throw new NotImplementedException();
|
||||
}
|
||||
|
||||
// Implement the remaining IFileStreamProvider members.
|
||||
}
|
||||
```
|
||||
|
||||
Then pass the provider to the factory:
|
||||
|
||||
```csharp
|
||||
var provider = new MyFileStreamProvider();
|
||||
|
||||
using var zoneTree = new ZoneTreeFactory<int, string>(provider)
|
||||
.SetDataDirectory("tenant-a")
|
||||
.OpenOrCreate();
|
||||
```
|
||||
|
||||
Focus first on the operations ZoneTree uses heavily:
|
||||
|
||||
* sequential WAL append and read,
|
||||
* random-access segment reads,
|
||||
* `SetLength`,
|
||||
* `Flush(bool)`,
|
||||
* `Replace`,
|
||||
* directory creation and recursive deletion,
|
||||
* small metadata reads through `ReadAllBytes` and `ReadAllText`.
|
||||
|
||||
The provider's durability model should be clear. If replacement or flush is not durable, document that and use the provider only for rebuildable workloads.
|
||||
@@ -0,0 +1,131 @@
|
||||
---
|
||||
title: File Stream Providers
|
||||
description: Choose local or RAM-backed files, reopen an in-memory tree correctly, and understand the stream and durability contracts required of custom providers.
|
||||
---
|
||||
|
||||
<SectionHeader class="zt-guide-intro" eyebrow="STORAGE / BACKING STORE" title="Choose where ZoneTree's files live" titleTag="h1" titleId="file-stream-providers-title" subtitle="The storage engine works through file and stream abstractions. Keep the same ordered API while choosing local persistence, RAM-backed files, or an application-specific provider."/>
|
||||
|
||||
<Grid columns="1" columnsMd="2" class="gap-3 zt-guide-outcomes">
|
||||
<Panel tone="accent" variant="surface" class="p-0" bodyClass="p-4"><strong>Local files</strong><p>The default provider uses the operating system's file APIs. Recovery also depends on WAL policy and the backing storage.</p></Panel>
|
||||
<Panel class="p-0" bodyClass="p-4"><strong>RAM-backed files</strong><p>The in-memory provider keeps the storage pipeline, but every file belongs to one volatile provider instance.</p></Panel>
|
||||
</Grid>
|
||||
|
||||
## Know what the provider controls
|
||||
|
||||
`IFileStreamProvider` supplies file operations and creates `IFileStream` objects. With the default storage components, it backs WALs, metadata, disk segments, and temporary replacement files.
|
||||
|
||||
Pass the provider to the `ZoneTreeFactory` constructor. `SetDataDirectory()` chooses the segment and default WAL location within that provider; it does not force files onto the local filesystem. `SetWriteAheadLogDirectory()` can select a separate WAL location.
|
||||
|
||||
| Choice | Backing store | Lifetime | Typical use |
|
||||
| --- | --- | --- | --- |
|
||||
| Default / `LocalFileStreamProvider` | Local filesystem. | Files outlive the tree and process when retained by the underlying storage. | Persistent application data. |
|
||||
| `InMemoryFileStreamProvider` | Process memory. | Files remain while the same provider instance retains them. | Tests, temporary indexes, replayable jobs. |
|
||||
| Custom `IFileStreamProvider` | Defined by the application. | Defined by its persistence and replacement semantics. | Specialized storage or fault-injection harnesses. |
|
||||
|
||||
## Use the default local provider
|
||||
|
||||
No provider argument is needed for local files. This complete program creates a tree in a chosen directory:
|
||||
|
||||
```csharp
|
||||
using ZoneTree;
|
||||
|
||||
using var tree = new ZoneTreeFactory<int, string>()
|
||||
.SetDataDirectory("data/local-storage")
|
||||
.OpenOrCreate();
|
||||
using var maintainer = tree.CreateMaintainer();
|
||||
|
||||
tree.Upsert(42, "Ada");
|
||||
|
||||
if (tree.TryGet(42, out var name))
|
||||
Console.WriteLine(name);
|
||||
```
|
||||
|
||||
The default provider delegates to local file and stream APIs. Whether an acknowledged write survives a particular failure also depends on the [WAL mode](../durability/wal-modes.md), flush behavior, and storage device. A local path alone does not define the recovery guarantee.
|
||||
|
||||
## Use RAM-backed files deliberately
|
||||
|
||||
`InMemoryFileStreamProvider` represents the same stream-based storage flow in RAM. A merge still produces serialized segment files; those bytes consume process memory instead of local disk space. The provider is not an instruction to keep all records in the mutable segment.
|
||||
|
||||
For a replayable job, disabling WAL avoids maintaining an additional volatile log. If you want to close and reopen that tree through the same provider, first finish merging the records you need to keep. The following complete program demonstrates that controlled boundary:
|
||||
|
||||
```csharp
|
||||
using ZoneTree;
|
||||
using ZoneTree.AbstractFileStream;
|
||||
using ZoneTree.Options;
|
||||
|
||||
var provider = new InMemoryFileStreamProvider();
|
||||
|
||||
ZoneTreeFactory<long, string> CreateFactory() =>
|
||||
new ZoneTreeFactory<long, string>(provider)
|
||||
.SetDataDirectory("transfer-job")
|
||||
.ConfigureWriteAheadLogOptions(options =>
|
||||
options.WriteAheadLogMode = WriteAheadLogMode.None);
|
||||
|
||||
using (var tree = CreateFactory().OpenOrCreate())
|
||||
using (var maintainer = tree.CreateMaintainer())
|
||||
{
|
||||
tree.Upsert(42, "Ready for the next stage");
|
||||
|
||||
// No concurrent writers: finish serializing the records before closing.
|
||||
maintainer.EvictToDisk();
|
||||
await maintainer.WaitForBackgroundThreadsAsync();
|
||||
}
|
||||
|
||||
using var reopened = CreateFactory().Open();
|
||||
|
||||
if (!reopened.TryGet(42, out var value))
|
||||
throw new InvalidOperationException("Expected the merged record.");
|
||||
|
||||
Console.WriteLine(value);
|
||||
```
|
||||
|
||||
Creating another factory is fine; it must receive the **same provider instance** and directory to see those files. A new `InMemoryFileStreamProvider` starts empty.
|
||||
|
||||
<AlertBox class="zt-guide-callout" tone="accent" title="WAL disabled means no replay of unmerged writes" icon="tabler:shield-check">
|
||||
<p>With <code>WriteAheadLogMode.None</code>, disposing a tree does not automatically merge its mutable records into segment files. Those unmerged writes cannot be recovered on reopen. Explicit eviction above preserves them only inside the surviving provider; it provides no protection against process loss.</p>
|
||||
</AlertBox>
|
||||
|
||||
## Decide whether an in-memory WAL is useful
|
||||
|
||||
| Goal | WAL choice |
|
||||
| --- | --- |
|
||||
| Run a disposable job whose input can be replayed. | `None` can avoid volatile log work. Keep the tree open for the job, or merge at an explicitly coordinated handoff. |
|
||||
| Exercise WAL creation, replay, or recovery behavior in tests. | Keep the relevant WAL mode enabled and retain the provider instance between opens. |
|
||||
| Preserve data after process termination or machine failure. | Use a durable backing provider and choose an appropriate WAL mode. RAM-backed WAL files cannot provide this. |
|
||||
|
||||
An in-memory provider with WAL enabled can recover retained log files when a tree is reopened in that process. Losing the provider loses both the log and its segment files.
|
||||
|
||||
## Respect the in-memory provider's boundaries
|
||||
|
||||
The built-in RAM-backed provider is optimized for ZoneTree's stream usage:
|
||||
|
||||
- Large files use chunked backing storage; `ReadAllBytes()` still needs a single array and can fail for very large files.
|
||||
- `Replace()` hands off the backing file object instead of copying the whole payload.
|
||||
- `FileShare` is intentionally ignored. It is not a filesystem locking simulator.
|
||||
- Open streams can retain an old file object after deletion or replacement.
|
||||
- A stream's mutable `Position` is not safe for unsynchronized concurrent use.
|
||||
|
||||
This makes it useful for storage-pipeline tests, but not a substitute for testing filesystem sharing, flush durability, or failure recovery on your deployment's actual storage.
|
||||
|
||||
## Implement the full custom-provider contract
|
||||
|
||||
Use `IFileStreamProvider` when neither built-in provider fits. Return `IFileStream` objects and implement the operations used by metadata, logs, and random-access segments—not just sequential append.
|
||||
|
||||
| Area | Required behavior to design and test |
|
||||
| --- | --- |
|
||||
| Stream access | Correct modes, access checks, positioning, seeking, partial reads, writes, truncation, length, and stream conversion. |
|
||||
| File lifecycle | Existence, directory creation, deletion, replacement, and any backup destination. Preserve the visibility and failure behavior expected by the engine. |
|
||||
| Metadata | Small-file text/byte reads, path combination, directory enumeration, and durable-file-writer integration. |
|
||||
| Legacy WAL migration | File enumeration through `GetFiles()` and non-overwriting renames through `MoveFile()`. Their interface defaults throw if not implemented. |
|
||||
| Durability | Explicit semantics for `Flush()`, `Flush(bool)`, replacement, and interrupted operations. A method named “flush” does not establish a storage guarantee by itself. |
|
||||
| Concurrency and failures | Coordinate access and surface failures consistently. Test interrupted appends, replacements, reopening, and merges. |
|
||||
|
||||
A remote object store usually needs a buffering and consistency design to satisfy seekable, mutable stream behavior. Implementing an upload method alone is not sufficient.
|
||||
|
||||
Advanced integrations can replace the higher-level components with `SetWriteAheadLogProvider()` or `SetRandomAccessDeviceManager()`. Those components then own their storage behavior; constructor injection describes the factory's default pipeline.
|
||||
|
||||
<Panel class="p-0 zt-guide-next-panel" bodyClass="p-5">
|
||||
<strong>Separate backing storage from recovery policy</strong>
|
||||
<p>Choose where files live here. Then decide which failure scenarios the WAL must cover and how backups capture the whole tree.</p>
|
||||
<Flex wrap="true" class="gap-3"><BtnLink href="/docs/durability/wal-modes/" tone="accent" icon="tabler:arrow-right" iconPosition="end">WAL modes</BtnLink><BtnLink href="/docs/storage/memory-usage/" tone="neutral" variant="outline">Memory usage</BtnLink></Flex>
|
||||
</Panel>
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
title: Storage
|
||||
description: Understand ZoneTree's physical files, backing providers, memory ownership, and independent WAL and disk compression settings.
|
||||
nav:
|
||||
icon: iconoir:database
|
||||
layout:
|
||||
fullWidth: true
|
||||
showToc: false
|
||||
---
|
||||
|
||||
<div class="zt-docs-overview">
|
||||
<SectionHeader class="zt-guide-intro" eyebrow="STORAGE / FROM RECORDS TO RESOURCES" title="Understand where your data lives and what it costs" titleTag="h1" titleId="storage-title" subtitle="Connect the logical tree to its physical files, backing store, and memory footprint. Then choose compression and cache settings with a clear view of their scope."/>
|
||||
|
||||
<Grid columns="1" columnsMd="3" class="gap-3 zt-guide-outcomes">
|
||||
<Panel class="p-0" bodyClass="p-4"><strong>Representation</strong><p>Sorted records become block-addressable files, indexes, and reusable parts.</p></Panel>
|
||||
<Panel tone="accent" variant="surface" class="p-0" bodyClass="p-4"><strong>Ownership</strong><p>Providers own file bytes. Segments, caches, and readers retain working memory.</p></Panel>
|
||||
<Panel class="p-0" bodyClass="p-4"><strong>Tradeoffs</strong><p>Compression and retention exchange CPU, I/O, storage space, and RAM.</p></Panel>
|
||||
</Grid>
|
||||
|
||||
<Grid columns="1" columnsMd="2" class="gap-4 zt-guide-next zt-docs-paths">
|
||||
<DocCard number="01 / FILE LAYOUT" title="Disk segment files" summary="Recognize data, headers, sparse indexes, and multipart descriptors. Understand reuse, inspection, and file lifetime." href="/docs/storage/disk-segments/" icon="tabler:database" label="Read the physical layout"/>
|
||||
<DocCard number="02 / BACKING STORE" title="File stream providers" summary="Choose local or RAM-backed files, reopen an in-memory tree correctly, and define custom storage contracts." href="/docs/storage/file-stream-providers/" icon="tabler:stack-2" label="Choose where files live"/>
|
||||
<DocCard number="03 / MEMORY BUDGET" title="Memory usage" summary="Account for mutable records, merge backlogs, sampled values, block caches, and iterator lifetimes." href="/docs/storage/memory-usage/" icon="tabler:settings" label="Budget retained memory"/>
|
||||
<DocCard number="04 / CPU AND I/O" title="Compression" summary="Configure WAL and disk profiles independently and measure block size, cache costs, and workload-specific results." href="/docs/storage/compression/" icon="tabler:arrows-sort" label="Compare compression choices"/>
|
||||
</Grid>
|
||||
|
||||
<Panel tone="accent" variant="surface" class="p-0 zt-guide-next-panel" bodyClass="p-5">
|
||||
<strong>Start with the resource you need to understand</strong>
|
||||
<p>Inspecting a data directory? Begin with file layout. Choosing a deployment's backing store? Read providers and WAL modes together. Investigating RAM or read latency? Start with memory ownership, then compression.</p>
|
||||
<Flex wrap="true" class="gap-3"><BtnLink href="/docs/concepts/lsm-tree/" tone="accent" icon="tabler:arrow-right" iconPosition="end">Review the architecture</BtnLink><BtnLink href="/docs/durability/wal-modes/" tone="neutral" variant="outline">Choose a WAL mode</BtnLink></Flex>
|
||||
</Panel>
|
||||
</div>
|
||||
@@ -1,101 +0,0 @@
|
||||
# Memory Usage
|
||||
|
||||
ZoneTree does not load the entire database into RAM. Its live memory is shaped
|
||||
by the active write set, pending maintenance, read caches, iterator lifetimes,
|
||||
and the application's key/value objects.
|
||||
|
||||
## Main Consumers
|
||||
|
||||
* active mutable-segment B+Tree and Bloom filter,
|
||||
* frozen in-memory segments waiting for merge,
|
||||
* sparse disk indexes,
|
||||
* decompressed disk blocks,
|
||||
* materialized 16-entry key/value chunks,
|
||||
* circular per-record key/value caches,
|
||||
* thread-local sequential search-hint buffers,
|
||||
* iterator prefetch buffers and segment leases,
|
||||
* merge, compression, and WAL buffers,
|
||||
* user key/value objects retained by in-memory segments.
|
||||
|
||||
## Mutable Segment
|
||||
|
||||
`MutableSegmentMaxItemCount` defaults to `1_000_000`. It is a record limit, not
|
||||
a byte budget. A million compact structs and a million documents have very
|
||||
different memory footprints.
|
||||
|
||||
```csharp
|
||||
using var zoneTree = new ZoneTreeFactory<int, string>()
|
||||
.SetMutableSegmentMaxItemCount(100_000)
|
||||
.OpenOrCreate();
|
||||
```
|
||||
|
||||
The Bloom filter requests `MutableSegmentBloomFilterBitsPerItem` times the
|
||||
configured mutable-segment capacity, rounds the result up to a power of two,
|
||||
and caps it at `2^30` bits. The default request is `8` bits per item. The filter
|
||||
uses a managed `long[]`, so include array overhead and power-of-two rounding in
|
||||
capacity estimates.
|
||||
|
||||
Read-only in-memory segments retain their completed Bloom filters and records
|
||||
until maintenance merges them. A backlog multiplies both record and filter
|
||||
memory.
|
||||
|
||||
## Disk Read Memory
|
||||
|
||||
The decompressed-block cache is usually the largest read-side consumer. A
|
||||
cached block retains its decompressed bytes and any materialized-entry cache
|
||||
attached to it.
|
||||
|
||||
`MaterializedEntryCacheSize` defaults to 4096 chunk slots per block. Each
|
||||
published chunk owns arrays for 16 keys and 16 values. The maximum number of
|
||||
resident chunks is bounded, but actual byte cost depends strongly on `TKey`,
|
||||
`TValue`, block contents, collisions, and how many positions have been read.
|
||||
|
||||
Circular key and value caches are bounded per disk segment. Search-hint buffers
|
||||
are allocated lazily per disk segment and calling thread, so many threads
|
||||
touching many segments can create a larger footprint than the prefetch number
|
||||
alone suggests.
|
||||
|
||||
See [read-path caching](../tuning/read-path-caching.md) for ownership and disable options.
|
||||
|
||||
## Iterators
|
||||
|
||||
Each low-level disk iterator can allocate prefetch arrays according to
|
||||
`IteratorOptions.DiskSegmentPrefetchSize`. A ZoneTree iterator may merge several
|
||||
segment iterators, so total buffer memory depends on the number of active
|
||||
segments as well as the configured prefetch size.
|
||||
|
||||
Iterators also lease the segments they scan. A long-lived iterator can keep a
|
||||
retired segment and its caches reachable until disposal.
|
||||
|
||||
## Maintainer Cleanup
|
||||
|
||||
A maintainer automates merge and inactive-cache cleanup work according to
|
||||
configurable policies. For custom control, `zoneTree.Maintenance` exposes
|
||||
maintenance state, events, and operations.
|
||||
|
||||
```csharp
|
||||
using var maintainer = zoneTree.CreateMaintainer();
|
||||
|
||||
maintainer.BlockCacheLifeTime = TimeSpan.FromMinutes(1);
|
||||
maintainer.InactiveBlockCacheCleanupInterval = TimeSpan.FromSeconds(30);
|
||||
```
|
||||
|
||||
Shorter block lifetime lowers retained read memory but can increase I/O and
|
||||
decompression. It does not override a block currently pinned by a read.
|
||||
|
||||
## .NET Process Memory
|
||||
|
||||
The .NET runtime may keep previously used memory reserved for reuse. Process
|
||||
working set and peak memory therefore do not directly equal live ZoneTree
|
||||
objects. Use managed heap diagnostics, allocation profiles, and retained-object
|
||||
analysis together with process counters.
|
||||
|
||||
## Capacity Checklist
|
||||
|
||||
* Estimate bytes per mutable record, not only record count.
|
||||
* Include every simultaneously frozen segment during a maintenance backlog.
|
||||
* Include Bloom-filter power-of-two rounding.
|
||||
* Multiply disk caches by the number of active disk segments and hot blocks.
|
||||
* Multiply search hints by active threads and touched disk segments.
|
||||
* Multiply iterator buffers by active iterators and their segment fan-in.
|
||||
* Measure after warmup and after inactive-cache cleanup.
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
title: Memory Usage
|
||||
description: Budget ZoneTree memory across mutable records, merge backlogs, sparse indexes, disk caches, iterators, and storage providers, and configure cleanup correctly.
|
||||
---
|
||||
|
||||
<SectionHeader class="zt-guide-intro" eyebrow="STORAGE / MEMORY BUDGET" title="Account for the memory around your records" titleTag="h1" titleId="memory-usage-title" subtitle="With local storage, ZoneTree can query a database larger than RAM. Its live footprint still includes the write set, pending merges, sampled indexes, read caches, and the objects your application retains."/>
|
||||
|
||||
<Grid columns="1" columnsMd="3" class="gap-3 zt-guide-outcomes">
|
||||
<Panel tone="accent" variant="surface" class="p-0" bodyClass="p-4"><strong>Write memory</strong><p>Record size and the number of frozen segments determine how much data awaits a merge.</p></Panel>
|
||||
<Panel class="p-0" bodyClass="p-4"><strong>Read memory</strong><p>Hot blocks, materialized objects, sampled records, and calling threads multiply cache costs.</p></Panel>
|
||||
<Panel class="p-0" bodyClass="p-4"><strong>Lifetime</strong><p>Maintenance and prompt iterator disposal let inactive buffers and retired segments be reclaimed.</p></Panel>
|
||||
</Grid>
|
||||
|
||||
## Build a budget by owner
|
||||
|
||||
Start with the allocations that stay reachable, then allow headroom for concurrent work and temporary allocations. There is no single setting here that caps the tree's total bytes.
|
||||
|
||||
| Owner | Retained memory | What multiplies it |
|
||||
| --- | --- | --- |
|
||||
| Mutable segment | Ordered tree nodes, keys, values, and Bloom filter. | Record size and configured record capacity. |
|
||||
| Read-only memory segments | Frozen records and their retained indexing structures. | Merge backlog and write rate. |
|
||||
| Physical disk parts | Sparse key/value samples, circular caches, thread-local search hints. | Active parts, sampled value size, and calling threads. |
|
||||
| Read block cache | Decompressed bytes and any materialized-entry chunks. | Blocks touched and retained across data/header devices and parts. |
|
||||
| Active iterators | Per-segment prefetch arrays and segment leases. | Iterator count, segment fan-in, and prefetch settings. |
|
||||
| Background work | Merge, compression, and WAL buffers or queued records. | Workload, selected modes, and concurrent operations. |
|
||||
| Application / provider | Retained results and any RAM-backed files. | Application ownership and provider choice. |
|
||||
|
||||
With `InMemoryFileStreamProvider`, add **all retained file bytes** to the process budget. Merging records into a RAM-backed “disk” segment changes representation; it does not move their storage outside process memory.
|
||||
|
||||
## Size the active write set
|
||||
|
||||
`MutableSegmentMaxItemCount` defaults to `1_000_000`. This limits records in the active mutable segment, not bytes in the whole tree. A million small structs and a million document objects have very different costs.
|
||||
|
||||
Moving a full mutable segment forward freezes it into the read-only layer. That memory remains reachable until merging and reader lifetimes permit its release. Lowering the mutable limit rotates segments sooner; it cannot compensate for a merge backlog that grows faster than storage can drain it.
|
||||
|
||||
Use `ReadOnlySegmentsCount`, `ReadOnlySegmentsRecordCount`, and `InMemoryRecordCount` to track that backlog. These count physical records, potentially including older versions and deletion markers; they are not byte measurements or live-key counts.
|
||||
|
||||
### Include Bloom-filter rounding
|
||||
|
||||
When enabled, the Bloom filter requests capacity × `MutableSegmentBloomFilterBitsPerItem`, rounds up to a power of two, and bounds the bit array between 64 and `2^30` bits. The default is 8 requested bits per item; setting the option to `0` disables the filter.
|
||||
|
||||
For the default one-million-record capacity:
|
||||
|
||||
```text
|
||||
requested bits = 1,000,000 × 8 = 8,000,000
|
||||
allocated bits = 8,388,608
|
||||
bit storage = 1,048,576 bytes = 1 MiB, plus object/array overhead
|
||||
```
|
||||
|
||||
Newly frozen mutable segments retain their filters alongside their records. Include those allocations for every simultaneously retained generation. See [Bloom filters](../concepts/bloom-filters.mdx) for filtering behavior and recovery differences.
|
||||
|
||||
## Count disk caches at the right scope
|
||||
|
||||
These defaults are capacities or retention controls, not a shared byte allowance:
|
||||
|
||||
| Control | Default | Allocation scope |
|
||||
| --- | --- | --- |
|
||||
| `CompressionBlockSize` | `4 MiB` | Disk block unit. A cold access can retain decompressed block bytes; a short final block can be smaller. |
|
||||
| `DefaultSparseArrayStepSize` | `1024` | Sampled key/value/index entries for each physical part. A smaller step retains more samples. |
|
||||
| `KeyCacheSize` / `ValueCacheSize` | `1024` each | Circular record-cache slots per physical single-part segment. |
|
||||
| Key / value cache record lifetime | `10 seconds` each | Expired circular-cache records become eligible for cleanup. |
|
||||
| `MaterializedEntryCacheSize` | `4096` | Chunk slots per decompressed block when that cache is created. Each published chunk has arrays for 16 keys and 16 values. |
|
||||
| `SearchHintPrefetchSize` | `16` | Lazily allocated prefetch buffers per physical segment and calling thread. |
|
||||
|
||||
A multipart logical segment contains multiple physical parts, each with its own applicable caches. Counting only logical segment objects underestimates the allocation scopes.
|
||||
|
||||
Materialized-entry caching retains deserialized objects in addition to block bytes. The slot count bounds the chunks held by that cache, but it is not a byte bound: key/value types, referenced objects, filled chunks, and replacement allocations determine the cost. Setting `MaterializedEntryCacheSize` to `0` disables that cache; setting `SearchHintPrefetchSize` to `0` disables search-hint prefetch.
|
||||
|
||||
For a first block-byte estimate, 100 fully populated 4 MiB cached blocks account for about 400 MiB **before** materialized objects, indexes, other caches, and temporary buffers. Use this to check the scale of a measurement, not as a total-memory prediction.
|
||||
|
||||
Read [compression](compression.mdx) for block-size tradeoffs and [read-path caching](../tuning/read-path-caching.md) for individual cache controls.
|
||||
|
||||
## Give iterators a short lifetime
|
||||
|
||||
`IteratorOptions.DiskSegmentPrefetchSize` controls the prefetch capacity of low-level disk iterators. A top-level scan can merge several segment iterators, so multiply buffers by segment fan-in as well as by concurrent scans.
|
||||
|
||||
Iterators also lease the segments they read. A long-lived scan can keep retired segments, associated objects, and old files alive after a merge. Dispose an iterator when its scan finishes; do not keep one idle as an application-wide cursor. See [iteration and range scans](../usage/iteration-and-range-scans.mdx).
|
||||
|
||||
## Configure retention and observe the write set
|
||||
|
||||
This complete example uses a smaller mutable capacity and materialized cache as an explicit starting profile. The values illustrate the controls; measure them against your workload before adopting them.
|
||||
|
||||
```csharp
|
||||
using ZoneTree;
|
||||
|
||||
using var tree = new ZoneTreeFactory<int, string>()
|
||||
.SetDataDirectory("data/memory-budget")
|
||||
.SetMutableSegmentMaxItemCount(100_000)
|
||||
.ConfigureDiskSegmentOptions(options =>
|
||||
{
|
||||
options.MaterializedEntryCacheSize = 1024;
|
||||
options.SearchHintPrefetchSize = 8;
|
||||
})
|
||||
.OpenOrCreate();
|
||||
using var maintainer = tree.CreateMaintainer();
|
||||
|
||||
// Restart the cleanup job so its timer uses the new interval.
|
||||
maintainer.EnableJobForCleaningInactiveCaches = false;
|
||||
maintainer.BlockCacheLifeTime = TimeSpan.FromSeconds(30);
|
||||
maintainer.InactiveBlockCacheCleanupInterval = TimeSpan.FromSeconds(10);
|
||||
maintainer.EnableJobForCleaningInactiveCaches = true;
|
||||
|
||||
for (var key = 0; key < 1000; ++key)
|
||||
tree.Upsert(key, $"Record {key}");
|
||||
|
||||
Console.WriteLine($"Mutable: {tree.Maintenance.MutableSegmentRecordCount}");
|
||||
Console.WriteLine($"Frozen: {tree.Maintenance.ReadOnlySegmentsRecordCount}");
|
||||
Console.WriteLine($"In memory: {tree.Maintenance.InMemoryRecordCount}");
|
||||
```
|
||||
|
||||
The standard maintainer starts cleanup automatically with a one-minute block lifetime and a 30-second interval. It reads `BlockCacheLifeTime` on each pass, but captures `InactiveBlockCacheCleanupInterval` when creating its timer. Changing the interval requires the stop/configure/start sequence above.
|
||||
|
||||
<AlertBox class="zt-guide-callout" tone="info" title="Cleanup is a retention policy, not a hard RAM limit" icon="tabler:settings">
|
||||
<p>Shorter retention can reduce inactive read buffers while increasing later I/O and decompression. Buffers still referenced by a reader remain reachable even if removed from the shared cache. Cleanup does not release application-owned objects or force the .NET runtime to return memory to the operating system.</p>
|
||||
</AlertBox>
|
||||
|
||||
## Measure steady state and peaks separately
|
||||
|
||||
Run a representative write/read mix, warm its caches, and observe both sustained operation and merge peaks. Then stop work, dispose scans, allow maintenance and cleanup to finish, and compare retained objects.
|
||||
|
||||
| Observation | Investigate first |
|
||||
| --- | --- |
|
||||
| Frozen record count keeps increasing. | Merge throughput, storage latency, and maintainer policy. |
|
||||
| Memory rises during reads with a stable write set. | Hot block count, materialized values, sparse indexes, threads, and active iterators. |
|
||||
| Memory peaks during merges, then stabilizes. | Temporary merge/compression allocations and overlapping old/new segment lifetimes. |
|
||||
| Live managed objects fall but process memory stays high. | GC heap reservation, native allocations, and operating-system working-set behavior. |
|
||||
| RAM-backed files keep growing after merges. | Retained provider files and reader leases, not just mutable record counts. |
|
||||
|
||||
Use managed heap and allocation profiles alongside process counters. Working set, peak working set, and `GC.GetTotalMemory()` answer different questions; none is a precise ZoneTree ownership report by itself.
|
||||
|
||||
<Panel class="p-0 zt-guide-next-panel" bodyClass="p-5">
|
||||
<strong>Tune the owner you actually measured</strong>
|
||||
<p>Start with backlog and cache ownership. Then adjust maintenance, block sizes, or cache capacities one variable at a time.</p>
|
||||
<Flex wrap="true" class="gap-3"><BtnLink href="/docs/tuning/read-path-caching/" tone="accent" icon="tabler:arrow-right" iconPosition="end">Read-path caching</BtnLink><BtnLink href="/docs/usage/maintenance/" tone="neutral" variant="outline">Maintenance</BtnLink><BtnLink href="/docs/operations/diagnostics/" tone="neutral" variant="outline">Diagnostics</BtnLink></Flex>
|
||||
</Panel>
|
||||
@@ -113,5 +113,5 @@ using var zoneTree = new ZoneTreeFactory<long, string>()
|
||||
```
|
||||
|
||||
See [read-path caching](read-path-caching.md),
|
||||
[memory usage](../storage/memory-usage.md), and
|
||||
[memory usage](../storage/memory-usage.mdx), and
|
||||
[write amplification](write-amplification.md).
|
||||
|
||||
@@ -107,7 +107,7 @@ using var zoneTree = new ZoneTreeFactory<int, string>()
|
||||
|
||||
Benchmark with representative values.
|
||||
|
||||
See [compression](../storage/compression.md) and [read-path caching](read-path-caching.md).
|
||||
See [compression](../storage/compression.mdx) and [read-path caching](read-path-caching.md).
|
||||
|
||||
## WAL Cost
|
||||
|
||||
|
||||
@@ -143,7 +143,7 @@ The cleanup interval is captured when the periodic timer is created; assigning t
|
||||
|
||||
Longer retention can help repeated reads and also retain more memory. Cleanup releases inactive block buffers and expired circular key/value cache entries. The .NET runtime may retain released memory for reuse rather than immediately returning it to the operating system.
|
||||
|
||||
See [memory usage](../storage/memory-usage.md) and [read-path caching](../tuning/read-path-caching.md) when interpreting process memory or changing the cache policy.
|
||||
See [memory usage](../storage/memory-usage.mdx) and [read-path caching](../tuning/read-path-caching.md) when interpreting process memory or changing the cache policy.
|
||||
|
||||
## Observe progress and failures
|
||||
|
||||
|
||||
@@ -31,8 +31,9 @@ function registerGuideTypography({ root, colors }: GuideStyleContext) {
|
||||
root.select('.zt-guide-callout').marginBlock('28px').fontSize('14px').lineHeight('1.8')
|
||||
root.select('.zt-guide-callout p').margin('0')
|
||||
root.select('.zt-guide-callout .flex').minWidth('0')
|
||||
root.select('.doc-content:has(.zt-guide-intro) table').fontSize('13px').lineHeight('1.7')
|
||||
root.select('.doc-content:has(.zt-guide-intro) table').width('100%').fontSize('13px').lineHeight('1.7')
|
||||
root.select('.doc-content:has(.zt-guide-intro) :is(th, td)').verticalAlign('top')
|
||||
.minWidth('8rem').whiteSpace('normal')
|
||||
root.select('.doc-content:has(.zt-guide-intro) td code').fontSize('12px').overflowWrap('anywhere').whiteSpace('normal')
|
||||
root.media('(max-width: 600px)').select('.zt-guide-intro > .text-tagline').fontSize('16px')
|
||||
root.media('(max-width: 600px)').select('.doc-content:has(.zt-guide-intro) > h2').fontSize('22px')
|
||||
|
||||
Reference in New Issue
Block a user