Merge pull request #150 from ZoneTree/improve-docs-for-wal-mode-tradeoffs

Clarifies WAL mode tradeoffs in docs and comments.

README.md

@@ -252,9 +252,9 @@ ZoneTree uses async compressed WAL by default. It is the recommended starting po
 
 | Need | Consider |
 | --- | --- |
-| Default safe high-throughput WAL mode | Async compressed WAL |
-| Synchronous compressed WAL acknowledgment | Sync compressed WAL |
-| Simplest synchronous WAL path | Sync WAL |
+| Recommended high-throughput persistent WAL mode | Async compressed WAL |
+| Plain synchronous WAL write path | Sync WAL |
+| Compressed WAL with a synchronous caller write path and tail durability tradeoff | Sync compressed WAL |
 | Intentional no-WAL boundary for cache/temp/rebuildable data | No WAL |
 
 Disk segments can also use compression and different segment layouts. These options help tune disk space, read patterns, merge behavior, and file-size boundaries.

docs/durability/recovery.md

@@ -99,7 +99,8 @@ These APIs are low-level. Recovery tools should load the persisted metadata or p
 For production systems:
 
 * keep the default async compressed WAL unless you have a specific reason to change it,
-* use sync WAL modes only when synchronous WAL acknowledgment is required,
+* use Sync WAL when the application specifically needs the plain synchronous WAL path,
+* use sync-compressed WAL only when its compressed-block tail durability tradeoff is acceptable,
 * use `No WAL` only for cache, temporary, or intentionally rebuildable data,
 * keep maintenance running for long-lived write-heavy workloads,
 * dispose ZoneTree instances during graceful shutdown,

docs/durability/wal-modes.md

@@ -10,22 +10,22 @@ Compression method, compression level, and block-size details are covered in [co
 
 | Need | Consider |
 | --- | --- |
-| Default safe high-throughput WAL mode | Async compressed WAL |
-| Synchronous compressed WAL acknowledgment | Sync compressed WAL |
-| Simplest synchronous WAL path | Sync WAL |
+| Recommended high-throughput persistent WAL mode | Async compressed WAL |
+| Plain synchronous WAL write path | Sync WAL |
+| Compressed WAL with a synchronous caller write path and tail durability tradeoff | Sync compressed WAL |
 | Intentional no-WAL boundary for cache/temp/rebuildable data | No WAL |
 
 ## Sync WAL
 
-Sync WAL writes records directly to the log path before the write is considered complete.
+Sync WAL writes records through the plain WAL stream before the write is considered complete. It favors simple synchronous WAL recovery semantics over write throughput.
 
-Use it when you want the simplest synchronous WAL path and can accept lower throughput.
+Use it when the application specifically needs the plain synchronous WAL path and can accept lower throughput. Full power-loss durability still depends on the operating system, file system, and storage device.
 
 ## Sync Compressed WAL
 
-Sync compressed WAL stores log records in compressed form. It balances durability and smaller WAL files with compression overhead.
+Sync compressed WAL stores log records in a compressed-block WAL format. The current tail block is stored separately and can be written by the tail writer job.
 
-Use it when WAL size matters and you want synchronous WAL acknowledgment.
+Use it when WAL size or write throughput matters and you can accept weaker crash durability than Sync WAL.
 
 In sync-compressed mode, the tail writer job is enabled by default and runs every `500 ms`.
 
@@ -43,6 +43,6 @@ Use it for caches, rebuildable indexes, temporary stores, and data that can be r
 
 ## Durability Boundary
 
-ZoneTree's WAL protects against process-level failures according to the selected mode and flush behavior. Hardware, operating system, and storage-device behavior still matter for full power-loss guarantees.
+ZoneTree's WAL protects against process-level failures according to the selected mode and flush behavior. Hardware, operating system, file-system, and storage-device behavior still matter for full power-loss guarantees.
 
-Choose `No WAL` only when the data-loss boundary is intentional. For most persistent data, start with the default async compressed WAL and move to a sync mode only when the application specifically needs synchronous WAL acknowledgment.
+Choose `No WAL` only when the data-loss boundary is intentional. For most persistent data, start with the default async compressed WAL and move to Sync WAL only when the application specifically needs the plain synchronous WAL path.

docs/operations/production-checklist.md

@@ -7,7 +7,8 @@ Use this checklist before putting ZoneTree behind production traffic.
 * Choose the data directory intentionally.
 * Ensure the storage volume has enough free space.
 * Start with the default async compressed WAL for persistent data.
-* Use sync WAL modes only when synchronous WAL acknowledgment is required.
+* Use Sync WAL when the application specifically needs the plain synchronous WAL path.
+* Use sync-compressed WAL only when its compressed-block tail durability tradeoff is acceptable.
 * Use `No WAL` only for cache, temporary, or intentionally rebuildable data.
 * Choose disk segment mode based on database size and operational file-size needs.
 * Tune sparse arrays, block cache lifetime, and circular key/value caches only after measuring read behavior.

docs/reference/configuration.md

@@ -139,7 +139,7 @@ Important WAL options:
 | `AsyncCompressedModeOptions` | async writer polling behavior |
 | `EnableIncrementalBackup` | preserves WAL content during WAL replacement/compaction |
 
-Use sync modes when the application specifically needs synchronous WAL acknowledgment. Use `No WAL` only for cache, temporary, or intentionally rebuildable data.
+Use the default async compressed WAL as the starting point for most persistent data. Use Sync WAL when the application specifically needs the plain synchronous WAL path. Use sync-compressed WAL when the compressed WAL tradeoff is acceptable. Use `No WAL` only for cache, temporary, or intentionally rebuildable data.
 
 The default WAL profile uses async compressed WAL, `256 KB` compression blocks, Zstd level `0` compression, and a `100 ms` async empty-queue poll interval.
 

docs/storage/compression.md

@@ -33,7 +33,7 @@ block:  256 KB
 
 Compressed WALs reduce log size and can improve write-path IO when data compresses well. They also add compression work. The async compressed mode is the default because it keeps WAL protection enabled while preserving high write throughput for most applications.
 
-Sync compressed WAL uses the same compressed WAL file shape with synchronous acknowledgment behavior. Use it when the application specifically needs synchronous WAL acknowledgment and wants compressed WAL storage.
+Sync compressed WAL uses the same compressed-block WAL file shape and tail behavior. Use Sync WAL when the application specifically needs the plain synchronous WAL path; use sync-compressed WAL when compressed WAL storage or throughput matters and its tail durability tradeoff is acceptable.
 
 ## Disk Segment Compression
 

docs/usage/atomic-operations.md

@@ -1,40 +1,126 @@
 # Atomic Operations
 
-Atomic methods are for same-key read-modify-write operations across LSM-tree segments.
+Atomic methods are ZoneTree's single-key read-modify-write tools.
 
-They are useful for:
+Use them when the next value depends on the value that is already visible for the same key:
 
-* counters,
-* compare-and-set logic,
-* appending to a value,
-* updating an aggregate,
-* initializing a value if absent and updating if present.
+| Need | Use |
+| --- | --- |
+| Increment a counter | atomic method |
+| Append to the current value | atomic method |
+| Compare current value before replacing it | atomic method |
+| Initialize if absent, update if present | `TryAtomicAddOrUpdate` |
+| Simple insert or replace | `Upsert` |
+| Several keys must change together | transaction |
 
-They are not the default write API. Use `Upsert` for simple inserts and replacements.
+`Upsert`, `TryAdd`, `TryDelete`, and `ForceDelete` are the normal high-throughput write APIs. They are thread-safe, but they do not join the atomic-method lock. If a key's correctness depends on read-modify-write behavior, keep every write for that key on the atomic path.
 
-## Add Or Update
+## What Atomic Means
+
+ZoneTree is an LSM-tree. The newest value for a key may be in the mutable segment, a read-only in-memory segment, the disk segment, or a bottom segment until maintenance merges older layers away.
+
+Atomic methods synchronize this sequence:
+
+1. Read the current visible value across the LSM-tree layers.
+2. Decide whether to add, update, cancel, or replace.
+3. Write the new value to the mutable segment.
+
+Atomic methods are ordered with other atomic methods on the same tree. Regular writes do not participate in that ordering. That is why `AtomicUpsert` exists: it is the upsert form to use when a key also participates in atomic read-modify-write operations.
+
+## Counter
+
+```csharp
+const string key = "stats:user:42:views";
+
+zoneTree.TryAtomicAddOrUpdate(
+    key,
+    valueToAdd: 1,
+    valueUpdater: (ref long value) =>
+    {
+        value++;
+        return true;
+    });
+```
+
+If the key is absent, ZoneTree writes `1`. If it is present, the updater receives the current value and writes back the incremented value.
+
+## Compare And Set
+
+```csharp
+var found = zoneTree.TryAtomicGetAndUpdate(
+    "order:123:state",
+    out var current,
+    (ref OrderState state) =>
+    {
+        if (state != OrderState.Pending)
+            return false;
+
+        state = OrderState.Confirmed;
+        return true;
+    });
+```
+
+`found` is `false` only when the key is not found. If the key is found and the delegate returns `false`, the method still returns `true` because the read succeeded, but no new value is written.
+
+## Initialize Or Update With A Callback
 
 ```csharp
 zoneTree.TryAtomicAddOrUpdate(
-    key: 42,
+    key: "counter:global",
     valueToAdd: 1,
-    valueUpdater: (ref int value) =>
+    valueUpdater: (ref long value) =>
     {
         value++;
         return true;
     },
-    result: (in int value, long opIndex, OperationResult result) =>
+    result: (in long value, long opIndex, OperationResult result) =>
     {
         Console.WriteLine($"{result}: {value} at {opIndex}");
     });
 ```
 
-The updater decides whether the existing value should change. Returning `false` cancels the update. For `TryAtomicGetAndUpdate`, that means the key was found but no new value was written; for add-or-update APIs, cancellation is reported as `OperationResult.Cancelled`.
+The result callback reports:
 
-`ValueUpdaterDelegate<TValue>` receives a local value by `ref`, and it is designed to update that value. For mutable reference types, in-place mutation is valid when the delegate commits by returning `true`. If the delegate may cancel, decide first, then mutate or assign.
+| Result | Meaning |
+| --- | --- |
+| `Added` | The key was absent and a new value was written. |
+| `Updated` | The key existed and the updated value was written. |
+| `Cancelled` | The adder or updater returned `false`; no value was written. |
+
+When an operation is cancelled, the callback receives `opIndex` `0`.
+
+`TryAtomicAddOrUpdate` returns `true` when it adds a new key and `false` when it updates or cancels. Use the result callback when the caller needs to distinguish `Updated` from `Cancelled`.
+
+## Method Guide
+
+| Method | Behavior |
+| --- | --- |
+| `TryAtomicAdd(key, value, out opIndex)` | Adds only when the key is absent. Returns `false` and `opIndex = 0` when the key already exists. |
+| `TryAtomicUpdate(key, value, out opIndex)` | Replaces only when the key exists. Returns `false` and `opIndex = 0` when the key is absent. |
+| `AtomicUpsert(key, value)` | Adds or replaces under the atomic-method lock and returns the operation index. |
+| `TryAtomicGetAndUpdate(key, out value, updater)` | Reads the current value and lets the updater decide whether to write a replacement. Returns `false` when the key is absent. |
+| `TryAtomicAddOrUpdate(key, valueToAdd, updater, result)` | Adds `valueToAdd` when absent; otherwise updates the current value. |
+| `TryAtomicAddOrUpdate(key, adder, updater, result)` | Lets the adder create the absent value and the updater change the existing value. Either delegate can cancel by returning `false`. |
+
+All successful writes are assigned a normal operation index. Operation indexes preserve ZoneTree's producer write order and are useful for replay, replication, audit, and restore workflows.
+
+## Delegate Rules
+
+`ValueUpdaterDelegate<TValue>` and `ValueAdderDelegate<TValue>` receive a local `TValue` variable by `ref`.
 
 ```csharp
-zoneTree.TryAtomicGetAndUpdate(42, out var user, (ref User value) =>
+public delegate bool ValueUpdaterDelegate<TValue>(ref TValue value);
+public delegate bool ValueAdderDelegate<TValue>(ref TValue value);
+```
+
+Return `true` to commit the local value. Return `false` to cancel the write.
+
+Keep delegates and result callbacks short and deterministic. `TryAtomicAddOrUpdate` may retry when the mutable segment is frozen or full, so adder/updater delegates can be invoked more than once before the method finishes. Avoid external side effects inside those delegates unless repeating the side effect is acceptable.
+
+For mutable reference types, decide before mutating. Returning `false` prevents ZoneTree from writing a new value, but it cannot undo in-place changes already made to a shared object reference.
+
+```csharp
+zoneTree.TryAtomicGetAndUpdate(1, out var user, (ref UserSnapshot value) =>
 {
     if (!ShouldRename(value))
         return false;
@@ -46,22 +132,36 @@ zoneTree.TryAtomicGetAndUpdate(42, out var user, (ref User value) =>
 
 See [value mutability](../concepts/value-mutability.md).
 
-## Why Atomic Methods Exist
+## Mixing Write Modes
 
-ZoneTree is an LSM-tree. A key can exist in several layers before compaction removes older versions. Atomic methods synchronize the read-decision-write sequence so other atomic methods observe a consistent ordering.
+Mixing write modes is fine when they protect different keys or different invariants:
 
-## Mixing Atomic And Regular Writes
+* `stats:user:123` uses atomic increments,
+* `profile:user:123` uses regular `Upsert`,
+* secondary index entries use regular `Upsert` or `ForceDelete`.
 
-You can use atomic methods for specific hot keys while the rest of the tree uses `Upsert`.
+Do not mix regular writes and atomic writes for the same read-modify-write invariant:
 
-Example:
+```csharp
+// Avoid this shape for the same counter key.
+zoneTree.TryAtomicAddOrUpdate("counter", 1, (ref long value) =>
+{
+    value++;
+    return true;
+});
 
-* `stats:user:123` uses atomic increment,
-* `profile:user:123` uses normal `Upsert`,
-* index entries use normal `Upsert` or `ForceDelete`.
+zoneTree.Upsert("counter", 0);
+```
 
-Avoid mixing regular writes and atomic writes for the same invariant. If a value must be protected by atomic read-modify-write, keep all writes for that value on the atomic path.
+The `Upsert` is thread-safe, but it is not synchronized with the atomic read-decision-write sequence. If the key belongs to an atomic workflow, use `AtomicUpsert` for unconditional replacement.
 
-## When To Use Transactions Instead
+## When To Use Transactions
 
-Atomic methods coordinate one key. Use transactions when the correctness rule spans multiple keys.
+Atomic methods coordinate one key's current value. Use transactions when a correctness rule spans multiple keys:
+
+* transfer from one account key to another,
+* append an event and update a separate stream pointer,
+* update a record and several secondary indexes as one unit,
+* reserve a global sequence number and write the payload in the same commit.
+
+For one-key counters, compare-and-set logic, and initialize-or-update behavior, atomic methods are the smaller and faster tool.

src/ZoneTree/Directory.Build.props

@@ -5,8 +5,8 @@
     <Authors>Ahmed Yasin Koculu</Authors>
     <PackageId>ZoneTree</PackageId>
     <Title>ZoneTree</Title>
-    <ProductVersion>1.9.2.0</ProductVersion>
-    <Version>1.9.2.0</Version>
+    <ProductVersion>1.9.3.0</ProductVersion>
+    <Version>1.9.3.0</Version>
     <AssemblyTitle>ZoneTree</AssemblyTitle>
     <Description>ZoneTree is a persistent, high-performance, transactional, ACID-compliant ordered key-value database and LSM-tree storage engine for .NET.</Description>
     <Summary>

src/ZoneTree/Options/WriteAheadLogMode.cs

@@ -6,33 +6,33 @@ namespace ZoneTree.Options;
 public enum WriteAheadLogMode : byte
 {
   /// <summary>
-  /// Sync mode write ahead log ensures that the data is flushed to the device
-  /// immediately.
-  /// It provides maximum durability in case of a crash,
-  /// but slower write speed.
+  /// Sync mode write ahead log writes each record through the plain WAL stream
+  /// before the ZoneTree write is considered complete.
+  /// It favors simple synchronous WAL recovery semantics over write speed.
+  /// Full power-loss durability still depends on the
+  /// operating system, file system, and storage device.
   /// </summary>
   Sync = 0,
 
   /// <summary>
-  /// Sync mode with compression.
-  /// It provides faster write speed but less durability.
-  /// (crashes might cause data loss.)
+  /// Sync mode with compressed-block WAL storage.
+  /// It is faster than Sync and can reduce WAL size, but it keeps the current
+  /// compressed tail block separately and has weaker crash durability than
+  /// Sync.
   /// </summary>
   SyncCompressed = 1,
 
   /// <summary>
-  /// AsyncCompressed mode write ahead log does not directly write and flush
-  /// the log entries to the device. Entries are kept in memory and
-  /// written to the device in another thread.
-  /// AsyncCompressed mode improves the performance with the cost of reduced durability 
-  /// (crashes might cause data loss.)
-  /// AsyncCompressed mode writes to the WAL with compression enabled.
+  /// AsyncCompressed mode writes compressed WAL records through a background
+  /// path. It is the recommended high-throughput persistent default for most
+  /// applications, but recent writes can be lost if the process terminates
+  /// before the background WAL work is durable.
   /// </summary>
   AsyncCompressed = 2,
 
   /// <summary>
   /// No Write Ahead Log. Nothing is saved to the WAL file.
-  /// Every inserts stay in memory. Data in memory can disappear on 
+  /// Inserts stay in memory. Data in memory can disappear on
   /// process crashes / terminations or power cuts.
   /// It is still possible to save in memory data to the disk segments
   /// using MoveMutableSegmentForward and StartMergeOperation

src/ZoneTree/Options/ZoneTreeOptions.cs

@@ -165,16 +165,6 @@ public sealed class ZoneTreeOptions<TKey, TValue>
       MarkValueDeleted = ComponentsForKnownTypes.GetMarkValueDeleted<TValue>();
   }
 
-  static bool IsAssignableToNull(Type type)
-  {
-    if (!type.IsValueType || Nullable.GetUnderlyingType(type) != null)
-    {
-      return true;
-    }
-
-    return false;
-  }
-
   /// <summary>
   /// Disables deletion to be able to insert default values of the value type.
   /// Databases created with this option are not able to delete records.

src/ZoneTree/docs/ZoneTree/README-NUGET.md

@@ -251,9 +251,9 @@ ZoneTree uses async compressed WAL by default. It is the recommended starting po
 
 | Need | Consider |
 | --- | --- |
-| Default safe high-throughput WAL mode | Async compressed WAL |
-| Synchronous compressed WAL acknowledgment | Sync compressed WAL |
-| Simplest synchronous WAL path | Sync WAL |
+| Recommended high-throughput persistent WAL mode | Async compressed WAL |
+| Plain synchronous WAL write path | Sync WAL |
+| Compressed WAL with a synchronous caller write path and tail durability tradeoff | Sync compressed WAL |
 | Intentional no-WAL boundary for cache/temp/rebuildable data | No WAL |
 
 Disk segments can also use compression and different segment layouts. These options help tune disk space, read patterns, merge behavior, and file-size boundaries.