Implement insertion strategy for merge update in timeseries

[vasil-pashov]

Merge update - Insert

High level overview

This PR implements both strategies MergeStrategy("do_nothing", "insert") and MergeStrategy("update", "insert"). Both the source and the target must be DateTime-indexed and ordered. Insertion is stable, meaning that new values will come after values already in the target.

Implementation details

Structure by time slice

Only source rows that are not matched by anything in the target are inserted. This means that if an index value appears in more than one segment all segments containing that index value must be loaded by the processing unit. This is implemented in structure_by_time_slice. This function structures together all segments that contain the same index values. With this change one segment can appear in multiple groups. Example

[[1, 2, 3],[4, 4, 4], [4, 5, 5], [5, 5, 5], [5, 6, 7], [7, 8, 9]]

Will produce the following groups:

[[[1, 2, 3]], [[4, 4, 4], [4, 5, 5]], [[4, 5, 5], [5, 5, 5], [5, 6, 7]], [[5, 6, 7], [7, 8, 9]]]

With corresponding time ranges

[[1, 4), [4, 5), [4, 8), [7, 10)]

Note 1. that the procedure intentionally splits the groups so that when segments are chained like so [a, b] [b, c] [c, d], [d, e] not all segments are loaded in the same processing unit.
Note 2. With this approach different processing unit not only can have overlapping index ranges. Two different processing units can have the same index range. Example:

[[10], [10, 10, 30], [30, 30]]

Will create two groups

[[[10], [10, 10, 30]], [[10, 10, 30], [30, 30]]]

Both groups span the same index range [10;31)
Note 3. Only the first and last row slice of a group can be requested by more than one processing unit
Note 4. Two processing units will never operate on the same set of data (see get_target_start_end). If the first row slice of a group has entity fetch count > 1 this means that it's the last row slice of a previous group. This means that the first index value of this row slice was repeated in the previous group. That group will handle this index value. This processing unit must skip the repeated index value. Analogous if the last segment has entity fetch count > 1, it must be cut at the end. Using the example above.

[[4, 5, 5], [5, 5, 5], [5, 6, 7]] // [5, 6, 7] has entity fetch count > 1. The processing unit will process all vaues in range [4;6)
[[5, 6, 7], [7, 8, 9]] // [5, 6, 7] has entity fetch count > 1. The processing unit will process all values in range [6; 10)

Performance consideration

Structure for processing still marks all segments in a time sliced group for loading if there's an index value appearing in the time range of the index group. When this is the value that spans multiple segments this will create more work, as some segments that are not matched will be loaded. Using the example above if the source contains the value 5 we'll end up with the following processing units

[4, 4, 4], [4, 5, 5]
[4, 5, 5], [5, 5, 5], [5, 6, 7]
[5, 6, 7], [7, 8, 9]

Clearly only the one in the middle needs actual work.

Merge

The merge function performs both inserting new data and updating matched rows. It produces a single one column. Essentially it is merging of two sorted lists, with the caveat that the target is actually list of lists, not a single list. However all values in concatenate(target_index) are sorted. The nuisance is that there's additional bookkeeping of which is the current target list.
Since the target rows to be updated can be in arbitrary order updating happens in two steps

1. Write all data values corresponding to index value x (including new values)
2. Iterate over the rows to update and overwrite with the new value
   1. This must keep track of how many values were inserted up to now in order to get the proper offset
   2. This makes strings sub-optimal. It first adds all strings in the global merged pool but overwriting might orphan some string values. The pool needs rebuilding at the end.
      Merge will also truncate columns so that the output contains only values for which the processing unit is responsible. Using the the example above if the source contains value 5 the processing units will be

[4, 4, 4], [4, 5, 5]
[4, 5, 5], [5, 5, 5], [5, 6, 7]
[5, 6, 7], [7, 8, 9]

assuming no insertion the output will be

[4, 4, 4, 4]
[5, 5, 5, 5, 5, 5]
[6, 7, 7, 8, 9]

Further work

Currently merging uses the same algorithm for the index and the data columns. A better approach would be to run the current algorithm for the index once and make it produce a recipe of the sort:

enum DataSource {SOURCE, TARGET};
struct Entry {
	DataSource src;
	size_t offset_from_source_start;
	size_t count;
};
std::vector<Entry> merge_recipe;

and apply this to all columns.

Index reconstruction

The reconstruction of the index key has changed a bit. Up to now there was a 1:1 mapping in the slices and keys of the old keys and the new keys because update did not create new data. Now the processing unit will emit different row ranges. The new row ranges will represent the combined row range of the whole group of row slices processed by the segment, it will not account the added rows. The added rows are reported separately in the MergeUpdateInsertedRowsEntity structure fetched from the component manager.

MatchRecord

Helper structure to handle all the matching logic together and encapsulated. The biggest change is that the structure of matched rows is no longer a list of lists (for every source row all target rows that match it). It's list of list of lists. For each row slice for each source source row all target rows that match it.

[claude]

ArcticDB Code Review Summary

This PR enables the insert merge strategies (MergeStrategy("do_nothing","insert") and MergeStrategy("update","insert")) for timeseries-indexed data in merge_experimental, replacing the previous hard "Merge cannot perform insertion at the moment" rejection. The core merge / structure_by_time_slice / index-reconstruction logic is intricate but is backed by rapidcheck (rapidcheck_merge_update.cpp, test_split_by_time_slice.cpp) and hypothesis (test_merge_update.py) tests, which is the right approach for this kind of change. Only items needing attention are listed below.

PR Title & Description

• PR is unlabelled. This is a user-facing behavioural change (a previously-rejected operation now succeeds), so it should carry an enhancement label and must NOT be marked no-release-notes — release notes are required. Please add the appropriate label(s).
• Title scope may be inaccurate. The title says "...insertion for numeric columns", but the merge path (merge() -> set_string_from_source / rebuild_sequence_column_in_new_pool) and the hypothesis tests (DTYPES includes "object") exercise dynamic-string columns too. Either broaden the title or clarify in the description what "numeric columns" excludes (only fixed-string is explicitly rejected). Minor grammar: "Partial implementation insertion" -> "Partial implementation of insertion".

Documentation

• Resolved in latest commits. The ArcticDB_merge.ipynb notebook now documents both newly-enabled insert strategies (insert-only and update+insert) with runnable examples, and the stale "Only ... do_nothing is implemented" warnings were removed from the library.py / _store.py docstrings. The limitations list correctly notes insert is timeseries-only.

Code Quality (minor)

• ✅ Resolved in latest commits. The new nested type has been renamed from MatchRechord to MatchRecord throughout clause.hpp and clause_merge_update.cpp (the latest delta is exactly this rename). Good to have this cleaned up before merge.

Correctness (informational — not blocking)

• ✅ Earlier commits fixed a latent validation bug. In MatchRecord::validate_rows_to_update, matched_rows is now sized to the target slice row count (row_range.diff()) instead of the source-row count, and matched_rows[target_row] = true is now set. Previously the bitset was mis-sized and never marked, so the "multiple source rows match the same target row" ambiguity check was effectively a no-op — good catch.
• NativeTensor::span(offset, count) computes bounds as nbytes() / sizeof(T) - offset; if offset ever exceeded the element count this would underflow (only the count bound is ARCTICDB_DEBUG_CHECK-guarded, not offset), producing an oversized span in release builds. All current callers pass in-range offsets derived from get_source_start_end, so this is latent rather than an active bug, but an explicit offset <= size guard would make the utility safer for future reuse.