SQL: Query decoded keys and headers on Iceberg topics - #658
Conversation
✅ Deploy Preview for rp-cloud ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
📝 WalkthroughWalkthroughDocuments SQL support for querying decoded keys and header values from Iceberg-enabled topics. It adds Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
kbatuigas
changed the title
![coderabbitai[bot]](https://cdn.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@modules/reference/pages/sql/sql-statements/create-table.adoc`:
- Around line 163-164: Revise the paragraph describing raw key and header values
so it states that key_decode_mode enables decoding, while key_schema_subject,
key_schema_message_full_name, and key_confluent_wire_protocol only configure
schema-mode decoding. Clarify that header_value_type controls header decoding,
and avoid implying auxiliary schema options alone change metadata types.
- Around line 91-94: Resolve the contradictory description for
key_schema_message_full_name in the CREATE TABLE option reference by checking
the implementation’s behavior and documenting either that the option is required
for schemas with multiple messages or that it defaults to the first message.
Update the column description so it states only the implemented rule and removes
the conflicting fallback or requirement.
In `@modules/sql/pages/query-data/query-iceberg-topics.adoc`:
- Around line 134-141: Update the CREATE TABLE example for
default_redpanda_catalog=>orders to set header_value_type to 'string', so the
subsequent (redpanda).headers query returns decoded string values and matches
the surrounding description.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 00f17fc7-cfce-4e4e-b96b-8fb766f02875
📒 Files selected for processing (3)
modules/get-started/pages/whats-new-cloud.adocmodules/reference/pages/sql/sql-statements/create-table.adocmodules/sql/pages/query-data/query-iceberg-topics.adoc
| |`key_schema_message_full_name` | ||
| |STRING | ||
| |No | ||
| |Full Protobuf message name for the key schema. Required when the key schema defines more than one message. Defaults to the first message in the schema. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Resolve the contradictory Protobuf key message-name rule.
Line 94 says key_schema_message_full_name is required for multi-message schemas, then says it defaults to the first message. Choose one behavior and align it with the implementation; otherwise users may omit a required option or rely on an ambiguous message selection.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@modules/reference/pages/sql/sql-statements/create-table.adoc` around lines 91
- 94, Resolve the contradictory description for key_schema_message_full_name in
the CREATE TABLE option reference by checking the implementation’s behavior and
documenting either that the option is required for schemas with multiple
messages or that it defaults to the first message. Update the column description
so it states only the implemented rule and removes the conflicting fallback or
requirement.
| By default, `key` is raw `bytea` and each header `value` is raw `bytea`. To decode them, set `key_decode_mode`, `key_schema_subject`, `key_confluent_wire_protocol`, or `header_value_type` (see <<options>>): | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Clarify which options actually enable decoding.
key_decode_mode selects the decoding mode; key_schema_subject, key_schema_message_full_name, and key_confluent_wire_protocol only provide schema-mode configuration. Reword this paragraph so users do not assume setting an auxiliary option alone changes the metadata type.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@modules/reference/pages/sql/sql-statements/create-table.adoc` around lines
163 - 164, Revise the paragraph describing raw key and header values so it
states that key_decode_mode enables decoding, while key_schema_subject,
key_schema_message_full_name, and key_confluent_wire_protocol only configure
schema-mode decoding. Clarify that header_value_type controls header decoding,
and avoid implying auxiliary schema options alone change metadata types.
| CREATE TABLE default_redpanda_catalog=>orders WITH ( | ||
| topic = 'orders', | ||
| schema_subject = 'orders-value', | ||
| key_decode_mode = 'schema_id_prefix' | ||
| ); | ||
|
|
||
| SELECT ((redpanda).key).customer_id, (redpanda).headers | ||
| FROM default_redpanda_catalog=>orders; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Make the example actually decode headers.
The CREATE TABLE statement configures only key_decode_mode, while header_value_type defaults to binary. Therefore (redpanda).headers still contains raw bytea values, despite the preceding text saying the example accesses decoded key and header values. Add header_value_type = 'string' or change the prose to describe headers as raw bytes.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@modules/sql/pages/query-data/query-iceberg-topics.adoc` around lines 134 -
141, Update the CREATE TABLE example for default_redpanda_catalog=>orders to set
header_value_type to 'string', so the subsequent (redpanda).headers query
returns decoded string values and matches the surrounding description.
1 participant
Description
This pull request adds support for decoding record keys and header values when querying Iceberg-enabled topics with Redpanda SQL. Users can now specify how keys and headers are decoded and exposed in query results by setting new options on
CREATE TABLE. Documentation is updated to describe these options, their effects, and usage examples.New SQL options for key and header decoding:
key_decode_modeoption toCREATE TABLEfor controlling how record keys are interpreted (as raw bytes, UTF-8 string, or decoded with a schema) (modules/reference/pages/sql/sql-statements/create-table.adoc, modules/reference/pages/sql/sql-statements/create-table.adocR73-R123).header_value_typeoption toCREATE TABLEfor decoding header values as either raw bytes or UTF-8 strings (modules/reference/pages/sql/sql-statements/create-table.adoc, modules/reference/pages/sql/sql-statements/create-table.adocR73-R123).key_schema_subject,key_schema_message_full_name, andkey_confluent_wire_protocolfor advanced key decoding scenarios (modules/reference/pages/sql/sql-statements/create-table.adoc, modules/reference/pages/sql/sql-statements/create-table.adocR73-R123).Documentation and usage examples:
modules/get-started/pages/whats-new-cloud.adoc, modules/get-started/pages/whats-new-cloud.adocR9-R14).modules/reference/pages/sql/sql-statements/create-table.adoc, modules/reference/pages/sql/sql-statements/create-table.adocR236-R250).redpandametadata column, including type changes and error handling (modules/reference/pages/sql/sql-statements/create-table.adoc, modules/reference/pages/sql/sql-statements/create-table.adocR163-R172).modules/sql/pages/query-data/query-iceberg-topics.adoc, modules/sql/pages/query-data/query-iceberg-topics.adocR103-R147).Resolves https://github.com/redpanda-data/documentation-private/issues/
Review deadline:
Page previews
Checks