From edfcbe122e9bd771213ff785f4e177a4a1ffab69 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Mon, 3 Mar 2025 16:11:03 -0700 Subject: [PATCH 01/30] feat(protect): add initial searchable encryption functionality --- README.md | 39 ++- docs/encrypt-config.md | 338 +++++++++++++++++++++ packages/protect/__tests__/protect.test.ts | 138 +++++---- packages/protect/package.json | 3 +- packages/protect/src/ffi/encrypt-config.ts | 77 +++++ packages/protect/src/ffi/index.ts | 49 ++- packages/protect/src/index.ts | 8 +- pnpm-lock.yaml | 58 ++-- 8 files changed, 604 insertions(+), 106 deletions(-) create mode 100644 docs/encrypt-config.md create mode 100644 packages/protect/src/ffi/encrypt-config.ts diff --git a/README.md b/README.md index 660d17d14..4bf726bff 100644 --- a/README.md +++ b/README.md @@ -114,20 +114,43 @@ You can read more about [configuration via toml file or environment variables he ### Initializing the EQL client -In your application, import the `protect` function from the `@cipherstash/protect` package, and initialize a client with your CipherStash credentials. +In your application, import the `protect` function and `EncryptConfig` type from the `@cipherstash/protect` package, and initialize a client with your encrypt config. -```typescript -const { protect } = require('@cipherstash/protect') -const protectClient = await protect() +To define your encrypt config, create an object that conforms to the `EncryptConfig` type. +This object determines how data will be encrypted and decrypted, and which searchable indexes will be added. + +You will need to provide the following information: + +1. Which **tables** should be encrypted and decrypted. +2. Which **columns** should be encrypted and decrypted. +3. Which **indexes** should be built on the encrypted columns. + +An example config might look like this: + +```ts +import { protect, type EncryptConfig } from '@cipherstash/protect' + +const config: EncryptConfig = { + v: 1, + tables: { + users: { + email: { + cast_as: 'text', + indexes: { ore: {}, match: {}, unique: {} }, + }, + }, + }, +} ``` -If you are using ES6: +Finally, initialize the client with your encrypt config: -```typescript -import { protect } from '@cipherstash/protect' -const protectClient = await protect() +```ts +const protectClient = await protect(config) ``` +To learn more about the `EncryptConfig` type, please refer to the [Encrypt Config docs](./docs/encrypt-config.md). + ### Encrypting data Use the `encrypt` function to encrypt data. diff --git a/docs/encrypt-config.md b/docs/encrypt-config.md new file mode 100644 index 000000000..dac34232a --- /dev/null +++ b/docs/encrypt-config.md @@ -0,0 +1,338 @@ +# Encrypt Config for Protect.js + +An **Encrypt Config** is a JSON (or TypeScript) structure that specifies: + +1. **Version** (the `"v"` field). +2. One or more **Tables** (the `"tables"` field). + +Inside each **Table**, you’ll define one or more **Columns**, specifying how each column is _cast_ (converted to a given type) and which **Indexes** (if any) should be built on that column. + +Once defined, use the config to initialize the Protect.js client: + +```ts +import { protect, type EncryptConfig } from "@cipherstash/protect"; + +const config: EncryptConfig = { + v: 1, + tables: { + users: { + email: { + cast_as: "text", + indexes: {}, + }, + }, + }, +}; + +const protectClient = protect(config); +``` + +## Top-Level Structure + +```jsonc +{ + "v": 1, // Required integer representing the config version + "tables": { + "table_name": { + "column_name": { + "cast_as": "...", // Optional, defaults to "text" + "indexes": { // Optional, each index is optional + "ore": { ... }, + "unique": { ... }, + "match": { ... }, + "ste_vec": { ... } + } + } + } + } +} +``` + +### `v` (Version) + +- **Type**: `number` +- **Required**: Yes +- **Purpose**: Used to mark the version of the configuration schema. +We only support version `1` at this time. + +### `tables` + +- **Type**: A map of **tableName** → **Table Definition**. +- **Required**: Yes +- **Example**: + ```json + { + "tables": { + "users": { ... }, + "orders": { ... } + } + } + ``` +- Each key in `"tables"` is the name of a table (e.g., `"users"`), whose value is another map of **columns**. + +## Table Definition + +```jsonc +"table_name": { + "column_name": { ... }, + "another_column": { ... } +} +``` + +- Each key (e.g., `"column_name"`) is the column name in that table. +- The value for each column is a **Column** object. + +## Column Definition + +```jsonc +{ + "cast_as": "text", // or "big_int", "int", "small_int", "boolean", "date", "real", "double", "jsonb" + "indexes": { + "ore": {}, + "unique": { + "token_filters": [ + { "kind": "downcase" } + ] + }, + "match": { + "tokenizer": { + "kind": "standard" // or "ngram" + }, + "token_filters": [], + "k": 6, + "m": 2048, + "include_original": false + }, + "ste_vec": { + "prefix": "..." + } + } +} +``` + +### `cast_as` + +Specifies how the column will be cast (typed) before encryption and indexing. This closely maps to the **Rust** enum `CastAs`. Possible values: + +- `"big_int"` +- `"small_int"` +- `"int"` +- `"boolean"` +- `"date"` +- `"real"` +- `"double"` +- `"text"` (default) +- `"jsonb"` + +**Default**: `"text"` + +### `indexes` + +An object defining which indexes (if any) to enable on this column. **All fields in `indexes` are optional**. If none are present, the column will not be indexed. + +#### 1. ORE Index + +Enables **Order-Revealing Encryption** (ORE). + +```json +"indexes": { + "ore": {} +} +``` + +- The `ore` property is an empty object (i.e., no extra settings). +- In Rust, this corresponds to `OreIndexOpts {}`. + +#### 2. Unique Index + +Enables **Unique** constraints. This typically uses specialized indexing so that each value in the column must be unique. + +```json +"indexes": { + "unique": { + "token_filters": [ + { "kind": "downcase" } + ] + } +} +``` + +- **`token_filters`**: An optional array of objects specifying transformations on the token before indexing. Currently, the example shows one possible filter: + - `{ "kind": "downcase" }` – transforms the string to lowercase before indexing. + +**Default**: `{ "token_filters": [] }` (an empty array of token filters). + +#### 3. Match Index + +Enables **Match** (full-text-ish) indexing, which can be used for substring/pattern matching. The default fields are: + +```json +"indexes": { + "match": { + "tokenizer": { + "kind": "standard" + }, + "token_filters": [], + "k": 6, + "m": 2048, + "include_original": false + } +} +``` + +**Field explanations**: + +1. **`tokenizer`** – How to chunk the text into tokens. + - `{"kind": "standard"}` (default) + - `{"kind": "ngram", "token_length": }` – e.g. `{"kind":"ngram","token_length":3}` +2. **`token_filters`** – Optional array of transforms, e.g. `[{"kind": "downcase"}]`. +3. **`k`** – Defaults to `6`. Used internally for chunk sizes or matching thresholds. +4. **`m`** – Defaults to `2048`. Another internal parameter for the cryptographic index. +5. **`include_original`** – Defaults to `false`. When `true`, also store the original tokens unmodified. + +#### 4. STE Vector Index + +Enables the **STE Vec** index, typically used for vector-like data or structured queries. + +```json +"indexes": { + "ste_vec": { + "prefix": "..." + } +} +``` + +**Fields**: + +- **`prefix`** – A required string that acts as a prefix for the index. + +## Example Configs + +### Minimal Config + +```json +{ + "v": 1, + "tables": { + "users": {} + } +} +``` + +- Creates a config for version `1`. +- Declares a `users` table with **no columns**. (In practice, you’d add columns.) + +### Simple Column Example + +```json +{ + "v": 1, + "tables": { + "users": { + "email": { + "cast_as": "text", + "indexes": {} + } + } + } +} +``` + +- `users.email` is cast as `"text"` (the default if omitted). +- **No** indexes are enabled on `email`. + +### ORE Index Example + +```json +{ + "v": 1, + "tables": { + "users": { + "age": { + "cast_as": "int", + "indexes": { + "ore": {} + } + } + } + } +} +``` + +- `users.age` is cast as `"int"`. +- ORE indexing is enabled. + +### Unique Index with Token Filters + +```json +{ + "v": 1, + "tables": { + "users": { + "email": { + "indexes": { + "unique": { + "token_filters": [ + { "kind": "downcase" } + ] + } + } + } + } + } +} +``` + +- `users.email` defaults to `"text"`. +- Unique index with a `downcase` token filter. + +### Match Index with Ngram Tokenizer + +```json +{ + "v": 1, + "tables": { + "users": { + "description": { + "indexes": { + "match": { + "tokenizer": { + "kind": "ngram", + "token_length": 3 + }, + "token_filters": [ + { "kind": "downcase" } + ], + "k": 8, + "m": 1024, + "include_original": true + } + } + } + } + } +} +``` + +- Enables a match index with an ngram tokenizer that uses 3-character slices. +- Applies `downcase`. +- Sets `k=8`, `m=1024`, `include_original=true`. + +### STE Vector Example + +```json +{ + "v": 1, + "tables": { + "users": { + "event_data": { + "indexes": { + "ste_vec": { + "prefix": "event-data" + } + } + } + } + } +} +``` \ No newline at end of file diff --git a/packages/protect/__tests__/protect.test.ts b/packages/protect/__tests__/protect.test.ts index 9128e57f3..8ba40adb6 100644 --- a/packages/protect/__tests__/protect.test.ts +++ b/packages/protect/__tests__/protect.test.ts @@ -3,6 +3,19 @@ import { describe, expect, it } from 'vitest' import { LockContext, createEqlPayload, getPlaintext, protect } from '../src' import type { CsPlaintextV1Schema } from '../src/cs_plaintext_v1' +import type { EncryptConfig } from '../src/ffi/encrypt-config' + +const config: EncryptConfig = { + v: 1, + tables: { + users: { + email: { + cast_as: 'text', + indexes: { ore: {}, match: {}, unique: {} }, + }, + }, + }, +} describe('createEqlPayload', () => { it('should create a payload with the correct default values', () => { @@ -113,10 +126,10 @@ describe('getPlaintext', () => { describe('encryption and decryption', () => { it('should encrypt and decrypt a payload', async () => { - const protectClient = await protect() + const protectClient = await protect(config) const ciphertext = await protectClient.encrypt('plaintext', { - column: 'column_name', + column: 'email', table: 'users', }) @@ -132,85 +145,90 @@ describe('encryption and decryption', () => { }, 30000) it('should return null if plaintext is null', async () => { - const protectClient = await protect() + const protectClient = await protect(config) const ciphertext = await protectClient.encrypt(null, { - column: 'column_name', + column: 'email', table: 'users', }) + console.log('ciphertext', ciphertext) + if (ciphertext.failure) { throw new Error(`[protect]: ${ciphertext.failure.message}`) } const plaintext = await protectClient.decrypt(ciphertext.data) + console.log('plaintext', plaintext) + expect(plaintext).toEqual({ data: null, }) }, 30000) }) -describe('bulk encryption', () => { - it('should bulk encrypt and decrypt a payload', async () => { - const protectClient = await protect() - const ciphertexts = await protectClient.bulkEncrypt( - [ - { - plaintext: 'test', - id: '1', - }, - { - plaintext: 'test2', - id: '2', - }, - ], - { - table: 'users', - column: 'column_name', - }, - ) +// TODO: Searchable encryption is not yet supported for bukl ops +// describe('bulk encryption', () => { +// it('should bulk encrypt and decrypt a payload', async () => { +// const protectClient = await protect() +// const ciphertexts = await protectClient.bulkEncrypt( +// [ +// { +// plaintext: 'test', +// id: '1', +// }, +// { +// plaintext: 'test2', +// id: '2', +// }, +// ], +// { +// table: 'users', +// column: 'column_name', +// }, +// ) - if (ciphertexts.failure) { - throw new Error(`[protect]: ${ciphertexts.failure.message}`) - } +// if (ciphertexts.failure) { +// throw new Error(`[protect]: ${ciphertexts.failure.message}`) +// } - const plaintexts = await protectClient.bulkDecrypt(ciphertexts.data) - - expect(plaintexts).toEqual({ - data: [ - { - plaintext: 'test', - id: '1', - }, - { - plaintext: 'test2', - id: '2', - }, - ], - }) - }, 30000) +// const plaintexts = await protectClient.bulkDecrypt(ciphertexts.data) - it('should return null if plaintexts is empty', async () => { - const protectClient = await protect() - const ciphertexts = await protectClient.bulkEncrypt([], { - table: 'users', - column: 'column_name', - }) - expect(ciphertexts).toEqual({ - data: null, - }) - }, 30000) +// expect(plaintexts).toEqual({ +// data: [ +// { +// plaintext: 'test', +// id: '1', +// }, +// { +// plaintext: 'test2', +// id: '2', +// }, +// ], +// }) +// }, 30000) - it('should return null if decrypting empty ciphertexts', async () => { - const protectClient = await protect() - const ciphertexts = null - const plaintexts = await protectClient.bulkDecrypt(ciphertexts) - expect(plaintexts).toEqual({ - data: null, - }) - }, 30000) -}) +// it('should return null if plaintexts is empty', async () => { +// const protectClient = await protect() +// const ciphertexts = await protectClient.bulkEncrypt([], { +// table: 'users', +// column: 'column_name', +// }) +// expect(ciphertexts).toEqual({ +// data: null, +// }) +// }, 30000) + +// it('should return null if decrypting empty ciphertexts', async () => { +// const protectClient = await protect() +// const ciphertexts = null +// const plaintexts = await protectClient.bulkDecrypt(ciphertexts) +// expect(plaintexts).toEqual({ +// data: null, +// }) +// }, 30000) +// }) // ------------------------ // TODO get bulk Encryption/Decryption working in CI. diff --git a/packages/protect/package.json b/packages/protect/package.json index 32496a79d..7514808e9 100644 --- a/packages/protect/package.json +++ b/packages/protect/package.json @@ -55,7 +55,8 @@ }, "dependencies": { "@byteslice/result": "^0.2.0", - "@cipherstash/protect-ffi": "0.11.0" + "@cipherstash/protect-ffi": "0.12.0-1", + "zod": "^3.24.2" }, "optionalDependencies": { "@rollup/rollup-linux-x64-gnu": "4.24.0" diff --git a/packages/protect/src/ffi/encrypt-config.ts b/packages/protect/src/ffi/encrypt-config.ts new file mode 100644 index 000000000..c70b30266 --- /dev/null +++ b/packages/protect/src/ffi/encrypt-config.ts @@ -0,0 +1,77 @@ +import { z } from 'zod' + +const castAsEnum = z + .enum([ + 'big_int', + 'boolean', + 'date', + 'real', + 'double', + 'int', + 'small_int', + 'text', + 'jsonb', + ]) + .default('text') + +const tokenFilterSchema = z.object({ + kind: z.literal('downcase'), +}) + +const tokenizerSchema = z + .union([ + z.object({ + kind: z.literal('standard'), + }), + z.object({ + kind: z.literal('ngram'), + token_length: z.number(), + }), + ]) + .default({ kind: 'standard' }) + .optional() + +const oreIndexOptsSchema = z.object({}) + +const uniqueIndexOptsSchema = z.object({ + token_filters: z.array(tokenFilterSchema).default([]).optional(), +}) + +const matchIndexOptsSchema = z.object({ + tokenizer: tokenizerSchema, + token_filters: z.array(tokenFilterSchema).default([]).optional(), + k: z.number().default(6).optional(), + m: z.number().default(2048).optional(), + include_original: z.boolean().default(false).optional(), +}) + +const steVecIndexOptsSchema = z.object({ + prefix: z.string(), +}) + +const indexesSchema = z + .object({ + ore: oreIndexOptsSchema.optional(), + unique: uniqueIndexOptsSchema.optional(), + match: matchIndexOptsSchema.optional(), + ste_vec: steVecIndexOptsSchema.optional(), + }) + .default({}) + +const columnSchema = z + .object({ + cast_as: castAsEnum, + indexes: indexesSchema, + }) + .default({}) + +const tableSchema = z.record(columnSchema).default({}) + +const tablesSchema = z.record(tableSchema).default({}) + +export const encryptConfigSchema = z.object({ + v: z.number(), + tables: tablesSchema, +}) + +export type EncryptConfig = z.infer diff --git a/packages/protect/src/ffi/index.ts b/packages/protect/src/ffi/index.ts index 1a57d9650..3b7e1c707 100644 --- a/packages/protect/src/ffi/index.ts +++ b/packages/protect/src/ffi/index.ts @@ -16,6 +16,7 @@ import { normalizeBulkEncryptPayloads, normalizeBulkEncryptPayloadsWithLockContext, } from './payload-helpers' +import { type EncryptConfig, encryptConfigSchema } from './encrypt-config' // ------------------------ // Type Definitions @@ -115,8 +116,14 @@ class EncryptOperation return null } - const val = await ffiEncrypt(this.client, this.plaintext, this.column) - return { c: val } + const val = await ffiEncrypt( + this.client, + this.plaintext, + this.column, + this.table, + ) + + return JSON.parse(val) }, (error) => ({ type: ProtectErrorTypes.EncryptionError, @@ -195,6 +202,7 @@ class EncryptOperationWithLockContext client, plaintext, column, + table, context.data.context, context.data.ctsToken, ) @@ -675,22 +683,43 @@ class BulkDecryptOperationWithLockContext // ------------------------ export class ProtectClient { private client: Client + private encryptConfig: EncryptConfig | undefined private workspaceId: string | undefined constructor() { const workspaceId = loadWorkSpaceId() - - logger.info( - 'Successfully initialized the EQL client with your defined environment variables.', - ) - - this.workspaceId = process.env.CS_WORKSPACE_ID + this.workspaceId = workspaceId } - async init(): Promise> { + async init( + encryptConifg?: EncryptConfig, + ): Promise> { return await withResult( async () => { - const c = await newClient() + let c: Client + + if (encryptConifg) { + const validated: EncryptConfig = + encryptConfigSchema.parse(encryptConifg) + + logger.debug( + 'Initializing the Protect.js client with the following encrypt config:', + { + encryptConfig: validated, + }, + ) + + c = await newClient(JSON.stringify(validated)) + this.encryptConfig = validated + } else { + logger.debug( + 'Initializing the Protect.js client with default encrypt config.', + ) + + c = await newClient() + } + + logger.info('Successfully initialized the Protect.js client.') this.client = c return this }, diff --git a/packages/protect/src/index.ts b/packages/protect/src/index.ts index ae8eb6049..ee8047b4e 100644 --- a/packages/protect/src/index.ts +++ b/packages/protect/src/index.ts @@ -1,4 +1,5 @@ import { ProtectClient } from './ffi' +import type { EncryptConfig } from './ffi/encrypt-config' export const ProtectErrorTypes = { ClientInitError: 'ClientInitError', @@ -13,9 +14,11 @@ export interface ProtectError { message: string } -export const protect = async (): Promise => { +export const protect = async ( + encryptConfig?: EncryptConfig, +): Promise => { const client = new ProtectClient() - const result = await client.init() + const result = await client.init(encryptConfig) if (result.failure) { throw new Error(`[protect]: ${result.failure.message}`) @@ -25,6 +28,7 @@ export const protect = async (): Promise => { } export type { Result } from '@byteslice/result' +export type { EncryptConfig } from './ffi/encrypt-config' export type { ProtectClient } from './ffi' export * from './cs_plaintext_v1' export * from './identify' diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index f83f730bc..62298a3a4 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -215,11 +215,14 @@ importers: specifier: ^0.2.0 version: 0.2.0 '@cipherstash/protect-ffi': - specifier: 0.11.0 - version: 0.11.0 + specifier: 0.12.0-1 + version: 0.12.0-1 typescript: specifier: ^5.0.0 version: 5.7.2 + zod: + specifier: ^3.24.2 + version: 3.24.2 optionalDependencies: '@rollup/rollup-linux-x64-gnu': specifier: 4.24.0 @@ -366,33 +369,33 @@ packages: '@changesets/write@0.3.2': resolution: {integrity: sha512-kDxDrPNpUgsjDbWBvUo27PzKX4gqeKOlhibaOXDJA6kuBisGqNHv/HwGJrAu8U/dSf8ZEFIeHIPtvSlZI1kULw==} - '@cipherstash/protect-ffi-darwin-arm64@0.11.0': - resolution: {integrity: sha512-dOmknDJxbH6RdmwSleNBgnHvM0nrOG/aAKs4pFpIBRSdRV0oFQDynaIAFa/hAcBtE37RRvZCOX6/zF1FQioDlg==} + '@cipherstash/protect-ffi-darwin-arm64@0.12.0-1': + resolution: {integrity: sha512-PY9A5+wyqRi1imAjPq8TvpMyb9z/eU/dnZ1TpRhAJeaQK3wLfOZH4r+A/IDR41HYBjPkRVKi9/IK2/c0ebAxqA==} cpu: [arm64] os: [darwin] - '@cipherstash/protect-ffi-darwin-x64@0.11.0': - resolution: {integrity: sha512-1eaXoY/RT3dhByZyErYocH7jGjxkXju9KsJ7Wx4sniwk5v+6bmmDrkhduORY7AdrQglNhLTKnUf48NIU+zkuiQ==} + '@cipherstash/protect-ffi-darwin-x64@0.12.0-1': + resolution: {integrity: sha512-V0QjUyYj8o0u9+FJjNw6Loa3NQdp8+fpPy0ODFszH15NntAblhjRrI2B6OK/Ou+3jdD/FOuC4J1L+QHIDg2wsw==} cpu: [x64] os: [darwin] - '@cipherstash/protect-ffi-linux-arm64-gnu@0.11.0': - resolution: {integrity: sha512-O2o2gZpIzp0ADLGcswl2jk2G18MgevTB0zv4ij1b3Ut+c+8h7QMWjtzZaJpG6+2PZK91OAgb3GBdSgIv5e2jFQ==} + '@cipherstash/protect-ffi-linux-arm64-gnu@0.12.0-1': + resolution: {integrity: sha512-fcI+tFjFDJWb2v9YyhkEFkV45JHZo9iB5+zYjXcfuCT2+3s3U0rlV4lgrgW6v1h8mqEf5yShc7DY/oO4gzEmMg==} cpu: [arm64] os: [linux] - '@cipherstash/protect-ffi-linux-x64-gnu@0.11.0': - resolution: {integrity: sha512-tp8hHR9fxTrcbv2E+MNN/e8uwJS4ncBorAlxksosnkYnz28FBmnagz0kQiltOkCieMEZv9c1EqaGjTt1s2tr0g==} + '@cipherstash/protect-ffi-linux-x64-gnu@0.12.0-1': + resolution: {integrity: sha512-7xrV1f2aNrZh1/x+e4Oq4ib7wwmDYY9DIYdWGG63FBx+Z9ZXa8CRgME9xor//w8fGuvoKYX6b2HR0Rqc4OeUAg==} cpu: [x64] os: [linux] - '@cipherstash/protect-ffi-win32-x64-msvc@0.11.0': - resolution: {integrity: sha512-DqUpDhsLSdO9JaDP7Jxfn5T++k4lM0fT+GoFQK5ITL+FBETN4eearV7BlPWNbH7iucAd624P88LKt98vg2orOg==} + '@cipherstash/protect-ffi-win32-x64-msvc@0.12.0-1': + resolution: {integrity: sha512-n8t1OWvlqQUD7dDuI/NGZi2xlQD0sfJSH6K3b7HqVAF8AMSvWmwox1kZmEhVz7/YuGz6+nOjCN2sSp1aCAR+Yg==} cpu: [x64] os: [win32] - '@cipherstash/protect-ffi@0.11.0': - resolution: {integrity: sha512-wyk9gEqQy1zFXUXU6Oaygcgy3NXx4M26sBN3HoRSmIVGU+C2agWDUqBqoGOdz2Q8v/YYApu8DnmVHDBaJeevgg==} + '@cipherstash/protect-ffi@0.12.0-1': + resolution: {integrity: sha512-WbOwotCRHRd8BtqQSdE5CxWiUt3+8CQPntDKq8L1iLGswThLD9J6vEqIiCifW2gChiooXxcbeOvigFmY5LaK+w==} '@clerk/backend@1.21.5': resolution: {integrity: sha512-fZ+HuHQkPngYp9vAEqsJ1XJkhWjSWTin90UixeV3jNrbuPY2jd70LKYTAIEvggnnG42pYxdVupaQDUfqj0dy4Q==} @@ -3647,6 +3650,9 @@ packages: resolution: {integrity: sha512-GQHQqAopRhwU8Kt1DDM8NjibDXHC8eoh1erhGAJPEyveY9qqVeXvVikNKrDz69sHowPMorbPUrH/mx8c50eiBQ==} engines: {node: '>=18'} + zod@3.24.2: + resolution: {integrity: sha512-lY7CDW43ECgW9u1TcT3IoXHflywfVqDYze4waEz812jR/bZ8FHDsl7pFQoSZTz5N+2NqRXs8GBwnAwo3ZNxqhQ==} + snapshots: '@alloc/quick-lru@5.2.0': {} @@ -3840,30 +3846,30 @@ snapshots: human-id: 1.0.2 prettier: 2.8.8 - '@cipherstash/protect-ffi-darwin-arm64@0.11.0': + '@cipherstash/protect-ffi-darwin-arm64@0.12.0-1': optional: true - '@cipherstash/protect-ffi-darwin-x64@0.11.0': + '@cipherstash/protect-ffi-darwin-x64@0.12.0-1': optional: true - '@cipherstash/protect-ffi-linux-arm64-gnu@0.11.0': + '@cipherstash/protect-ffi-linux-arm64-gnu@0.12.0-1': optional: true - '@cipherstash/protect-ffi-linux-x64-gnu@0.11.0': + '@cipherstash/protect-ffi-linux-x64-gnu@0.12.0-1': optional: true - '@cipherstash/protect-ffi-win32-x64-msvc@0.11.0': + '@cipherstash/protect-ffi-win32-x64-msvc@0.12.0-1': optional: true - '@cipherstash/protect-ffi@0.11.0': + '@cipherstash/protect-ffi@0.12.0-1': dependencies: '@neon-rs/load': 0.1.82 optionalDependencies: - '@cipherstash/protect-ffi-darwin-arm64': 0.11.0 - '@cipherstash/protect-ffi-darwin-x64': 0.11.0 - '@cipherstash/protect-ffi-linux-arm64-gnu': 0.11.0 - '@cipherstash/protect-ffi-linux-x64-gnu': 0.11.0 - '@cipherstash/protect-ffi-win32-x64-msvc': 0.11.0 + '@cipherstash/protect-ffi-darwin-arm64': 0.12.0-1 + '@cipherstash/protect-ffi-darwin-x64': 0.12.0-1 + '@cipherstash/protect-ffi-linux-arm64-gnu': 0.12.0-1 + '@cipherstash/protect-ffi-linux-x64-gnu': 0.12.0-1 + '@cipherstash/protect-ffi-win32-x64-msvc': 0.12.0-1 '@clerk/backend@1.21.5(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': dependencies: @@ -6539,3 +6545,5 @@ snapshots: optional: true yoctocolors@2.1.1: {} + + zod@3.24.2: {} From be55d3f23eb813e7f5140a71664ef94ea9369637 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Mon, 3 Mar 2025 16:15:40 -0700 Subject: [PATCH 02/30] feat(app-drizzle): add new encrypt config --- apps/drizzle/src/protect.ts | 16 ++++++++++++++-- apps/drizzle/src/select.ts | 1 - 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/apps/drizzle/src/protect.ts b/apps/drizzle/src/protect.ts index a992c4e98..1e2071ce2 100644 --- a/apps/drizzle/src/protect.ts +++ b/apps/drizzle/src/protect.ts @@ -1,4 +1,16 @@ import 'dotenv/config' -import { protect } from '@cipherstash/protect' +import { protect, type EncryptConfig } from '@cipherstash/protect' -export const protectClient = await protect() +const config: EncryptConfig = { + v: 1, + tables: { + users: { + email_encrypted: { + cast_as: 'text', + indexes: { ore: {}, match: {}, unique: {} }, + }, + }, + }, +} + +export const protectClient = await protect(config) diff --git a/apps/drizzle/src/select.ts b/apps/drizzle/src/select.ts index b13cf52e1..6a586b291 100644 --- a/apps/drizzle/src/select.ts +++ b/apps/drizzle/src/select.ts @@ -2,7 +2,6 @@ import 'dotenv/config' import { db } from './db' import { users } from './db/schema' import { protectClient } from './protect' -import type { ProtectError, Result } from '@cipherstash/protect' const sql = db .select({ From 0574b0cf118dc57f683af8966e2e88adbc2f7d94 Mon Sep 17 00:00:00 2001 From: Drew Thomas Date: Tue, 4 Mar 2025 09:52:05 +1100 Subject: [PATCH 03/30] Add compose file with service for PG --- docker-compose.yml | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 docker-compose.yml diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 000000000..b92bd5079 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,22 @@ +services: + postgres: &postgres + image: postgres:latest + environment: + PGPORT: 5432 + POSTGRES_DB: "cipherstash" + POSTGRES_USER: "cipherstash" + PGUSER: "cipherstash" + POSTGRES_PASSWORD: password + ports: + - 5432:5432 + deploy: + resources: + limits: + cpus: "${CPU_LIMIT:-2}" + memory: 2048mb + restart: always + healthcheck: + test: ["CMD-SHELL", "pg_isready"] + interval: 1s + timeout: 5s + retries: 10 From b2c271b7d4b2fecc31b20fb143bde3b2ecb7b8d2 Mon Sep 17 00:00:00 2001 From: Drew Thomas Date: Tue, 4 Mar 2025 09:52:41 +1100 Subject: [PATCH 04/30] Add example of EQL setup with mise --- mise.toml | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/mise.toml b/mise.toml index 44dcb556d..a9ad30528 100644 --- a/mise.toml +++ b/mise.toml @@ -1,3 +1,47 @@ [tools] node = "22.11.0" pnpm = "9.15.3" + +[tasks."postgres:eql:download"] +alias = 'e' +description = "Download latest EQL release" +outputs = [ + "{{config_root}}/sql/cipherstash-encrypt.sql", + "{{config_root}}/sql/cipherstash-encrypt-uninstall.sql", +] +run = """ +mkdir sql + +# install script +if [ -z "$CS_EQL_PATH" ]; then + curl -sLo sql/cipherstash-encrypt.sql https://github.com/cipherstash/encrypt-query-language/releases/latest/download/cipherstash-encrypt.sql +else + echo "Using EQL: ${CS_EQL_PATH}" + cp "$CS_EQL_PATH" sql/cipherstash-encrypt.sql +fi + +# uninstall script +if [ -z "$CS_EQL_UNINSTALL_PATH" ]; then + curl -sLo sql/cipherstash-encrypt-uninstall.sql https://github.com/cipherstash/encrypt-query-language/releases/latest/download/cipherstash-encrypt-uninstall.sql +else + echo "Using EQL: ${CS_EQL_PATH}" + cp "$CS_EQL_UNINSTALL_PATH" sql/cipherstash-encrypt-uninstall.sql +fi +""" + +[tasks."postgres:setup"] +alias = 's' +description = "Installs EQL and applies schema to database" +run = """ +#!/bin/bash + +mise run postgres:eql:download +cat sql/cipherstash-encrypt.sql | docker exec -i protectjs-postgres-1 psql postgresql://cipherstash:password@postgres:5432/cipherstash -f- +""" + +[tasks."postgres:psql"] +description = "Run psql (interactively) against the Postgres instance; assumes Postgres is already up" +run = """ +set -eu +docker exec -it protectjs-postgres-1 psql "postgresql://cipherstash:password@postgres:5432/cipherstash" +""" From 68f13a7d3da5f5f6f65ff6803c0254477b6970b6 Mon Sep 17 00:00:00 2001 From: Drew Thomas Date: Tue, 4 Mar 2025 09:52:53 +1100 Subject: [PATCH 05/30] Add EQL sql scripts to .gitignore --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index cc3a9e760..fc7ee4389 100644 --- a/.gitignore +++ b/.gitignore @@ -65,3 +65,4 @@ mise.local.toml # cipherstash cipherstash.toml cipherstash.secret.toml +sql/cipherstash-*.sql From 1582e1c48923b3b1a7ffa61a78f86f67aa5eda44 Mon Sep 17 00:00:00 2001 From: Drew Thomas Date: Tue, 4 Mar 2025 09:54:33 +1100 Subject: [PATCH 06/30] Start example of search on encrypted column Not working yet (needs Encrypt config support in Protect.js) --- apps/drizzle/src/select.ts | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/apps/drizzle/src/select.ts b/apps/drizzle/src/select.ts index 6a586b291..b5e236136 100644 --- a/apps/drizzle/src/select.ts +++ b/apps/drizzle/src/select.ts @@ -2,17 +2,41 @@ import 'dotenv/config' import { db } from './db' import { users } from './db/schema' import { protectClient } from './protect' +import { bindIfParam, sql } from 'drizzle-orm'; +import type { BinaryOperator, SQL, SQLWrapper } from 'drizzle-orm'; -const sql = db +// const csEq: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { +// return sql`cs_unique_v1(${left}) = cs_unique_v1(${bindIfParam(right, left)})`; +// }; + +// const csGt: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { +// return sql`cs_ore_64_8_v1(${left}) > cs_ore_64_8_v1(${bindIfParam(right, left)})`; +// }; + +// const csLt: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { +// return sql`cs_ore_64_8_v1(${left}) < cs_ore_64_8_v1(${bindIfParam(right, left)})`; +// }; + +const csMatch: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { + return sql`cs_match_v1(${left}) @> cs_match_v1(${bindIfParam(right, left)})`; +}; + +const filterInput= await protectClient.encrypt("abc", { + column: 'email', + table: 'users', +}) + +const query = db .select({ email: users.email_encrypted, }) .from(users) + .where(csMatch(users.email_encrypted, filterInput)) -const sqlResult = sql.toSQL() +const sqlResult = query.toSQL() console.log('[INFO] SQL statement:', sqlResult) -const data = await sql.execute() +const data = await query.execute() const emails = await Promise.all( data.map( From 932e06b0e60da7bc6e371c1a80bceecad372fc56 Mon Sep 17 00:00:00 2001 From: Drew Thomas Date: Tue, 4 Mar 2025 10:51:27 +1100 Subject: [PATCH 07/30] Working search example with equality on email --- apps/drizzle/src/select.ts | 44 ++++++++++++++++++++++++++++++-------- 1 file changed, 35 insertions(+), 9 deletions(-) diff --git a/apps/drizzle/src/select.ts b/apps/drizzle/src/select.ts index b5e236136..5b8ea3a9f 100644 --- a/apps/drizzle/src/select.ts +++ b/apps/drizzle/src/select.ts @@ -4,10 +4,32 @@ import { users } from './db/schema' import { protectClient } from './protect' import { bindIfParam, sql } from 'drizzle-orm'; import type { BinaryOperator, SQL, SQLWrapper } from 'drizzle-orm'; +import { parseArgs } from 'node:util' -// const csEq: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { -// return sql`cs_unique_v1(${left}) = cs_unique_v1(${bindIfParam(right, left)})`; -// }; +const getEmail = () => { + const { values, positionals } = parseArgs({ + args: process.argv, + options: { + email: { + type: 'string', + }, + }, + strict: true, + allowPositionals: true, + }) + + return values.email +} + +const email = getEmail() + +if (!email) { + throw new Error('Email is required') +} + +const csEq: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { + return sql`cs_unique_v1(${left}) = cs_unique_v1(${bindIfParam(right, left)})`; +}; // const csGt: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { // return sql`cs_ore_64_8_v1(${left}) > cs_ore_64_8_v1(${bindIfParam(right, left)})`; @@ -17,21 +39,25 @@ import type { BinaryOperator, SQL, SQLWrapper } from 'drizzle-orm'; // return sql`cs_ore_64_8_v1(${left}) < cs_ore_64_8_v1(${bindIfParam(right, left)})`; // }; -const csMatch: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { - return sql`cs_match_v1(${left}) @> cs_match_v1(${bindIfParam(right, left)})`; -}; +// const csMatch: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { +// return sql`cs_match_v1(${left}) @> cs_match_v1(${bindIfParam(right, left)})`; +// }; -const filterInput= await protectClient.encrypt("abc", { - column: 'email', +const filterInput = await protectClient.encrypt(email, { + column: 'email_encrypted', table: 'users', }) +if (filterInput.failure) { + throw new Error(`[protect]: ${filterInput.failure.message}`) +} + const query = db .select({ email: users.email_encrypted, }) .from(users) - .where(csMatch(users.email_encrypted, filterInput)) + .where(csEq(users.email_encrypted, filterInput.data)) const sqlResult = query.toSQL() console.log('[INFO] SQL statement:', sqlResult) From c70f9d23bfc8d171da1eb5d244cede0098edfe88 Mon Sep 17 00:00:00 2001 From: Drew Thomas Date: Tue, 4 Mar 2025 11:48:02 +1100 Subject: [PATCH 08/30] Add examples of matching and sorting to apps/drizzle --- apps/drizzle/src/protect.ts | 17 ++++++++++++++++- apps/drizzle/src/select.ts | 38 +++++++++++++++++++++++++++---------- 2 files changed, 44 insertions(+), 11 deletions(-) diff --git a/apps/drizzle/src/protect.ts b/apps/drizzle/src/protect.ts index 1e2071ce2..60f627803 100644 --- a/apps/drizzle/src/protect.ts +++ b/apps/drizzle/src/protect.ts @@ -7,7 +7,22 @@ const config: EncryptConfig = { users: { email_encrypted: { cast_as: 'text', - indexes: { ore: {}, match: {}, unique: {} }, + indexes: { + ore: {}, + match: { + tokenizer: { + kind: 'ngram', + token_length: 3 + }, + token_filters: [ + { + kind: 'downcase' + } + ], + k: 6, + m: 2048 + }, + unique: {} }, }, }, }, diff --git a/apps/drizzle/src/select.ts b/apps/drizzle/src/select.ts index 5b8ea3a9f..86632f906 100644 --- a/apps/drizzle/src/select.ts +++ b/apps/drizzle/src/select.ts @@ -6,27 +6,42 @@ import { bindIfParam, sql } from 'drizzle-orm'; import type { BinaryOperator, SQL, SQLWrapper } from 'drizzle-orm'; import { parseArgs } from 'node:util' -const getEmail = () => { +const getArgs = () => { const { values, positionals } = parseArgs({ args: process.argv, options: { - email: { + filter: { type: 'string', }, + op: { + type: 'string', + default: 'match' + }, }, strict: true, allowPositionals: true, }) - return values.email + return values } -const email = getEmail() +const {filter, op} = getArgs() -if (!email) { +if (!filter) { throw new Error('Email is required') } +const fnForOp: (op: string) => BinaryOperator = op => { + switch (op) { + case 'match': + return csMatch + case 'eq': + return csEq + default: + throw new Error(`unknown op: ${op}`) + } +} + const csEq: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { return sql`cs_unique_v1(${left}) = cs_unique_v1(${bindIfParam(right, left)})`; }; @@ -39,11 +54,11 @@ const csEq: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { // return sql`cs_ore_64_8_v1(${left}) < cs_ore_64_8_v1(${bindIfParam(right, left)})`; // }; -// const csMatch: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { -// return sql`cs_match_v1(${left}) @> cs_match_v1(${bindIfParam(right, left)})`; -// }; +const csMatch: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { + return sql`cs_match_v1(${left}) @> cs_match_v1(${bindIfParam(right, left)})`; +}; -const filterInput = await protectClient.encrypt(email, { +const filterInput = await protectClient.encrypt(filter, { column: 'email_encrypted', table: 'users', }) @@ -52,12 +67,15 @@ if (filterInput.failure) { throw new Error(`[protect]: ${filterInput.failure.message}`) } +const filterFn = fnForOp(op); + const query = db .select({ email: users.email_encrypted, }) .from(users) - .where(csEq(users.email_encrypted, filterInput.data)) + .where(filterFn(users.email_encrypted, filterInput.data)) + .orderBy(sql`cs_ore_64_8_v1(users.email_encrypted)`) const sqlResult = query.toSQL() console.log('[INFO] SQL statement:', sqlResult) From bd4c6b916d1e5c7272622c0f6dd03839d28f8580 Mon Sep 17 00:00:00 2001 From: Drew Thomas Date: Tue, 4 Mar 2025 11:53:27 +1100 Subject: [PATCH 09/30] Fix up error message in select.ts in Drizzle example --- apps/drizzle/src/select.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/drizzle/src/select.ts b/apps/drizzle/src/select.ts index 86632f906..e19119952 100644 --- a/apps/drizzle/src/select.ts +++ b/apps/drizzle/src/select.ts @@ -28,7 +28,7 @@ const getArgs = () => { const {filter, op} = getArgs() if (!filter) { - throw new Error('Email is required') + throw new Error('filter is required') } const fnForOp: (op: string) => BinaryOperator = op => { From ad827cf195a0bc8f233ea4a0746f0953f5567167 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Tue, 4 Mar 2025 10:11:52 -0700 Subject: [PATCH 10/30] feat(protect): implement a protect schema strategy --- CONTRIBUTE.md | 21 ++ README.md | 85 ++++--- apps/drizzle/src/insert.ts | 7 +- apps/drizzle/src/protect.ts | 36 +-- apps/drizzle/src/select.ts | 26 +-- docs/schema.md | 3 + packages/protect/__tests__/protect.test.ts | 158 +++++++------ packages/protect/package.json | 4 +- packages/protect/src/ffi/encrypt-config.ts | 77 ------- packages/protect/src/ffi/index.ts | 49 ++-- packages/protect/src/ffi/payload-helpers.ts | 2 + packages/protect/src/index.ts | 12 +- packages/protect/src/schema/index.ts | 243 ++++++++++++++++++++ pnpm-lock.yaml | 50 ++-- 14 files changed, 489 insertions(+), 284 deletions(-) create mode 100644 docs/schema.md delete mode 100644 packages/protect/src/ffi/encrypt-config.ts create mode 100644 packages/protect/src/schema/index.ts diff --git a/CONTRIBUTE.md b/CONTRIBUTE.md index 4b48eaa65..9363d06a6 100644 --- a/CONTRIBUTE.md +++ b/CONTRIBUTE.md @@ -98,6 +98,27 @@ We use [**Changesets**](https://github.com/changesets/changesets) to manage vers - Follow the prompts to indicate the type of version bump (patch, minor, major). - The [GitHub Actions](./.github/workflows/) (or other CI pipeline) will handle the **publish** step to npm once your PR is merged and the changeset is committed to `main`. +## Pre release process + +We currently use [changesets to manage pre-releasing](https://github.com/changesets/changesets/blob/main/docs/prereleases.md) the `next` version of the package, and the process is executed manually. + +To do so, you need to: + +1. Check out the `next` branch +2. Run `pnpm changeset pre enter next` +3. Run `pnpm changeset version` +4. Run `git add .` +5. Run `git commit -m "Enter prerelease mode and version packages"` +6. Run `pnpm changeset publish --tag next` +7. Run `git push --follow-tags` + +When you are ready to release, you can run `pnpm changeset pre exit` to exit prerelease mode and commit the changes. +When you merge the PR, the `next` branch will be merged into `main`, and the package will be published to npm without the prerelease tag. + +> [!IMPORTANT] +> This process can be dangerous, so please be careful when using it as it's difficult to undo mistakes. +> If you are unfamiliar with the process, please reach out to the maintainers for help. + ## Additional Resources - [Turborepo Documentation](https://turbo.build/repo/docs) diff --git a/README.md b/README.md index 447e48823..d3c4165f1 100644 --- a/README.md +++ b/README.md @@ -117,54 +117,70 @@ At the end of `stash setup`, you will have two files in your project: You can read more about [configuration via toml file or environment variables here](./docs/configuration.md). -### Initializing the EQL client +### Basic file structure -In your application, import the `protect` function and `EncryptConfig` type from the `@cipherstash/protect` package, and initialize a client with your encrypt config. +This is the basic file structure of the project. In the `src/protect` directory, we have table definition in `schema.ts` and the protect client in `index.ts`. -To define your encrypt config, create an object that conforms to the `EncryptConfig` type. -This object determines how data will be encrypted and decrypted, and which searchable indexes will be added. +``` +📦 + ├ 📂 src + │ ├ 📂 protect + │ │ ├ 📜 index.ts + │ │ └ 📜 schema.ts + │ └ 📜 index.ts + ├ 📜 .env + ├ 📜 cipherstash.toml + ├ 📜 cipherstash.secret.toml + ├ 📜 package.json + └ 📜 tsconfig.json +``` -You will need to provide the following information: +### Defining your schema -1. Which **tables** should be encrypted and decrypted. -2. Which **columns** should be encrypted and decrypted. -3. Which **indexes** should be built on the encrypted columns. +Protect.js uses a schema to define the tables and columns that you want to encrypt and decrypt. -An example config might look like this: +In the `src/protect/schema.ts` file, you can define your tables and columns. ```ts -import { protect, type EncryptConfig } from '@cipherstash/protect' - -const config: EncryptConfig = { - v: 1, - tables: { - users: { - email: { - cast_as: 'text', - indexes: { ore: {}, match: {}, unique: {} }, - }, - }, - }, -} +import { csTable, csColumn } from '@cipherstash/protect' + +export const users = csTable('users', { + email: csColumn('email').freeTextSearch().equality().orderAndSort(), +}) + +export const orders = csTable('orders', { + address: csColumn('address').freeTextSearch().orderAndSort(), +}) ``` -Finally, initialize the client with your encrypt config: +Read more about [defining your schema here](./docs/schema.md). + +### Initializing the EQL client + +To initialize the protect client, import the `protect` function and initialize a client with your defined schema. + +In the `src/protect/index.ts` file: ```ts -const protectClient = await protect(config) -``` +import { protect } from '@cipherstash/protect' +import { users } from './schema' -To learn more about the `EncryptConfig` type, please refer to the [Encrypt Config docs](./docs/encrypt-config.md). +// Pass all your tables to the protect function to initialize the client +export const protectClient = await protect(users, orders) +``` ### Encrypting data Use the `encrypt` function to encrypt data. -`encrypt` takes a plaintext string, and an object with the table and column name as parameters. +`encrypt` takes a plaintext string, and an object with the table and column as parameters. ```typescript +import { users } from './protect/schema' +import { protectClient } from './protect' + const encryptResult = await protectClient.encrypt('secret@squirrel.example', { - column: 'email', - table: 'users', + column: users.email, + table: users, }) if (encryptResult.failure) { @@ -203,6 +219,8 @@ Use the `decrypt` function to decrypt data. `decrypt` takes an encrypted data object, and an object with the lock context as parameters. ```typescript +import { protectClient } from './protect' + const decryptResult = await protectClient.decrypt(ciphertext) if (decryptResult.failure) { @@ -293,9 +311,12 @@ const lockContext = identifyResult.data To encrypt data with a lock context, call the optional `withLockContext` method on the `encrypt` function and pass the lock context object as a parameter: ```typescript +import { protectClient } from './protect' +import { users } from './protect/schema' + const encryptResult = await protectClient.encrypt('plaintext', { - table: 'users', - column: 'email', + table: users, + column: users.email, }).withLockContext(lockContext) if (encryptResult.failure) { @@ -310,6 +331,8 @@ const ciphertext = encryptResult.data To decrypt data with a lock context, call the optional `withLockContext` method on the `decrypt` function and pass the lock context object as a parameter: ```typescript +import { protectClient } from './protect' + const decryptResult = await protectClient.decrypt(ciphertext).withLockContext(lockContext) if (decryptResult.failure) { diff --git a/apps/drizzle/src/insert.ts b/apps/drizzle/src/insert.ts index b39efbe91..2f3bd1945 100644 --- a/apps/drizzle/src/insert.ts +++ b/apps/drizzle/src/insert.ts @@ -1,9 +1,8 @@ import 'dotenv/config' import { parseArgs } from 'node:util' -import { getTableName } from 'drizzle-orm' import { db } from './db' import { users } from './db/schema' -import { protectClient } from './protect' +import { protectClient, users as protectUsers } from './protect' const getEmail = () => { const { values, positionals } = parseArgs({ @@ -27,8 +26,8 @@ if (!email) { } const encryptedResult = await protectClient.encrypt(email, { - column: users.email_encrypted.name, - table: getTableName(users), + column: protectUsers.email_encrypted, + table: protectUsers, }) if (encryptedResult.failure) { diff --git a/apps/drizzle/src/protect.ts b/apps/drizzle/src/protect.ts index 60f627803..e26d4f935 100644 --- a/apps/drizzle/src/protect.ts +++ b/apps/drizzle/src/protect.ts @@ -1,31 +1,11 @@ import 'dotenv/config' -import { protect, type EncryptConfig } from '@cipherstash/protect' +import { protect, csColumn, csTable } from '@cipherstash/protect' -const config: EncryptConfig = { - v: 1, - tables: { - users: { - email_encrypted: { - cast_as: 'text', - indexes: { - ore: {}, - match: { - tokenizer: { - kind: 'ngram', - token_length: 3 - }, - token_filters: [ - { - kind: 'downcase' - } - ], - k: 6, - m: 2048 - }, - unique: {} }, - }, - }, - }, -} +export const users = csTable('users', { + email_encrypted: csColumn('email_encrypted') + .equality() + .orderAndSort() + .freeTextSearch(), +}) -export const protectClient = await protect(config) +export const protectClient = await protect(users) diff --git a/apps/drizzle/src/select.ts b/apps/drizzle/src/select.ts index e19119952..c26f7335a 100644 --- a/apps/drizzle/src/select.ts +++ b/apps/drizzle/src/select.ts @@ -1,9 +1,9 @@ import 'dotenv/config' import { db } from './db' import { users } from './db/schema' -import { protectClient } from './protect' -import { bindIfParam, sql } from 'drizzle-orm'; -import type { BinaryOperator, SQL, SQLWrapper } from 'drizzle-orm'; +import { protectClient, users as protectUsers } from './protect' +import { bindIfParam, sql } from 'drizzle-orm' +import type { BinaryOperator, SQL, SQLWrapper } from 'drizzle-orm' import { parseArgs } from 'node:util' const getArgs = () => { @@ -15,7 +15,7 @@ const getArgs = () => { }, op: { type: 'string', - default: 'match' + default: 'match', }, }, strict: true, @@ -25,13 +25,13 @@ const getArgs = () => { return values } -const {filter, op} = getArgs() +const { filter, op } = getArgs() if (!filter) { throw new Error('filter is required') } -const fnForOp: (op: string) => BinaryOperator = op => { +const fnForOp: (op: string) => BinaryOperator = (op) => { switch (op) { case 'match': return csMatch @@ -43,8 +43,8 @@ const fnForOp: (op: string) => BinaryOperator = op => { } const csEq: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { - return sql`cs_unique_v1(${left}) = cs_unique_v1(${bindIfParam(right, left)})`; -}; + return sql`cs_unique_v1(${left}) = cs_unique_v1(${bindIfParam(right, left)})` +} // const csGt: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { // return sql`cs_ore_64_8_v1(${left}) > cs_ore_64_8_v1(${bindIfParam(right, left)})`; @@ -55,19 +55,19 @@ const csEq: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { // }; const csMatch: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { - return sql`cs_match_v1(${left}) @> cs_match_v1(${bindIfParam(right, left)})`; -}; + return sql`cs_match_v1(${left}) @> cs_match_v1(${bindIfParam(right, left)})` +} const filterInput = await protectClient.encrypt(filter, { - column: 'email_encrypted', - table: 'users', + column: protectUsers.email_encrypted, + table: protectUsers, }) if (filterInput.failure) { throw new Error(`[protect]: ${filterInput.failure.message}`) } -const filterFn = fnForOp(op); +const filterFn = fnForOp(op) const query = db .select({ diff --git a/docs/schema.md b/docs/schema.md new file mode 100644 index 000000000..8f8ead684 --- /dev/null +++ b/docs/schema.md @@ -0,0 +1,3 @@ +# Protect.js Schema + +... \ No newline at end of file diff --git a/packages/protect/__tests__/protect.test.ts b/packages/protect/__tests__/protect.test.ts index 8ba40adb6..04c476833 100644 --- a/packages/protect/__tests__/protect.test.ts +++ b/packages/protect/__tests__/protect.test.ts @@ -1,21 +1,19 @@ import 'dotenv/config' import { describe, expect, it } from 'vitest' -import { LockContext, createEqlPayload, getPlaintext, protect } from '../src' +import { + LockContext, + createEqlPayload, + getPlaintext, + protect, + csTable, + csColumn, +} from '../src' import type { CsPlaintextV1Schema } from '../src/cs_plaintext_v1' -import type { EncryptConfig } from '../src/ffi/encrypt-config' - -const config: EncryptConfig = { - v: 1, - tables: { - users: { - email: { - cast_as: 'text', - indexes: { ore: {}, match: {}, unique: {} }, - }, - }, - }, -} + +const users = csTable('users', { + email: csColumn('email').freeTextSearch().equality().orderAndSort(), +}) describe('createEqlPayload', () => { it('should create a payload with the correct default values', () => { @@ -126,11 +124,11 @@ describe('getPlaintext', () => { describe('encryption and decryption', () => { it('should encrypt and decrypt a payload', async () => { - const protectClient = await protect(config) + const protectClient = await protect(users) const ciphertext = await protectClient.encrypt('plaintext', { - column: 'email', - table: 'users', + column: users.email, + table: users, }) if (ciphertext.failure) { @@ -145,11 +143,11 @@ describe('encryption and decryption', () => { }, 30000) it('should return null if plaintext is null', async () => { - const protectClient = await protect(config) + const protectClient = await protect(users) const ciphertext = await protectClient.encrypt(null, { - column: 'email', - table: 'users', + column: users.email, + table: users, }) console.log('ciphertext', ciphertext) @@ -168,67 +166,67 @@ describe('encryption and decryption', () => { }, 30000) }) -// TODO: Searchable encryption is not yet supported for bukl ops -// describe('bulk encryption', () => { -// it('should bulk encrypt and decrypt a payload', async () => { -// const protectClient = await protect() -// const ciphertexts = await protectClient.bulkEncrypt( -// [ -// { -// plaintext: 'test', -// id: '1', -// }, -// { -// plaintext: 'test2', -// id: '2', -// }, -// ], -// { -// table: 'users', -// column: 'column_name', -// }, -// ) - -// if (ciphertexts.failure) { -// throw new Error(`[protect]: ${ciphertexts.failure.message}`) -// } - -// const plaintexts = await protectClient.bulkDecrypt(ciphertexts.data) - -// expect(plaintexts).toEqual({ -// data: [ -// { -// plaintext: 'test', -// id: '1', -// }, -// { -// plaintext: 'test2', -// id: '2', -// }, -// ], -// }) -// }, 30000) - -// it('should return null if plaintexts is empty', async () => { -// const protectClient = await protect() -// const ciphertexts = await protectClient.bulkEncrypt([], { -// table: 'users', -// column: 'column_name', -// }) -// expect(ciphertexts).toEqual({ -// data: null, -// }) -// }, 30000) +// TODO: Still have some issues with bulk encryption +describe('bulk encryption', () => { + // it('should bulk encrypt and decrypt a payload', async () => { + // const protectClient = await protect(users) + // const ciphertexts = await protectClient.bulkEncrypt( + // [ + // { + // plaintext: 'test', + // id: '1', + // }, + // { + // plaintext: 'test2', + // id: '2', + // }, + // ], + // { + // table: users, + // column: users.email, + // }, + // ) + + // if (ciphertexts.failure) { + // throw new Error(`[protect]: ${ciphertexts.failure.message}`) + // } + + // const plaintexts = await protectClient.bulkDecrypt(ciphertexts.data) + + // expect(plaintexts).toEqual({ + // data: [ + // { + // plaintext: 'test', + // id: '1', + // }, + // { + // plaintext: 'test2', + // id: '2', + // }, + // ], + // }) + // }, 30000) + + it('should return null if plaintexts is empty', async () => { + const protectClient = await protect(users) + const ciphertexts = await protectClient.bulkEncrypt([], { + table: users, + column: users.email, + }) + expect(ciphertexts).toEqual({ + data: null, + }) + }, 30000) -// it('should return null if decrypting empty ciphertexts', async () => { -// const protectClient = await protect() -// const ciphertexts = null -// const plaintexts = await protectClient.bulkDecrypt(ciphertexts) -// expect(plaintexts).toEqual({ -// data: null, -// }) -// }, 30000) -// }) + it('should return null if decrypting empty ciphertexts', async () => { + const protectClient = await protect(users) + const ciphertexts = null + const plaintexts = await protectClient.bulkDecrypt(ciphertexts) + expect(plaintexts).toEqual({ + data: null, + }) + }, 30000) +}) // ------------------------ // TODO get bulk Encryption/Decryption working in CI. diff --git a/packages/protect/package.json b/packages/protect/package.json index 7514808e9..5e4a85161 100644 --- a/packages/protect/package.json +++ b/packages/protect/package.json @@ -1,7 +1,7 @@ { "name": "@cipherstash/protect", "version": "6.0.0", - "description": "CipherStash Protech for JavaScript", + "description": "CipherStash Protect for JavaScript", "keywords": [ "encrypted", "query", @@ -55,7 +55,7 @@ }, "dependencies": { "@byteslice/result": "^0.2.0", - "@cipherstash/protect-ffi": "0.12.0-1", + "@cipherstash/protect-ffi": "0.12.0-3", "zod": "^3.24.2" }, "optionalDependencies": { diff --git a/packages/protect/src/ffi/encrypt-config.ts b/packages/protect/src/ffi/encrypt-config.ts deleted file mode 100644 index c70b30266..000000000 --- a/packages/protect/src/ffi/encrypt-config.ts +++ /dev/null @@ -1,77 +0,0 @@ -import { z } from 'zod' - -const castAsEnum = z - .enum([ - 'big_int', - 'boolean', - 'date', - 'real', - 'double', - 'int', - 'small_int', - 'text', - 'jsonb', - ]) - .default('text') - -const tokenFilterSchema = z.object({ - kind: z.literal('downcase'), -}) - -const tokenizerSchema = z - .union([ - z.object({ - kind: z.literal('standard'), - }), - z.object({ - kind: z.literal('ngram'), - token_length: z.number(), - }), - ]) - .default({ kind: 'standard' }) - .optional() - -const oreIndexOptsSchema = z.object({}) - -const uniqueIndexOptsSchema = z.object({ - token_filters: z.array(tokenFilterSchema).default([]).optional(), -}) - -const matchIndexOptsSchema = z.object({ - tokenizer: tokenizerSchema, - token_filters: z.array(tokenFilterSchema).default([]).optional(), - k: z.number().default(6).optional(), - m: z.number().default(2048).optional(), - include_original: z.boolean().default(false).optional(), -}) - -const steVecIndexOptsSchema = z.object({ - prefix: z.string(), -}) - -const indexesSchema = z - .object({ - ore: oreIndexOptsSchema.optional(), - unique: uniqueIndexOptsSchema.optional(), - match: matchIndexOptsSchema.optional(), - ste_vec: steVecIndexOptsSchema.optional(), - }) - .default({}) - -const columnSchema = z - .object({ - cast_as: castAsEnum, - indexes: indexesSchema, - }) - .default({}) - -const tableSchema = z.record(columnSchema).default({}) - -const tablesSchema = z.record(tableSchema).default({}) - -export const encryptConfigSchema = z.object({ - v: z.number(), - tables: tablesSchema, -}) - -export type EncryptConfig = z.infer diff --git a/packages/protect/src/ffi/index.ts b/packages/protect/src/ffi/index.ts index 3b7e1c707..4a4cb7c55 100644 --- a/packages/protect/src/ffi/index.ts +++ b/packages/protect/src/ffi/index.ts @@ -16,7 +16,13 @@ import { normalizeBulkEncryptPayloads, normalizeBulkEncryptPayloadsWithLockContext, } from './payload-helpers' -import { type EncryptConfig, encryptConfigSchema } from './encrypt-config' +import { + type EncryptConfig, + encryptConfigSchema, + type ProtectTable, + type ProtectColumn, + type ProtectTableColumn, +} from '../schema' // ------------------------ // Type Definitions @@ -45,8 +51,8 @@ export type BulkDecryptedData = | null export type EncryptOptions = { - column: string - table: string + column: ProtectColumn + table: ProtectTable> } type Client = Awaited> | undefined @@ -67,8 +73,8 @@ class EncryptOperation { private client: Client private plaintext: EncryptPayload - private column: string - private table: string + private column: ProtectColumn + private table: ProtectTable constructor(client: Client, plaintext: EncryptPayload, opts: EncryptOptions) { this.client = client @@ -102,8 +108,8 @@ class EncryptOperation /** Actual encryption logic, deferred until `then()` is called. */ private async execute(): Promise> { logger.debug('Encrypting data WITHOUT a lock context', { - column: this.column, - table: this.table, + column: this.column.getName(), + table: this.table.tableName, }) return await withResult( @@ -119,8 +125,8 @@ class EncryptOperation const val = await ffiEncrypt( this.client, this.plaintext, - this.column, - this.table, + this.column.getName(), + this.table.tableName, ) return JSON.parse(val) @@ -135,8 +141,8 @@ class EncryptOperation public getOperation(): { client: Client plaintext: EncryptPayload - column: string - table: string + column: ProtectColumn + table: ProtectTable } { return { client: this.client, @@ -201,8 +207,8 @@ class EncryptOperationWithLockContext const val = await ffiEncrypt( client, plaintext, - column, - table, + column.getName(), + table.tableName, context.data.context, context.data.ctsToken, ) @@ -347,8 +353,8 @@ class BulkEncryptOperation { private client: Client private plaintexts: BulkEncryptPayload - private column: string - private table: string + private column: ProtectColumn + private table: ProtectTable constructor( client: Client, @@ -395,12 +401,12 @@ class BulkEncryptOperation const encryptPayloads = normalizeBulkEncryptPayloads( this.plaintexts, - this.column, + this.column.getName(), ) logger.debug('Bulk encrypting data WITHOUT a lock context', { - column: this.column, - table: this.table, + column: this.column.getName(), + table: this.table.tableName, }) const encryptedData = await ffiEncryptBulk(this.client, encryptPayloads) @@ -419,8 +425,8 @@ class BulkEncryptOperation public getOperation(): { client: Client plaintexts: BulkEncryptPayload - column: string - table: string + column: ProtectColumn + table: ProtectTable } { return { client: this.client, @@ -474,7 +480,8 @@ class BulkEncryptOperationWithLockContext const encryptPayloads = await normalizeBulkEncryptPayloadsWithLockContext( plaintexts, - column, + column.getName(), + table.tableName, this.lockContext, ) diff --git a/packages/protect/src/ffi/payload-helpers.ts b/packages/protect/src/ffi/payload-helpers.ts index 5dafeceae..8744c680a 100644 --- a/packages/protect/src/ffi/payload-helpers.ts +++ b/packages/protect/src/ffi/payload-helpers.ts @@ -62,6 +62,7 @@ export async function normalizeBulkDecryptPayloadsWithLockContext( export async function normalizeBulkEncryptPayloadsWithLockContext( plaintexts: BulkEncryptPayload, column: string, + table: string, lockContext: LockContext, ): Promise> { const lockContextPayload = await getLockContextPayload(lockContext) @@ -74,6 +75,7 @@ export async function normalizeBulkEncryptPayloadsWithLockContext( const payload = { plaintext: plaintext.plaintext, column, + table, ...lockContextPayload, } diff --git a/packages/protect/src/index.ts b/packages/protect/src/index.ts index ee8047b4e..87c7d0a6b 100644 --- a/packages/protect/src/index.ts +++ b/packages/protect/src/index.ts @@ -1,5 +1,9 @@ import { ProtectClient } from './ffi' -import type { EncryptConfig } from './ffi/encrypt-config' +import { + type ProtectTable, + type ProtectTableColumn, + buildEncryptConfig, +} from './schema' export const ProtectErrorTypes = { ClientInitError: 'ClientInitError', @@ -15,9 +19,11 @@ export interface ProtectError { } export const protect = async ( - encryptConfig?: EncryptConfig, + ...tables: Array> ): Promise => { const client = new ProtectClient() + const encryptConfig = buildEncryptConfig(...tables) + const result = await client.init(encryptConfig) if (result.failure) { @@ -28,8 +34,8 @@ export const protect = async ( } export type { Result } from '@byteslice/result' -export type { EncryptConfig } from './ffi/encrypt-config' export type { ProtectClient } from './ffi' +export { csTable, csColumn } from './schema' export * from './cs_plaintext_v1' export * from './identify' export * from './eql' diff --git a/packages/protect/src/schema/index.ts b/packages/protect/src/schema/index.ts new file mode 100644 index 000000000..36624ac07 --- /dev/null +++ b/packages/protect/src/schema/index.ts @@ -0,0 +1,243 @@ +import { z } from 'zod' + +// ------------------------ +// Zod schemas +// ------------------------ +const castAsEnum = z + .enum([ + 'big_int', + 'boolean', + 'date', + 'real', + 'double', + 'int', + 'small_int', + 'text', + 'jsonb', + ]) + .default('text') + +const tokenFilterSchema = z.object({ + kind: z.literal('downcase'), +}) + +const tokenizerSchema = z + .union([ + z.object({ + kind: z.literal('standard'), + }), + z.object({ + kind: z.literal('ngram'), + token_length: z.number(), + }), + ]) + .default({ kind: 'standard' }) + .optional() + +const oreIndexOptsSchema = z.object({}) + +const uniqueIndexOptsSchema = z.object({ + token_filters: z.array(tokenFilterSchema).default([]).optional(), +}) + +const matchIndexOptsSchema = z.object({ + tokenizer: tokenizerSchema, + token_filters: z.array(tokenFilterSchema).default([]).optional(), + k: z.number().default(6).optional(), + m: z.number().default(2048).optional(), + include_original: z.boolean().default(false).optional(), +}) + +const steVecIndexOptsSchema = z.object({ + prefix: z.string(), +}) + +const indexesSchema = z + .object({ + ore: oreIndexOptsSchema.optional(), + unique: uniqueIndexOptsSchema.optional(), + match: matchIndexOptsSchema.optional(), + ste_vec: steVecIndexOptsSchema.optional(), + }) + .default({}) + +const columnSchema = z + .object({ + cast_as: castAsEnum, + indexes: indexesSchema, + }) + .default({}) + +const tableSchema = z.record(columnSchema).default({}) + +const tablesSchema = z.record(tableSchema).default({}) + +export const encryptConfigSchema = z.object({ + v: z.number(), + tables: tablesSchema, +}) + +// ------------------------ +// Type definitions +// ------------------------ +type CastAs = z.infer +type TokenFilter = z.infer +type MatchIndexOpts = z.infer +type SteVecIndexOpts = z.infer +type UniqueIndexOpts = z.infer +type OreIndexOpts = z.infer +type ColumnSchema = z.infer + +export type ProtectTableColumn = Record +export type EncryptConfig = z.infer + +// ------------------------ +// Interface definitions +// ------------------------ +export class ProtectColumn { + private columnName: string + private castAsValue: CastAs + private indexesValue: { + ore?: OreIndexOpts + unique?: UniqueIndexOpts + match?: Required + ste_vec?: SteVecIndexOpts + } = {} + + constructor(columnName: string) { + this.columnName = columnName + this.castAsValue = 'text' + } + + /** + * Set or override the cast_as value. + */ + dataType(castAs: CastAs) { + this.castAsValue = castAs + return this + } + + /** + * Enable ORE indexing (Order-Revealing Encryption). + */ + orderAndSort() { + this.indexesValue.ore = {} + return this + } + + /** + * Enable a Unique index. Optionally pass tokenFilters. + */ + equality(tokenFilters?: TokenFilter[]) { + this.indexesValue.unique = { + token_filters: tokenFilters ?? [], + } + return this + } + + /** + * Enable a Match index. Allows passing of custom match options. + */ + freeTextSearch(opts?: MatchIndexOpts) { + // Provide defaults + this.indexesValue.match = { + tokenizer: opts?.tokenizer ?? { kind: 'ngram', token_length: 3 }, + token_filters: opts?.token_filters ?? [ + { + kind: 'downcase', + }, + ], + k: opts?.k ?? 6, + m: opts?.m ?? 2048, + include_original: opts?.include_original ?? true, + } + return this + } + + /** + * Enable a STE Vec index, requires a prefix. + * TODO: Determine a better way to describe ste_vec + */ + steVec(prefix: string) { + this.indexesValue.ste_vec = { prefix } + return this + } + + build() { + return { + cast_as: this.castAsValue, + indexes: this.indexesValue, + } + } + + getName() { + return this.columnName + } +} + +interface TableDefinition { + tableName: string + columns: Record +} + +export class ProtectTable { + constructor( + public readonly tableName: string, + private readonly columnBuilders: T, + ) {} + + /** + * Build a TableDefinition object: tableName + built column configs. + */ + build(): TableDefinition { + const builtColumns: Record = {} + for (const [colName, builder] of Object.entries(this.columnBuilders)) { + builtColumns[colName] = builder.build() + } + + return { + tableName: this.tableName, + columns: builtColumns, + } + } +} + +// ------------------------ +// User facing functions +// ------------------------ +export function csTable( + tableName: string, + columns: T, +): ProtectTable & T { + const tableBuilder = new ProtectTable(tableName, columns) as ProtectTable & + T + + for (const [colName, colBuilder] of Object.entries(columns)) { + ;(tableBuilder as ProtectTableColumn)[colName] = colBuilder + } + + return tableBuilder +} + +export function csColumn(columnName: string) { + return new ProtectColumn(columnName) +} + +// ------------------------ +// Internal functions +// ------------------------ +export function buildEncryptConfig( + ...protectTables: Array> +): EncryptConfig { + const config: EncryptConfig = { + v: 1, + tables: {}, + } + + for (const tb of protectTables) { + const tableDef = tb.build() + config.tables[tableDef.tableName] = tableDef.columns + } + + return config +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 62298a3a4..0c79fed6d 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -215,8 +215,8 @@ importers: specifier: ^0.2.0 version: 0.2.0 '@cipherstash/protect-ffi': - specifier: 0.12.0-1 - version: 0.12.0-1 + specifier: 0.12.0-3 + version: 0.12.0-3 typescript: specifier: ^5.0.0 version: 5.7.2 @@ -369,33 +369,33 @@ packages: '@changesets/write@0.3.2': resolution: {integrity: sha512-kDxDrPNpUgsjDbWBvUo27PzKX4gqeKOlhibaOXDJA6kuBisGqNHv/HwGJrAu8U/dSf8ZEFIeHIPtvSlZI1kULw==} - '@cipherstash/protect-ffi-darwin-arm64@0.12.0-1': - resolution: {integrity: sha512-PY9A5+wyqRi1imAjPq8TvpMyb9z/eU/dnZ1TpRhAJeaQK3wLfOZH4r+A/IDR41HYBjPkRVKi9/IK2/c0ebAxqA==} + '@cipherstash/protect-ffi-darwin-arm64@0.12.0-3': + resolution: {integrity: sha512-LRrNEecK54EtIHcyDDywiksAMTKZfXb012xAeiHP0/6KrvUI6SpNyoz1HW/cDTbVBmc4j5sZru/LDv7qAU7Tmw==} cpu: [arm64] os: [darwin] - '@cipherstash/protect-ffi-darwin-x64@0.12.0-1': - resolution: {integrity: sha512-V0QjUyYj8o0u9+FJjNw6Loa3NQdp8+fpPy0ODFszH15NntAblhjRrI2B6OK/Ou+3jdD/FOuC4J1L+QHIDg2wsw==} + '@cipherstash/protect-ffi-darwin-x64@0.12.0-3': + resolution: {integrity: sha512-eGCGchrZMkKcrwluXl787QP5d0GzmTP61dRDs3PyhYaMTgHUHlSIm/egVF1Tea+ru1D72KDajVSt6ApwgT77nA==} cpu: [x64] os: [darwin] - '@cipherstash/protect-ffi-linux-arm64-gnu@0.12.0-1': - resolution: {integrity: sha512-fcI+tFjFDJWb2v9YyhkEFkV45JHZo9iB5+zYjXcfuCT2+3s3U0rlV4lgrgW6v1h8mqEf5yShc7DY/oO4gzEmMg==} + '@cipherstash/protect-ffi-linux-arm64-gnu@0.12.0-3': + resolution: {integrity: sha512-RqkfgxPqLmoAjduryQPpnahDUH8fme0iUBoqswdVat2LHCim0BSV/UwpcffAiuHDtnMARnilRBN0cFT8WsmQBQ==} cpu: [arm64] os: [linux] - '@cipherstash/protect-ffi-linux-x64-gnu@0.12.0-1': - resolution: {integrity: sha512-7xrV1f2aNrZh1/x+e4Oq4ib7wwmDYY9DIYdWGG63FBx+Z9ZXa8CRgME9xor//w8fGuvoKYX6b2HR0Rqc4OeUAg==} + '@cipherstash/protect-ffi-linux-x64-gnu@0.12.0-3': + resolution: {integrity: sha512-p5VrXNNFq8y0ka4HaAvm/NIPKLPjdMnCXCkZccJ7iX6lR4oQlVxZeXcrnv9JnBm4ZkDx+Z0grZtsHUH+vX2uRg==} cpu: [x64] os: [linux] - '@cipherstash/protect-ffi-win32-x64-msvc@0.12.0-1': - resolution: {integrity: sha512-n8t1OWvlqQUD7dDuI/NGZi2xlQD0sfJSH6K3b7HqVAF8AMSvWmwox1kZmEhVz7/YuGz6+nOjCN2sSp1aCAR+Yg==} + '@cipherstash/protect-ffi-win32-x64-msvc@0.12.0-3': + resolution: {integrity: sha512-JKPFawWDLqeJ2rFCJerIW3K4SYj55QiR3TOaTImSYaetOHGA19Mt8eaBudNlA6GILkVLH0UmBX9VEOUvqd0wYg==} cpu: [x64] os: [win32] - '@cipherstash/protect-ffi@0.12.0-1': - resolution: {integrity: sha512-WbOwotCRHRd8BtqQSdE5CxWiUt3+8CQPntDKq8L1iLGswThLD9J6vEqIiCifW2gChiooXxcbeOvigFmY5LaK+w==} + '@cipherstash/protect-ffi@0.12.0-3': + resolution: {integrity: sha512-AnjZNKDXpOPvEKQYAFwERXFY/uL/hPrrn8t2yY/oQwdQUNay3v0PSc6f7q1iEP1T24J7tvyf0PtIIX/oQJVtqA==} '@clerk/backend@1.21.5': resolution: {integrity: sha512-fZ+HuHQkPngYp9vAEqsJ1XJkhWjSWTin90UixeV3jNrbuPY2jd70LKYTAIEvggnnG42pYxdVupaQDUfqj0dy4Q==} @@ -3846,30 +3846,30 @@ snapshots: human-id: 1.0.2 prettier: 2.8.8 - '@cipherstash/protect-ffi-darwin-arm64@0.12.0-1': + '@cipherstash/protect-ffi-darwin-arm64@0.12.0-3': optional: true - '@cipherstash/protect-ffi-darwin-x64@0.12.0-1': + '@cipherstash/protect-ffi-darwin-x64@0.12.0-3': optional: true - '@cipherstash/protect-ffi-linux-arm64-gnu@0.12.0-1': + '@cipherstash/protect-ffi-linux-arm64-gnu@0.12.0-3': optional: true - '@cipherstash/protect-ffi-linux-x64-gnu@0.12.0-1': + '@cipherstash/protect-ffi-linux-x64-gnu@0.12.0-3': optional: true - '@cipherstash/protect-ffi-win32-x64-msvc@0.12.0-1': + '@cipherstash/protect-ffi-win32-x64-msvc@0.12.0-3': optional: true - '@cipherstash/protect-ffi@0.12.0-1': + '@cipherstash/protect-ffi@0.12.0-3': dependencies: '@neon-rs/load': 0.1.82 optionalDependencies: - '@cipherstash/protect-ffi-darwin-arm64': 0.12.0-1 - '@cipherstash/protect-ffi-darwin-x64': 0.12.0-1 - '@cipherstash/protect-ffi-linux-arm64-gnu': 0.12.0-1 - '@cipherstash/protect-ffi-linux-x64-gnu': 0.12.0-1 - '@cipherstash/protect-ffi-win32-x64-msvc': 0.12.0-1 + '@cipherstash/protect-ffi-darwin-arm64': 0.12.0-3 + '@cipherstash/protect-ffi-darwin-x64': 0.12.0-3 + '@cipherstash/protect-ffi-linux-arm64-gnu': 0.12.0-3 + '@cipherstash/protect-ffi-linux-x64-gnu': 0.12.0-3 + '@cipherstash/protect-ffi-win32-x64-msvc': 0.12.0-3 '@clerk/backend@1.21.5(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': dependencies: From c2d2f9b92782383f81a312fc1acad7e698a80f8f Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Tue, 4 Mar 2025 12:27:01 -0700 Subject: [PATCH 11/30] fix(protect): bulk encryption with schema --- packages/protect/__tests__/protect.test.ts | 84 ++++++++++----------- packages/protect/src/ffi/index.ts | 9 ++- packages/protect/src/ffi/payload-helpers.ts | 10 ++- 3 files changed, 54 insertions(+), 49 deletions(-) diff --git a/packages/protect/__tests__/protect.test.ts b/packages/protect/__tests__/protect.test.ts index 04c476833..e483cb96a 100644 --- a/packages/protect/__tests__/protect.test.ts +++ b/packages/protect/__tests__/protect.test.ts @@ -150,16 +150,12 @@ describe('encryption and decryption', () => { table: users, }) - console.log('ciphertext', ciphertext) - if (ciphertext.failure) { throw new Error(`[protect]: ${ciphertext.failure.message}`) } const plaintext = await protectClient.decrypt(ciphertext.data) - console.log('plaintext', plaintext) - expect(plaintext).toEqual({ data: null, }) @@ -168,44 +164,48 @@ describe('encryption and decryption', () => { // TODO: Still have some issues with bulk encryption describe('bulk encryption', () => { - // it('should bulk encrypt and decrypt a payload', async () => { - // const protectClient = await protect(users) - // const ciphertexts = await protectClient.bulkEncrypt( - // [ - // { - // plaintext: 'test', - // id: '1', - // }, - // { - // plaintext: 'test2', - // id: '2', - // }, - // ], - // { - // table: users, - // column: users.email, - // }, - // ) - - // if (ciphertexts.failure) { - // throw new Error(`[protect]: ${ciphertexts.failure.message}`) - // } - - // const plaintexts = await protectClient.bulkDecrypt(ciphertexts.data) - - // expect(plaintexts).toEqual({ - // data: [ - // { - // plaintext: 'test', - // id: '1', - // }, - // { - // plaintext: 'test2', - // id: '2', - // }, - // ], - // }) - // }, 30000) + it('should bulk encrypt and decrypt a payload', async () => { + const protectClient = await protect(users) + const ciphertexts = await protectClient.bulkEncrypt( + [ + { + plaintext: 'test', + id: '1', + }, + { + plaintext: 'test2', + id: '2', + }, + ], + { + table: users, + column: users.email, + }, + ) + + if (ciphertexts.failure) { + throw new Error(`[protect]: ${ciphertexts.failure.message}`) + } + + const plaintextResult = await protectClient.bulkDecrypt(ciphertexts.data) + + if (plaintextResult.failure) { + throw new Error(`[protect]: ${plaintextResult.failure.message}`) + } + + const plaintexts = plaintextResult.data + + expect(plaintexts).toEqual([ + { + plaintext: 'test', + id: '1', + }, + { + plaintext: 'test2', + id: '2', + }, + ]) + }, 30000) it('should return null if plaintexts is empty', async () => { const protectClient = await protect(users) diff --git a/packages/protect/src/ffi/index.ts b/packages/protect/src/ffi/index.ts index 4a4cb7c55..f6f4f6204 100644 --- a/packages/protect/src/ffi/index.ts +++ b/packages/protect/src/ffi/index.ts @@ -38,7 +38,9 @@ export type BulkEncryptPayload = { export type BulkEncryptedData = | { - c: string + encryptedPayload: { + c: string + } id: string }[] | null @@ -402,6 +404,7 @@ class BulkEncryptOperation const encryptPayloads = normalizeBulkEncryptPayloads( this.plaintexts, this.column.getName(), + this.table.tableName, ) logger.debug('Bulk encrypting data WITHOUT a lock context', { @@ -411,7 +414,7 @@ class BulkEncryptOperation const encryptedData = await ffiEncryptBulk(this.client, encryptPayloads) return encryptedData.map((enc, index) => ({ - c: enc, + encryptedPayload: JSON.parse(enc), id: this.plaintexts[index].id, })) }, @@ -507,7 +510,7 @@ class BulkEncryptOperationWithLockContext ) return encryptedData.map((enc, index) => ({ - c: enc, + encryptedPayload: JSON.parse(enc), id: plaintexts[index].id, })) }, diff --git a/packages/protect/src/ffi/payload-helpers.ts b/packages/protect/src/ffi/payload-helpers.ts index 8744c680a..34ce166d6 100644 --- a/packages/protect/src/ffi/payload-helpers.ts +++ b/packages/protect/src/ffi/payload-helpers.ts @@ -14,9 +14,9 @@ const getLockContextPayload = async (lockContext: LockContext) => export const normalizeBulkDecryptPayloads = ( encryptedPayloads: BulkEncryptedData, ) => - encryptedPayloads?.reduce((acc, encryptedPayload) => { + encryptedPayloads?.reduce((acc, data) => { const payload = { - ciphertext: encryptedPayload.c, + ciphertext: data.encryptedPayload.c, } acc.push(payload) @@ -26,11 +26,13 @@ export const normalizeBulkDecryptPayloads = ( export const normalizeBulkEncryptPayloads = ( plaintexts: BulkEncryptPayload, column: string, + table: string, ) => plaintexts.reduce((acc, plaintext) => { const payload = { plaintext: plaintext.plaintext, column, + table, } acc.push(payload) @@ -47,9 +49,9 @@ export async function normalizeBulkDecryptPayloadsWithLockContext( if (!encryptedPayloads) return { data: [] } return { - data: encryptedPayloads?.reduce((acc, encryptedPayload) => { + data: encryptedPayloads?.reduce((acc, data) => { const payload = { - ciphertext: encryptedPayload.c, + ciphertext: data.encryptedPayload.c, ...lockContextPayload, } From 91338619eec27786d142ed56ec73d9221149eb64 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Tue, 4 Mar 2025 12:43:21 -0700 Subject: [PATCH 12/30] Enter prerelease mode and version packages --- .changeset/pre.json | 11 +++++++++++ README.md | 8 ++------ docker-compose.yml | 2 +- 3 files changed, 14 insertions(+), 7 deletions(-) create mode 100644 .changeset/pre.json diff --git a/.changeset/pre.json b/.changeset/pre.json new file mode 100644 index 000000000..1fec061d8 --- /dev/null +++ b/.changeset/pre.json @@ -0,0 +1,11 @@ +{ + "mode": "pre", + "tag": "next", + "initialVersions": { + "@cipherstash/basic-example": "1.0.3", + "@cipherstash/nextjs-clerk-example": "0.0.1", + "@cipherstash/nextjs": "3.0.0", + "@cipherstash/protect": "6.0.0" + }, + "changesets": [] +} diff --git a/README.md b/README.md index d3c4165f1..2b4d1061b 100644 --- a/README.md +++ b/README.md @@ -465,12 +465,8 @@ Until support for other data types are available in `@cipherstash/protect`, you ## Searchable encryption -`@cipherstash/protect` does not currently support searching encrypted data. -Searchable encryption is an extremely well supported capability in other CipherStash products, and will be coming to `@cipherstash/protect`. -Until searchable encryption support is released in `@cipherstash/protect`, you can: - -- Read [about how searchable encryption works in EQL](https://github.com/cipherstash/encrypt-query-language) -- Express interest in this feature by adding a :+1: on this [GitHub Issue](https://github.com/cipherstash/protectjs/issues/46). +The searchable encryption documentation is in the works and will be available soon. +Express interest in this feature by adding a :+1: on this [GitHub Issue](https://github.com/cipherstash/protectjs/issues/46). ## Logging diff --git a/docker-compose.yml b/docker-compose.yml index b92bd5079..45f5ab1cf 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -16,7 +16,7 @@ services: memory: 2048mb restart: always healthcheck: - test: ["CMD-SHELL", "pg_isready"] + test: [ "CMD-SHELL", "pg_isready" ] interval: 1s timeout: 5s retries: 10 From 43e1acb96d1f3700ad4d3bd9a929f00f2094c73d Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Tue, 4 Mar 2025 12:53:14 -0700 Subject: [PATCH 13/30] Exit prerelease --- .changeset/pre.json | 2 +- .changeset/young-waves-worry.md | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) create mode 100644 .changeset/young-waves-worry.md diff --git a/.changeset/pre.json b/.changeset/pre.json index 1fec061d8..4476ccae4 100644 --- a/.changeset/pre.json +++ b/.changeset/pre.json @@ -1,5 +1,5 @@ { - "mode": "pre", + "mode": "exit", "tag": "next", "initialVersions": { "@cipherstash/basic-example": "1.0.3", diff --git a/.changeset/young-waves-worry.md b/.changeset/young-waves-worry.md new file mode 100644 index 000000000..46a94395b --- /dev/null +++ b/.changeset/young-waves-worry.md @@ -0,0 +1,9 @@ +--- +"@cipherstash/protect": major +--- + +Released version 7. + +* Added support for searching encrypted data +* Added a schema strategy for defining your schema +* Required schema to initialize the protect client From ca7e8d3cdc2f66713348b5f56f06f32754642e15 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Tue, 4 Mar 2025 12:59:54 -0700 Subject: [PATCH 14/30] fix: hot fix for prerelease --- packages/protect/.npmignore | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 packages/protect/.npmignore diff --git a/packages/protect/.npmignore b/packages/protect/.npmignore new file mode 100644 index 000000000..3490e24db --- /dev/null +++ b/packages/protect/.npmignore @@ -0,0 +1,5 @@ +.env +.turbo +node_modules +cipherstash.secret.toml +cipherstash.toml \ No newline at end of file From dde8203b52df07d91e58a59ab5b8960d6e961509 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Wed, 5 Mar 2025 07:35:08 -0700 Subject: [PATCH 15/30] feat(protect): add eql data types --- .changeset/pre.json | 11 - .changeset/young-waves-worry.md | 4 +- README.md | 72 +++- apps/basic/{index.mjs => index.ts} | 27 +- apps/basic/package.json | 4 +- apps/basic/protect.ts | 8 + apps/basic/tsconfig.json | 27 ++ apps/drizzle/package.json | 8 +- apps/drizzle/src/select.ts | 3 +- apps/hono-supabase/README.md | 107 ------ apps/hono-supabase/src/index.ts | 19 +- apps/nextjs-clerk/src/app/page.tsx | 7 +- apps/nextjs-clerk/src/core/protect/index.ts | 15 +- apps/nextjs-clerk/src/lib/actions.ts | 6 +- docs/README.md | 8 + docs/bulk-encryption-decryption.md | 45 ++- docs/encrypt-config.md | 338 ------------------- docs/searchable-encryption.md | 0 packages/protect/__tests__/protect.test.ts | 192 +++-------- packages/protect/cs_plaintext_v1.schema.json | 44 --- packages/protect/eql.schema.json | 102 ++++++ packages/protect/generateEqlSchema.ts | 28 ++ packages/protect/generateSchema.mjs | 26 -- packages/protect/package.json | 9 +- packages/protect/src/cs_plaintext_v1.ts | 33 -- packages/protect/src/eql.schema.ts | 51 +++ packages/protect/src/eql/index.ts | 66 ---- packages/protect/src/ffi/index.ts | 117 ++++--- packages/protect/src/ffi/payload-helpers.ts | 34 +- packages/protect/src/index.ts | 13 +- pnpm-lock.yaml | 335 ++++++++++++++++-- 31 files changed, 819 insertions(+), 940 deletions(-) delete mode 100644 .changeset/pre.json rename apps/basic/{index.mjs => index.ts} (68%) create mode 100644 apps/basic/protect.ts create mode 100644 apps/basic/tsconfig.json delete mode 100644 docs/encrypt-config.md create mode 100644 docs/searchable-encryption.md delete mode 100644 packages/protect/cs_plaintext_v1.schema.json create mode 100644 packages/protect/eql.schema.json create mode 100644 packages/protect/generateEqlSchema.ts delete mode 100644 packages/protect/generateSchema.mjs delete mode 100644 packages/protect/src/cs_plaintext_v1.ts create mode 100644 packages/protect/src/eql.schema.ts delete mode 100644 packages/protect/src/eql/index.ts diff --git a/.changeset/pre.json b/.changeset/pre.json deleted file mode 100644 index 4476ccae4..000000000 --- a/.changeset/pre.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "mode": "exit", - "tag": "next", - "initialVersions": { - "@cipherstash/basic-example": "1.0.3", - "@cipherstash/nextjs-clerk-example": "0.0.1", - "@cipherstash/nextjs": "3.0.0", - "@cipherstash/protect": "6.0.0" - }, - "changesets": [] -} diff --git a/.changeset/young-waves-worry.md b/.changeset/young-waves-worry.md index 46a94395b..988c36db4 100644 --- a/.changeset/young-waves-worry.md +++ b/.changeset/young-waves-worry.md @@ -1,9 +1,7 @@ --- -"@cipherstash/protect": major +"@cipherstash/protect": minor --- -Released version 7. - * Added support for searching encrypted data * Added a schema strategy for defining your schema * Required schema to initialize the protect client diff --git a/README.md b/README.md index 2b4d1061b..046fc4278 100644 --- a/README.md +++ b/README.md @@ -3,11 +3,16 @@ [![Tests](https://github.com/cipherstash/protectjs/actions/workflows/tests.yml/badge.svg)](https://github.com/cipherstash/protectjs/actions/workflows/tests.yml) [![Built by CipherStash](https://raw.githubusercontent.com/cipherstash/meta/refs/heads/main/csbadge.svg)](https://cipherstash.com) -Protect.js is a JavaScript/TypeScript package for encrypting and decrypting data in PostgreSQL databases. -Encryption operations happen directly in your app, and the ciphertext is stored in your PostgreSQL database. +Protect.js is a TypeScript package for encrypting and decrypting data. +Encryption operations happen directly in your app, and the ciphertext is stored in your database. -Every value you encrypt with Protect.js has a unique key, made possible by CipherStash [ZeroKMS](https://cipherstash.com/products/zerokms)'s blazing fast bulk key operations. -Under the hood Protect.js uses CipherStash [Encrypt Query Language (EQL)](https://github.com/cipherstash/encrypt-query-language), and all ZeroKMS data keys are backed by a root key in [AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html). +Every value you encrypt with Protect.js has a unique key, made possible by CipherStash [ZeroKMS](https://cipherstash.com/products/zerokms)'s blazing fast bulk key operations, and backed by a root key in [AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html). + +The encrypted data is structured as an [Encrypt Query Language (EQL)](https://github.com/cipherstash/encrypt-query-language) JSON payload, and can be stored in any database that supports JSONB. + +> [!IMPORTANT] +> Searching, sorting, and filtering on encrypted data is only supported in PostgreSQL at the moment. +> Read more about [searching encrypted data](./docs/searchable-encryption.md). ## Table of contents @@ -29,7 +34,7 @@ For more specific documentation, please refer to the [docs](https://github.com/c ## Features -Protect.js protects data in PostgreSQL databases using industry-standard AES encryption. +Protect.js protects data in using industry-standard AES encryption. Protect.js uses [ZeroKMS](https://cipherstash.com/products/zerokms) for bulk encryption and decryption operations. This enables every encrypted value, in every column, in every row in your database to have a unique key — without sacrificing performance. @@ -39,6 +44,7 @@ This enables every encrypted value, in every column, in every row in your databa - **Really fast:** ZeroKMS's performance makes using millions of unique keys feasible and performant for real-world applications built with Protect.js. - **Identity-aware encryption**: Lock down access to sensitive data by requiring a valid JWT to perform a decryption. - **TypeScript support**: Strongly typed with TypeScript interfaces and types. +- **Searchable encryption**: Protect.js supports searching encrypted data in PostgreSQL. **Use cases:** - **Trusted data access**: make sure only your end-users can access their sensitive data stored in your product. @@ -145,11 +151,23 @@ In the `src/protect/schema.ts` file, you can define your tables and columns. import { csTable, csColumn } from '@cipherstash/protect' export const users = csTable('users', { - email: csColumn('email').freeTextSearch().equality().orderAndSort(), + email: csColumn('email'), }) export const orders = csTable('orders', { - address: csColumn('address').freeTextSearch().orderAndSort(), + address: csColumn('address'), +}) +``` + +**Searchable encryption** + +If you are looking to enable searchable encryption in a PostgreSQL database, you must declaratively enable the indexes in your schema. + +```ts +import { csTable, csColumn } from '@cipherstash/protect' + +export const users = csTable('users', { + email: csColumn('email').freeTextSearch().equality().orderAndSort(), }) ``` @@ -169,6 +187,8 @@ import { users } from './schema' export const protectClient = await protect(users, orders) ``` +The `protect` function requires at least one `csTable` to be passed in. + ### Encrypting data Use the `encrypt` function to encrypt data. @@ -253,7 +273,9 @@ The `decryptResult` will return one of the following: ### Storing encrypted data in a database -To store the encrypted data in PostgreSQL, you will need to specify the column type as `jsonb`. +Encrypted data can be stored in any database that supports JSONB, noting that searchable encryption is only supported in PostgreSQL at the moment. + +To store the encrypted data, you will need to specify the column type as `jsonb`. ```sql CREATE TABLE users ( @@ -262,6 +284,31 @@ CREATE TABLE users ( ); ``` +**Searchable encryption** + +To enable searchable encryption in PostgreSQL, you need to [install the EQL custom types and functions](https://github.com/cipherstash/encrypt-query-language?tab=readme-ov-file#installation). + +1. Download the latest EQL install script: + + ```sh + curl -sLo cipherstash-encrypt.sql https://github.com/cipherstash/encrypt-query-language/releases/latest/download/cipherstash-encrypt.sql + ``` + +2. Run this command to install the custom types and functions: + + ```sh + psql -f cipherstash-encrypt.sql + ``` + +EQL is now installed in your database and you can enable searchable encryption by adding the `cs_encrypted_v1` type to a column. + +```sql +CREATE TABLE users ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email cs_encrypted_v1 +); +``` + ## Identity-aware encryption Protect.js can add an additional layer of protection to your data by requiring a valid JWT to perform a decryption. @@ -384,8 +431,8 @@ const encryptedValues = encryptedResults.data // encryptedValues might look like: // [ -// { c: 'ENCRYPTED_VALUE_1', id: '1' }, -// { c: 'ENCRYPTED_VALUE_2', id: '2' }, +// { encryptedData: { c: 'ENCRYPTED_VALUE_1', k: 'ct' }, id: '1' }, +// { encryptedData: { c: 'ENCRYPTED_VALUE_2', k: 'ct' }, id: '2' }, // ] ``` @@ -396,7 +443,7 @@ encryptedValues.forEach((result) => { // Find the corresponding user const user = users.find((u) => u.id === result.id) if (user) { - user.email = result.c // Store ciphertext back into the user object + user.email = result.encryptedData // Store the encrypted data back into the user object } }) ``` @@ -465,8 +512,7 @@ Until support for other data types are available in `@cipherstash/protect`, you ## Searchable encryption -The searchable encryption documentation is in the works and will be available soon. -Express interest in this feature by adding a :+1: on this [GitHub Issue](https://github.com/cipherstash/protectjs/issues/46). +Read more about [searching encrypted data](./docs/searchable-encryption.md) in the docs. ## Logging diff --git a/apps/basic/index.mjs b/apps/basic/index.ts similarity index 68% rename from apps/basic/index.mjs rename to apps/basic/index.ts index af64d49c9..d9233cbd7 100644 --- a/apps/basic/index.mjs +++ b/apps/basic/index.ts @@ -1,27 +1,26 @@ import 'dotenv/config' -import { protect } from '@cipherstash/protect' -import readline from 'node:readline'; +import { protectClient, users } from './protect' +import readline from 'node:readline' const rl = readline.createInterface({ input: process.stdin, output: process.stdout, -}); +}) -const askQuestion = () => { +const askQuestion = (): Promise => { return new Promise((resolve) => { - rl.question("\n👋Hello\n\nWhat is your name? ", (answer) => { - resolve(answer); - }); - }); -}; + rl.question('\n👋Hello\n\nWhat is your name? ', (answer) => { + resolve(answer) + }) + }) +} async function main() { - const protectClient = await protect() - const input = await askQuestion(); + const input = await askQuestion() const encryptResult = await protectClient.encrypt(input, { - column: 'column_name', - table: 'users', + column: users.name, + table: users, }) if (encryptResult.failure) { @@ -44,7 +43,7 @@ async function main() { console.log('Decrypting the ciphertext...') console.log('The plaintext is:', plaintext) - rl.close(); + rl.close() } main() diff --git a/apps/basic/package.json b/apps/basic/package.json index ac4968c96..9299d3d48 100644 --- a/apps/basic/package.json +++ b/apps/basic/package.json @@ -4,7 +4,7 @@ "version": "1.0.3", "main": "index.mjs", "scripts": { - "start": "node index.mjs" + "start": "tsx index.ts" }, "keywords": [], "author": "", @@ -12,6 +12,8 @@ "description": "", "dependencies": { "@cipherstash/protect": "workspace:*", + "typescript": "^5.0.0", + "tsx": "^4.19.2", "dotenv": "^16.4.7" } } diff --git a/apps/basic/protect.ts b/apps/basic/protect.ts new file mode 100644 index 000000000..40e1fc8cf --- /dev/null +++ b/apps/basic/protect.ts @@ -0,0 +1,8 @@ +import 'dotenv/config' +import { protect, csColumn, csTable } from '@cipherstash/protect' + +export const users = csTable('users', { + name: csColumn('name'), +}) + +export const protectClient = await protect(users) diff --git a/apps/basic/tsconfig.json b/apps/basic/tsconfig.json new file mode 100644 index 000000000..238655f2c --- /dev/null +++ b/apps/basic/tsconfig.json @@ -0,0 +1,27 @@ +{ + "compilerOptions": { + // Enable latest features + "lib": ["ESNext", "DOM"], + "target": "ESNext", + "module": "ESNext", + "moduleDetection": "force", + "jsx": "react-jsx", + "allowJs": true, + + // Bundler mode + "moduleResolution": "bundler", + "allowImportingTsExtensions": true, + "verbatimModuleSyntax": true, + "noEmit": true, + + // Best practices + "strict": true, + "skipLibCheck": true, + "noFallthroughCasesInSwitch": true, + + // Some stricter flags (disabled by default) + "noUnusedLocals": false, + "noUnusedParameters": false, + "noPropertyAccessFromIndexSignature": false + } +} diff --git a/apps/drizzle/package.json b/apps/drizzle/package.json index f0e6eccc5..f99652e6d 100644 --- a/apps/drizzle/package.json +++ b/apps/drizzle/package.json @@ -8,17 +8,15 @@ "@types/pg": "^8.11.10", "dotenv": "^16.4.7", "drizzle-kit": "^0.24.2", + "typescript": "^5.0.0", "tsx": "^4.19.2" }, "scripts": { - "insert": "ts-node src/insert.ts", - "select": "ts-node src/select.ts", + "insert": "tsx src/insert.ts", + "select": "tsx src/select.ts", "db:generate": "drizzle-kit generate", "db:migrate": "drizzle-kit migrate" }, - "peerDependencies": { - "typescript": "^5.0.0" - }, "dependencies": { "@cipherstash/protect": "workspace:*", "drizzle-orm": "^0.33.0", diff --git a/apps/drizzle/src/select.ts b/apps/drizzle/src/select.ts index c26f7335a..7eccea4d3 100644 --- a/apps/drizzle/src/select.ts +++ b/apps/drizzle/src/select.ts @@ -5,6 +5,7 @@ import { protectClient, users as protectUsers } from './protect' import { bindIfParam, sql } from 'drizzle-orm' import type { BinaryOperator, SQL, SQLWrapper } from 'drizzle-orm' import { parseArgs } from 'node:util' +import type { EncryptedData } from '@cipherstash/protect' const getArgs = () => { const { values, positionals } = parseArgs({ @@ -84,7 +85,7 @@ const data = await query.execute() const emails = await Promise.all( data.map( - async (row) => await protectClient.decrypt(row.email as { c: string }), + async (row) => await protectClient.decrypt(row.email as EncryptedData), ), ) diff --git a/apps/hono-supabase/README.md b/apps/hono-supabase/README.md index cad790902..b34ed0b6a 100644 --- a/apps/hono-supabase/README.md +++ b/apps/hono-supabase/README.md @@ -14,7 +14,6 @@ This project demonstrates how to encrypt data using [@cipherstash/protect](https - [API Endpoints](#api-endpoints) - [GET /users](#get-users) - [POST /users](#post-users) -- [Explanation of the Code](#explanation-of-the-code) - [Additional Resources](#additional-resources) - [License](#license) @@ -156,112 +155,6 @@ Creates a new user with an **encrypted** email field. } ``` ---- - -## Explanation of the Code - -```js -import 'dotenv/config' -import { serve } from '@hono/node-server' -import { createClient } from '@supabase/supabase-js' -import { Hono } from 'hono' -import { createRequire } from 'node:module' - -// We use ES6 require for @cipherstash/protect due to dynamic import limitations -const require = createRequire(import.meta.url) -const { protect } = require('@cipherstash/protect') - -// 1. Initialize the CipherStash EQL client using environment variables -const protectClient = await protect() - -// 2. Initialize Supabase client -const supabaseUrl = process.env.SUPABASE_URL -const supabaseKey = process.env.SUPABASE_ANON_KEY -export const supabase = createClient(supabaseUrl, supabaseKey) - -// 3. Create your Hono application -const app = new Hono() - -// 4. GET /users -// - Pulls records from the `users` table -// - Decrypts the `email` field -app.get('/users', async (c) => { - const { data: users } = await supabase.from('users').select() - if (users && users.length > 0) { - const decryptedusers = await Promise.all( - users.map(async (user) => { - const plaintextResult = await protectClient.decrypt(user.email) - - if (plaintextResult.failure) { - console.error( - 'Failed to decrypt the email for user', - user.id, - plaintextResult.failure.message, - ) - - return user - } - - const plaintext = plaintextResult.data - return { ...user, email: plaintext } - }) - ) - return c.json({ users: decryptedusers }) - } - return c.json({ users: [] }) -}) - -// 5. POST /users -// - Encrypts the `email` field using jsprotect -// - Inserts the encrypted data into the `users` table -app.post('/users', async (c) => { - const { email, name } = await c.req.json() - if (!email || !name) { - return c.json({ message: 'Email and name are required to create a user' }, 400) - } - - // Encrypt the email - const encryptedResult = await protectClient.encrypt(email, { - column: 'email', - table: 'users', - }) - - if (encryptedResult.failure) { - return c.json({ message: 'Failed to encrypt the email' }, 500) - } - - const encryptedEmail = encryptedResult.data - - // Insert the encrypted data - const result = await supabase - .from('users') - .insert({ email: encryptedEmail, name, role: 'admin' }) - - if (result.statusText === 'Created') { - return c.json({ message: 'User created successfully' }) - } - - // Log and return an error message if the insertion fails - console.error('User creation failed:', result) - return c.json({ message: 'User creation failed. Please check the logs' }, 500) -}) - -// 6. Start the server on port 3000 -serve({ - fetch: app.fetch, - port: 3000, -}) -``` - -**Key points to note:** -- `@cipherstash/protect` provides two primary functions: `encrypt()` and `decrypt()`. -- The encrypted field is stored as JSON in the format `{ c: "ciphertext" }`. -- `@hono/node-server` is used to run the Hono application as a Node.js server. -- `dotenv/config` automatically loads environment variables from your `.env` file. -- We leverage Supabase’s client (`@supabase/supabase-js`) to insert and select data. - ---- - ## Additional Resources - [@cipherstash/protect Documentation](https://github.com/cipherstash/protectjs) diff --git a/apps/hono-supabase/src/index.ts b/apps/hono-supabase/src/index.ts index e7f31db69..0ab806d4c 100644 --- a/apps/hono-supabase/src/index.ts +++ b/apps/hono-supabase/src/index.ts @@ -2,12 +2,15 @@ import 'dotenv/config' import { serve } from '@hono/node-server' import { createClient } from '@supabase/supabase-js' import { Hono } from 'hono' -import { protect } from '@cipherstash/protect' -// Initialize the EQL client -// Make sure you have the following environment variables defined in your .env file: -// CS_CLIENT_ID, CS_CLIENT_KEY, CS_CLIENT_ACCESS_KEY, CS_WORKSPACE_ID -const protectClient = await protect() +// Consolidated protect and it's schemas into a single file +import { protect, csColumn, csTable } from '@cipherstash/protect' + +export const users = csTable('users', { + email: csColumn('email'), +}) + +export const protectClient = await protect(users) // Create a single supabase client for interacting with the database const supabaseUrl = process.env.SUPABASE_URL @@ -71,8 +74,8 @@ app.post('/users', async (c) => { // and the second argument to be an object with the table and column // names of the table where you are storing the data. const encryptedResult = await protectClient.encrypt(email, { - column: 'email', - table: 'users', + column: users.email, + table: users, }) if (encryptedResult.failure) { @@ -85,8 +88,6 @@ app.post('/users', async (c) => { const encryptedEmail = encryptedResult.data - // The encrypt function will return an object with a c key, which is the encrypted data. - // We are logging the encrypted data to the console for demonstration purposes. console.log( 'Encrypted email that will be stored in the database:', encryptedEmail, diff --git a/apps/nextjs-clerk/src/app/page.tsx b/apps/nextjs-clerk/src/app/page.tsx index 2f1c6d992..b8b220025 100644 --- a/apps/nextjs-clerk/src/app/page.tsx +++ b/apps/nextjs-clerk/src/app/page.tsx @@ -5,10 +5,7 @@ import { db } from '@/core/db' import { protectClient, getLockContext } from '@/core/protect' import { auth, currentUser } from '@clerk/nextjs/server' import { getCtsToken } from '@cipherstash/nextjs' - -export type EqlPayload = { - c: string // Ciphertext -} +import type { EncryptedData } from '@cipherstash/protect' export type EncryptedUser = { id: number @@ -29,7 +26,7 @@ async function getUsers(): Promise { const promises = results.map(async (row) => { const decryptResult = await protectClient - .decrypt(row.email as { c: string }) + .decrypt(row.email as EncryptedData) .withLockContext(lockContext) if (decryptResult.failure) { diff --git a/apps/nextjs-clerk/src/core/protect/index.ts b/apps/nextjs-clerk/src/core/protect/index.ts index 298716715..18a4d4e6d 100644 --- a/apps/nextjs-clerk/src/core/protect/index.ts +++ b/apps/nextjs-clerk/src/core/protect/index.ts @@ -1,6 +1,17 @@ import 'dotenv/config' -import { protect, LockContext, type CtsToken } from '@cipherstash/protect' -export const protectClient = await protect() +import { + protect, + LockContext, + type CtsToken, + csColumn, + csTable, +} from '@cipherstash/protect' + +export const users = csTable('users', { + email: csColumn('email'), +}) + +export const protectClient = await protect(users) export const getLockContext = (cts_token?: CtsToken) => { if (!cts_token) { diff --git a/apps/nextjs-clerk/src/lib/actions.ts b/apps/nextjs-clerk/src/lib/actions.ts index 27f213215..c1d505a6e 100644 --- a/apps/nextjs-clerk/src/lib/actions.ts +++ b/apps/nextjs-clerk/src/lib/actions.ts @@ -2,7 +2,7 @@ import { users } from '@/core/db/schema' import { db } from '@/core/db' -import { protectClient } from '@/core/protect' +import { protectClient, users as protectUsers } from '@/core/protect' import { getLockContext } from '@/core/protect' import { getCtsToken } from '@cipherstash/nextjs' import { revalidatePath } from 'next/cache' @@ -32,8 +32,8 @@ export async function addUser(formData: FormData) { const lockContext = getLockContext(ctsToken.ctsToken) const encryptedResult = await protectClient .encrypt(email, { - column: users.email.name, - table: 'users', + column: protectUsers.email, + table: protectUsers, }) .withLockContext(lockContext) diff --git a/docs/README.md b/docs/README.md index 4837ddb88..0db30994d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,8 +4,16 @@ The main documentation for Protect.js is located in the [main README](https://gi ## Docs table of contents +### Setup + - [Configuration and production deployment](configuration.md) + +### Usage + - [Bulk encryption and decryption examples and best practices](bulk-encryption-and-decryption.md) - [Lock contexts with Clerk and Next.js](lock-context.md) + +### Reference + - [Next.js build notes](nextjs.md) - [SST and serverless function notes](sst.md) \ No newline at end of file diff --git a/docs/bulk-encryption-decryption.md b/docs/bulk-encryption-decryption.md index 9d9e223f5..c65aaa882 100644 --- a/docs/bulk-encryption-decryption.md +++ b/docs/bulk-encryption-decryption.md @@ -3,12 +3,17 @@ If you have a large list of items to encrypt or decrypt, you can use the **`bulkEncrypt`** and **`bulkDecrypt`** methods to batch encryption/decryption. `bulkEncrypt` and `bulkDecrypt` give your app significantly better throughput than the single-item [`encrypt`](../README.md#encrypting-data) / [`decrypt`](../README.md#decrypting-data) methods. +Assuming you've [defined your schema and initialized the client](../README.md#defining-your-schema), you can use the `bulkEncrypt` and `bulkDecrypt` methods to batch encryption/decryption. + ### Bulk encrypting data ```ts +import { users } from './protect/schema' +import { protectClient } from './protect' + const encryptedResults = await protectClient.bulkEncrypt(plaintextsToEncrypt, { - column: 'email', - table: 'Users', + column: users.email, + table: users, }) if (encryptedResults.failure) { @@ -20,8 +25,8 @@ const encryptedValues = encryptedResults.data // or with lock context const encryptedResults = await protectClient.bulkEncrypt(plaintextsToEncrypt, { - column: 'email', - table: 'Users', + column: users.email, + table: users, }).withLockContext(lockContext) if (encryptedResults.failure) { @@ -52,10 +57,10 @@ const encryptedValues = encryptedResults.data **Return value** -- **Type**: `Promise | null, ProtectError>>` +- **Type**: `Promise | null, ProtectError>>` - Returns a `Result` object, where: - **`data`** is an array of objects or `null`, where: - - **`c`** is the ciphertext. + - **`encryptedData`** is the encrypted data. - **`id`** is the same **id** you passed in, so you can correlate which ciphertext matches which original plaintext. - **`failure`** is an object with the following properties: - **`type`** is a string with the error type. @@ -64,22 +69,25 @@ const encryptedValues = encryptedResults.data #### Example usage ```ts +import { users } from './protect/schema' +import { protectClient } from './protect' + // 1) Gather your data. For example, a list of users with plaintext fields. -const users = [ +const examples = [ { id: '1', name: 'CJ', email: 'cj@example.com' }, { id: '2', name: 'Alex', email: 'alex@example.com' }, ] // 2) Prepare the array for bulk encryption (only encrypting the "email" field here). -const plaintextsToEncrypt = users.map((user) => ({ +const plaintextsToEncrypt = examples.map((user) => ({ plaintext: user.email, // The data to encrypt id: user.id, // Keep track by user ID })) // 3) Call bulkEncrypt const encryptedResults = await bulkEncrypt(plaintextsToEncrypt, { - column: 'email', - table: 'Users', + column: users.email, + table: users, }) if (encryptedResults.failure) { @@ -90,8 +98,8 @@ const encryptedValues = encryptedResults.data // encryptedValues might look like: // [ -// { c: 'ENCRYPTED_VALUE_1', id: '1' }, -// { c: 'ENCRYPTED_VALUE_2', id: '2' }, +// { encryptedData: { c: 'ENCRYPTED_VALUE_1', k: 'ct' }, id: '1' }, +// { encryptedData: { c: 'ENCRYPTED_VALUE_2', k: 'ct' }, id: '2' }, // ] // 4) Reassemble data by matching IDs @@ -100,7 +108,7 @@ if (encryptedValues) { // Find the corresponding user const user = users.find((u) => u.id === result.id) if (user) { - user.email = result.c // Store ciphertext back into the user object + user.email = result.encryptedData // Store the encrypted data back into the user object } }) } @@ -119,9 +127,10 @@ const decryptedResults = await protectClient.bulkDecrypt(encryptedPayloads).with **Parameters** 1. **`encryptedPayloads`** - - **Type**: `Array<{ c: string; id: string }> | null` + - **Type**: `Array<{ encryptedData: EncryptedData; id: string }> | null` - **Description**: - An array of objects containing the **ciphertext** (`c`) and the **id**. If this array is empty or `null`, the function returns `null`. + An array of objects containing the **encrypted data** (`encryptedData`) and the **id**. + If this array is empty or `null`, the function returns `null`. **Return value** @@ -138,14 +147,14 @@ const decryptedResults = await protectClient.bulkDecrypt(encryptedPayloads).with ```ts // Suppose you've retrieved an array of users where their email fields are ciphertext: -const users = [ +const myUsers = [ { id: '1', name: 'CJ', email: 'ENCRYPTED_VALUE_1' }, { id: '2', name: 'Alex', email: 'ENCRYPTED_VALUE_2' }, ] // 1) Prepare the array for bulk decryption -const encryptedPayloads = users.map((user) => ({ - c: user.email, +const encryptedPayloads = myUsers.map((user) => ({ + encryptedData: user.email, id: user.id, })) diff --git a/docs/encrypt-config.md b/docs/encrypt-config.md deleted file mode 100644 index dac34232a..000000000 --- a/docs/encrypt-config.md +++ /dev/null @@ -1,338 +0,0 @@ -# Encrypt Config for Protect.js - -An **Encrypt Config** is a JSON (or TypeScript) structure that specifies: - -1. **Version** (the `"v"` field). -2. One or more **Tables** (the `"tables"` field). - -Inside each **Table**, you’ll define one or more **Columns**, specifying how each column is _cast_ (converted to a given type) and which **Indexes** (if any) should be built on that column. - -Once defined, use the config to initialize the Protect.js client: - -```ts -import { protect, type EncryptConfig } from "@cipherstash/protect"; - -const config: EncryptConfig = { - v: 1, - tables: { - users: { - email: { - cast_as: "text", - indexes: {}, - }, - }, - }, -}; - -const protectClient = protect(config); -``` - -## Top-Level Structure - -```jsonc -{ - "v": 1, // Required integer representing the config version - "tables": { - "table_name": { - "column_name": { - "cast_as": "...", // Optional, defaults to "text" - "indexes": { // Optional, each index is optional - "ore": { ... }, - "unique": { ... }, - "match": { ... }, - "ste_vec": { ... } - } - } - } - } -} -``` - -### `v` (Version) - -- **Type**: `number` -- **Required**: Yes -- **Purpose**: Used to mark the version of the configuration schema. -We only support version `1` at this time. - -### `tables` - -- **Type**: A map of **tableName** → **Table Definition**. -- **Required**: Yes -- **Example**: - ```json - { - "tables": { - "users": { ... }, - "orders": { ... } - } - } - ``` -- Each key in `"tables"` is the name of a table (e.g., `"users"`), whose value is another map of **columns**. - -## Table Definition - -```jsonc -"table_name": { - "column_name": { ... }, - "another_column": { ... } -} -``` - -- Each key (e.g., `"column_name"`) is the column name in that table. -- The value for each column is a **Column** object. - -## Column Definition - -```jsonc -{ - "cast_as": "text", // or "big_int", "int", "small_int", "boolean", "date", "real", "double", "jsonb" - "indexes": { - "ore": {}, - "unique": { - "token_filters": [ - { "kind": "downcase" } - ] - }, - "match": { - "tokenizer": { - "kind": "standard" // or "ngram" - }, - "token_filters": [], - "k": 6, - "m": 2048, - "include_original": false - }, - "ste_vec": { - "prefix": "..." - } - } -} -``` - -### `cast_as` - -Specifies how the column will be cast (typed) before encryption and indexing. This closely maps to the **Rust** enum `CastAs`. Possible values: - -- `"big_int"` -- `"small_int"` -- `"int"` -- `"boolean"` -- `"date"` -- `"real"` -- `"double"` -- `"text"` (default) -- `"jsonb"` - -**Default**: `"text"` - -### `indexes` - -An object defining which indexes (if any) to enable on this column. **All fields in `indexes` are optional**. If none are present, the column will not be indexed. - -#### 1. ORE Index - -Enables **Order-Revealing Encryption** (ORE). - -```json -"indexes": { - "ore": {} -} -``` - -- The `ore` property is an empty object (i.e., no extra settings). -- In Rust, this corresponds to `OreIndexOpts {}`. - -#### 2. Unique Index - -Enables **Unique** constraints. This typically uses specialized indexing so that each value in the column must be unique. - -```json -"indexes": { - "unique": { - "token_filters": [ - { "kind": "downcase" } - ] - } -} -``` - -- **`token_filters`**: An optional array of objects specifying transformations on the token before indexing. Currently, the example shows one possible filter: - - `{ "kind": "downcase" }` – transforms the string to lowercase before indexing. - -**Default**: `{ "token_filters": [] }` (an empty array of token filters). - -#### 3. Match Index - -Enables **Match** (full-text-ish) indexing, which can be used for substring/pattern matching. The default fields are: - -```json -"indexes": { - "match": { - "tokenizer": { - "kind": "standard" - }, - "token_filters": [], - "k": 6, - "m": 2048, - "include_original": false - } -} -``` - -**Field explanations**: - -1. **`tokenizer`** – How to chunk the text into tokens. - - `{"kind": "standard"}` (default) - - `{"kind": "ngram", "token_length": }` – e.g. `{"kind":"ngram","token_length":3}` -2. **`token_filters`** – Optional array of transforms, e.g. `[{"kind": "downcase"}]`. -3. **`k`** – Defaults to `6`. Used internally for chunk sizes or matching thresholds. -4. **`m`** – Defaults to `2048`. Another internal parameter for the cryptographic index. -5. **`include_original`** – Defaults to `false`. When `true`, also store the original tokens unmodified. - -#### 4. STE Vector Index - -Enables the **STE Vec** index, typically used for vector-like data or structured queries. - -```json -"indexes": { - "ste_vec": { - "prefix": "..." - } -} -``` - -**Fields**: - -- **`prefix`** – A required string that acts as a prefix for the index. - -## Example Configs - -### Minimal Config - -```json -{ - "v": 1, - "tables": { - "users": {} - } -} -``` - -- Creates a config for version `1`. -- Declares a `users` table with **no columns**. (In practice, you’d add columns.) - -### Simple Column Example - -```json -{ - "v": 1, - "tables": { - "users": { - "email": { - "cast_as": "text", - "indexes": {} - } - } - } -} -``` - -- `users.email` is cast as `"text"` (the default if omitted). -- **No** indexes are enabled on `email`. - -### ORE Index Example - -```json -{ - "v": 1, - "tables": { - "users": { - "age": { - "cast_as": "int", - "indexes": { - "ore": {} - } - } - } - } -} -``` - -- `users.age` is cast as `"int"`. -- ORE indexing is enabled. - -### Unique Index with Token Filters - -```json -{ - "v": 1, - "tables": { - "users": { - "email": { - "indexes": { - "unique": { - "token_filters": [ - { "kind": "downcase" } - ] - } - } - } - } - } -} -``` - -- `users.email` defaults to `"text"`. -- Unique index with a `downcase` token filter. - -### Match Index with Ngram Tokenizer - -```json -{ - "v": 1, - "tables": { - "users": { - "description": { - "indexes": { - "match": { - "tokenizer": { - "kind": "ngram", - "token_length": 3 - }, - "token_filters": [ - { "kind": "downcase" } - ], - "k": 8, - "m": 1024, - "include_original": true - } - } - } - } - } -} -``` - -- Enables a match index with an ngram tokenizer that uses 3-character slices. -- Applies `downcase`. -- Sets `k=8`, `m=1024`, `include_original=true`. - -### STE Vector Example - -```json -{ - "v": 1, - "tables": { - "users": { - "event_data": { - "indexes": { - "ste_vec": { - "prefix": "event-data" - } - } - } - } - } -} -``` \ No newline at end of file diff --git a/docs/searchable-encryption.md b/docs/searchable-encryption.md new file mode 100644 index 000000000..e69de29bb diff --git a/packages/protect/__tests__/protect.test.ts b/packages/protect/__tests__/protect.test.ts index e483cb96a..98fe10e09 100644 --- a/packages/protect/__tests__/protect.test.ts +++ b/packages/protect/__tests__/protect.test.ts @@ -1,127 +1,12 @@ import 'dotenv/config' import { describe, expect, it } from 'vitest' -import { - LockContext, - createEqlPayload, - getPlaintext, - protect, - csTable, - csColumn, -} from '../src' -import type { CsPlaintextV1Schema } from '../src/cs_plaintext_v1' +import { LockContext, protect, csTable, csColumn } from '../src' const users = csTable('users', { email: csColumn('email').freeTextSearch().equality().orderAndSort(), }) -describe('createEqlPayload', () => { - it('should create a payload with the correct default values', () => { - const result = createEqlPayload({ - plaintext: 'test', - table: 'users', - column: 'email', - }) - - const expectedPayload: CsPlaintextV1Schema = { - v: 1, - k: 'pt', - p: 'test', - i: { - t: 'users', - c: 'email', - }, - } - - expect(result).toEqual(expectedPayload) - }) - - it('should set custom schemaVersion and queryType values when provided', () => { - const result = createEqlPayload({ - plaintext: 'test', - table: 'users', - column: 'email', - schemaVersion: 2, - queryType: 'match', - }) - - const expectedPayload: CsPlaintextV1Schema = { - v: 2, - k: 'pt', - p: 'test', - i: { - t: 'users', - c: 'email', - }, - q: 'match', - } - - expect(result).toEqual(expectedPayload) - }) - - it('should set plaintext to an empty string if undefined', () => { - const result = createEqlPayload({ - plaintext: '', - table: 'users', - column: 'email', - }) - - expect(result.p).toBe('') - }) -}) - -describe('getPlaintext', () => { - it('should return plaintext if payload is valid and key is "pt"', () => { - const payload: CsPlaintextV1Schema = { - v: 1, - k: 'pt', - p: 'test', - i: { - t: 'users', - c: 'email', - }, - } - - const result = getPlaintext(payload) - - expect(result).toEqual({ - failure: false, - plaintext: 'test', - }) - }) - - it('should return an error if payload is missing "p" or key is not "pt"', () => { - const invalidPayload = { - v: 1, - k: 'ct', - c: 'ciphertext', - p: '', - i: { - t: 'users', - c: 'email', - }, - } - - const result = getPlaintext( - invalidPayload as unknown as CsPlaintextV1Schema, - ) - - expect(result).toEqual({ - failure: true, - error: new Error('No plaintext data found in the EQL payload'), - }) - }) - - it('should return an error and log if payload is invalid', () => { - const result = getPlaintext(null as unknown as CsPlaintextV1Schema) - - expect(result).toEqual({ - failure: true, - error: new Error('No plaintext data found in the EQL payload'), - }) - }) -}) - describe('encryption and decryption', () => { it('should encrypt and decrypt a payload', async () => { const protectClient = await protect(users) @@ -233,43 +118,64 @@ describe('bulk encryption', () => { // These tests pass locally, given you provide a valid JWT. // To manually test locally, uncomment the following lines and provide a valid JWT in the userJwt variable. // ------------------------ -// const userJwt = '' +// const userJwt = +// '' // describe('encryption and decryption with lock context', () => { // it('should encrypt and decrypt a payload with lock context', async () => { -// const protectClient = await protect() +// const protectClient = await protect(users) // const lc = new LockContext() // const lockContext = await lc.identify(userJwt) -// const ciphertext = await protectClient +// if (lockContext.failure) { +// throw new Error(`[protect]: ${lockContext.failure.message}`) +// } + +// const encryptResult = await protectClient // .encrypt('plaintext', { -// column: 'column_name', -// table: 'users', +// column: users.email, +// table: users, // }) -// .withLockContext(lockContext) +// .withLockContext(lockContext.data) + +// if (encryptResult.failure) { +// throw new Error(`[protect]: ${encryptResult.failure.message}`) +// } // const plaintext = await protectClient -// .decrypt(ciphertext) -// .withLockContext(lockContext) +// .decrypt(encryptResult.data) +// .withLockContext(lockContext.data) -// expect(plaintext).toEqual('plaintext') +// if (plaintext.failure) { +// throw new Error(`[protect]: ${plaintext.failure.message}`) +// } + +// expect(plaintext.data).toEqual('plaintext') // }, 30000) // it('should encrypt with context and be unable to decrypt without context', async () => { -// const protectClient = await protect() +// const protectClient = await protect(users) // const lc = new LockContext() // const lockContext = await lc.identify(userJwt) +// if (lockContext.failure) { +// throw new Error(`[protect]: ${lockContext.failure.message}`) +// } + // const ciphertext = await protectClient // .encrypt('plaintext', { -// column: 'column_name', -// table: 'users', +// column: users.email, +// table: users, // }) -// .withLockContext(lockContext) +// .withLockContext(lockContext.data) + +// if (ciphertext.failure) { +// throw new Error(`[protect]: ${ciphertext.failure.message}`) +// } // try { -// await protectClient.decrypt(ciphertext) +// await protectClient.decrypt(ciphertext.data) // } catch (error) { // const e = error as Error // expect(e.message.startsWith('Failed to retrieve key')).toEqual(true) @@ -277,11 +183,15 @@ describe('bulk encryption', () => { // }, 30000) // it('should bulk encrypt and decrypt a payload with lock context', async () => { -// const protectClient = await protect() +// const protectClient = await protect(users) // const lc = new LockContext() // const lockContext = await lc.identify(userJwt) +// if (lockContext.failure) { +// throw new Error(`[protect]: ${lockContext.failure.message}`) +// } + // const ciphertexts = await protectClient // .bulkEncrypt( // [ @@ -295,17 +205,25 @@ describe('bulk encryption', () => { // }, // ], // { -// table: 'users', -// column: 'column_name', +// table: users, +// column: users.email, // }, // ) -// .withLockContext(lockContext) +// .withLockContext(lockContext.data) + +// if (ciphertexts.failure) { +// throw new Error(`[protect]: ${ciphertexts.failure.message}`) +// } // const plaintexts = await protectClient -// .bulkDecrypt(ciphertexts) -// .withLockContext(lockContext) +// .bulkDecrypt(ciphertexts.data) +// .withLockContext(lockContext.data) + +// if (plaintexts.failure) { +// throw new Error(`[protect]: ${plaintexts.failure.message}`) +// } -// expect(plaintexts).toEqual([ +// expect(plaintexts.data).toEqual([ // { // plaintext: 'test', // id: '1', diff --git a/packages/protect/cs_plaintext_v1.schema.json b/packages/protect/cs_plaintext_v1.schema.json deleted file mode 100644 index 56bec1b75..000000000 --- a/packages/protect/cs_plaintext_v1.schema.json +++ /dev/null @@ -1,44 +0,0 @@ -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "description": "The EQL plaintext JSON payload sent by a client (such as an application) to CipherStash Proxy.", - "type": "object", - "properties": { - "v": { - "title": "Schema version", - "type": "integer" - }, - "k": { - "title": "kind", - "type": "string", - "const": "pt" - }, - "i": { - "title": "ident", - "type": "object", - "properties": { - "t": { - "title": "table", - "type": "string", - "pattern": "^[a-zA-Z_]{1}[0-9a-zA-Z_]*$" - }, - "c": { - "title": "column", - "type": "string", - "pattern": "^[a-zA-Z_]{1}[0-9a-zA-Z_]*$" - } - }, - "required": ["t", "c"] - }, - "p": { - "title": "plaintext", - "type": "string" - }, - "q": { - "title": "for query", - "description": "Specifies that the plaintext should be encrypted for a specific query operation. If null, source encryption and encryption for all indexes will be performed.", - "type": "string", - "enum": ["match", "ore", "unique", "ste_vec", "ejson_path"] - } - }, - "required": ["v", "k", "i", "p"] -} diff --git a/packages/protect/eql.schema.json b/packages/protect/eql.schema.json new file mode 100644 index 000000000..1481c16b2 --- /dev/null +++ b/packages/protect/eql.schema.json @@ -0,0 +1,102 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "description": "The EQL encrypted JSON payload used for storage.", + "type": "object", + "properties": { + "v": { + "title": "Schema version", + "type": "integer" + }, + "k": { + "title": "kind", + "type": "string", + "enum": [ + "ct", + "sv" + ] + }, + "i": { + "title": "ident", + "type": "object", + "properties": { + "t": { + "title": "table", + "type": "string", + "pattern": "^[a-zA-Z_]{1}[0-9a-zA-Z_]*$" + }, + "c": { + "title": "column", + "type": "string", + "pattern": "^[a-zA-Z_]{1}[0-9a-zA-Z_]*$" + } + }, + "required": [ + "t", + "c" + ] + } + }, + "oneOf": [ + { + "properties": { + "k": { + "const": "ct" + }, + "c": { + "title": "ciphertext", + "type": "string" + }, + "u": { + "title": "unique index", + "type": "string" + }, + "o": { + "title": "ore index", + "type": "array", + "minItems": 1, + "items": { + "type": "string" + } + }, + "m": { + "title": "match index", + "type": "array", + "minItems": 1, + "items": { + "type": "number" + } + } + }, + "required": [ + "c" + ] + }, + { + "properties": { + "k": { + "const": "sv" + }, + "sv": { + "title": "Structured Encryption vector", + "type": "array", + "items": { + "type": "array", + "items": { + "type": "string", + "minItems": 3, + "maxItems": 3 + } + } + } + }, + "required": [ + "sv" + ] + } + ], + "required": [ + "v", + "k", + "i" + ] +} \ No newline at end of file diff --git a/packages/protect/generateEqlSchema.ts b/packages/protect/generateEqlSchema.ts new file mode 100644 index 000000000..6888195ae --- /dev/null +++ b/packages/protect/generateEqlSchema.ts @@ -0,0 +1,28 @@ +import fs from 'node:fs/promises' +import { execa } from 'execa' + +async function main() { + const url = + 'https://raw.githubusercontent.com/cipherstash/encrypt-query-language/main/sql/schemas/cs_encrypted_storage_v1.schema.json' + + const response = await fetch(url) + + if (!response.ok) { + throw new Error(`Failed to fetch schema, status = ${response.status}`) + } + + const data = await response.json() + + await fs.writeFile('./eql.schema.json', JSON.stringify(data, null, 2)) + + await execa('pnpm', ['run', 'eql:generate'], { stdio: 'inherit' }) + + console.log( + 'The EQL schema has been updated from the source repo and the types have been generated. See the `eql.schema.json` file for the latest schema.', + ) +} + +main().catch((err) => { + console.error(err) + process.exit(1) +}) diff --git a/packages/protect/generateSchema.mjs b/packages/protect/generateSchema.mjs deleted file mode 100644 index 34b635abe..000000000 --- a/packages/protect/generateSchema.mjs +++ /dev/null @@ -1,26 +0,0 @@ -#!/usr/bin/env node - -import fs from 'node:fs/promises' -import { execa } from 'execa' - -async function main() { - const url = - 'https://raw.githubusercontent.com/cipherstash/encrypt-query-language/main/sql/schemas/cs_plaintext_v1.schema.json' - - const response = await fetch(url) - const data = await response.json() - - await fs.writeFile( - './cs_plaintext_v1.schema.json', - JSON.stringify(data, null, 2), - ) - - await execa('pnpm', ['run', 'generate-types'], { stdio: 'inherit' }) - - console.log('Types generated!') -} - -main().catch((err) => { - console.error(err) - process.exit(1) -}) diff --git a/packages/protect/package.json b/packages/protect/package.json index 5e4a85161..c123465e3 100644 --- a/packages/protect/package.json +++ b/packages/protect/package.json @@ -1,6 +1,6 @@ { "name": "@cipherstash/protect", - "version": "6.0.0", + "version": "6.0.0-next.0", "description": "CipherStash Protect for JavaScript", "keywords": [ "encrypted", @@ -35,8 +35,8 @@ "scripts": { "build": "tsup", "dev": "tsup --watch", - "update-schema": "node ./generateSchema.mjs", - "generate-types": "json2ts ./cs_plaintext_v1.schema.json --output ./src/cs_plaintext_v1.ts", + "eql:update": "tsx ./generateEqlSchema.ts", + "eql:generate": "json2ts ./eql.schema.json --output ./src/eql.schema.ts", "test": "vitest run", "release": "tsup" }, @@ -45,6 +45,7 @@ "execa": "^9.5.2", "json-schema-to-typescript": "^15.0.2", "tsup": "^8.3.0", + "tsx": "^4.19.3", "vitest": "^2.1.9" }, "peerDependencies": { @@ -55,7 +56,7 @@ }, "dependencies": { "@byteslice/result": "^0.2.0", - "@cipherstash/protect-ffi": "0.12.0-3", + "@cipherstash/protect-ffi": "0.12.0-4", "zod": "^3.24.2" }, "optionalDependencies": { diff --git a/packages/protect/src/cs_plaintext_v1.ts b/packages/protect/src/cs_plaintext_v1.ts deleted file mode 100644 index 37c80d510..000000000 --- a/packages/protect/src/cs_plaintext_v1.ts +++ /dev/null @@ -1,33 +0,0 @@ -/* eslint-disable */ -/** - * This file was automatically generated by json-schema-to-typescript. - * DO NOT MODIFY IT BY HAND. Instead, modify the source JSONSchema file, - * and run json-schema-to-typescript to regenerate this file. - */ - -export type SchemaVersion = number -export type Kind = 'pt' -export type Table = string -export type Column = string -export type Plaintext = string -/** - * Specifies that the plaintext should be encrypted for a specific query operation. If null, source encryption and encryption for all indexes will be performed. - */ -export type ForQuery = 'match' | 'ore' | 'unique' | 'ste_vec' | 'ejson_path' - -/** - * The EQL plaintext JSON payload sent by a client (such as an application) to CipherStash Proxy. - */ -export interface CsPlaintextV1Schema { - v: SchemaVersion - k: Kind - i: Ident - p: Plaintext - q?: ForQuery - [k: string]: unknown -} -export interface Ident { - t: Table - c: Column - [k: string]: unknown -} diff --git a/packages/protect/src/eql.schema.ts b/packages/protect/src/eql.schema.ts new file mode 100644 index 000000000..8add2fc08 --- /dev/null +++ b/packages/protect/src/eql.schema.ts @@ -0,0 +1,51 @@ +/* eslint-disable */ +/** + * This file was automatically generated by json-schema-to-typescript. + * DO NOT MODIFY IT BY HAND. Instead, modify the source JSONSchema file, + * and run json-schema-to-typescript to regenerate this file. + */ + +/** + * The EQL encrypted JSON payload used for storage. + */ +export type EqlSchema = { + v: SchemaVersion; + k: Kind; + i: Ident; + [k: string]: unknown; +} & ( + | { + k?: "ct"; + c: Ciphertext; + u?: UniqueIndex; + o?: OreIndex; + m?: MatchIndex; + [k: string]: unknown; + } + | { + k?: "sv"; + sv: StructuredEncryptionVector; + [k: string]: unknown; + } +); +export type SchemaVersion = number; +export type Kind = "ct" | "sv"; +export type Table = string; +export type Column = string; +export type Ciphertext = string; +export type UniqueIndex = string; +/** + * @minItems 1 + */ +export type OreIndex = [string, ...string[]]; +/** + * @minItems 1 + */ +export type MatchIndex = [number, ...number[]]; +export type StructuredEncryptionVector = string[][]; + +export interface Ident { + t: Table; + c: Column; + [k: string]: unknown; +} diff --git a/packages/protect/src/eql/index.ts b/packages/protect/src/eql/index.ts deleted file mode 100644 index 62b154d1d..000000000 --- a/packages/protect/src/eql/index.ts +++ /dev/null @@ -1,66 +0,0 @@ -import { logger } from '../../../utils/logger' -import type { - Column, - CsPlaintextV1Schema, - ForQuery, - Plaintext, - SchemaVersion, - Table, -} from '../cs_plaintext_v1' - -export type CreateEqlPayload = { - plaintext: Plaintext - table: Table - column: Column - schemaVersion?: SchemaVersion - queryType?: ForQuery | null -} - -export type GetPlaintextResult = { - failure?: boolean - error?: Error - plaintext?: Plaintext -} - -export const createEqlPayload = ({ - plaintext, - table, - column, - schemaVersion = 1, - queryType = null, -}: CreateEqlPayload): CsPlaintextV1Schema => { - const payload: CsPlaintextV1Schema = { - v: schemaVersion, - k: 'pt', - p: plaintext ?? '', - i: { - t: table, - c: column, - }, - } - - if (queryType) { - payload.q = queryType - } - - logger.debug('Creating the EQL payload', payload) - return payload -} - -export const getPlaintext = ( - payload: CsPlaintextV1Schema, -): GetPlaintextResult => { - if (payload?.p && payload?.k === 'pt') { - logger.debug('Returning the plaintext data from the EQL payload', payload) - return { - failure: false, - plaintext: payload.p, - } - } - - logger.error('No plaintext data found in the EQL payload', payload ?? {}) - return { - failure: true, - error: new Error('No plaintext data found in the EQL payload'), - } -} diff --git a/packages/protect/src/ffi/index.ts b/packages/protect/src/ffi/index.ts index f6f4f6204..848a81851 100644 --- a/packages/protect/src/ffi/index.ts +++ b/packages/protect/src/ffi/index.ts @@ -10,6 +10,7 @@ import { type ProtectError, ProtectErrorTypes } from '..' import { loadWorkSpaceId } from '../../../utils/config' import { logger } from '../../../utils/logger' import type { LockContext } from '../identify' +import type { EqlSchema } from '../eql.schema' import { normalizeBulkDecryptPayloads, normalizeBulkDecryptPayloadsWithLockContext, @@ -28,8 +29,7 @@ import { // Type Definitions // ------------------------ export type EncryptPayload = string | null - -export type EncryptedPayload = { c: string } | null +export type EncryptedData = EqlSchema | null export type BulkEncryptPayload = { plaintext: string @@ -38,9 +38,7 @@ export type BulkEncryptPayload = { export type BulkEncryptedData = | { - encryptedPayload: { - c: string - } + encryptedData: EncryptedData id: string }[] | null @@ -71,7 +69,7 @@ const noClientError = () => // Encrhyption operation implementations // ------------------------ class EncryptOperation - implements PromiseLike> + implements PromiseLike> { private client: Client private plaintext: EncryptPayload @@ -92,13 +90,10 @@ class EncryptOperation } /** Implement the PromiseLike interface so `await` works. */ - public then< - TResult1 = Result, - TResult2 = never, - >( + public then, TResult2 = never>( onfulfilled?: | (( - value: Result, + value: Result, ) => TResult1 | PromiseLike) | null, // biome-ignore lint/suspicious/noExplicitAny: Rejections require an any type @@ -108,7 +103,7 @@ class EncryptOperation } /** Actual encryption logic, deferred until `then()` is called. */ - private async execute(): Promise> { + private async execute(): Promise> { logger.debug('Encrypting data WITHOUT a lock context', { column: this.column.getName(), table: this.table.tableName, @@ -156,7 +151,7 @@ class EncryptOperation } class EncryptOperationWithLockContext - implements PromiseLike> + implements PromiseLike> { private operation: EncryptOperation private lockContext: LockContext @@ -166,13 +161,10 @@ class EncryptOperationWithLockContext this.lockContext = lockContext } - public then< - TResult1 = Result, - TResult2 = never, - >( + public then, TResult2 = never>( onfulfilled?: | (( - value: Result, + value: Result, ) => TResult1 | PromiseLike) | null, // biome-ignore lint/suspicious/noExplicitAny: Rejections require an any type @@ -181,7 +173,7 @@ class EncryptOperationWithLockContext return this.execute().then(onfulfilled, onrejected) } - private async execute(): Promise> { + private async execute(): Promise> { return await withResult( async () => { const { client, plaintext, column, table } = @@ -200,7 +192,7 @@ class EncryptOperationWithLockContext return null } - const context = await this.lockContext?.getLockContext() + const context = await this.lockContext.getLockContext() if (context.failure) { throw new Error(`[protect]: ${context.failure.message}`) @@ -214,7 +206,8 @@ class EncryptOperationWithLockContext context.data.context, context.data.ctsToken, ) - return { c: val } + + return JSON.parse(val) }, (error) => ({ type: ProtectErrorTypes.EncryptionError, @@ -231,11 +224,11 @@ class DecryptOperation implements PromiseLike> { private client: Client - private encryptedPayload: EncryptedPayload + private encryptedData: EncryptedData - constructor(client: Client, encryptedPayload: EncryptedPayload) { + constructor(client: Client, encryptedData: EncryptedData) { this.client = client - this.encryptedPayload = encryptedPayload + this.encryptedData = encryptedData } public withLockContext( @@ -263,12 +256,18 @@ class DecryptOperation throw noClientError() } - if (this.encryptedPayload === null) { + if (this.encryptedData === null) { return null } + if (this.encryptedData.k !== 'ct') { + throw new Error( + 'The encrypted data is not compliant with the EQL schema', + ) + } + logger.debug('Decrypting data WITHOUT a lock context') - return await ffiDecrypt(this.client, this.encryptedPayload.c) + return await ffiDecrypt(this.client, this.encryptedData.c) }, (error) => ({ type: ProtectErrorTypes.DecryptionError, @@ -279,11 +278,11 @@ class DecryptOperation public getOperation(): { client: Client - encryptedPayload: EncryptedPayload + encryptedData: EncryptedData } { return { client: this.client, - encryptedPayload: this.encryptedPayload, + encryptedData: this.encryptedData, } } } @@ -314,27 +313,33 @@ class DecryptOperationWithLockContext private async execute(): Promise> { return await withResult( async () => { - const { client, encryptedPayload } = this.operation.getOperation() + const { client, encryptedData } = this.operation.getOperation() if (!client) { throw noClientError() } - if (encryptedPayload === null) { + if (encryptedData === null) { return null } logger.debug('Decrypting data WITH a lock context') - const context = await this.lockContext?.getLockContext() + const context = await this.lockContext.getLockContext() if (context.failure) { throw new Error(`[protect]: ${context.failure.message}`) } + if (encryptedData.k !== 'ct') { + throw new Error( + 'The encrypted data is not compliant with the EQL schema', + ) + } + return await ffiDecrypt( client, - encryptedPayload.c, + encryptedData.c, context.data.context, context.data.ctsToken, ) @@ -414,7 +419,7 @@ class BulkEncryptOperation const encryptedData = await ffiEncryptBulk(this.client, encryptPayloads) return encryptedData.map((enc, index) => ({ - encryptedPayload: JSON.parse(enc), + encryptedData: JSON.parse(enc), id: this.plaintexts[index].id, })) }, @@ -510,7 +515,7 @@ class BulkEncryptOperationWithLockContext ) return encryptedData.map((enc, index) => ({ - encryptedPayload: JSON.parse(enc), + encryptedData: JSON.parse(enc), id: plaintexts[index].id, })) }, @@ -529,11 +534,11 @@ class BulkDecryptOperation implements PromiseLike> { private client: Client - private encryptedPayloads: BulkEncryptedData + private encryptedDatas: BulkEncryptedData - constructor(client: Client, encryptedPayloads: BulkEncryptedData) { + constructor(client: Client, encryptedDatas: BulkEncryptedData) { this.client = client - this.encryptedPayloads = encryptedPayloads + this.encryptedDatas = encryptedDatas } public withLockContext( @@ -564,12 +569,12 @@ class BulkDecryptOperation throw noClientError() } - if (!this.encryptedPayloads) { + if (!this.encryptedDatas) { return null } const decryptPayloads = normalizeBulkDecryptPayloads( - this.encryptedPayloads, + this.encryptedDatas, ) if (!decryptPayloads) { @@ -580,10 +585,10 @@ class BulkDecryptOperation const decryptedData = await ffiDecryptBulk(this.client, decryptPayloads) return decryptedData.map((dec, index) => { - if (!this.encryptedPayloads) return null + if (!this.encryptedDatas) return null return { plaintext: dec, - id: this.encryptedPayloads[index].id, + id: this.encryptedDatas[index].id, } }) }, @@ -596,11 +601,11 @@ class BulkDecryptOperation public getOperation(): { client: Client - encryptedPayloads: BulkEncryptedData + encryptedDatas: BulkEncryptedData } { return { client: this.client, - encryptedPayloads: this.encryptedPayloads, + encryptedDatas: this.encryptedDatas, } } } @@ -634,19 +639,19 @@ class BulkDecryptOperationWithLockContext private async execute(): Promise> { return await withResult( async () => { - const { client, encryptedPayloads } = this.operation.getOperation() + const { client, encryptedDatas } = this.operation.getOperation() if (!client) { throw noClientError() } - if (!encryptedPayloads) { + if (!encryptedDatas) { return null } const decryptPayloads = await normalizeBulkDecryptPayloadsWithLockContext( - encryptedPayloads, + encryptedDatas, this.lockContext, ) @@ -673,10 +678,10 @@ class BulkDecryptOperationWithLockContext ) return decryptedData.map((dec, index) => { - if (!encryptedPayloads) return null + if (!encryptedDatas) return null return { plaintext: dec, - id: encryptedPayloads[index].id, + id: encryptedDatas[index].id, } }) }, @@ -753,11 +758,11 @@ export class ProtectClient { /** * Decryption - returns a thenable object. * Usage: - * await eqlClient.decrypt(encryptedPayload) - * await eqlClient.decrypt(encryptedPayload).withLockContext(lockContext) + * await eqlClient.decrypt(encryptedData) + * await eqlClient.decrypt(encryptedData).withLockContext(lockContext) */ - decrypt(encryptedPayload: EncryptedPayload): DecryptOperation { - return new DecryptOperation(this.client, encryptedPayload) + decrypt(encryptedData: EncryptedData): DecryptOperation { + return new DecryptOperation(this.client, encryptedData) } /** @@ -778,11 +783,11 @@ export class ProtectClient { /** * Bulk Decrypt - returns a thenable object. * Usage: - * await eqlClient.bulkDecrypt(encryptedPayloads) - * await eqlClient.bulkDecrypt(encryptedPayloads).withLockContext(lockContext) + * await eqlClient.bulkDecrypt(encryptedDatas) + * await eqlClient.bulkDecrypt(encryptedDatas).withLockContext(lockContext) */ - bulkDecrypt(encryptedPayloads: BulkEncryptedData): BulkDecryptOperation { - return new BulkDecryptOperation(this.client, encryptedPayloads) + bulkDecrypt(encryptedDatas: BulkEncryptedData): BulkDecryptOperation { + return new BulkDecryptOperation(this.client, encryptedDatas) } /** e.g., debugging or environment info */ diff --git a/packages/protect/src/ffi/payload-helpers.ts b/packages/protect/src/ffi/payload-helpers.ts index 34ce166d6..67f6bf46d 100644 --- a/packages/protect/src/ffi/payload-helpers.ts +++ b/packages/protect/src/ffi/payload-helpers.ts @@ -11,12 +11,18 @@ import type { BulkEncryptPayload, BulkEncryptedData } from './index' const getLockContextPayload = async (lockContext: LockContext) => await lockContext.getLockContext() -export const normalizeBulkDecryptPayloads = ( - encryptedPayloads: BulkEncryptedData, -) => - encryptedPayloads?.reduce((acc, data) => { +export const normalizeBulkDecryptPayloads = (payload: BulkEncryptedData) => + payload?.reduce((acc, data) => { + if (!data.encryptedData) { + return acc + } + + if (data.encryptedData.k !== 'ct') { + throw new Error('The encrypted data is not compliant with the EQL schema') + } + const payload = { - ciphertext: data.encryptedPayload.c, + ciphertext: data.encryptedData.c, } acc.push(payload) @@ -40,18 +46,28 @@ export const normalizeBulkEncryptPayloads = ( }, [] as InternalBulkEncryptPayload[]) export async function normalizeBulkDecryptPayloadsWithLockContext( - encryptedPayloads: BulkEncryptedData, + payloads: BulkEncryptedData, lockContext: LockContext, ): Promise> { const lockContextPayload = await getLockContextPayload(lockContext) if (lockContextPayload.failure) return lockContextPayload - if (!encryptedPayloads) return { data: [] } + if (!payloads) return { data: [] } return { - data: encryptedPayloads?.reduce((acc, data) => { + data: payloads.reduce((acc, data) => { + if (!data.encryptedData) { + return acc + } + + if (data.encryptedData.k !== 'ct') { + throw new Error( + 'The encrypted data is not compliant with the EQL schema', + ) + } + const payload = { - ciphertext: data.encryptedPayload.c, + ciphertext: data.encryptedData.c, ...lockContextPayload, } diff --git a/packages/protect/src/index.ts b/packages/protect/src/index.ts index 87c7d0a6b..bb78eb438 100644 --- a/packages/protect/src/index.ts +++ b/packages/protect/src/index.ts @@ -18,9 +18,16 @@ export interface ProtectError { message: string } +type AtLeastOneCsTable = [T, ...T[]] export const protect = async ( - ...tables: Array> + ...tables: AtLeastOneCsTable> ): Promise => { + if (!tables.length) { + throw new Error( + '[protect]: At least one csTable must be provided to initialize the protect client', + ) + } + const client = new ProtectClient() const encryptConfig = buildEncryptConfig(...tables) @@ -34,8 +41,6 @@ export const protect = async ( } export type { Result } from '@byteslice/result' -export type { ProtectClient } from './ffi' +export type { ProtectClient, EncryptedData } from './ffi' export { csTable, csColumn } from './schema' -export * from './cs_plaintext_v1' export * from './identify' -export * from './eql' diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 0c79fed6d..6eaf38b08 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -204,7 +204,7 @@ importers: version: 16.4.7 tsup: specifier: ^8.3.0 - version: 8.3.5(jiti@1.21.7)(postcss@8.5.1)(tsx@4.19.2)(typescript@5.7.2)(yaml@2.7.0) + version: 8.3.5(jiti@1.21.7)(postcss@8.5.1)(tsx@4.19.3)(typescript@5.7.2)(yaml@2.7.0) vitest: specifier: ^2.1.9 version: 2.1.9(@types/node@22.10.5) @@ -215,8 +215,8 @@ importers: specifier: ^0.2.0 version: 0.2.0 '@cipherstash/protect-ffi': - specifier: 0.12.0-3 - version: 0.12.0-3 + specifier: 0.12.0-4 + version: 0.12.0-4 typescript: specifier: ^5.0.0 version: 5.7.2 @@ -239,7 +239,10 @@ importers: version: 15.0.3 tsup: specifier: ^8.3.0 - version: 8.3.5(jiti@1.21.7)(postcss@8.5.1)(tsx@4.19.2)(typescript@5.7.2)(yaml@2.7.0) + version: 8.3.5(jiti@1.21.7)(postcss@8.5.1)(tsx@4.19.3)(typescript@5.7.2)(yaml@2.7.0) + tsx: + specifier: ^4.19.3 + version: 4.19.3 vitest: specifier: ^2.1.9 version: 2.1.9(@types/node@22.10.5) @@ -369,33 +372,33 @@ packages: '@changesets/write@0.3.2': resolution: {integrity: sha512-kDxDrPNpUgsjDbWBvUo27PzKX4gqeKOlhibaOXDJA6kuBisGqNHv/HwGJrAu8U/dSf8ZEFIeHIPtvSlZI1kULw==} - '@cipherstash/protect-ffi-darwin-arm64@0.12.0-3': - resolution: {integrity: sha512-LRrNEecK54EtIHcyDDywiksAMTKZfXb012xAeiHP0/6KrvUI6SpNyoz1HW/cDTbVBmc4j5sZru/LDv7qAU7Tmw==} + '@cipherstash/protect-ffi-darwin-arm64@0.12.0-4': + resolution: {integrity: sha512-F5r/RWs3JDkOKbWqL8uQf3Y62SsKCzE/QzwBdw52RTgsL+4o+09kUGT6y7f6q9ZT0vrjq1eftbf2m/tLj9x0Rg==} cpu: [arm64] os: [darwin] - '@cipherstash/protect-ffi-darwin-x64@0.12.0-3': - resolution: {integrity: sha512-eGCGchrZMkKcrwluXl787QP5d0GzmTP61dRDs3PyhYaMTgHUHlSIm/egVF1Tea+ru1D72KDajVSt6ApwgT77nA==} + '@cipherstash/protect-ffi-darwin-x64@0.12.0-4': + resolution: {integrity: sha512-Klzva/2SoaQv+fq8RXg2r+dJEb0a2N6gDs+RwQRCQackgwYgemZIz4mxRR67qww4DSddVE9d8zEKerO6O7Ip/g==} cpu: [x64] os: [darwin] - '@cipherstash/protect-ffi-linux-arm64-gnu@0.12.0-3': - resolution: {integrity: sha512-RqkfgxPqLmoAjduryQPpnahDUH8fme0iUBoqswdVat2LHCim0BSV/UwpcffAiuHDtnMARnilRBN0cFT8WsmQBQ==} + '@cipherstash/protect-ffi-linux-arm64-gnu@0.12.0-4': + resolution: {integrity: sha512-0cLxOX4QGvlhnjQwtYkgDm11m12ucF8sNZmFTt891jXocuwxYjYS6jvHZJySMQE96YjOEkHVLQBylD91SCWxMg==} cpu: [arm64] os: [linux] - '@cipherstash/protect-ffi-linux-x64-gnu@0.12.0-3': - resolution: {integrity: sha512-p5VrXNNFq8y0ka4HaAvm/NIPKLPjdMnCXCkZccJ7iX6lR4oQlVxZeXcrnv9JnBm4ZkDx+Z0grZtsHUH+vX2uRg==} + '@cipherstash/protect-ffi-linux-x64-gnu@0.12.0-4': + resolution: {integrity: sha512-TgF+Wr9QDWovW3JItnCrwzc2ddMFWC8lyKhT9aSdQyrE8yYvscCMnY2bSqQQiganud0fYL5ixYGydddmmfHyIg==} cpu: [x64] os: [linux] - '@cipherstash/protect-ffi-win32-x64-msvc@0.12.0-3': - resolution: {integrity: sha512-JKPFawWDLqeJ2rFCJerIW3K4SYj55QiR3TOaTImSYaetOHGA19Mt8eaBudNlA6GILkVLH0UmBX9VEOUvqd0wYg==} + '@cipherstash/protect-ffi-win32-x64-msvc@0.12.0-4': + resolution: {integrity: sha512-mi4WJhQS/zy1kkSTh+a75KlDc0vDfyUTm0rxNSX50hjCwvKjyBlVEWXDbbWBE5hOAU+Cjpa3JpH8GCLcOzHk6w==} cpu: [x64] os: [win32] - '@cipherstash/protect-ffi@0.12.0-3': - resolution: {integrity: sha512-AnjZNKDXpOPvEKQYAFwERXFY/uL/hPrrn8t2yY/oQwdQUNay3v0PSc6f7q1iEP1T24J7tvyf0PtIIX/oQJVtqA==} + '@cipherstash/protect-ffi@0.12.0-4': + resolution: {integrity: sha512-87UJe4Pl115kP8bCzxSYjw00ktp0hniYgopME0N0mTpkQhp7yOrdqWPBh0FMZNbQppgUs+cGmFEDTh5X7X/M6A==} '@clerk/backend@1.21.5': resolution: {integrity: sha512-fZ+HuHQkPngYp9vAEqsJ1XJkhWjSWTin90UixeV3jNrbuPY2jd70LKYTAIEvggnnG42pYxdVupaQDUfqj0dy4Q==} @@ -509,6 +512,12 @@ packages: cpu: [ppc64] os: [aix] + '@esbuild/aix-ppc64@0.25.0': + resolution: {integrity: sha512-O7vun9Sf8DFjH2UtqK8Ku3LkquL9SZL8OLY1T5NZkA34+wG3OQF7cl4Ql8vdNzM6fzBbYfLaiRLIOZ+2FOCgBQ==} + engines: {node: '>=18'} + cpu: [ppc64] + os: [aix] + '@esbuild/android-arm64@0.18.20': resolution: {integrity: sha512-Nz4rJcchGDtENV0eMKUNa6L12zz2zBDXuhj/Vjh18zGqB44Bi7MBMSXjgunJgjRhCmKOjnPuZp4Mb6OKqtMHLQ==} engines: {node: '>=12'} @@ -539,6 +548,12 @@ packages: cpu: [arm64] os: [android] + '@esbuild/android-arm64@0.25.0': + resolution: {integrity: sha512-grvv8WncGjDSyUBjN9yHXNt+cq0snxXbDxy5pJtzMKGmmpPxeAmAhWxXI+01lU5rwZomDgD3kJwulEnhTRUd6g==} + engines: {node: '>=18'} + cpu: [arm64] + os: [android] + '@esbuild/android-arm@0.18.20': resolution: {integrity: sha512-fyi7TDI/ijKKNZTUJAQqiG5T7YjJXgnzkURqmGj13C6dCqckZBLdl4h7bkhHt/t0WP+zO9/zwroDvANaOqO5Sw==} engines: {node: '>=12'} @@ -569,6 +584,12 @@ packages: cpu: [arm] os: [android] + '@esbuild/android-arm@0.25.0': + resolution: {integrity: sha512-PTyWCYYiU0+1eJKmw21lWtC+d08JDZPQ5g+kFyxP0V+es6VPPSUhM6zk8iImp2jbV6GwjX4pap0JFbUQN65X1g==} + engines: {node: '>=18'} + cpu: [arm] + os: [android] + '@esbuild/android-x64@0.18.20': resolution: {integrity: sha512-8GDdlePJA8D6zlZYJV/jnrRAi6rOiNaCC/JclcXpB+KIuvfBN4owLtgzY2bsxnx666XjJx2kDPUmnTtR8qKQUg==} engines: {node: '>=12'} @@ -599,6 +620,12 @@ packages: cpu: [x64] os: [android] + '@esbuild/android-x64@0.25.0': + resolution: {integrity: sha512-m/ix7SfKG5buCnxasr52+LI78SQ+wgdENi9CqyCXwjVR2X4Jkz+BpC3le3AoBPYTC9NHklwngVXvbJ9/Akhrfg==} + engines: {node: '>=18'} + cpu: [x64] + os: [android] + '@esbuild/darwin-arm64@0.18.20': resolution: {integrity: sha512-bxRHW5kHU38zS2lPTPOyuyTm+S+eobPUnTNkdJEfAddYgEcll4xkT8DB9d2008DtTbl7uJag2HuE5NZAZgnNEA==} engines: {node: '>=12'} @@ -629,6 +656,12 @@ packages: cpu: [arm64] os: [darwin] + '@esbuild/darwin-arm64@0.25.0': + resolution: {integrity: sha512-mVwdUb5SRkPayVadIOI78K7aAnPamoeFR2bT5nszFUZ9P8UpK4ratOdYbZZXYSqPKMHfS1wdHCJk1P1EZpRdvw==} + engines: {node: '>=18'} + cpu: [arm64] + os: [darwin] + '@esbuild/darwin-x64@0.18.20': resolution: {integrity: sha512-pc5gxlMDxzm513qPGbCbDukOdsGtKhfxD1zJKXjCCcU7ju50O7MeAZ8c4krSJcOIJGFR+qx21yMMVYwiQvyTyQ==} engines: {node: '>=12'} @@ -659,6 +692,12 @@ packages: cpu: [x64] os: [darwin] + '@esbuild/darwin-x64@0.25.0': + resolution: {integrity: sha512-DgDaYsPWFTS4S3nWpFcMn/33ZZwAAeAFKNHNa1QN0rI4pUjgqf0f7ONmXf6d22tqTY+H9FNdgeaAa+YIFUn2Rg==} + engines: {node: '>=18'} + cpu: [x64] + os: [darwin] + '@esbuild/freebsd-arm64@0.18.20': resolution: {integrity: sha512-yqDQHy4QHevpMAaxhhIwYPMv1NECwOvIpGCZkECn8w2WFHXjEwrBn3CeNIYsibZ/iZEUemj++M26W3cNR5h+Tw==} engines: {node: '>=12'} @@ -689,6 +728,12 @@ packages: cpu: [arm64] os: [freebsd] + '@esbuild/freebsd-arm64@0.25.0': + resolution: {integrity: sha512-VN4ocxy6dxefN1MepBx/iD1dH5K8qNtNe227I0mnTRjry8tj5MRk4zprLEdG8WPyAPb93/e4pSgi1SoHdgOa4w==} + engines: {node: '>=18'} + cpu: [arm64] + os: [freebsd] + '@esbuild/freebsd-x64@0.18.20': resolution: {integrity: sha512-tgWRPPuQsd3RmBZwarGVHZQvtzfEBOreNuxEMKFcd5DaDn2PbBxfwLcj4+aenoh7ctXcbXmOQIn8HI6mCSw5MQ==} engines: {node: '>=12'} @@ -719,6 +764,12 @@ packages: cpu: [x64] os: [freebsd] + '@esbuild/freebsd-x64@0.25.0': + resolution: {integrity: sha512-mrSgt7lCh07FY+hDD1TxiTyIHyttn6vnjesnPoVDNmDfOmggTLXRv8Id5fNZey1gl/V2dyVK1VXXqVsQIiAk+A==} + engines: {node: '>=18'} + cpu: [x64] + os: [freebsd] + '@esbuild/linux-arm64@0.18.20': resolution: {integrity: sha512-2YbscF+UL7SQAVIpnWvYwM+3LskyDmPhe31pE7/aoTMFKKzIc9lLbyGUpmmb8a8AixOL61sQ/mFh3jEjHYFvdA==} engines: {node: '>=12'} @@ -749,6 +800,12 @@ packages: cpu: [arm64] os: [linux] + '@esbuild/linux-arm64@0.25.0': + resolution: {integrity: sha512-9QAQjTWNDM/Vk2bgBl17yWuZxZNQIF0OUUuPZRKoDtqF2k4EtYbpyiG5/Dk7nqeK6kIJWPYldkOcBqjXjrUlmg==} + engines: {node: '>=18'} + cpu: [arm64] + os: [linux] + '@esbuild/linux-arm@0.18.20': resolution: {integrity: sha512-/5bHkMWnq1EgKr1V+Ybz3s1hWXok7mDFUMQ4cG10AfW3wL02PSZi5kFpYKrptDsgb2WAJIvRcDm+qIvXf/apvg==} engines: {node: '>=12'} @@ -779,6 +836,12 @@ packages: cpu: [arm] os: [linux] + '@esbuild/linux-arm@0.25.0': + resolution: {integrity: sha512-vkB3IYj2IDo3g9xX7HqhPYxVkNQe8qTK55fraQyTzTX/fxaDtXiEnavv9geOsonh2Fd2RMB+i5cbhu2zMNWJwg==} + engines: {node: '>=18'} + cpu: [arm] + os: [linux] + '@esbuild/linux-ia32@0.18.20': resolution: {integrity: sha512-P4etWwq6IsReT0E1KHU40bOnzMHoH73aXp96Fs8TIT6z9Hu8G6+0SHSw9i2isWrD2nbx2qo5yUqACgdfVGx7TA==} engines: {node: '>=12'} @@ -809,6 +872,12 @@ packages: cpu: [ia32] os: [linux] + '@esbuild/linux-ia32@0.25.0': + resolution: {integrity: sha512-43ET5bHbphBegyeqLb7I1eYn2P/JYGNmzzdidq/w0T8E2SsYL1U6un2NFROFRg1JZLTzdCoRomg8Rvf9M6W6Gg==} + engines: {node: '>=18'} + cpu: [ia32] + os: [linux] + '@esbuild/linux-loong64@0.18.20': resolution: {integrity: sha512-nXW8nqBTrOpDLPgPY9uV+/1DjxoQ7DoB2N8eocyq8I9XuqJ7BiAMDMf9n1xZM9TgW0J8zrquIb/A7s3BJv7rjg==} engines: {node: '>=12'} @@ -839,6 +908,12 @@ packages: cpu: [loong64] os: [linux] + '@esbuild/linux-loong64@0.25.0': + resolution: {integrity: sha512-fC95c/xyNFueMhClxJmeRIj2yrSMdDfmqJnyOY4ZqsALkDrrKJfIg5NTMSzVBr5YW1jf+l7/cndBfP3MSDpoHw==} + engines: {node: '>=18'} + cpu: [loong64] + os: [linux] + '@esbuild/linux-mips64el@0.18.20': resolution: {integrity: sha512-d5NeaXZcHp8PzYy5VnXV3VSd2D328Zb+9dEq5HE6bw6+N86JVPExrA6O68OPwobntbNJ0pzCpUFZTo3w0GyetQ==} engines: {node: '>=12'} @@ -869,6 +944,12 @@ packages: cpu: [mips64el] os: [linux] + '@esbuild/linux-mips64el@0.25.0': + resolution: {integrity: sha512-nkAMFju7KDW73T1DdH7glcyIptm95a7Le8irTQNO/qtkoyypZAnjchQgooFUDQhNAy4iu08N79W4T4pMBwhPwQ==} + engines: {node: '>=18'} + cpu: [mips64el] + os: [linux] + '@esbuild/linux-ppc64@0.18.20': resolution: {integrity: sha512-WHPyeScRNcmANnLQkq6AfyXRFr5D6N2sKgkFo2FqguP44Nw2eyDlbTdZwd9GYk98DZG9QItIiTlFLHJHjxP3FA==} engines: {node: '>=12'} @@ -899,6 +980,12 @@ packages: cpu: [ppc64] os: [linux] + '@esbuild/linux-ppc64@0.25.0': + resolution: {integrity: sha512-NhyOejdhRGS8Iwv+KKR2zTq2PpysF9XqY+Zk77vQHqNbo/PwZCzB5/h7VGuREZm1fixhs4Q/qWRSi5zmAiO4Fw==} + engines: {node: '>=18'} + cpu: [ppc64] + os: [linux] + '@esbuild/linux-riscv64@0.18.20': resolution: {integrity: sha512-WSxo6h5ecI5XH34KC7w5veNnKkju3zBRLEQNY7mv5mtBmrP/MjNBCAlsM2u5hDBlS3NGcTQpoBvRzqBcRtpq1A==} engines: {node: '>=12'} @@ -929,6 +1016,12 @@ packages: cpu: [riscv64] os: [linux] + '@esbuild/linux-riscv64@0.25.0': + resolution: {integrity: sha512-5S/rbP5OY+GHLC5qXp1y/Mx//e92L1YDqkiBbO9TQOvuFXM+iDqUNG5XopAnXoRH3FjIUDkeGcY1cgNvnXp/kA==} + engines: {node: '>=18'} + cpu: [riscv64] + os: [linux] + '@esbuild/linux-s390x@0.18.20': resolution: {integrity: sha512-+8231GMs3mAEth6Ja1iK0a1sQ3ohfcpzpRLH8uuc5/KVDFneH6jtAJLFGafpzpMRO6DzJ6AvXKze9LfFMrIHVQ==} engines: {node: '>=12'} @@ -959,6 +1052,12 @@ packages: cpu: [s390x] os: [linux] + '@esbuild/linux-s390x@0.25.0': + resolution: {integrity: sha512-XM2BFsEBz0Fw37V0zU4CXfcfuACMrppsMFKdYY2WuTS3yi8O1nFOhil/xhKTmE1nPmVyvQJjJivgDT+xh8pXJA==} + engines: {node: '>=18'} + cpu: [s390x] + os: [linux] + '@esbuild/linux-x64@0.18.20': resolution: {integrity: sha512-UYqiqemphJcNsFEskc73jQ7B9jgwjWrSayxawS6UVFZGWrAAtkzjxSqnoclCXxWtfwLdzU+vTpcNYhpn43uP1w==} engines: {node: '>=12'} @@ -989,12 +1088,24 @@ packages: cpu: [x64] os: [linux] + '@esbuild/linux-x64@0.25.0': + resolution: {integrity: sha512-9yl91rHw/cpwMCNytUDxwj2XjFpxML0y9HAOH9pNVQDpQrBxHy01Dx+vaMu0N1CKa/RzBD2hB4u//nfc+Sd3Cw==} + engines: {node: '>=18'} + cpu: [x64] + os: [linux] + '@esbuild/netbsd-arm64@0.24.2': resolution: {integrity: sha512-wuLK/VztRRpMt9zyHSazyCVdCXlpHkKm34WUyinD2lzK07FAHTq0KQvZZlXikNWkDGoT6x3TD51jKQ7gMVpopw==} engines: {node: '>=18'} cpu: [arm64] os: [netbsd] + '@esbuild/netbsd-arm64@0.25.0': + resolution: {integrity: sha512-RuG4PSMPFfrkH6UwCAqBzauBWTygTvb1nxWasEJooGSJ/NwRw7b2HOwyRTQIU97Hq37l3npXoZGYMy3b3xYvPw==} + engines: {node: '>=18'} + cpu: [arm64] + os: [netbsd] + '@esbuild/netbsd-x64@0.18.20': resolution: {integrity: sha512-iO1c++VP6xUBUmltHZoMtCUdPlnPGdBom6IrO4gyKPFFVBKioIImVooR5I83nTew5UOYrk3gIJhbZh8X44y06A==} engines: {node: '>=12'} @@ -1025,6 +1136,12 @@ packages: cpu: [x64] os: [netbsd] + '@esbuild/netbsd-x64@0.25.0': + resolution: {integrity: sha512-jl+qisSB5jk01N5f7sPCsBENCOlPiS/xptD5yxOx2oqQfyourJwIKLRA2yqWdifj3owQZCL2sn6o08dBzZGQzA==} + engines: {node: '>=18'} + cpu: [x64] + os: [netbsd] + '@esbuild/openbsd-arm64@0.23.1': resolution: {integrity: sha512-3x37szhLexNA4bXhLrCC/LImN/YtWis6WXr1VESlfVtVeoFJBRINPJ3f0a/6LV8zpikqoUg4hyXw0sFBt5Cr+Q==} engines: {node: '>=18'} @@ -1037,6 +1154,12 @@ packages: cpu: [arm64] os: [openbsd] + '@esbuild/openbsd-arm64@0.25.0': + resolution: {integrity: sha512-21sUNbq2r84YE+SJDfaQRvdgznTD8Xc0oc3p3iW/a1EVWeNj/SdUCbm5U0itZPQYRuRTW20fPMWMpcrciH2EJw==} + engines: {node: '>=18'} + cpu: [arm64] + os: [openbsd] + '@esbuild/openbsd-x64@0.18.20': resolution: {integrity: sha512-e5e4YSsuQfX4cxcygw/UCPIEP6wbIL+se3sxPdCiMbFLBWu0eiZOJ7WoD+ptCLrmjZBK1Wk7I6D/I3NglUGOxg==} engines: {node: '>=12'} @@ -1067,6 +1190,12 @@ packages: cpu: [x64] os: [openbsd] + '@esbuild/openbsd-x64@0.25.0': + resolution: {integrity: sha512-2gwwriSMPcCFRlPlKx3zLQhfN/2WjJ2NSlg5TKLQOJdV0mSxIcYNTMhk3H3ulL/cak+Xj0lY1Ym9ysDV1igceg==} + engines: {node: '>=18'} + cpu: [x64] + os: [openbsd] + '@esbuild/sunos-x64@0.18.20': resolution: {integrity: sha512-kDbFRFp0YpTQVVrqUd5FTYmWo45zGaXe0X8E1G/LKFC0v8x0vWrhOWSLITcCn63lmZIxfOMXtCfti/RxN/0wnQ==} engines: {node: '>=12'} @@ -1097,6 +1226,12 @@ packages: cpu: [x64] os: [sunos] + '@esbuild/sunos-x64@0.25.0': + resolution: {integrity: sha512-bxI7ThgLzPrPz484/S9jLlvUAHYMzy6I0XiU1ZMeAEOBcS0VePBFxh1JjTQt3Xiat5b6Oh4x7UC7IwKQKIJRIg==} + engines: {node: '>=18'} + cpu: [x64] + os: [sunos] + '@esbuild/win32-arm64@0.18.20': resolution: {integrity: sha512-ddYFR6ItYgoaq4v4JmQQaAI5s7npztfV4Ag6NrhiaW0RrnOXqBkgwZLofVTlq1daVTQNhtI5oieTvkRPfZrePg==} engines: {node: '>=12'} @@ -1127,6 +1262,12 @@ packages: cpu: [arm64] os: [win32] + '@esbuild/win32-arm64@0.25.0': + resolution: {integrity: sha512-ZUAc2YK6JW89xTbXvftxdnYy3m4iHIkDtK3CLce8wg8M2L+YZhIvO1DKpxrd0Yr59AeNNkTiic9YLf6FTtXWMw==} + engines: {node: '>=18'} + cpu: [arm64] + os: [win32] + '@esbuild/win32-ia32@0.18.20': resolution: {integrity: sha512-Wv7QBi3ID/rROT08SABTS7eV4hX26sVduqDOTe1MvGMjNd3EjOz4b7zeexIR62GTIEKrfJXKL9LFxTYgkyeu7g==} engines: {node: '>=12'} @@ -1157,6 +1298,12 @@ packages: cpu: [ia32] os: [win32] + '@esbuild/win32-ia32@0.25.0': + resolution: {integrity: sha512-eSNxISBu8XweVEWG31/JzjkIGbGIJN/TrRoiSVZwZ6pkC6VX4Im/WV2cz559/TXLcYbcrDN8JtKgd9DJVIo8GA==} + engines: {node: '>=18'} + cpu: [ia32] + os: [win32] + '@esbuild/win32-x64@0.18.20': resolution: {integrity: sha512-kTdfRcSiDfQca/y9QIkng02avJ+NCaQvrMejlsB3RRv5sE9rRoeBPISaZpKxHELzRxZyLvNts1P27W3wV+8geQ==} engines: {node: '>=12'} @@ -1187,6 +1334,12 @@ packages: cpu: [x64] os: [win32] + '@esbuild/win32-x64@0.25.0': + resolution: {integrity: sha512-ZENoHJBxA20C2zFzh6AI4fT6RraMzjYw4xKWemRTRmRVtN9c5DcH9r/f2ihEkMjOW5eGgrwCslG/+Y/3bL+DHQ==} + engines: {node: '>=18'} + cpu: [x64] + os: [win32] + '@floating-ui/core@1.6.9': resolution: {integrity: sha512-uMXCuQ3BItDUbAMhIXw7UPXRfAlOAvZzdK9BWpE60MCn+Svt3aLn9jsPTi/WNGlRUu2uI0v5S7JiIUsbsvh3fw==} @@ -2488,6 +2641,11 @@ packages: engines: {node: '>=18'} hasBin: true + esbuild@0.25.0: + resolution: {integrity: sha512-BXq5mqc8ltbaN34cDqWuYKyNhX8D/Z0J1xdtdQ8UcIIIyJyz+ZMKUt58tF3SrZ85jcfN/PZYhjR5uDQAYNVbuw==} + engines: {node: '>=18'} + hasBin: true + esprima@4.0.1: resolution: {integrity: sha512-eGuFFw7Upda+g4p+QHvnW0RyTX/SVeJBDM/gCtMARO0cLuT2HcEKnTPvhjV6aGeqrCB/sbNop0Kszm0jsaWU4A==} engines: {node: '>=4'} @@ -3438,6 +3596,11 @@ packages: engines: {node: '>=18.0.0'} hasBin: true + tsx@4.19.3: + resolution: {integrity: sha512-4H8vUNGNjQ4V2EOoGw005+c+dGuPSnhpPBPHBtsZdGZBk/iJb4kguGlPWaZTZ3q5nMtFOEsY0nRDlh9PJyd6SQ==} + engines: {node: '>=18.0.0'} + hasBin: true + turbo-darwin-64@2.1.1: resolution: {integrity: sha512-aYNuJpZlCoi0Htd79fl/2DywpewGKijdXeOfg9KzNuPVKzSMYlAXuAlNGh0MKjiOcyqxQGL7Mq9LFhwA0VpDpQ==} cpu: [x64] @@ -3846,30 +4009,30 @@ snapshots: human-id: 1.0.2 prettier: 2.8.8 - '@cipherstash/protect-ffi-darwin-arm64@0.12.0-3': + '@cipherstash/protect-ffi-darwin-arm64@0.12.0-4': optional: true - '@cipherstash/protect-ffi-darwin-x64@0.12.0-3': + '@cipherstash/protect-ffi-darwin-x64@0.12.0-4': optional: true - '@cipherstash/protect-ffi-linux-arm64-gnu@0.12.0-3': + '@cipherstash/protect-ffi-linux-arm64-gnu@0.12.0-4': optional: true - '@cipherstash/protect-ffi-linux-x64-gnu@0.12.0-3': + '@cipherstash/protect-ffi-linux-x64-gnu@0.12.0-4': optional: true - '@cipherstash/protect-ffi-win32-x64-msvc@0.12.0-3': + '@cipherstash/protect-ffi-win32-x64-msvc@0.12.0-4': optional: true - '@cipherstash/protect-ffi@0.12.0-3': + '@cipherstash/protect-ffi@0.12.0-4': dependencies: '@neon-rs/load': 0.1.82 optionalDependencies: - '@cipherstash/protect-ffi-darwin-arm64': 0.12.0-3 - '@cipherstash/protect-ffi-darwin-x64': 0.12.0-3 - '@cipherstash/protect-ffi-linux-arm64-gnu': 0.12.0-3 - '@cipherstash/protect-ffi-linux-x64-gnu': 0.12.0-3 - '@cipherstash/protect-ffi-win32-x64-msvc': 0.12.0-3 + '@cipherstash/protect-ffi-darwin-arm64': 0.12.0-4 + '@cipherstash/protect-ffi-darwin-x64': 0.12.0-4 + '@cipherstash/protect-ffi-linux-arm64-gnu': 0.12.0-4 + '@cipherstash/protect-ffi-linux-x64-gnu': 0.12.0-4 + '@cipherstash/protect-ffi-win32-x64-msvc': 0.12.0-4 '@clerk/backend@1.21.5(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': dependencies: @@ -4001,6 +4164,9 @@ snapshots: '@esbuild/aix-ppc64@0.24.2': optional: true + '@esbuild/aix-ppc64@0.25.0': + optional: true + '@esbuild/android-arm64@0.18.20': optional: true @@ -4016,6 +4182,9 @@ snapshots: '@esbuild/android-arm64@0.24.2': optional: true + '@esbuild/android-arm64@0.25.0': + optional: true + '@esbuild/android-arm@0.18.20': optional: true @@ -4031,6 +4200,9 @@ snapshots: '@esbuild/android-arm@0.24.2': optional: true + '@esbuild/android-arm@0.25.0': + optional: true + '@esbuild/android-x64@0.18.20': optional: true @@ -4046,6 +4218,9 @@ snapshots: '@esbuild/android-x64@0.24.2': optional: true + '@esbuild/android-x64@0.25.0': + optional: true + '@esbuild/darwin-arm64@0.18.20': optional: true @@ -4061,6 +4236,9 @@ snapshots: '@esbuild/darwin-arm64@0.24.2': optional: true + '@esbuild/darwin-arm64@0.25.0': + optional: true + '@esbuild/darwin-x64@0.18.20': optional: true @@ -4076,6 +4254,9 @@ snapshots: '@esbuild/darwin-x64@0.24.2': optional: true + '@esbuild/darwin-x64@0.25.0': + optional: true + '@esbuild/freebsd-arm64@0.18.20': optional: true @@ -4091,6 +4272,9 @@ snapshots: '@esbuild/freebsd-arm64@0.24.2': optional: true + '@esbuild/freebsd-arm64@0.25.0': + optional: true + '@esbuild/freebsd-x64@0.18.20': optional: true @@ -4106,6 +4290,9 @@ snapshots: '@esbuild/freebsd-x64@0.24.2': optional: true + '@esbuild/freebsd-x64@0.25.0': + optional: true + '@esbuild/linux-arm64@0.18.20': optional: true @@ -4121,6 +4308,9 @@ snapshots: '@esbuild/linux-arm64@0.24.2': optional: true + '@esbuild/linux-arm64@0.25.0': + optional: true + '@esbuild/linux-arm@0.18.20': optional: true @@ -4136,6 +4326,9 @@ snapshots: '@esbuild/linux-arm@0.24.2': optional: true + '@esbuild/linux-arm@0.25.0': + optional: true + '@esbuild/linux-ia32@0.18.20': optional: true @@ -4151,6 +4344,9 @@ snapshots: '@esbuild/linux-ia32@0.24.2': optional: true + '@esbuild/linux-ia32@0.25.0': + optional: true + '@esbuild/linux-loong64@0.18.20': optional: true @@ -4166,6 +4362,9 @@ snapshots: '@esbuild/linux-loong64@0.24.2': optional: true + '@esbuild/linux-loong64@0.25.0': + optional: true + '@esbuild/linux-mips64el@0.18.20': optional: true @@ -4181,6 +4380,9 @@ snapshots: '@esbuild/linux-mips64el@0.24.2': optional: true + '@esbuild/linux-mips64el@0.25.0': + optional: true + '@esbuild/linux-ppc64@0.18.20': optional: true @@ -4196,6 +4398,9 @@ snapshots: '@esbuild/linux-ppc64@0.24.2': optional: true + '@esbuild/linux-ppc64@0.25.0': + optional: true + '@esbuild/linux-riscv64@0.18.20': optional: true @@ -4211,6 +4416,9 @@ snapshots: '@esbuild/linux-riscv64@0.24.2': optional: true + '@esbuild/linux-riscv64@0.25.0': + optional: true + '@esbuild/linux-s390x@0.18.20': optional: true @@ -4226,6 +4434,9 @@ snapshots: '@esbuild/linux-s390x@0.24.2': optional: true + '@esbuild/linux-s390x@0.25.0': + optional: true + '@esbuild/linux-x64@0.18.20': optional: true @@ -4241,9 +4452,15 @@ snapshots: '@esbuild/linux-x64@0.24.2': optional: true + '@esbuild/linux-x64@0.25.0': + optional: true + '@esbuild/netbsd-arm64@0.24.2': optional: true + '@esbuild/netbsd-arm64@0.25.0': + optional: true + '@esbuild/netbsd-x64@0.18.20': optional: true @@ -4259,12 +4476,18 @@ snapshots: '@esbuild/netbsd-x64@0.24.2': optional: true + '@esbuild/netbsd-x64@0.25.0': + optional: true + '@esbuild/openbsd-arm64@0.23.1': optional: true '@esbuild/openbsd-arm64@0.24.2': optional: true + '@esbuild/openbsd-arm64@0.25.0': + optional: true + '@esbuild/openbsd-x64@0.18.20': optional: true @@ -4280,6 +4503,9 @@ snapshots: '@esbuild/openbsd-x64@0.24.2': optional: true + '@esbuild/openbsd-x64@0.25.0': + optional: true + '@esbuild/sunos-x64@0.18.20': optional: true @@ -4295,6 +4521,9 @@ snapshots: '@esbuild/sunos-x64@0.24.2': optional: true + '@esbuild/sunos-x64@0.25.0': + optional: true + '@esbuild/win32-arm64@0.18.20': optional: true @@ -4310,6 +4539,9 @@ snapshots: '@esbuild/win32-arm64@0.24.2': optional: true + '@esbuild/win32-arm64@0.25.0': + optional: true + '@esbuild/win32-ia32@0.18.20': optional: true @@ -4325,6 +4557,9 @@ snapshots: '@esbuild/win32-ia32@0.24.2': optional: true + '@esbuild/win32-ia32@0.25.0': + optional: true + '@esbuild/win32-x64@0.18.20': optional: true @@ -4340,6 +4575,9 @@ snapshots: '@esbuild/win32-x64@0.24.2': optional: true + '@esbuild/win32-x64@0.25.0': + optional: true + '@floating-ui/core@1.6.9': dependencies: '@floating-ui/utils': 0.2.9 @@ -5434,6 +5672,34 @@ snapshots: '@esbuild/win32-ia32': 0.24.2 '@esbuild/win32-x64': 0.24.2 + esbuild@0.25.0: + optionalDependencies: + '@esbuild/aix-ppc64': 0.25.0 + '@esbuild/android-arm': 0.25.0 + '@esbuild/android-arm64': 0.25.0 + '@esbuild/android-x64': 0.25.0 + '@esbuild/darwin-arm64': 0.25.0 + '@esbuild/darwin-x64': 0.25.0 + '@esbuild/freebsd-arm64': 0.25.0 + '@esbuild/freebsd-x64': 0.25.0 + '@esbuild/linux-arm': 0.25.0 + '@esbuild/linux-arm64': 0.25.0 + '@esbuild/linux-ia32': 0.25.0 + '@esbuild/linux-loong64': 0.25.0 + '@esbuild/linux-mips64el': 0.25.0 + '@esbuild/linux-ppc64': 0.25.0 + '@esbuild/linux-riscv64': 0.25.0 + '@esbuild/linux-s390x': 0.25.0 + '@esbuild/linux-x64': 0.25.0 + '@esbuild/netbsd-arm64': 0.25.0 + '@esbuild/netbsd-x64': 0.25.0 + '@esbuild/openbsd-arm64': 0.25.0 + '@esbuild/openbsd-x64': 0.25.0 + '@esbuild/sunos-x64': 0.25.0 + '@esbuild/win32-arm64': 0.25.0 + '@esbuild/win32-ia32': 0.25.0 + '@esbuild/win32-x64': 0.25.0 + esprima@4.0.1: {} estree-walker@3.0.3: @@ -5902,13 +6168,13 @@ snapshots: postcss: 8.5.1 ts-node: 10.9.2(@types/node@20.17.12)(typescript@5.7.2) - postcss-load-config@6.0.1(jiti@1.21.7)(postcss@8.5.1)(tsx@4.19.2)(yaml@2.7.0): + postcss-load-config@6.0.1(jiti@1.21.7)(postcss@8.5.1)(tsx@4.19.3)(yaml@2.7.0): dependencies: lilconfig: 3.1.3 optionalDependencies: jiti: 1.21.7 postcss: 8.5.1 - tsx: 4.19.2 + tsx: 4.19.3 yaml: 2.7.0 postcss-nested@6.2.0(postcss@8.5.1): @@ -6338,7 +6604,7 @@ snapshots: tslib@2.8.1: {} - tsup@8.3.5(jiti@1.21.7)(postcss@8.5.1)(tsx@4.19.2)(typescript@5.7.2)(yaml@2.7.0): + tsup@8.3.5(jiti@1.21.7)(postcss@8.5.1)(tsx@4.19.3)(typescript@5.7.2)(yaml@2.7.0): dependencies: bundle-require: 5.1.0(esbuild@0.24.2) cac: 6.7.14 @@ -6348,7 +6614,7 @@ snapshots: esbuild: 0.24.2 joycon: 3.1.1 picocolors: 1.1.1 - postcss-load-config: 6.0.1(jiti@1.21.7)(postcss@8.5.1)(tsx@4.19.2)(yaml@2.7.0) + postcss-load-config: 6.0.1(jiti@1.21.7)(postcss@8.5.1)(tsx@4.19.3)(yaml@2.7.0) resolve-from: 5.0.0 rollup: 4.30.1 source-map: 0.8.0-beta.0 @@ -6372,6 +6638,13 @@ snapshots: optionalDependencies: fsevents: 2.3.3 + tsx@4.19.3: + dependencies: + esbuild: 0.25.0 + get-tsconfig: 4.8.1 + optionalDependencies: + fsevents: 2.3.3 + turbo-darwin-64@2.1.1: optional: true From 2ecd0a38302ecb3ee49956af968eb3defc8c7085 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Wed, 5 Mar 2025 07:35:29 -0700 Subject: [PATCH 16/30] chore: update lock file --- pnpm-lock.yaml | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 6eaf38b08..4e5307bb7 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -32,6 +32,12 @@ importers: dotenv: specifier: ^16.4.7 version: 16.4.7 + tsx: + specifier: ^4.19.2 + version: 4.19.3 + typescript: + specifier: ^5.0.0 + version: 5.7.2 apps/drizzle: dependencies: @@ -47,9 +53,6 @@ importers: postgres: specifier: ^3.4.4 version: 3.4.5 - typescript: - specifier: ^5.0.0 - version: 5.7.2 devDependencies: '@types/node': specifier: ^22.10.2 @@ -66,6 +69,9 @@ importers: tsx: specifier: ^4.19.2 version: 4.19.2 + typescript: + specifier: ^5.0.0 + version: 5.7.2 apps/hono-supabase: dependencies: From b75cf97b4960ae6d655e71e07f874a7f47669c64 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Wed, 5 Mar 2025 11:15:25 -0700 Subject: [PATCH 17/30] docs(branding): improve heading in the main README --- README.md | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 046fc4278..995274f32 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,25 @@ -# Protect.js +

+ CipherStash Logo +
-[![Tests](https://github.com/cipherstash/protectjs/actions/workflows/tests.yml/badge.svg)](https://github.com/cipherstash/protectjs/actions/workflows/tests.yml) -[![Built by CipherStash](https://raw.githubusercontent.com/cipherstash/meta/refs/heads/main/csbadge.svg)](https://cipherstash.com) + Protect.js

+

+ Implement robust data security without sacrificing performance or usability +
+ Built by CipherStash +

+
+ + + +## What's Protect.js? Protect.js is a TypeScript package for encrypting and decrypting data. Encryption operations happen directly in your app, and the ciphertext is stored in your database. Every value you encrypt with Protect.js has a unique key, made possible by CipherStash [ZeroKMS](https://cipherstash.com/products/zerokms)'s blazing fast bulk key operations, and backed by a root key in [AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html). -The encrypted data is structured as an [Encrypt Query Language (EQL)](https://github.com/cipherstash/encrypt-query-language) JSON payload, and can be stored in any database that supports JSONB. +The encrypted data is structured as an [EQL](https://github.com/cipherstash/encrypt-query-language) JSON payload, and can be stored in any database that supports JSONB. > [!IMPORTANT] > Searching, sorting, and filtering on encrypted data is only supported in PostgreSQL at the moment. @@ -43,8 +54,9 @@ This enables every encrypted value, in every column, in every row in your databa - **Single item encryption and decryption**: Just looking for a way to encrypt and decrypt single values? Protect.js has you covered. - **Really fast:** ZeroKMS's performance makes using millions of unique keys feasible and performant for real-world applications built with Protect.js. - **Identity-aware encryption**: Lock down access to sensitive data by requiring a valid JWT to perform a decryption. -- **TypeScript support**: Strongly typed with TypeScript interfaces and types. +- **Audit trail**: Every decryption event will be logged in ZeroKMS to help you prove compliance. - **Searchable encryption**: Protect.js supports searching encrypted data in PostgreSQL. +- **TypeScript support**: Strongly typed with TypeScript interfaces and types. **Use cases:** - **Trusted data access**: make sure only your end-users can access their sensitive data stored in your product. @@ -60,7 +72,7 @@ Check out the example applications: - [Drizzle example](/apps/drizzle) demonstrates how to use Protect.js with an ORM - [Next.js and lock contexts example using Clerk](/apps/nextjs-clerk) demonstrates how to protect data with identity-aware encryption -`@cipherstash/protect` can be used with most ORMs that support PostgreSQL. +`@cipherstash/protect` can be used with most ORMs. If you're interested in using `@cipherstash/protect` with a specific ORM, please [create an issue](https://github.com/cipherstash/protectjs/issues/new). ## Installing Protect.js From ee78a1fbb8a2e5cbdf66b6724245b0316cb001f4 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Wed, 5 Mar 2025 11:45:01 -0700 Subject: [PATCH 18/30] build(protect): bump ffi to 0.12.0 --- packages/protect/package.json | 4 +-- pnpm-lock.yaml | 50 +++++++++++++++++------------------ 2 files changed, 27 insertions(+), 27 deletions(-) diff --git a/packages/protect/package.json b/packages/protect/package.json index c123465e3..de29da312 100644 --- a/packages/protect/package.json +++ b/packages/protect/package.json @@ -1,6 +1,6 @@ { "name": "@cipherstash/protect", - "version": "6.0.0-next.0", + "version": "6.0.0", "description": "CipherStash Protect for JavaScript", "keywords": [ "encrypted", @@ -56,7 +56,7 @@ }, "dependencies": { "@byteslice/result": "^0.2.0", - "@cipherstash/protect-ffi": "0.12.0-4", + "@cipherstash/protect-ffi": "0.12.0", "zod": "^3.24.2" }, "optionalDependencies": { diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 4e5307bb7..c0f5a8585 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -221,8 +221,8 @@ importers: specifier: ^0.2.0 version: 0.2.0 '@cipherstash/protect-ffi': - specifier: 0.12.0-4 - version: 0.12.0-4 + specifier: 0.12.0 + version: 0.12.0 typescript: specifier: ^5.0.0 version: 5.7.2 @@ -378,33 +378,33 @@ packages: '@changesets/write@0.3.2': resolution: {integrity: sha512-kDxDrPNpUgsjDbWBvUo27PzKX4gqeKOlhibaOXDJA6kuBisGqNHv/HwGJrAu8U/dSf8ZEFIeHIPtvSlZI1kULw==} - '@cipherstash/protect-ffi-darwin-arm64@0.12.0-4': - resolution: {integrity: sha512-F5r/RWs3JDkOKbWqL8uQf3Y62SsKCzE/QzwBdw52RTgsL+4o+09kUGT6y7f6q9ZT0vrjq1eftbf2m/tLj9x0Rg==} + '@cipherstash/protect-ffi-darwin-arm64@0.12.0': + resolution: {integrity: sha512-Oh7A9Mbn17QDHLLbugbT5bo/hZRrtgzcJZBTvXNTQpW9FtNmscCz1Wn79CJX+IhpdVrVxIQk7GOnG7JhTxJ/JA==} cpu: [arm64] os: [darwin] - '@cipherstash/protect-ffi-darwin-x64@0.12.0-4': - resolution: {integrity: sha512-Klzva/2SoaQv+fq8RXg2r+dJEb0a2N6gDs+RwQRCQackgwYgemZIz4mxRR67qww4DSddVE9d8zEKerO6O7Ip/g==} + '@cipherstash/protect-ffi-darwin-x64@0.12.0': + resolution: {integrity: sha512-1UkwOm5lUukIFPXYaI1KvlBy+WafMGX0qw5lYBq1ybFMOhQgAdCloRxON60yM1aEM6MCWgqw6J6eQbisLms/Ow==} cpu: [x64] os: [darwin] - '@cipherstash/protect-ffi-linux-arm64-gnu@0.12.0-4': - resolution: {integrity: sha512-0cLxOX4QGvlhnjQwtYkgDm11m12ucF8sNZmFTt891jXocuwxYjYS6jvHZJySMQE96YjOEkHVLQBylD91SCWxMg==} + '@cipherstash/protect-ffi-linux-arm64-gnu@0.12.0': + resolution: {integrity: sha512-zIOQoAgWXqPXE+wXQwiTu0+GNkLvXhN9US49nKezL2P+X97PuzZd0XaL+e5AXuDdrt5wC1WjUDvKvP1cQpNYPg==} cpu: [arm64] os: [linux] - '@cipherstash/protect-ffi-linux-x64-gnu@0.12.0-4': - resolution: {integrity: sha512-TgF+Wr9QDWovW3JItnCrwzc2ddMFWC8lyKhT9aSdQyrE8yYvscCMnY2bSqQQiganud0fYL5ixYGydddmmfHyIg==} + '@cipherstash/protect-ffi-linux-x64-gnu@0.12.0': + resolution: {integrity: sha512-wSJPi+pNR9+7DgcLZz/6V1VZAyejeEByIh+URfuRDaqbv8oxhh957WPfZTrWZXP5paIhXEdzkH62g6rnAxKaiQ==} cpu: [x64] os: [linux] - '@cipherstash/protect-ffi-win32-x64-msvc@0.12.0-4': - resolution: {integrity: sha512-mi4WJhQS/zy1kkSTh+a75KlDc0vDfyUTm0rxNSX50hjCwvKjyBlVEWXDbbWBE5hOAU+Cjpa3JpH8GCLcOzHk6w==} + '@cipherstash/protect-ffi-win32-x64-msvc@0.12.0': + resolution: {integrity: sha512-45icvAH+3QUaZVzCkpz4Xb8+EZ4jcQs0pd9Cvk6CMX4srkzdzGcN61NaEJmcWmYaq4XQqGSHYQi9afznpfvJow==} cpu: [x64] os: [win32] - '@cipherstash/protect-ffi@0.12.0-4': - resolution: {integrity: sha512-87UJe4Pl115kP8bCzxSYjw00ktp0hniYgopME0N0mTpkQhp7yOrdqWPBh0FMZNbQppgUs+cGmFEDTh5X7X/M6A==} + '@cipherstash/protect-ffi@0.12.0': + resolution: {integrity: sha512-h1EdM+erAWBCy37jq4XExTQY4Pheg+wqLRyBGDRkS9vgxOQTt2g4TquqYVaS+aFZsYE1pMw76aSJNdca9sP2jg==} '@clerk/backend@1.21.5': resolution: {integrity: sha512-fZ+HuHQkPngYp9vAEqsJ1XJkhWjSWTin90UixeV3jNrbuPY2jd70LKYTAIEvggnnG42pYxdVupaQDUfqj0dy4Q==} @@ -4015,30 +4015,30 @@ snapshots: human-id: 1.0.2 prettier: 2.8.8 - '@cipherstash/protect-ffi-darwin-arm64@0.12.0-4': + '@cipherstash/protect-ffi-darwin-arm64@0.12.0': optional: true - '@cipherstash/protect-ffi-darwin-x64@0.12.0-4': + '@cipherstash/protect-ffi-darwin-x64@0.12.0': optional: true - '@cipherstash/protect-ffi-linux-arm64-gnu@0.12.0-4': + '@cipherstash/protect-ffi-linux-arm64-gnu@0.12.0': optional: true - '@cipherstash/protect-ffi-linux-x64-gnu@0.12.0-4': + '@cipherstash/protect-ffi-linux-x64-gnu@0.12.0': optional: true - '@cipherstash/protect-ffi-win32-x64-msvc@0.12.0-4': + '@cipherstash/protect-ffi-win32-x64-msvc@0.12.0': optional: true - '@cipherstash/protect-ffi@0.12.0-4': + '@cipherstash/protect-ffi@0.12.0': dependencies: '@neon-rs/load': 0.1.82 optionalDependencies: - '@cipherstash/protect-ffi-darwin-arm64': 0.12.0-4 - '@cipherstash/protect-ffi-darwin-x64': 0.12.0-4 - '@cipherstash/protect-ffi-linux-arm64-gnu': 0.12.0-4 - '@cipherstash/protect-ffi-linux-x64-gnu': 0.12.0-4 - '@cipherstash/protect-ffi-win32-x64-msvc': 0.12.0-4 + '@cipherstash/protect-ffi-darwin-arm64': 0.12.0 + '@cipherstash/protect-ffi-darwin-x64': 0.12.0 + '@cipherstash/protect-ffi-linux-arm64-gnu': 0.12.0 + '@cipherstash/protect-ffi-linux-x64-gnu': 0.12.0 + '@cipherstash/protect-ffi-win32-x64-msvc': 0.12.0 '@clerk/backend@1.21.5(react-dom@19.0.0(react@19.0.0))(react@19.0.0)': dependencies: From b2d4163e42eca74b7792ec61b3bca3706a41cdf6 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Wed, 5 Mar 2025 12:25:15 -0700 Subject: [PATCH 19/30] docs(schema): add initial schema docs --- docs/schema.md | 94 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 92 insertions(+), 2 deletions(-) diff --git a/docs/schema.md b/docs/schema.md index 8f8ead684..ccbf4d092 100644 --- a/docs/schema.md +++ b/docs/schema.md @@ -1,3 +1,93 @@ -# Protect.js Schema +# Protect.js schema -... \ No newline at end of file +Protect.js lets you define a schema in TypeScript with properties that map to your database columns, and define indexes and casting for each column which are used when searching on encrypted data. + +## Creating schema files + +You can declare your Protect.js schema directly in TypeScript either in a single `schema.ts` file, or you can split your schema into multiple files. It's up to you. + +Example in a single file: + +``` +📦 + ├ 📂 src + │ ├ 📂 protect + │ │ └ 📜 schema.ts +``` + +or in multiple files: + +``` +📦 + ├ 📂 src + │ ├ 📂 protect + │ | └ 📂 schemas + │ │ └ 📜 users.ts + │ │ └ 📜 posts.ts +``` + +## Understanding schema files + +A schema represents a mapping of your database, and which columns you want to encrypt and index. Thus, it's a collection of tables and columns represented with `csTable` and `csColumn`. + +The below is pseudo-code for how these mappings are defined: + +```ts +import { csTable, csColumn } from "@cipherstash/protect"; + +export const tableNameInTypeScript = csTable("tableNameInDatabase", { + columnNameInTypeScript: csColumn("columnNameInDatabase"), +}); +``` + +## Defining your schema + +Now that you understand how your schema is defined, let's dive into how you can configure your schema. + +Start by importing the `csTable` and `csColumn` functions from `@cipherstash/protect` and create a new table with a column. + +```ts +import { csTable, csColumn } from "@cipherstash/protect"; + +export const protectedUsers = csTable("users", { + email: csColumn("email"), +}); +``` + +**Searchable encryption** + +If you are looking to enable searchable encryption in a PostgreSQL database, you must declaratively enable the indexes in your schema by chanining the index options to the column. + +```ts +import { csTable, csColumn } from "@cipherstash/protect"; + +export const protectedUsers = csTable("users", { + email: csColumn("email").freeTextSearch().equality().orderAndSort(), +}); +``` + +## Available index options + +The following index options are available for your schema: + +| **Method** | **Description** | **SQL equivalent** | +| ----------- | --------------- | ------------------ | +| equality | Enables a unique index for equality queries. | `WHERE email = 'example@example.com'` | +| freeTextSearch | Enables a match index for free text queries. | `WHERE description LIKE '%example%'` | +| orderAndSort | Enables an sorting and range queries index. | `ORDER BY price ASC` | + +You can chain these methods to your column to configure them in any combination. + +## Initializing the EQL client + +You will use your defined schemas to initialize the EQL client. +Simply import your schemas and pass them to the `protect` function. + +```ts +import { protect } from "@cipherstash/protect"; +import { protectedUsers } from "./schemas/users"; + +const protectClient = await protect(protectedUsers, ...); +``` + +The `protect` function requires at least one `csTable` to be passed in. \ No newline at end of file From f456d1e40102071cac784f210dcf82deaae8121c Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Wed, 5 Mar 2025 12:29:36 -0700 Subject: [PATCH 20/30] docs: add schema to root docs readme --- docs/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/README.md b/docs/README.md index 0db30994d..cc7fd9223 100644 --- a/docs/README.md +++ b/docs/README.md @@ -12,8 +12,9 @@ The main documentation for Protect.js is located in the [main README](https://gi - [Bulk encryption and decryption examples and best practices](bulk-encryption-and-decryption.md) - [Lock contexts with Clerk and Next.js](lock-context.md) +- [Schema definition](schema.md) -### Reference +### How-to guides - [Next.js build notes](nextjs.md) - [SST and serverless function notes](sst.md) \ No newline at end of file From 54e54d7e5140c42450b4ca4f37828688817cdbed Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Wed, 5 Mar 2025 12:31:12 -0700 Subject: [PATCH 21/30] docs: add coming soon for searchable encryption --- docs/searchable-encryption.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/searchable-encryption.md b/docs/searchable-encryption.md index e69de29bb..6832cd516 100644 --- a/docs/searchable-encryption.md +++ b/docs/searchable-encryption.md @@ -0,0 +1,3 @@ +# Searchable encryption + +Docs coming soon... sorry for the wait this is high priority. \ No newline at end of file From c2b18719f27a4f4fb07c7d785d550a88b7e298aa Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Wed, 5 Mar 2025 12:46:34 -0700 Subject: [PATCH 22/30] docs(configuration): correct file name --- docs/{configration.md => configuration.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/{configration.md => configuration.md} (100%) diff --git a/docs/configration.md b/docs/configuration.md similarity index 100% rename from docs/configration.md rename to docs/configuration.md From 393f3118c3dfb9b9c3168757b64a9552f76490aa Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Wed, 5 Mar 2025 14:56:35 -0700 Subject: [PATCH 23/30] docs: clean up supported data types --- README.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 995274f32..9fdb66653 100644 --- a/README.md +++ b/README.md @@ -248,7 +248,7 @@ The `encryptResult` will return one of the following: ### Decrypting data Use the `decrypt` function to decrypt data. -`decrypt` takes an encrypted data object, and an object with the lock context as parameters. +`decrypt` takes an encrypted data object as a parameter. ```typescript import { protectClient } from './protect' @@ -515,12 +515,10 @@ Learn more about [bulk decryption](./docs/bulk-encryption-decryption.md#bulk-dec ## Supported data types -`@cipherstash/protect` currently supports encrypting and decrypting text. -Other data types like booleans, dates, ints, floats, and JSON are extremely well supported in other CipherStash products, and will be coming to `@cipherstash/protect`. -Until support for other data types are available in `@cipherstash/protect`, you can: +Protect.js currently supports encrypting and decrypting text. +Other data types like booleans, dates, ints, floats, and JSON are well supported in other CipherStash products, and will be coming to Protect.js soon. -- Read [about how these data types work in EQL](https://github.com/cipherstash/encrypt-query-language/blob/main/docs/reference/INDEX.md) -- Express interest in this feature by adding a :+1: on this [GitHub Issue](https://github.com/cipherstash/protectjs/issues/48). +Until support for other data types are available, you can express interest in this feature by adding a :+1: on this [GitHub Issue](https://github.com/cipherstash/protectjs/issues/48). ## Searchable encryption From 32ca245a1342bc4100c9446fb786dffa01f122c3 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Wed, 5 Mar 2025 15:08:36 -0700 Subject: [PATCH 24/30] docs: update header image --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9fdb66653..6724bee23 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@

- CipherStash Logo + CipherStash Logo
Protect.js

From 414a080d356f1a341054d43ca2923b922973d982 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Thu, 6 Mar 2025 06:37:42 -0700 Subject: [PATCH 25/30] docs: add structure and searchable encryption concepts --- README.md | 18 ++--- apps/drizzle/src/select.ts | 12 ++-- docs/README.md | 23 ++++--- docs/concepts/searchable-encryption.md | 69 +++++++++++++++++++ .../lock-contexts-with-clerk.md} | 0 .../nextjs-external-packages.md} | 0 .../sst-external-packages.md} | 0 .../bulk-encryption-decryption.md | 0 docs/{ => reference}/configuration.md | 0 docs/{ => reference}/schema.md | 0 docs/searchable-encryption.md | 3 - 11 files changed, 96 insertions(+), 29 deletions(-) create mode 100644 docs/concepts/searchable-encryption.md rename docs/{lock-context.md => how-to/lock-contexts-with-clerk.md} (100%) rename docs/{nextjs.md => how-to/nextjs-external-packages.md} (100%) rename docs/{sst.md => how-to/sst-external-packages.md} (100%) rename docs/{ => reference}/bulk-encryption-decryption.md (100%) rename docs/{ => reference}/configuration.md (100%) rename docs/{ => reference}/schema.md (100%) delete mode 100644 docs/searchable-encryption.md diff --git a/README.md b/README.md index 6724bee23..f9d4f190f 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ The encrypted data is structured as an [EQL](https://github.com/cipherstash/encr > [!IMPORTANT] > Searching, sorting, and filtering on encrypted data is only supported in PostgreSQL at the moment. -> Read more about [searching encrypted data](./docs/searchable-encryption.md). +> Read more about [searching encrypted data](./docs/concepts/searchable-encryption.md). ## Table of contents @@ -133,7 +133,7 @@ At the end of `stash setup`, you will have two files in your project: > `cipherstash.secret.toml` should not be committed to git, because it contains sensitive credentials. > The `stash setup` command will attempt to append to your `.gitignore` file with the `cipherstash.secret.toml` file. -You can read more about [configuration via toml file or environment variables here](./docs/configuration.md). +You can read more about [configuration via toml file or environment variables here](./docs/reference/configuration.md). ### Basic file structure @@ -183,7 +183,7 @@ export const users = csTable('users', { }) ``` -Read more about [defining your schema here](./docs/schema.md). +Read more about [defining your schema here](./docs/reference/schema.md). ### Initializing the EQL client @@ -460,7 +460,7 @@ encryptedValues.forEach((result) => { }) ``` -Learn more about [bulk encryption](./docs/bulk-encryption-decryption.md#bulk-encrypting-data) +Learn more about [bulk encryption](./docs/reference/bulk-encryption-decryption.md#bulk-encrypting-data) ### Bulk decrypting data @@ -511,7 +511,7 @@ decryptedValues.forEach((result) => { }) ``` -Learn more about [bulk decryption](./docs/bulk-encryption-decryption.md#bulk-decrypting-data) +Learn more about [bulk decryption](./docs/reference/bulk-encryption-decryption.md#bulk-decrypting-data) ## Supported data types @@ -522,7 +522,7 @@ Until support for other data types are available, you can express interest in th ## Searchable encryption -Read more about [searching encrypted data](./docs/searchable-encryption.md) in the docs. +Read more about [searching encrypted data](./docs/concepts/searchable-encryption.md) in the docs. ## Logging @@ -544,7 +544,7 @@ PROTECT_LOG_LEVEL=error # Enable error logging Protect.js is built on top of the CipherStash Client Rust SDK which is embedded with the `@cipherstash/protect-ffi` package. The `@cipherstash/protect-ffi` source code is available on [GitHub](https://github.com/cipherstash/protectjs-ffi). -Read more about configuring the CipherStash client in the [configuration docs](./docs/configuration.md). +Read more about configuring the CipherStash client in the [configuration docs](./docs/reference/configuration.md). ## Builds and bundling @@ -552,8 +552,8 @@ Read more about configuring the CipherStash client in the [configuration docs](. Here are a few resources to help based on your tool set: -- [Required Next.js configuration](./docs/nextjs.md). -- [SST and AWS serverless functions](./docs/sst.md). +- [Required Next.js configuration](./docs/how-to/nextjs-external-packages.md). +- [SST and AWS serverless functions](./docs/how-to/sst-external-packages.md). ## Contributing diff --git a/apps/drizzle/src/select.ts b/apps/drizzle/src/select.ts index 7eccea4d3..414ab4042 100644 --- a/apps/drizzle/src/select.ts +++ b/apps/drizzle/src/select.ts @@ -47,13 +47,13 @@ const csEq: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { return sql`cs_unique_v1(${left}) = cs_unique_v1(${bindIfParam(right, left)})` } -// const csGt: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { -// return sql`cs_ore_64_8_v1(${left}) > cs_ore_64_8_v1(${bindIfParam(right, left)})`; -// }; +const csGt: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { + return sql`cs_ore_64_8_v1(${left}) > cs_ore_64_8_v1(${bindIfParam(right, left)})` +} -// const csLt: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { -// return sql`cs_ore_64_8_v1(${left}) < cs_ore_64_8_v1(${bindIfParam(right, left)})`; -// }; +const csLt: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { + return sql`cs_ore_64_8_v1(${left}) < cs_ore_64_8_v1(${bindIfParam(right, left)})` +} const csMatch: BinaryOperator = (left: SQLWrapper, right: unknown): SQL => { return sql`cs_match_v1(${left}) @> cs_match_v1(${bindIfParam(right, left)})` diff --git a/docs/README.md b/docs/README.md index cc7fd9223..4213a1685 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,20 +1,21 @@ # Protect.js Documentation -The main documentation for Protect.js is located in the [main README](https://github.com/cipherstash/protectjs). +The documentation for Protect.js is organized into the following sections: -## Docs table of contents +- [Getting started](../README.md) -### Setup +## Concepts -- [Configuration and production deployment](configuration.md) +- [Searchable encryption](./concepts/searchable-encryption.md) -### Usage +## Reference -- [Bulk encryption and decryption examples and best practices](bulk-encryption-and-decryption.md) -- [Lock contexts with Clerk and Next.js](lock-context.md) -- [Schema definition](schema.md) +- [Configuration and production deployment](./reference/configuration.md) +- [Bulk encryption and decryption](./reference/bulk-encryption-decryption.md) +- [Protect.js schemas](./reference/schema.md) -### How-to guides +## How-to guides -- [Next.js build notes](nextjs.md) -- [SST and serverless function notes](sst.md) \ No newline at end of file +- [Lock contexts with Clerk and Next.js](./how-to/lock-contexts-with-clerk.md) +- [Next.js build notes](./how-to/nextjs-external-packages.md) +- [SST and serverless function notes](./how-to/sst-external-packages.md) diff --git a/docs/concepts/searchable-encryption.md b/docs/concepts/searchable-encryption.md new file mode 100644 index 000000000..46402c06e --- /dev/null +++ b/docs/concepts/searchable-encryption.md @@ -0,0 +1,69 @@ +# Searchable encryption + +Protect.js supports searching encrypted data, which enabled trusted data access. + +## What does searchable encryption even mean? + +The best way to describe searchable encryption is with an example. +Let's say you have a table of users in your database, and you want to search for a user by their email address. + +```sql +SELECT * FROM users WHERE email = 'example@example.com'; +``` + +This is a pretty basic example, and you'd expect it to return something like: + +| id | name | email | +| --- | --- | --- | +| 1 | John | example@example.com | + +Whether you executed this query directly in the database, or through an application ORM, you'd expect the result to be the same. + +**But what if the email address is encrypted before it's stored in the database?** + +Executing: + +```sql +SELECT * FROM users; +``` + +Would return something like: + +| id | name | email | +| --- | --- | --- | +| 1 | John | mBbKmsMMkbKBSN... +| 2 | Jane | s1THy_NfQN892... +| 3 | Bob | 892!dercyd0s... + +The same equality query would return nothing, because `example@example.com` does not equal `mBbKmsMMkbKBSN...`. + +## How do you search on encrypted data? + +There is prior art for this, and it's called [Homomorphic Encryption](https://en.wikipedia.org/wiki/Homomorphic_encryption), and is defined as: + +> "a form of encryption that allows computations to be performed on encrypted data without first having to decrypt it" + +The issue with homomorphic encryption isn't around the functionality, but rather performance in a modern application use case. + +CipherStash's approach to searchable encryption solves the performance problem without sacrificing security, usability, or functionality. + +Based on some [benchmarks](https://github.com/cipherstash/tfhe-ore-bench?tab=readme-ov-file#results) CipherStash's approach is ***410,000x faster*** than homomorphic encryption: + +| Operation | Homomorphic | CipherStash | Speedup | +|--------------------|----------------|---------------|-------------| +| **Encrypt** | 1.97 ms | 48 µs | ~41× | +| **a == b** | 111 ms | 238 ns | ~466 000× | +| **a > b** | 192 ms | 238 ns | ~807 000× | +| **a < b** | 190 ms | 240 ns | ~792 000× | +| **a >= a** | 44 ms | 221 ns | ~199 000× | +| **a <= a** | 44 ms | 226 ns | ~195 000× | + +## Bringing it all together + +With searchable encryption: + +1. Data can be stored in the database encrypted. +2. Encrypted data can be searched using equality, free text search, and range queries. +3. Data remains encrypted, and will be decrypted using the Protect.js library in your application. +4. Queries are blazing fast, and won't slow down your application experience. +5. Every decryption event is logged, giving you an audit trail of data access events to help you prove compliance with your data protection policies. \ No newline at end of file diff --git a/docs/lock-context.md b/docs/how-to/lock-contexts-with-clerk.md similarity index 100% rename from docs/lock-context.md rename to docs/how-to/lock-contexts-with-clerk.md diff --git a/docs/nextjs.md b/docs/how-to/nextjs-external-packages.md similarity index 100% rename from docs/nextjs.md rename to docs/how-to/nextjs-external-packages.md diff --git a/docs/sst.md b/docs/how-to/sst-external-packages.md similarity index 100% rename from docs/sst.md rename to docs/how-to/sst-external-packages.md diff --git a/docs/bulk-encryption-decryption.md b/docs/reference/bulk-encryption-decryption.md similarity index 100% rename from docs/bulk-encryption-decryption.md rename to docs/reference/bulk-encryption-decryption.md diff --git a/docs/configuration.md b/docs/reference/configuration.md similarity index 100% rename from docs/configuration.md rename to docs/reference/configuration.md diff --git a/docs/schema.md b/docs/reference/schema.md similarity index 100% rename from docs/schema.md rename to docs/reference/schema.md diff --git a/docs/searchable-encryption.md b/docs/searchable-encryption.md deleted file mode 100644 index 6832cd516..000000000 --- a/docs/searchable-encryption.md +++ /dev/null @@ -1,3 +0,0 @@ -# Searchable encryption - -Docs coming soon... sorry for the wait this is high priority. \ No newline at end of file From 26691d37203c8f1748e29311fa3272626de1bed0 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Thu, 6 Mar 2025 06:45:08 -0700 Subject: [PATCH 26/30] docs(getting-started): add note about clerk support for lock contexts --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index f9d4f190f..b18bfc8fb 100644 --- a/README.md +++ b/README.md @@ -323,6 +323,10 @@ CREATE TABLE users ( ## Identity-aware encryption +> [!IMPORTANT] +> Identity-aware encryption is only supported if you are using [Clerk](https://clerk.com/) as your identity provider. +> Read more about [lock contexts with Clerk and Next.js](./docs/how-to/lock-contexts-with-clerk.md). + Protect.js can add an additional layer of protection to your data by requiring a valid JWT to perform a decryption. This ensures that only the user who encrypted data is able to decrypt it. From b980fbf6c84590bc326d670e073aedb21756d928 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Thu, 6 Mar 2025 08:01:37 -0700 Subject: [PATCH 27/30] docs(reference): add searchable encryption reference --- .../searchable-encryption-postgres.md | 111 ++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 docs/reference/searchable-encryption-postgres.md diff --git a/docs/reference/searchable-encryption-postgres.md b/docs/reference/searchable-encryption-postgres.md new file mode 100644 index 000000000..8a87d618c --- /dev/null +++ b/docs/reference/searchable-encryption-postgres.md @@ -0,0 +1,111 @@ +# Searchable encryption with Protect.js and PostgreSQL + +This reference guide outlines the different query patterns you can use to search encrypted data with Protect.js. + +You will have needed to [define your schema and initialized the protect client](../../README.md#defining-your-schema), and have [installed the EQL custom types and functions](../../README.md#searchable-encryption-in-postgresql). + +The below examples assume you have a schema defined: + +```ts +import { csTable, csColumn } from '@cipherstash/protect' + +export const protectedUsers = csTable('users', { + email: csColumn('email').equality().freeTextSearch().orderAndRange(), +}) +``` + +> [!TIP] +> To see an example using the [Drizzle ORM](https://github.com/drizzle-team/drizzle-orm) see the example [here](../../apps/drizzle/src/select.ts). + +## Equality query + +For an equality query, use the `cs_unique_v1` function. + +```ts +// 1) Encrypt the search term +const searchTerm = 'alice@example.com' + +const encryptedParam = await protectClient.encrypt(searchTerm, { + column: protectedUsers.email, // Your Protect column definition + table: protectedUsers, // Reference to the table schema +}) + +if (encryptedParam.failure) { + // Handle the failure +} + +// 2) Build an equality query (cs_unique_v1) +const equalitySQL = ` + SELECT email + FROM users + WHERE cs_unique_v1($1) = cs_unique_v1($2) +` + +// Use your flavor or ORM to execute the query. `client` is a PostgreSQL client. +const result = await client.query(equalitySQL, [ protectedUser.email.getName(), encryptedParam.data ]) +``` + +### Explanation + +`WHERE cs_unique_v1($1) = cs_unique_v1($2)` + +The first `$1` is the Postgres column name, and the second `$2` is the encrypted search term. + +## Free-Text search + +For partial matches or full-text searches, use the `cs_match_v1` function. + +```ts +// Suppose you're searching for emails containing "alice" +const searchTerm = 'alice' + +const encryptedParam = await protectClient.encrypt(searchTerm, { + column: protectedUsers.email, + table: protectedUsers, +}) + +if (encryptedParam.failure) { + // Handle the failure +} + +// Build and execute a "match" query (cs_match_v1) +const matchSQL = ` + SELECT email + FROM users + WHERE cs_match_v1($1) @> cs_match_v1($2) +` + +// Use your flavor or ORM to execute the query. `client` is a PostgreSQL client. +const result = await client.query(matchSQL, [ protectedUser.email.getName(), encryptedParam.data ]) +``` + +### Explanation + +`WHERE cs_match_v1($1) @> cs_match_v1($2)` + +The first `$1` is the Postgres column name, and the second `$2` is the encrypted search term. + +## Sorting data + +For `order by` queries, use the `cs_ore_64_8_v1` function. + +```ts +// Suppose you're sorting by email address +const orderSQL = ` + SELECT email + FROM users + ORDER BY cs_ore_64_8_v1(email) ASC +` +const orderedResult = await client.query(orderSQL) + +// Use your flavor or ORM to execute the query. `client` is a PostgreSQL client. +const result = await client.query(orderSQL) +``` + +### Explanation + +The `cs_ore_64_8_v1` function doesn't require any encrypted parameters, but rather the name of the Postgres column you want to sort by. + +## Range queries + +TODO: flesh this out (sorry it's not done yet) \ No newline at end of file From 7b7f308ab6448e95b057829167c6bc9a7c48697e Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Thu, 6 Mar 2025 08:09:08 -0700 Subject: [PATCH 28/30] feat(protect): rename orderAndSort to orderAndRange --- README.md | 8 ++++---- apps/drizzle/src/protect.ts | 2 +- docs/reference/schema.md | 8 ++++---- packages/protect/__tests__/protect.test.ts | 3 +-- packages/protect/src/schema/index.ts | 9 ++++----- 5 files changed, 14 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index b18bfc8fb..db4b250b8 100644 --- a/README.md +++ b/README.md @@ -179,13 +179,13 @@ If you are looking to enable searchable encryption in a PostgreSQL database, you import { csTable, csColumn } from '@cipherstash/protect' export const users = csTable('users', { - email: csColumn('email').freeTextSearch().equality().orderAndSort(), + email: csColumn('email').freeTextSearch().equality().orderAndRange(), }) ``` Read more about [defining your schema here](./docs/reference/schema.md). -### Initializing the EQL client +### Initializing the Protect client To initialize the protect client, import the `protect` function and initialize a client with your defined schema. @@ -296,7 +296,7 @@ CREATE TABLE users ( ); ``` -**Searchable encryption** +#### Searchable encryption in PostgreSQL To enable searchable encryption in PostgreSQL, you need to [install the EQL custom types and functions](https://github.com/cipherstash/encrypt-query-language?tab=readme-ov-file#installation). @@ -324,7 +324,7 @@ CREATE TABLE users ( ## Identity-aware encryption > [!IMPORTANT] -> Identity-aware encryption is only supported if you are using [Clerk](https://clerk.com/) as your identity provider. +> Right now identity-aware encryption is only supported if you are using [Clerk](https://clerk.com/) as your identity provider. > Read more about [lock contexts with Clerk and Next.js](./docs/how-to/lock-contexts-with-clerk.md). Protect.js can add an additional layer of protection to your data by requiring a valid JWT to perform a decryption. diff --git a/apps/drizzle/src/protect.ts b/apps/drizzle/src/protect.ts index e26d4f935..8e683cd50 100644 --- a/apps/drizzle/src/protect.ts +++ b/apps/drizzle/src/protect.ts @@ -4,7 +4,7 @@ import { protect, csColumn, csTable } from '@cipherstash/protect' export const users = csTable('users', { email_encrypted: csColumn('email_encrypted') .equality() - .orderAndSort() + .orderAndRange() .freeTextSearch(), }) diff --git a/docs/reference/schema.md b/docs/reference/schema.md index ccbf4d092..fc718481d 100644 --- a/docs/reference/schema.md +++ b/docs/reference/schema.md @@ -62,7 +62,7 @@ If you are looking to enable searchable encryption in a PostgreSQL database, you import { csTable, csColumn } from "@cipherstash/protect"; export const protectedUsers = csTable("users", { - email: csColumn("email").freeTextSearch().equality().orderAndSort(), + email: csColumn("email").freeTextSearch().equality().orderAndRange(), }); ``` @@ -72,13 +72,13 @@ The following index options are available for your schema: | **Method** | **Description** | **SQL equivalent** | | ----------- | --------------- | ------------------ | -| equality | Enables a unique index for equality queries. | `WHERE email = 'example@example.com'` | +| equality | Enables a exact index for equality queries. | `WHERE email = 'example@example.com'` | | freeTextSearch | Enables a match index for free text queries. | `WHERE description LIKE '%example%'` | -| orderAndSort | Enables an sorting and range queries index. | `ORDER BY price ASC` | +| orderAndRange | Enables an sorting and range queries index. | `ORDER BY price ASC` | You can chain these methods to your column to configure them in any combination. -## Initializing the EQL client +## Initializing the Protect client You will use your defined schemas to initialize the EQL client. Simply import your schemas and pass them to the `protect` function. diff --git a/packages/protect/__tests__/protect.test.ts b/packages/protect/__tests__/protect.test.ts index 98fe10e09..d658e2314 100644 --- a/packages/protect/__tests__/protect.test.ts +++ b/packages/protect/__tests__/protect.test.ts @@ -4,7 +4,7 @@ import { describe, expect, it } from 'vitest' import { LockContext, protect, csTable, csColumn } from '../src' const users = csTable('users', { - email: csColumn('email').freeTextSearch().equality().orderAndSort(), + email: csColumn('email').freeTextSearch().equality().orderAndRange(), }) describe('encryption and decryption', () => { @@ -47,7 +47,6 @@ describe('encryption and decryption', () => { }, 30000) }) -// TODO: Still have some issues with bulk encryption describe('bulk encryption', () => { it('should bulk encrypt and decrypt a payload', async () => { const protectClient = await protect(users) diff --git a/packages/protect/src/schema/index.ts b/packages/protect/src/schema/index.ts index 36624ac07..e71a71027 100644 --- a/packages/protect/src/schema/index.ts +++ b/packages/protect/src/schema/index.ts @@ -31,7 +31,7 @@ const tokenizerSchema = z token_length: z.number(), }), ]) - .default({ kind: 'standard' }) + .default({ kind: 'ngram', token_length: 3 }) .optional() const oreIndexOptsSchema = z.object({}) @@ -120,13 +120,13 @@ export class ProtectColumn { /** * Enable ORE indexing (Order-Revealing Encryption). */ - orderAndSort() { + orderAndRange() { this.indexesValue.ore = {} return this } /** - * Enable a Unique index. Optionally pass tokenFilters. + * Enable an Exact index. Optionally pass tokenFilters. */ equality(tokenFilters?: TokenFilter[]) { this.indexesValue.unique = { @@ -156,9 +156,8 @@ export class ProtectColumn { /** * Enable a STE Vec index, requires a prefix. - * TODO: Determine a better way to describe ste_vec */ - steVec(prefix: string) { + josn(prefix: string) { this.indexesValue.ste_vec = { prefix } return this } From 5ae2d9cf632ad29169f2db57780c98827321cce5 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Thu, 6 Mar 2025 08:46:37 -0700 Subject: [PATCH 29/30] docs(searchable-encryption): improve readability of concept --- docs/concepts/searchable-encryption.md | 103 ++++++++++++++++++++----- 1 file changed, 82 insertions(+), 21 deletions(-) diff --git a/docs/concepts/searchable-encryption.md b/docs/concepts/searchable-encryption.md index 46402c06e..69500aee3 100644 --- a/docs/concepts/searchable-encryption.md +++ b/docs/concepts/searchable-encryption.md @@ -1,41 +1,46 @@ # Searchable encryption -Protect.js supports searching encrypted data, which enabled trusted data access. +Protect.js supports searching encrypted data, which enabled trusted data access so that you can: + +1. Prove to your customers that you can track exactly what data is being accessed in your application. +2. Provide evidence for compliance requirements, such as [SOC2](https://cipherstash.com/compliance/soc2) and [BDSG](https://cipherstash.com/compliance/bdsg). ## What does searchable encryption even mean? The best way to describe searchable encryption is with an example. -Let's say you have a table of users in your database, and you want to search for a user by their email address. +Let's say you have a table of users in your database, and you want to search for a user by their email address: ```sql -SELECT * FROM users WHERE email = 'example@example.com'; +# SELECT * FROM users WHERE email = 'alice.johnson@example.com'; + id | name | email +----+----------------+---------------------------- + 1 | Alice Johnson | alice.johnson@example.com ``` -This is a pretty basic example, and you'd expect it to return something like: - -| id | name | email | -| --- | --- | --- | -| 1 | John | example@example.com | - Whether you executed this query directly in the database, or through an application ORM, you'd expect the result to be the same. **But what if the email address is encrypted before it's stored in the database?** -Executing: +Executing the following query will return all the rows in the table with the encrypted email address: ```sql -SELECT * FROM users; +# SELECT * FROM users; + id | name | email +----+----------------+---------------------------- + 1 | Alice Johnson | mBbKmsMMkbKBSN... + 2 | Jane Doe | s1THy_NfQdN892... + 3 | Bob Smith | 892!dercydsd0s... ``` -Would return something like: +Now, what's the issue if you execute the equality query with this data set? -| id | name | email | -| --- | --- | --- | -| 1 | John | mBbKmsMMkbKBSN... -| 2 | Jane | s1THy_NfQN892... -| 3 | Bob | 892!dercyd0s... +```sql +# SELECT * FROM users WHERE email = 'alice.johnson@example.com'; + id | name | email +----+----------------+---------------------------- +``` -The same equality query would return nothing, because `example@example.com` does not equal `mBbKmsMMkbKBSN...`. +There would be no results returned, because `alice.johnson@example.com` does not equal `mBbKmsMMkbKBSN...`! ## How do you search on encrypted data? @@ -47,6 +52,51 @@ The issue with homomorphic encryption isn't around the functionality, but rather CipherStash's approach to searchable encryption solves the performance problem without sacrificing security, usability, or functionality. +### Using Encrypt Query Language (EQL) + +CipherStash uses [EQL](https://github.com/cipherstash/encrypt-query-language) to perform queries on encrypted data, and Protect.js makes it easy to use EQL with any TypeScipt application. + +```ts +// 1) Encrypt the search term +const searchTerm = 'alice.johnson@example.com' + +const encryptedParam = await protectClient.encrypt(searchTerm, { + column: protectedUsers.email, // Your Protect column definition + table: protectedUsers, // Reference to the table schema +}) + +if (encryptedParam.failure) { + // Handle the failure +} + +// 2) Build an equality query using EQL +const equalitySQL = ` + SELECT email + FROM users + WHERE cs_unique_v1($1) = cs_unique_v1($2) +` + +// 3) Execute the query, passing in the Postgres column name +// and the encrypted search term as the second parameter +// (client is an arbitrary Postgres client) +const result = await client.query(equalitySQL, [ protectedUser.email.getName(), encryptedParam.data ]) +``` + +Using the above approach, Protect.js is generating the EQL payloads and which means you never have to drop down to writing complex SQL queries. + +So does this solve the original problem of searching on encrypted data? + +```sql +# SELECT * FROM users WHERE WHERE cs_unique_v1(email) = cs_unique_v1(eql_payload_created_by_protect); + id | name | email +----+----------------+---------------------------- + 1 | Alice Johnson | mBbKmsMMkbKBSN... +``` + +The answer is yes! And you can use Protect.js to [decrypt the results in your application](../../README.md#decrypting-data). + +## How fast is CipherStash's searchable encryption? + Based on some [benchmarks](https://github.com/cipherstash/tfhe-ore-bench?tab=readme-ov-file#results) CipherStash's approach is ***410,000x faster*** than homomorphic encryption: | Operation | Homomorphic | CipherStash | Speedup | @@ -58,12 +108,23 @@ Based on some [benchmarks](https://github.com/cipherstash/tfhe-ore-bench?tab=rea | **a >= a** | 44 ms | 221 ns | ~199 000× | | **a <= a** | 44 ms | 226 ns | ~195 000× | -## Bringing it all together +## How does searchable encryption help me? + +Every single decryption event is logged in CipherStash [ZeroKMS](https://cipherstash.com/products/zerokms), giving you an audit trail of data access events to help you prove compliance with your data protection policies. + +With searchable encryption, you can: + +1. Prove to your customers that you can track exactly what data is being accessed in your application. +2. Provide evidence for compliance requirements, such as [SOC2](https://cipherstash.com/compliance/soc2) and [BDSG](https://cipherstash.com/compliance/bdsg). + +## Bringing everything together With searchable encryption: -1. Data can be stored in the database encrypted. +1. Data can be encrypted, stored, and searched in your existing PostgreSQL database. 2. Encrypted data can be searched using equality, free text search, and range queries. 3. Data remains encrypted, and will be decrypted using the Protect.js library in your application. 4. Queries are blazing fast, and won't slow down your application experience. -5. Every decryption event is logged, giving you an audit trail of data access events to help you prove compliance with your data protection policies. \ No newline at end of file +5. Every decryption event is logged, giving you an audit trail of data access events. + +Read more about the [implementation details](../reference/searchable-encryption-postgres.md) to get started. \ No newline at end of file From fea3a67298e1a3481ad956d59cc26b8b0f708010 Mon Sep 17 00:00:00 2001 From: CJ Brewer Date: Thu, 6 Mar 2025 08:51:08 -0700 Subject: [PATCH 30/30] docs: update table of contents --- docs/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/README.md b/docs/README.md index 4213a1685..8ad3a1906 100644 --- a/docs/README.md +++ b/docs/README.md @@ -12,6 +12,7 @@ The documentation for Protect.js is organized into the following sections: - [Configuration and production deployment](./reference/configuration.md) - [Bulk encryption and decryption](./reference/bulk-encryption-decryption.md) +- [Searchable encryption with PostgreSQL](./reference/searchable-encryption-postgres.md)q - [Protect.js schemas](./reference/schema.md) ## How-to guides