diff --git a/docs/commands/extract.md b/docs/commands/extract.md index de214343..d9208a7b 100644 --- a/docs/commands/extract.md +++ b/docs/commands/extract.md @@ -121,11 +121,11 @@ diagnostics: ### Filterable resource types -All 16 supported filter keys: +All 17 supported filter keys: | Filter key | Resource type | |------------|---------------| -| `apis` | APIs | +| `apis` | APIs (accepts nested object entries — see [API sub-filters](#api-sub-filters) below) | | `backends` | Backends | | `products` | Products | | `namedValues` | Named values | @@ -138,9 +138,60 @@ All 16 supported filter keys: | `groups` | Groups | | `subscriptions` | Subscriptions | | `schemas` | Schemas | +| `policies` | Service-level policy (singleton — use `- 'policy'` to include, `[]` to exclude) | | `policyRestrictions` | Policy restrictions | | `documentations` | Documentation resources | -| `workspaces` | Workspaces | +| `workspaces` | Workspaces (accepts nested object entries — see [Workspace sub-filters](#workspace-sub-filters) below) | + +For every key, the value semantics are: + +- **Key omitted** → include all resources of that type (default) +- **`key: []`** → include none of that type +- **`key: [name1, name2]`** → include only the named resources (exact match, case-insensitive; supports `*` and `?` wildcards) + +> [!TIP] +> Each key is independent. Setting only `apis: [my-api]` still extracts every backend, named value, product, tag, workspace, and other resource type, because those keys are omitted and default to "include all". To extract just one API plus its transitive dependencies, set every other key to `[]`. See [How To: Extract Just One API](../guides/filtering-resources.md#how-to-extract-just-one-api) in the filtering guide. + +### API sub-filters + +To restrict which child resources of an API are extracted (operations, +diagnostics, schemas, releases), use the **nested object entry** form under +`apis:`. Each entry becomes a `name: { subKey: [...] }` map: + +```yaml +apis: + - 'echo-api' # plain entry — all child resources included + - 'petstore-api': # nested entry — only listed children + operations: + - 'listPets' + - 'getPet' + diagnostics: [] # exclude all diagnostics for this API + schemas: + - 'pet-schema' + releases: [] +``` + +Supported API sub-filter keys: `operations`, `diagnostics`, `schemas`, `releases`. +Same value semantics as top-level keys (omitted = include all, `[]` = include none, list = allowlist). + +### Workspace sub-filters + +Workspaces accept the same nested object entry form. Each workspace can have +its own inner allowlists for the workspace-scoped resource types: + +```yaml +workspaces: + - 'team-a-workspace': + apis: + - 'team-a-orders' + backends: [] # exclude all workspace backends + namedValues: + - 'team-a-secret' +``` + +Supported workspace sub-filter keys: `apis`, `backends`, `diagnostics`, +`groups`, `loggers`, `namedValues`, `policyFragments`, `products`, `schemas`, +`subscriptions`, `tags`, `versionSets`. ### Transitive dependencies diff --git a/docs/guides/filtering-resources.md b/docs/guides/filtering-resources.md index 4beb4e7d..83959359 100644 --- a/docs/guides/filtering-resources.md +++ b/docs/guides/filtering-resources.md @@ -32,7 +32,47 @@ apiops extract \ --filter configuration.extractor.yaml ``` -Only `petstore-api`, `orders-api`, and their transitive dependencies are extracted. +`petstore-api`, `orders-api`, and their transitive dependencies are extracted — along with every backend, named value, product, tag, workspace, and every other resource type, because those keys are omitted and therefore default to "include all". To narrow the extract to just these APIs, see [How To: Extract Just One API](#how-to-extract-just-one-api) below. + +--- + +## How To: Extract Just One API + +Each top-level filter key is **independent**. Setting `apis:` narrows only the `apis` type — it does not implicitly exclude other resource types. Every key that is **omitted** from the file defaults to "include all resources of that type". + +To extract a single API (plus whatever transitive dependencies it needs) and nothing else, set every other type to `[]`: + +```yaml +# configuration.extractor.yaml — extract only my-own-api and its transitive deps +apis: + - my-own-api +backends: [] +namedValues: [] +products: [] +tags: [] +versionSets: [] +loggers: [] +diagnostics: [] +groups: [] +policyFragments: [] +gateways: [] +schemas: [] +subscriptions: [] +policies: [] +policyRestrictions: [] +documentations: [] +workspaces: [] +``` + +With transitive resolution enabled (the default), any version set, backend, named value, policy fragment, or tag directly referenced by `my-own-api` or its policies is still pulled in automatically — even though those keys are set to `[]`. Use `--no-transitive` to disable that behavior. + +The three states for every key are: + +| Value | Meaning | +|-------|---------| +| Key omitted | Include **all** resources of that type (default) | +| `key: []` | Include **none** of that type | +| `key: [name1, name2]` | Include only the named resources (case-insensitive, supports `*` and `?` wildcards) | --- diff --git a/docs/reference/dependency-graph.md b/docs/reference/dependency-graph.md index 076ee51b..bb45884a 100644 --- a/docs/reference/dependency-graph.md +++ b/docs/reference/dependency-graph.md @@ -161,7 +161,17 @@ If a Tier 2 resource fails to publish, its dependents in Tier 3 and 4 will also ### Filtering Considerations -When using `filter.yaml` to include specific resources, be aware of dependencies. For example, including an **ApiPolicy** without its parent **Api** will cause a publish failure. apiops-cli does **not** auto-include dependencies — your filter must explicitly include all required parents. +When using `filter.yaml` to include specific resources, be aware of dependencies. + +**During publish**, apiops-cli does **not** auto-include dependencies at the +artifact level — your filter (and hence your extracted artifact tree) must +already contain every required parent. For example, publishing an **ApiPolicy** +without its parent **Api** present in the artifact tree will fail. + +**During extract**, apiops-cli **does** auto-follow a limited set of transitive +references when a filter is applied and `--no-transitive` is not passed. See +[`apiops extract` — Transitive dependencies](../commands/extract.md#transitive-dependencies) +for the full list of reference types the extractor scans. ## Related Docs diff --git a/src/services/extract-service.ts b/src/services/extract-service.ts index e39e1245..e48eb20a 100644 --- a/src/services/extract-service.ts +++ b/src/services/extract-service.ts @@ -30,6 +30,7 @@ import { findTransitiveDependencies, } from './transitive-resolver.js'; import { redactAndWarnPolicySecrets } from './secret-redactor.js'; +import { shouldIncludeResource } from './filter-service.js'; import { logger } from '../lib/logger.js'; import { buildResourceLabel } from '../lib/resource-uri.js'; import { EXIT_SUCCESS, EXIT_PARTIAL, EXIT_FATAL } from '../lib/exit-codes.js'; @@ -122,8 +123,8 @@ export async function runExtraction( // Phase 4: Tier 4 resources are extracted within API-specific extraction // (ApiOperationPolicy, GraphQLResolverPolicy are handled in api-extractor) - // Phase 5: Extract service-level policy - await extractServicePolicy(client, store, service, outputDir, result); + // Phase 5: Extract service-level policy (respects `policies` filter key) + await extractServicePolicy(client, store, service, outputDir, filter, result); // Phase 6: Transitive dependency resolution (if enabled) if (config.includeTransitive && filter) { @@ -380,12 +381,18 @@ async function extractProductSubResources( /** * Extract service-level global policy. + * Honors the `policies` filter key: when a filter is supplied and its + * `policies` field is an empty array (explicit exclude), the service policy + * is skipped. When the filter omits `policies` entirely, the policy is + * included (undefined = include-all, matching the semantics of every other + * filter field). */ async function extractServicePolicy( client: IApimClient, store: IArtifactStore, context: ApimServiceContext, outputDir: string, + filter: FilterConfig | undefined, result: ExtractionResult ): Promise { const descriptor: ResourceDescriptor = { @@ -393,6 +400,11 @@ async function extractServicePolicy( nameParts: [], }; + if (!shouldIncludeResource(descriptor, filter)) { + logger.debug('Skipping service-level policy (excluded by filter)'); + return; + } + const policyJson = await client.getResource(context, descriptor); if (!policyJson) { return; diff --git a/src/templates/configs/filter-config.yaml b/src/templates/configs/filter-config.yaml index 7a935ebd..c27e74c0 100644 --- a/src/templates/configs/filter-config.yaml +++ b/src/templates/configs/filter-config.yaml @@ -3,6 +3,40 @@ # Customize this file to control which resources are extracted # For full format details and examples, see: # https://github.com/Azure/apiops-cli/blob/main/docs/guides/filtering-resources.md +# +# HOW FILTERING WORKS (read this before uncommenting sections below): +# Each top-level key is INDEPENDENT. The three states are: +# - key omitted → include ALL resources of that type (default) +# - key: [] → include NONE of that type +# - key: [names] → include only the named resources (case-insensitive, +# supports * and ? wildcards) +# +# Tip: to extract JUST one API plus its transitive dependencies, set the +# `apis` key to that API and set every other type to []. Every key you +# omit still extracts everything of that type. Example: +# +# apis: +# - my-own-api +# backends: [] +# namedValues: [] +# products: [] +# tags: [] +# versionSets: [] +# loggers: [] +# diagnostics: [] +# groups: [] +# policyFragments: [] +# gateways: [] +# schemas: [] +# subscriptions: [] +# policies: [] +# policyRestrictions: [] +# documentations: [] +# workspaces: [] +# +# Transitive dependencies (referenced backends, named values, policy +# fragments, tags, version sets) are still pulled in automatically even +# when the parent key is []. Use `--no-transitive` to disable that. # Extract only specific APIs by name (or wildcard pattern) # apis: diff --git a/tests/unit/services/extract-service.test.ts b/tests/unit/services/extract-service.test.ts index c4c630da..3aeccc42 100644 --- a/tests/unit/services/extract-service.test.ts +++ b/tests/unit/services/extract-service.test.ts @@ -150,6 +150,96 @@ describe('extract-service', () => { expect(result.collectedPolicies.has('service-policy')).toBe(true); }); + it('should skip service policy when filter.policies is an empty array', async () => { + // Regression: service policy previously bypassed the filter entirely and + // was extracted even when `policies: []` explicitly excluded it. + const client = createMockClient({}); + client.getResource = vi.fn().mockImplementation(async (_ctx, descriptor) => { + if (descriptor.type === ResourceType.ServicePolicy) { + return { + name: 'policy', + properties: { value: '' }, + }; + } + return undefined; + }); + const store = createMockStore(); + + const filter: FilterConfig = { policies: [] }; + const config: ExtractConfig = { + service: testContext, + outputDir: '/output', + filter, + includeTransitive: false, + logLevel: LogLevel.INFO, + }; + + const result = await runExtraction(client, store, config); + + // The mock's getResource may still be called for other types, but writeContent + // must NOT have been invoked for the ServicePolicy descriptor. + const serviceWrites = (store.writeContent as ReturnType).mock.calls.filter( + (call) => (call[1] as ResourceDescriptor)?.type === ResourceType.ServicePolicy + ); + expect(serviceWrites).toHaveLength(0); + expect(result.collectedPolicies.has('service-policy')).toBe(false); + }); + + it('should extract service policy when filter omits policies key entirely (undefined = include all)', async () => { + const client = createMockClient({}); + client.getResource = vi.fn().mockImplementation(async (_ctx, descriptor) => { + if (descriptor.type === ResourceType.ServicePolicy) { + return { + name: 'policy', + properties: { value: '' }, + }; + } + return undefined; + }); + const store = createMockStore(); + + // Filter with no `policies` field — semantics: include all singleton types + const filter: FilterConfig = { apis: ['some-api'] }; + const config: ExtractConfig = { + service: testContext, + outputDir: '/output', + filter, + includeTransitive: false, + logLevel: LogLevel.INFO, + }; + + const result = await runExtraction(client, store, config); + + expect(result.collectedPolicies.has('service-policy')).toBe(true); + }); + + it('should extract service policy when filter.policies contains the singleton name "policy"', async () => { + const client = createMockClient({}); + client.getResource = vi.fn().mockImplementation(async (_ctx, descriptor) => { + if (descriptor.type === ResourceType.ServicePolicy) { + return { + name: 'policy', + properties: { value: '' }, + }; + } + return undefined; + }); + const store = createMockStore(); + + const filter: FilterConfig = { policies: ['policy'] }; + const config: ExtractConfig = { + service: testContext, + outputDir: '/output', + filter, + includeTransitive: false, + logLevel: LogLevel.INFO, + }; + + const result = await runExtraction(client, store, config); + + expect(result.collectedPolicies.has('service-policy')).toBe(true); + }); + it('should redact inline secrets in service policy before storing', async () => { const policyContent = 'abc123'; const redactedPolicy = `${REDACTION_MARKER}`;