From 32cf558f32b277db178929316dbdabf9a6c5bcd5 Mon Sep 17 00:00:00 2001 From: Peter Hauge Date: Thu, 2 Jul 2026 10:58:14 -0700 Subject: [PATCH 1/3] fix: honor filter for service-level policy; document policies key and sub-filters The extractServicePolicy function was called without a filter argument and never consulted shouldIncludeResource, so a filter of policies: [] still emitted apim-artifacts/policy.xml. Thread the filter through and add the guard so an empty list truly excludes the singleton. Documentation gaps found in the same investigation: - docs/commands/extract.md was missing the policies filter key and did not describe the nested apiSubFilters / workspaceSubFilters syntax accepted by parseFilterArrayWithNested. - docs/reference/dependency-graph.md claimed extract does not auto-include dependencies, which contradicted the transitive resolver. Changes: - extractServicePolicy now takes filter and returns early via shouldIncludeResource, matching every other extract-* helper. - Add three unit tests covering policies: [], omitted, and ["policy"]. - docs/commands/extract.md: add policies row plus apiSubFilters and workspaceSubFilters sections with nested YAML examples. - docs/reference/dependency-graph.md: rewrite the Filtering Considerations paragraph to separate publish-time vs extract-time behavior. Verified with 45 live filter scenarios against a source APIM instance; the previously leaked policy.xml no longer appears in the exclude-all run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/commands/extract.md | 54 ++++++++++++- docs/reference/dependency-graph.md | 12 ++- src/services/extract-service.ts | 16 +++- tests/unit/services/extract-service.test.ts | 90 +++++++++++++++++++++ 4 files changed, 166 insertions(+), 6 deletions(-) diff --git a/docs/commands/extract.md b/docs/commands/extract.md index de214343..08705e5e 100644 --- a/docs/commands/extract.md +++ b/docs/commands/extract.md @@ -121,11 +121,11 @@ diagnostics: ### Filterable resource types -All 16 supported filter keys: +All 17 supported filter keys: | Filter key | Resource type | |------------|---------------| -| `apis` | APIs | +| `apis` | APIs (accepts nested object entries — see [API sub-filters](#api-sub-filters) below) | | `backends` | Backends | | `products` | Products | | `namedValues` | Named values | @@ -138,9 +138,57 @@ All 16 supported filter keys: | `groups` | Groups | | `subscriptions` | Subscriptions | | `schemas` | Schemas | +| `policies` | Service-level policy (singleton — use `- 'policy'` to include, `[]` to exclude) | | `policyRestrictions` | Policy restrictions | | `documentations` | Documentation resources | -| `workspaces` | Workspaces | +| `workspaces` | Workspaces (accepts nested object entries — see [Workspace sub-filters](#workspace-sub-filters) below) | + +For every key, the value semantics are: + +- **Key omitted** → include all resources of that type (default) +- **`key: []`** → include none of that type +- **`key: [name1, name2]`** → include only the named resources (exact match, case-insensitive; supports `*` and `?` wildcards) + +### API sub-filters + +To restrict which child resources of an API are extracted (operations, +diagnostics, schemas, releases), use the **nested object entry** form under +`apis:`. Each entry becomes a `name: { subKey: [...] }` map: + +```yaml +apis: + - 'echo-api' # plain entry — all child resources included + - 'petstore-api': # nested entry — only listed children + operations: + - 'listPets' + - 'getPet' + diagnostics: [] # exclude all diagnostics for this API + schemas: + - 'pet-schema' + releases: [] +``` + +Supported API sub-filter keys: `operations`, `diagnostics`, `schemas`, `releases`. +Same value semantics as top-level keys (omitted = include all, `[]` = include none, list = allowlist). + +### Workspace sub-filters + +Workspaces accept the same nested object entry form. Each workspace can have +its own inner allowlists for the workspace-scoped resource types: + +```yaml +workspaces: + - 'team-a-workspace': + apis: + - 'team-a-orders' + backends: [] # exclude all workspace backends + namedValues: + - 'team-a-secret' +``` + +Supported workspace sub-filter keys: `apis`, `backends`, `diagnostics`, +`groups`, `loggers`, `namedValues`, `policyFragments`, `products`, `schemas`, +`subscriptions`, `tags`, `versionSets`. ### Transitive dependencies diff --git a/docs/reference/dependency-graph.md b/docs/reference/dependency-graph.md index 076ee51b..bb45884a 100644 --- a/docs/reference/dependency-graph.md +++ b/docs/reference/dependency-graph.md @@ -161,7 +161,17 @@ If a Tier 2 resource fails to publish, its dependents in Tier 3 and 4 will also ### Filtering Considerations -When using `filter.yaml` to include specific resources, be aware of dependencies. For example, including an **ApiPolicy** without its parent **Api** will cause a publish failure. apiops-cli does **not** auto-include dependencies — your filter must explicitly include all required parents. +When using `filter.yaml` to include specific resources, be aware of dependencies. + +**During publish**, apiops-cli does **not** auto-include dependencies at the +artifact level — your filter (and hence your extracted artifact tree) must +already contain every required parent. For example, publishing an **ApiPolicy** +without its parent **Api** present in the artifact tree will fail. + +**During extract**, apiops-cli **does** auto-follow a limited set of transitive +references when a filter is applied and `--no-transitive` is not passed. See +[`apiops extract` — Transitive dependencies](../commands/extract.md#transitive-dependencies) +for the full list of reference types the extractor scans. ## Related Docs diff --git a/src/services/extract-service.ts b/src/services/extract-service.ts index e39e1245..e48eb20a 100644 --- a/src/services/extract-service.ts +++ b/src/services/extract-service.ts @@ -30,6 +30,7 @@ import { findTransitiveDependencies, } from './transitive-resolver.js'; import { redactAndWarnPolicySecrets } from './secret-redactor.js'; +import { shouldIncludeResource } from './filter-service.js'; import { logger } from '../lib/logger.js'; import { buildResourceLabel } from '../lib/resource-uri.js'; import { EXIT_SUCCESS, EXIT_PARTIAL, EXIT_FATAL } from '../lib/exit-codes.js'; @@ -122,8 +123,8 @@ export async function runExtraction( // Phase 4: Tier 4 resources are extracted within API-specific extraction // (ApiOperationPolicy, GraphQLResolverPolicy are handled in api-extractor) - // Phase 5: Extract service-level policy - await extractServicePolicy(client, store, service, outputDir, result); + // Phase 5: Extract service-level policy (respects `policies` filter key) + await extractServicePolicy(client, store, service, outputDir, filter, result); // Phase 6: Transitive dependency resolution (if enabled) if (config.includeTransitive && filter) { @@ -380,12 +381,18 @@ async function extractProductSubResources( /** * Extract service-level global policy. + * Honors the `policies` filter key: when a filter is supplied and its + * `policies` field is an empty array (explicit exclude), the service policy + * is skipped. When the filter omits `policies` entirely, the policy is + * included (undefined = include-all, matching the semantics of every other + * filter field). */ async function extractServicePolicy( client: IApimClient, store: IArtifactStore, context: ApimServiceContext, outputDir: string, + filter: FilterConfig | undefined, result: ExtractionResult ): Promise { const descriptor: ResourceDescriptor = { @@ -393,6 +400,11 @@ async function extractServicePolicy( nameParts: [], }; + if (!shouldIncludeResource(descriptor, filter)) { + logger.debug('Skipping service-level policy (excluded by filter)'); + return; + } + const policyJson = await client.getResource(context, descriptor); if (!policyJson) { return; diff --git a/tests/unit/services/extract-service.test.ts b/tests/unit/services/extract-service.test.ts index c4c630da..3aeccc42 100644 --- a/tests/unit/services/extract-service.test.ts +++ b/tests/unit/services/extract-service.test.ts @@ -150,6 +150,96 @@ describe('extract-service', () => { expect(result.collectedPolicies.has('service-policy')).toBe(true); }); + it('should skip service policy when filter.policies is an empty array', async () => { + // Regression: service policy previously bypassed the filter entirely and + // was extracted even when `policies: []` explicitly excluded it. + const client = createMockClient({}); + client.getResource = vi.fn().mockImplementation(async (_ctx, descriptor) => { + if (descriptor.type === ResourceType.ServicePolicy) { + return { + name: 'policy', + properties: { value: '' }, + }; + } + return undefined; + }); + const store = createMockStore(); + + const filter: FilterConfig = { policies: [] }; + const config: ExtractConfig = { + service: testContext, + outputDir: '/output', + filter, + includeTransitive: false, + logLevel: LogLevel.INFO, + }; + + const result = await runExtraction(client, store, config); + + // The mock's getResource may still be called for other types, but writeContent + // must NOT have been invoked for the ServicePolicy descriptor. + const serviceWrites = (store.writeContent as ReturnType).mock.calls.filter( + (call) => (call[1] as ResourceDescriptor)?.type === ResourceType.ServicePolicy + ); + expect(serviceWrites).toHaveLength(0); + expect(result.collectedPolicies.has('service-policy')).toBe(false); + }); + + it('should extract service policy when filter omits policies key entirely (undefined = include all)', async () => { + const client = createMockClient({}); + client.getResource = vi.fn().mockImplementation(async (_ctx, descriptor) => { + if (descriptor.type === ResourceType.ServicePolicy) { + return { + name: 'policy', + properties: { value: '' }, + }; + } + return undefined; + }); + const store = createMockStore(); + + // Filter with no `policies` field — semantics: include all singleton types + const filter: FilterConfig = { apis: ['some-api'] }; + const config: ExtractConfig = { + service: testContext, + outputDir: '/output', + filter, + includeTransitive: false, + logLevel: LogLevel.INFO, + }; + + const result = await runExtraction(client, store, config); + + expect(result.collectedPolicies.has('service-policy')).toBe(true); + }); + + it('should extract service policy when filter.policies contains the singleton name "policy"', async () => { + const client = createMockClient({}); + client.getResource = vi.fn().mockImplementation(async (_ctx, descriptor) => { + if (descriptor.type === ResourceType.ServicePolicy) { + return { + name: 'policy', + properties: { value: '' }, + }; + } + return undefined; + }); + const store = createMockStore(); + + const filter: FilterConfig = { policies: ['policy'] }; + const config: ExtractConfig = { + service: testContext, + outputDir: '/output', + filter, + includeTransitive: false, + logLevel: LogLevel.INFO, + }; + + const result = await runExtraction(client, store, config); + + expect(result.collectedPolicies.has('service-policy')).toBe(true); + }); + it('should redact inline secrets in service policy before storing', async () => { const policyContent = 'abc123'; const redactedPolicy = `${REDACTION_MARKER}`; From 0226899df5a914fe0b696b2d9d092ec72fc90daa Mon Sep 17 00:00:00 2001 From: Peter Hauge Date: Thu, 2 Jul 2026 11:15:28 -0700 Subject: [PATCH 2/3] docs: clarify that omitted filter keys mean include-all MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Issue #218 reported that a filter with only pis: [my-api] set still extracted every backend, named value, product, etc. That is the documented behavior — each top-level key is independent, and omitted keys default to "include all" — but the docs and the init-generated template did not make this clear enough, and the filtering guide's Quick Start actively claimed the opposite. Changes: - docs/guides/filtering-resources.md: fix misleading Quick Start claim and add a "Common Pitfall: Narrowing to Just One API" section with a complete worked example that sets every other type to []. - docs/commands/extract.md: add a callout under the value-semantics list pointing at the new pitfall section. - src/templates/configs/filter-config.yaml: prepend a "HOW FILTERING WORKS" block so the first thing a user sees in their init-generated configuration.extractor.yaml explains the three states and shows the narrow-to-one-API pattern. - Regenerated src/templates/generated/embedded-markdown.ts (prelint). Refs #218 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/commands/extract.md | 2 ++ docs/guides/filtering-resources.md | 42 +++++++++++++++++++++++- src/templates/configs/filter-config.yaml | 35 ++++++++++++++++++++ 3 files changed, 78 insertions(+), 1 deletion(-) diff --git a/docs/commands/extract.md b/docs/commands/extract.md index 08705e5e..32754906 100644 --- a/docs/commands/extract.md +++ b/docs/commands/extract.md @@ -149,6 +149,8 @@ For every key, the value semantics are: - **`key: []`** → include none of that type - **`key: [name1, name2]`** → include only the named resources (exact match, case-insensitive; supports `*` and `?` wildcards) +> **Common pitfall.** Each key is independent. A filter that only sets `apis: [my-api]` still extracts every backend, named value, product, tag, workspace, and other resource type, because those keys are omitted and default to "include all". To narrow to a single API plus its transitive dependencies, set every other key to `[]` explicitly. See [Common Pitfall: Narrowing to Just One API](../guides/filtering-resources.md#common-pitfall-narrowing-to-just-one-api) in the filtering guide. + ### API sub-filters To restrict which child resources of an API are extracted (operations, diff --git a/docs/guides/filtering-resources.md b/docs/guides/filtering-resources.md index 4beb4e7d..2133f68d 100644 --- a/docs/guides/filtering-resources.md +++ b/docs/guides/filtering-resources.md @@ -32,7 +32,47 @@ apiops extract \ --filter configuration.extractor.yaml ``` -Only `petstore-api`, `orders-api`, and their transitive dependencies are extracted. +`petstore-api`, `orders-api`, and their transitive dependencies are extracted — **plus every backend, named value, product, tag, workspace, and every other resource type**, because those keys are omitted from the file and therefore default to "include all". If that is not what you want, see the next section. + +--- + +## Common Pitfall: Narrowing to Just One API + +Each top-level filter key is **independent**. Setting `apis:` does not implicitly exclude other resource types — it only narrows the `apis` type. Every key that is **omitted** from the file defaults to "include all resources of that type". + +If you want to extract just one API (plus whatever transitive dependencies it needs) and nothing else, you must **explicitly set every other type to `[]`**: + +```yaml +# configuration.extractor.yaml — extract only my-own-api and its transitive deps +apis: + - my-own-api +backends: [] +namedValues: [] +products: [] +tags: [] +versionSets: [] +loggers: [] +diagnostics: [] +groups: [] +policyFragments: [] +gateways: [] +schemas: [] +subscriptions: [] +policies: [] +policyRestrictions: [] +documentations: [] +workspaces: [] +``` + +With transitive resolution enabled (the default), any version set, backend, named value, policy fragment, or tag directly referenced by `my-own-api` or its policies is still pulled in automatically — even though those keys are set to `[]`. Use `--no-transitive` to disable that behavior. + +The three states for every key are: + +| Value | Meaning | +|-------|---------| +| Key omitted | Include **all** resources of that type (default) | +| `key: []` | Include **none** of that type | +| `key: [name1, name2]` | Include only the named resources (case-insensitive, supports `*` and `?` wildcards) | --- diff --git a/src/templates/configs/filter-config.yaml b/src/templates/configs/filter-config.yaml index 7a935ebd..2a44d441 100644 --- a/src/templates/configs/filter-config.yaml +++ b/src/templates/configs/filter-config.yaml @@ -3,6 +3,41 @@ # Customize this file to control which resources are extracted # For full format details and examples, see: # https://github.com/Azure/apiops-cli/blob/main/docs/guides/filtering-resources.md +# +# HOW FILTERING WORKS (read this before uncommenting sections below): +# Each top-level key is INDEPENDENT. The three states are: +# - key omitted → include ALL resources of that type (default) +# - key: [] → include NONE of that type +# - key: [names] → include only the named resources (case-insensitive, +# supports * and ? wildcards) +# +# Common pitfall: setting only `apis: [my-api]` does NOT narrow the whole +# extract to that API — every other type is still extracted because those +# keys are omitted. To extract JUST one API plus its transitive dependencies, +# set every other type to [] explicitly. Example: +# +# apis: +# - my-own-api +# backends: [] +# namedValues: [] +# products: [] +# tags: [] +# versionSets: [] +# loggers: [] +# diagnostics: [] +# groups: [] +# policyFragments: [] +# gateways: [] +# schemas: [] +# subscriptions: [] +# policies: [] +# policyRestrictions: [] +# documentations: [] +# workspaces: [] +# +# Transitive dependencies (referenced backends, named values, policy +# fragments, tags, version sets) are still pulled in automatically even +# when the parent key is []. Use `--no-transitive` to disable that. # Extract only specific APIs by name (or wildcard pattern) # apis: From ec5011e38290deff7306e49b588dc860b83aa575 Mon Sep 17 00:00:00 2001 From: Peter Hauge Date: Thu, 2 Jul 2026 12:38:06 -0700 Subject: [PATCH 3/3] docs: address review feedback on framing and formatting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per PR review from @EMaher: - docs/commands/extract.md: promote the plain blockquote to a GitHub markdown alert (> [!TIP]) so it renders as a visual callout. - docs/guides/filtering-resources.md: rename "Common Pitfall: Narrowing to Just One API" to "How To: Extract Just One API" — the section is a how-to, not a warning. - src/templates/configs/filter-config.yaml: soften the "Common pitfall" wording in the init-generated template to "Tip:"; drop the negative framing from the leading comment block. Also updated the extract.md anchor and the Quick Start callout in the filtering guide to match the renamed section. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/commands/extract.md | 3 ++- docs/guides/filtering-resources.md | 8 ++++---- src/templates/configs/filter-config.yaml | 7 +++---- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/commands/extract.md b/docs/commands/extract.md index 32754906..d9208a7b 100644 --- a/docs/commands/extract.md +++ b/docs/commands/extract.md @@ -149,7 +149,8 @@ For every key, the value semantics are: - **`key: []`** → include none of that type - **`key: [name1, name2]`** → include only the named resources (exact match, case-insensitive; supports `*` and `?` wildcards) -> **Common pitfall.** Each key is independent. A filter that only sets `apis: [my-api]` still extracts every backend, named value, product, tag, workspace, and other resource type, because those keys are omitted and default to "include all". To narrow to a single API plus its transitive dependencies, set every other key to `[]` explicitly. See [Common Pitfall: Narrowing to Just One API](../guides/filtering-resources.md#common-pitfall-narrowing-to-just-one-api) in the filtering guide. +> [!TIP] +> Each key is independent. Setting only `apis: [my-api]` still extracts every backend, named value, product, tag, workspace, and other resource type, because those keys are omitted and default to "include all". To extract just one API plus its transitive dependencies, set every other key to `[]`. See [How To: Extract Just One API](../guides/filtering-resources.md#how-to-extract-just-one-api) in the filtering guide. ### API sub-filters diff --git a/docs/guides/filtering-resources.md b/docs/guides/filtering-resources.md index 2133f68d..83959359 100644 --- a/docs/guides/filtering-resources.md +++ b/docs/guides/filtering-resources.md @@ -32,15 +32,15 @@ apiops extract \ --filter configuration.extractor.yaml ``` -`petstore-api`, `orders-api`, and their transitive dependencies are extracted — **plus every backend, named value, product, tag, workspace, and every other resource type**, because those keys are omitted from the file and therefore default to "include all". If that is not what you want, see the next section. +`petstore-api`, `orders-api`, and their transitive dependencies are extracted — along with every backend, named value, product, tag, workspace, and every other resource type, because those keys are omitted and therefore default to "include all". To narrow the extract to just these APIs, see [How To: Extract Just One API](#how-to-extract-just-one-api) below. --- -## Common Pitfall: Narrowing to Just One API +## How To: Extract Just One API -Each top-level filter key is **independent**. Setting `apis:` does not implicitly exclude other resource types — it only narrows the `apis` type. Every key that is **omitted** from the file defaults to "include all resources of that type". +Each top-level filter key is **independent**. Setting `apis:` narrows only the `apis` type — it does not implicitly exclude other resource types. Every key that is **omitted** from the file defaults to "include all resources of that type". -If you want to extract just one API (plus whatever transitive dependencies it needs) and nothing else, you must **explicitly set every other type to `[]`**: +To extract a single API (plus whatever transitive dependencies it needs) and nothing else, set every other type to `[]`: ```yaml # configuration.extractor.yaml — extract only my-own-api and its transitive deps diff --git a/src/templates/configs/filter-config.yaml b/src/templates/configs/filter-config.yaml index 2a44d441..c27e74c0 100644 --- a/src/templates/configs/filter-config.yaml +++ b/src/templates/configs/filter-config.yaml @@ -11,10 +11,9 @@ # - key: [names] → include only the named resources (case-insensitive, # supports * and ? wildcards) # -# Common pitfall: setting only `apis: [my-api]` does NOT narrow the whole -# extract to that API — every other type is still extracted because those -# keys are omitted. To extract JUST one API plus its transitive dependencies, -# set every other type to [] explicitly. Example: +# Tip: to extract JUST one API plus its transitive dependencies, set the +# `apis` key to that API and set every other type to []. Every key you +# omit still extracts everything of that type. Example: # # apis: # - my-own-api