Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 54 additions & 3 deletions docs/commands/extract.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,11 +121,11 @@ diagnostics:

### Filterable resource types

All 16 supported filter keys:
All 17 supported filter keys:

| Filter key | Resource type |
|------------|---------------|
| `apis` | APIs |
| `apis` | APIs (accepts nested object entries — see [API sub-filters](#api-sub-filters) below) |
| `backends` | Backends |
| `products` | Products |
| `namedValues` | Named values |
Expand All @@ -138,9 +138,60 @@ All 16 supported filter keys:
| `groups` | Groups |
| `subscriptions` | Subscriptions |
| `schemas` | Schemas |
| `policies` | Service-level policy (singleton — use `- 'policy'` to include, `[]` to exclude) |
| `policyRestrictions` | Policy restrictions |
| `documentations` | Documentation resources |
| `workspaces` | Workspaces |
| `workspaces` | Workspaces (accepts nested object entries — see [Workspace sub-filters](#workspace-sub-filters) below) |

For every key, the value semantics are:

- **Key omitted** → include all resources of that type (default)
- **`key: []`** → include none of that type
- **`key: [name1, name2]`** → include only the named resources (exact match, case-insensitive; supports `*` and `?` wildcards)

> [!TIP]
> Each key is independent. Setting only `apis: [my-api]` still extracts every backend, named value, product, tag, workspace, and other resource type, because those keys are omitted and default to "include all". To extract just one API plus its transitive dependencies, set every other key to `[]`. See [How To: Extract Just One API](../guides/filtering-resources.md#how-to-extract-just-one-api) in the filtering guide.

### API sub-filters

To restrict which child resources of an API are extracted (operations,
diagnostics, schemas, releases), use the **nested object entry** form under
`apis:`. Each entry becomes a `name: { subKey: [...] }` map:

```yaml
apis:
- 'echo-api' # plain entry — all child resources included
- 'petstore-api': # nested entry — only listed children
operations:
- 'listPets'
- 'getPet'
diagnostics: [] # exclude all diagnostics for this API
schemas:
- 'pet-schema'
releases: []
```

Supported API sub-filter keys: `operations`, `diagnostics`, `schemas`, `releases`.
Same value semantics as top-level keys (omitted = include all, `[]` = include none, list = allowlist).

### Workspace sub-filters

Workspaces accept the same nested object entry form. Each workspace can have
its own inner allowlists for the workspace-scoped resource types:

```yaml
workspaces:
- 'team-a-workspace':
apis:
- 'team-a-orders'
backends: [] # exclude all workspace backends
namedValues:
- 'team-a-secret'
```

Supported workspace sub-filter keys: `apis`, `backends`, `diagnostics`,
`groups`, `loggers`, `namedValues`, `policyFragments`, `products`, `schemas`,
`subscriptions`, `tags`, `versionSets`.

### Transitive dependencies

Expand Down
42 changes: 41 additions & 1 deletion docs/guides/filtering-resources.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,47 @@ apiops extract \
--filter configuration.extractor.yaml
```

Only `petstore-api`, `orders-api`, and their transitive dependencies are extracted.
`petstore-api`, `orders-api`, and their transitive dependencies are extracted — along with every backend, named value, product, tag, workspace, and every other resource type, because those keys are omitted and therefore default to "include all". To narrow the extract to just these APIs, see [How To: Extract Just One API](#how-to-extract-just-one-api) below.

---

## How To: Extract Just One API

Each top-level filter key is **independent**. Setting `apis:` narrows only the `apis` type — it does not implicitly exclude other resource types. Every key that is **omitted** from the file defaults to "include all resources of that type".

To extract a single API (plus whatever transitive dependencies it needs) and nothing else, set every other type to `[]`:

```yaml
# configuration.extractor.yaml — extract only my-own-api and its transitive deps
apis:
- my-own-api
backends: []
namedValues: []
products: []
tags: []
versionSets: []
loggers: []
diagnostics: []
groups: []
policyFragments: []
gateways: []
schemas: []
subscriptions: []
policies: []
policyRestrictions: []
documentations: []
workspaces: []
```

With transitive resolution enabled (the default), any version set, backend, named value, policy fragment, or tag directly referenced by `my-own-api` or its policies is still pulled in automatically — even though those keys are set to `[]`. Use `--no-transitive` to disable that behavior.

The three states for every key are:

| Value | Meaning |
|-------|---------|
| Key omitted | Include **all** resources of that type (default) |
| `key: []` | Include **none** of that type |
| `key: [name1, name2]` | Include only the named resources (case-insensitive, supports `*` and `?` wildcards) |

---

Expand Down
12 changes: 11 additions & 1 deletion docs/reference/dependency-graph.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,17 @@ If a Tier 2 resource fails to publish, its dependents in Tier 3 and 4 will also

### Filtering Considerations

When using `filter.yaml` to include specific resources, be aware of dependencies. For example, including an **ApiPolicy** without its parent **Api** will cause a publish failure. apiops-cli does **not** auto-include dependencies — your filter must explicitly include all required parents.
When using `filter.yaml` to include specific resources, be aware of dependencies.

**During publish**, apiops-cli does **not** auto-include dependencies at the
artifact level — your filter (and hence your extracted artifact tree) must
already contain every required parent. For example, publishing an **ApiPolicy**
without its parent **Api** present in the artifact tree will fail.

**During extract**, apiops-cli **does** auto-follow a limited set of transitive
references when a filter is applied and `--no-transitive` is not passed. See
[`apiops extract` — Transitive dependencies](../commands/extract.md#transitive-dependencies)
for the full list of reference types the extractor scans.

## Related Docs

Expand Down
16 changes: 14 additions & 2 deletions src/services/extract-service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import {
findTransitiveDependencies,
} from './transitive-resolver.js';
import { redactAndWarnPolicySecrets } from './secret-redactor.js';
import { shouldIncludeResource } from './filter-service.js';
import { logger } from '../lib/logger.js';
import { buildResourceLabel } from '../lib/resource-uri.js';
import { EXIT_SUCCESS, EXIT_PARTIAL, EXIT_FATAL } from '../lib/exit-codes.js';
Expand Down Expand Up @@ -122,8 +123,8 @@ export async function runExtraction(
// Phase 4: Tier 4 resources are extracted within API-specific extraction
// (ApiOperationPolicy, GraphQLResolverPolicy are handled in api-extractor)

// Phase 5: Extract service-level policy
await extractServicePolicy(client, store, service, outputDir, result);
// Phase 5: Extract service-level policy (respects `policies` filter key)
await extractServicePolicy(client, store, service, outputDir, filter, result);

// Phase 6: Transitive dependency resolution (if enabled)
if (config.includeTransitive && filter) {
Expand Down Expand Up @@ -380,19 +381,30 @@ async function extractProductSubResources(

/**
* Extract service-level global policy.
* Honors the `policies` filter key: when a filter is supplied and its
* `policies` field is an empty array (explicit exclude), the service policy
* is skipped. When the filter omits `policies` entirely, the policy is
* included (undefined = include-all, matching the semantics of every other
* filter field).
*/
async function extractServicePolicy(
client: IApimClient,
store: IArtifactStore,
context: ApimServiceContext,
outputDir: string,
filter: FilterConfig | undefined,
result: ExtractionResult
): Promise<void> {
const descriptor: ResourceDescriptor = {
type: ResourceType.ServicePolicy,
nameParts: [],
};

if (!shouldIncludeResource(descriptor, filter)) {
logger.debug('Skipping service-level policy (excluded by filter)');
return;
}

const policyJson = await client.getResource(context, descriptor);
if (!policyJson) {
return;
Expand Down
34 changes: 34 additions & 0 deletions src/templates/configs/filter-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,40 @@
# Customize this file to control which resources are extracted
# For full format details and examples, see:
# https://github.com/Azure/apiops-cli/blob/main/docs/guides/filtering-resources.md
#
# HOW FILTERING WORKS (read this before uncommenting sections below):
# Each top-level key is INDEPENDENT. The three states are:
# - key omitted → include ALL resources of that type (default)
# - key: [] → include NONE of that type
# - key: [names] → include only the named resources (case-insensitive,
# supports * and ? wildcards)
#
# Tip: to extract JUST one API plus its transitive dependencies, set the
# `apis` key to that API and set every other type to []. Every key you
# omit still extracts everything of that type. Example:
#
# apis:
# - my-own-api
# backends: []
# namedValues: []
# products: []
# tags: []
# versionSets: []
# loggers: []
# diagnostics: []
# groups: []
# policyFragments: []
# gateways: []
# schemas: []
# subscriptions: []
# policies: []
# policyRestrictions: []
# documentations: []
# workspaces: []
#
# Transitive dependencies (referenced backends, named values, policy
# fragments, tags, version sets) are still pulled in automatically even
# when the parent key is []. Use `--no-transitive` to disable that.

# Extract only specific APIs by name (or wildcard pattern)
# apis:
Expand Down
90 changes: 90 additions & 0 deletions tests/unit/services/extract-service.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,96 @@ describe('extract-service', () => {
expect(result.collectedPolicies.has('service-policy')).toBe(true);
});

it('should skip service policy when filter.policies is an empty array', async () => {
// Regression: service policy previously bypassed the filter entirely and
// was extracted even when `policies: []` explicitly excluded it.
const client = createMockClient({});
client.getResource = vi.fn().mockImplementation(async (_ctx, descriptor) => {
if (descriptor.type === ResourceType.ServicePolicy) {
return {
name: 'policy',
properties: { value: '<policies><inbound><base /></inbound></policies>' },
};
}
return undefined;
});
const store = createMockStore();

const filter: FilterConfig = { policies: [] };
const config: ExtractConfig = {
service: testContext,
outputDir: '/output',
filter,
includeTransitive: false,
logLevel: LogLevel.INFO,
};

const result = await runExtraction(client, store, config);

// The mock's getResource may still be called for other types, but writeContent
// must NOT have been invoked for the ServicePolicy descriptor.
const serviceWrites = (store.writeContent as ReturnType<typeof vi.fn>).mock.calls.filter(
(call) => (call[1] as ResourceDescriptor)?.type === ResourceType.ServicePolicy
);
expect(serviceWrites).toHaveLength(0);
expect(result.collectedPolicies.has('service-policy')).toBe(false);
});

it('should extract service policy when filter omits policies key entirely (undefined = include all)', async () => {
const client = createMockClient({});
client.getResource = vi.fn().mockImplementation(async (_ctx, descriptor) => {
if (descriptor.type === ResourceType.ServicePolicy) {
return {
name: 'policy',
properties: { value: '<policies><inbound><base /></inbound></policies>' },
};
}
return undefined;
});
const store = createMockStore();

// Filter with no `policies` field — semantics: include all singleton types
const filter: FilterConfig = { apis: ['some-api'] };
const config: ExtractConfig = {
service: testContext,
outputDir: '/output',
filter,
includeTransitive: false,
logLevel: LogLevel.INFO,
};

const result = await runExtraction(client, store, config);

expect(result.collectedPolicies.has('service-policy')).toBe(true);
});

it('should extract service policy when filter.policies contains the singleton name "policy"', async () => {
const client = createMockClient({});
client.getResource = vi.fn().mockImplementation(async (_ctx, descriptor) => {
if (descriptor.type === ResourceType.ServicePolicy) {
return {
name: 'policy',
properties: { value: '<policies><inbound><base /></inbound></policies>' },
};
}
return undefined;
});
const store = createMockStore();

const filter: FilterConfig = { policies: ['policy'] };
const config: ExtractConfig = {
service: testContext,
outputDir: '/output',
filter,
includeTransitive: false,
logLevel: LogLevel.INFO,
};

const result = await runExtraction(client, store, config);

expect(result.collectedPolicies.has('service-policy')).toBe(true);
});

it('should redact inline secrets in service policy before storing', async () => {
const policyContent = '<policies><inbound><set-query-parameter name="sig"><value>abc123</value></set-query-parameter></inbound></policies>';
const redactedPolicy = `<policies><inbound><set-query-parameter name="sig"><value>${REDACTION_MARKER}</value></set-query-parameter></inbound></policies>`;
Expand Down