Feature Request: Respect .gitignore and support configurable file/directory exclusions during indexing

[kgdemir]

Problem

When indexing a repository, MemSearch currently scans all files under the target directory. In practice, repositories often contain directories and files that should not be indexed, such as:

node_modules/
dist/
build/
.next/
coverage/
.venv/
Generated assets
Large binary files
Project-specific cache directories

These unnecessary files increase indexing time, consume additional storage, and may reduce search quality by introducing content that users are unlikely to search.

Additionally, most developers already maintain a .gitignore file that defines which files are not part of the source code. Not respecting these ignore rules means MemSearch performs work that has already been explicitly excluded by the project.

Proposed solution

Add support for configurable exclusion rules, ideally with both automatic .gitignore support and user-defined exclusions.

Possible implementation:

--respect-gitignore
Skip files and directories ignored by .gitignore.
--exclude
Allow multiple exclude patterns from the command line.

Example:

memsearch index .
--respect-gitignore
--exclude "*.min.js"
--exclude "docs/generated"
Optional configuration file (e.g. .memsearchignore or configuration section in the existing config) for persistent project-specific exclusions.

This would give users flexibility while keeping the default behavior unchanged unless the option is enabled.

Alternatives considered

One option would be to delegate file discovery to a mature tool such as fd, which already supports:

.gitignore
.ignore
hidden file handling
efficient directory traversal
glob-based exclusions

Using fd could reduce maintenance by relying on a well-tested implementation of ignore semantics. However, introducing an external dependency may not be desirable for all platforms or installation methods, so implementing native ignore handling may be the better long-term solution.

Either approach would significantly improve indexing performance and relevance for real-world repositories.

[zc277584121]

Thanks for the feature request. One thing I want to clarify before deciding the right product behavior: MemSearch currently only indexes Markdown-like files (.md / .markdown) and skips hidden files/directories by default. So directories such as node_modules, dist, build, .venv, or generated asset folders should only affect indexing if they contain Markdown files.

Could you share a bit more about the concrete case you are seeing?

• Are the unwanted indexed files generated or vendored Markdown files inside ignored directories?
• Do they come from package dependencies, docs generators, coverage/build outputs, or something else?
• Roughly how many extra Markdown files are being picked up, and how much does that affect indexing time or recall quality?
• Would you expect ignore rules to apply only when indexing directories, or also when a user explicitly passes a specific file path?

That context would help us choose between a narrow exclusion option, .gitignore support, a .memsearchignore-style file, or some combination of those without surprising users who intentionally index ignored local memory files.

[kgdemir]

Thanks for the clarification.

In our case, the issue isn't generated Markdown from build artifacts. We intentionally keep a large number of Markdown files in the repository that are unrelated to the project's source code or documentation.

For example, we have directories containing:

• AI assistant plugins/extensions
• Skills and prompt libraries
• Installed MCPs/tools
• Third-party templates
• Personal knowledge bases
• Other project-specific resources that happen to be Markdown

These files are committed to the repository because they are part of our development environment or workflow, but they are not relevant when indexing the project's codebase. They can easily outnumber the actual project documentation.

Since they are committed, hidden-directory filtering doesn't help, and these directories are often intentionally listed in .gitignore (or could be excluded through a project-specific ignore file) because they shouldn't participate in indexing.

The main goal isn't only reducing indexing time, but also improving search quality by avoiding unrelated Markdown content. Without exclusions, searches may return results from plugin documentation, prompt libraries, or other auxiliary resources instead of the project's own documentation.

I think configurable exclusions would provide the most flexibility. Supporting .gitignore as an optional feature (rather than the default) together with explicit --exclude patterns or a .memsearchignore file would allow users to tailor indexing to their repository layout without changing existing behavior.

Regarding your last question, my expectation would be:

• Ignore rules should apply when indexing a directory recursively.
• If the user explicitly specifies a file path to index, that file should still be indexed, regardless of ignore rules, since that represents an explicit user choice.