noseyparker-scan(1)

📦 version 0.24.0, in nixos 26.05, nixpkgs unstable

✨ NAME ✨

noseyparker-scan(1) — okay bestie, this is the General Commands Manual page for noseyparker-scan, and it is literally here to scan content for secrets. That's the whole assignment. 💅

🦋 SYNOPSIS 🦋

bestie, here's the whole flag parade in one place:

noseyparker scan [-d|--datastore] [-j|--jobs] [--rules-path] [--ruleset]
[--load-builtins] [--git-url] [--enumerator] [--github-user]
[--github-organization] [--all-github-organizations] [--github-api-url]
[--github-repo-type] [--git-clone] [--git-history] [--max-file-size]
[-i|--ignore] [--blob-metadata] [--git-blob-provenance]
[--snippet-length] [--copy-blobs] [--copy-blobs-format]
[-v|--verbose]... [-q|--quiet] [--color] [--progress] [--ignore-certs]
[--rlimit-nofile] [--sqlite-cache-size] [--enable-backtraces]
[-h|--help] [INPUT]

💖 DESCRIPTION 💖

Scan content for secrets. Yes, that's the vibe. 🌸

This command uses regex-based rules to identify hardcoded secrets and other potentially sensitive information in textual content — or in inputs that can have textual content extracted from them. It's giving detective but make it regex. 🔍

The findings from scanning are recorded into a datastore. The recorded findings can later be reported in several formats using the summarize and report commands. So like, scan first, then spill the tea later. 💅

Several types of inputs can be specified:

  • Positional input arguments can be either files or directories. Files are scanned directly; directories are recursively enumerated and scanned. Any directories encountered that are Git repositories will have their entire history scanned. No cap, the whole history. 😭
  • A Git repository URL can be specified with the --git-repo=URL argument. This will cause Nosey Parker to clone that repository to its datastore and scan its history.
  • A GitHub user can be specified with the --github-user=NAME argument. This will cause Nosey Parker to enumerate accessible repositories belonging to that user, clone them to its datastore, and scan their entire history.
  • A GitHub organization can be specified with the --github-org=NAME argument. This will cause Nosey Parker to enumerate accessible repositories belonging to that organization, clone them to its datastore, and scan their entire history.

The git binary on the PATH is used to clone any required Git repositories. It is carefully invoked to avoid using any system-wide or user-specific configuration. Iconic, honestly. 🫶

By default, when cloning repositories from GitHub or enumerating GitHub users or organizations, unauthenticated access is used. An optional personal access token can be specified using the NP_GITHUB_TOKEN environment variable. Using a personal access token gives higher rate limits and may make additional content accessible. Rate limits who? Not us with a token, bestie. 🔥

💅 OPTIONS 💅

  • -d, --datastore=PATH [default: datastore.np] — Use the specified datastore. The datastore will be created if it does not exist. May also be specified with the NP_DATASTORE environment variable. Cute and flexible. ✨
  • -j, --jobs=N [default: 80] — Use N parallel scanning threads. Eighty threads default? She's ambitious. 💀
  • -h, --help — Print help (see a summary with '-h'). The page literally says that, bestie. I'm just the messenger. 🎀

🎀 RULE SELECTION OPTIONS 🎀

  • --rules-path=PATH — Load additional rules and rulesets from the specified file or directory. The paths can be either files or directories. Directories are recursively walked and all discovered YAML files of rules and rulesets will be loaded. This option can be repeated. Stack those rules, bestie. 🦋
  • --ruleset=ID [default: default] — Enable the ruleset with the specified ID. The ID must resolve to a built-in ruleset or to an additional ruleset loaded with the --rules=PATH option. The special all ID causes all loaded rules to be used. This option can be repeated.

Specifying this option disables the default ruleset. If you want to use a custom ruleset in addition to the default ruleset, specify this option twice, e.g., --ruleset default --ruleset CUSTOM_ID. Yes, twice. She said what she said. 💅

  • --load-builtins=BOOL [default: true] — Control whether built-in rules and rulesets are loaded. Possible values: true, false. Simple, clean, built-in behavior check. ⭐

🌈 INPUT SPECIFIER OPTIONS 🌈

  • --git-url=URL — Clone and scan the Git repository at the specified URL. Only https URLs without credentials, query parameters, or fragment identifiers are supported. This option can be repeated. Keep it https and keep it clean. 🧚
  • --enumerator=PATH — Read inputs from a JSONL enumerator file (experimental). This can be used to stream inputs from other processes without having to write them to disk. Shell process substitution (e.g., --enumerator=<(my-enumerator-program)) can make this ergonomic.

Each line of the enumerator file should be a JSON object with one of the following forms:

{ "content_base64": "base64-encoded bytestring to scan", "provenance": <arbitrary object> }
{ "content": "utf8 string to scan", "provenance": <arbitrary object> }

This option can be repeated. Streaming inputs? Obsessed. 🍓

  • --github-user=NAME — Clone and scan accessible repositories belonging to the specified GitHub user. This option can be repeated.
  • --github-organization=NAME — Clone and scan accessible repositories belonging to the specified GitHub organization. This option can be repeated.
  • --all-github-organizations — Clone and scan accessible repositories from all accessible GitHub organizations. This only works with a GitHub Enterprise Server instance. A non-default option for the --github-api-url option must be specified. GHES girlies, this one's for you. 👑
  • --github-api-url=URL [default: https://api.github.com/] — Use the specified URL for GitHub API access. If accessing a GitHub Enterprise Server instance, this value should be the entire base URL and include the api/v3 portion, e.g., https://github.example.com/api/v3.
  • --github-repo-type=TYPE [default: source] — Clone and scan GitHub repos only of the given type. Possible values:
    • all: Select both source repositories and fork repositories
    • source: Only source repositories, i.e., ones that are not forks
    • fork: Only fork repositories

Choose your fighter. Or repo type. Same energy. 💜

  • --git-clone=MODE [default: bare] — Use the specified method for cloning Git repositories. Possible values:
    • bare: Match the behavior of git clone --bare
    • mirror: Match the behavior of git clone --mirror
  • --git-history=MODE [default: full] — Use the specified mode for handling Git history.

Git history can be completely ignored when scanning by using --git-history=none. Note that this will interfere with other input specifiers that cause Git repositories to be automatically cloned. For example, specifying an input with --git-url=<URL> while simultaneously using --git-history=none will not result in useful scanning. So like, don't do that and expect magic. 🙈

Possible values:

  • full: Scan all history
  • none: Scan no history
  • [INPUT] — Scan the specified file, directory, or local Git repository. The main character input, honestly. ✨

💖 CONTENT FILTERING OPTIONS 💖

  • --max-file-size=MEGABYTES [default: 100] — Do not scan files larger than the specified size. The value is parsed as a floating point literal, and hence fractional values can be supplied. A non-positive value means "no limit". Note that scanning requires reading the entire contents of each file into memory, so using an excessively large limit may be problematic. Big files are not automatically a flex, bestie. 😭
  • -i, --ignore=FILE — Use custom path-based ignore rules from the specified file. The ignore file should contain gitignore-style rules. This option can be repeated. Set boundaries, queen. 💅

🦋 METADATA COLLECTION OPTIONS 🦋

  • --blob-metadata=MODE [default: matching] — Specify which blobs will have metadata recorded. Possible values:
    • all: Record metadata for all encountered blobs
    • matching: Record metadata only for blobs with matches
    • none: Record metadata for no blobs
  • --git-blob-provenance=MODE [default: first-seen] — Specify which Git commit provenance metadata will be collected. This should not need to be changed unless you are running into performance problems on a problematic Git repository input. Possible values:
    • first-seen: The Git repository and set of commits and accompanying pathnames in which a blob is first seen
    • minimal: Only the Git repository in which a blob is seen

Don't touch this unless the repo is acting up, bestie. 🫶

  • --copy-blobs-format=FORMAT [default: parquet] — Specify the format for blobs copied by the --copy-blobs option. Possible values:
    • parquet: Parquet format
    • files: Plain files, similar to Git's loose object format

🌸 DATA COLLECTION OPTIONS 🌸

  • --snippet-length=BYTES [default: 256] — Include up to the specified number of bytes before and after each match. The default value typically gives between 4 and 7 lines of context before and after each match. Context is cute. Context is important. Context is everything. 💅
  • --copy-blobs=MODE [default: none] — Specify which blobs will be copied in entirety to the datastore.

If this option is enabled, corresponding blobs will be written to the blobs directory within the datastore. The format of that directory is similar to Git's "loose" object format: the first 2 characters of the hex-encoded blob ID name a subdirectory, and the remaining characters are used as the filename.

This mechanism exists to aid in ad-hoc downstream investigation. Copied blobs are not used elsewhere in Nosey Parker at this point. So like, they're for your investigation era, not for Nosey Parker's main-character behavior. 🔥

Possible values:

  • all: Copy all encountered blobs
  • matching: Copy only blobs with matches
  • none: Copy no blobs

🌍 GLOBAL OPTIONS

  • -v, --verbose — ok so this turns on verbose output 💬 and yes bestie you can stack it, up to 3 times, each one giving you successively more output. More -v = more tea ✨
  • -q, --quiet — shhh 🙈 this suppresses the non-error feedback messages. It silences WARNING, INFO, DEBUG, and TRACE messages and disables progress bars entirely. And listen — this overrides any provided verbosity and progress reporting options, so it's the main character here, periodt. 👑
  • --color=MODE [default: auto] — enable or disable colored output 🌈 When this is auto, colors get enabled for stdout and stderr when they are terminals.
heads up: if the NO_COLOR environment variable is set, it takes precedence and is equivalent to --color=never 💅

[possible values: auto, never, always]

  • --progress=MODE [default: auto] — enable or disable progress bars 🎀 When this is auto, progress bars are enabled when stderr is a terminal. [possible values: auto, never, always]
  • --ignore-certs — ignore validation of TLS certificates 🚨 like, just don't check 'em. iykyk.

🧪 ADVANCED GLOBAL OPTIONS

okay these are the deep-lore flags, the ones you don't touch unless you need to 🧚

  • --rlimit-nofile=LIMIT [default: 16384] — set the rlimit for number of open files to LIMIT 🗂️ Fr fr, this should not need to be changed from the default unless you run into crashes from running out of file descriptors. That's the only vibe check that justifies touching it.
  • --sqlite-cache-size=SIZE [default: -1048576] — set the cache size for SQLite connections to SIZE 💾 This has the effect of setting SQLite's pragma cache_size=SIZE. The default value is set to use a maximum of 1GiB for database cache. See https://sqlite.org/pragma.html#pragma_cache_size for more details. slay, very demure, very mindful caching 🫶
  • --enable-backtraces=BOOL [default: true] — enable or disable backtraces on panic 😭 This has the effect of setting the RUST_BACKTRACE environment variable to 1. [possible values: true, false]

okay that's the whole page bestie, you're officially a noseyparker-scan(1) scholar 🎓✨ this one's from the scan section of the docs, page noseyparker-scan(1) — go forth and scan responsibly 💖🦋👑

📦 where this page comes from