8000 Bring-your own-plugins (BYOP), via --custom-plugins option by KevinHock · Pull Request #255 · Yelp/detect-secrets · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Bring-your own-plugins (BYOP), via --custom-plugins option #255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Bring-your own-plugins (BYOP), via --custom-plugins option #255

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f5f9295
:snake: Refactor some useless wrapper functions
KevinHock Oct 11, 2019
4b5405c
:mortar_board: Consistency nit
KevinHock Oct 11, 2019
0b7ae7f
:telescope: Improve `get_secret_access_keys` detection
KevinHock Oct 18, 2019
332d9ef
:mortar_board: Add succinct comment :D
KevinHock Oct 18, 2019
a836eae
:tada: Add `--custom-plugins` feature
KevinHock Oct 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ For a look at recent changes, please see [CHANGELOG.md](https://github.com/Yelp/

If you are looking 8000 to contribute, please see [CONTRIBUTING.md](https://github.com/Yelp/detect-secrets/blob/master/CONTRIBUTING.md).


## Example Usage

### Setting Up a Baseline
Expand All @@ -44,6 +45,7 @@ If you are looking to contribute, please see [CONTRIBUTING.md](https://github.co
$ detect-secrets scan > .secrets.baseline
```


### pre-commit Hook

```
Expand All @@ -56,12 +58,14 @@ $ cat .pre-commit-config.yaml
exclude: .*/tests/.*
```


### Auditing a Baseline

```
$ detect-secrets audit .secrets.baseline
```


### Upgrading Baselines

This is only applicable for upgrading baselines that have been created after version 0.9.
Expand All @@ -71,6 +75,7 @@ For upgrading baselines lower than that version, just recreate it.
$ detect-secrets scan --update .secrets.baseline
```


### Command Line

`detect-secrets` is designed to be used as a git pre-commit hook, but you can also invoke `detect-secrets scan [path]` directly being `path` the file(s) and/or directory(ies) to scan (`path` defaults to `.` if not specified).
Expand All @@ -93,6 +98,7 @@ either the client-side pre-commit hook, or the server-side secret scanner.
3. **Secrets Baseline**, to allowlist pre-existing secrets in the repository,
so that they won't be continuously caught through scan iterations.


### Client-side `pre-commit` Hook

See [pre-commit](https://github.com/pre-commit/pre-commit) for instructions
Expand All @@ -104,11 +110,13 @@ Hooks available:
- `detect-secrets`: This hook detects and prevents high entropy strings from
entering the codebase.


### Server-side Secret Scanning

Please see the [detect-secrets-server](https://github.com/Yelp/detect-secrets-server)
repository for installation instructions.


### Secrets Baseline

```
Expand Down Expand Up @@ -144,6 +152,7 @@ This may be a convenient way for you to allowlist secrets, without having to
regenerate the entire baseline again. Furthermore, this makes the allowlisted
secrets easily searchable, auditable, and maintainable.


## Currently Supported Plugins

The current heuristic searches we implement out of the box include:
Expand Down Expand Up @@ -175,18 +184,23 @@ See [detect_secrets/
plugins](https://github.com/Yelp/detect-secrets/tree/master/detect_secrets/plugins)
for more details.

There is also a `--custom-plugins` option in which you can bring your own plugins, e.g. `detect-secrets scan --custom-plugins testing/custom_plugins_dir/ --custom-plugins testing/hippo_plugin.py`.


## Caveats

This is not meant to be a sure-fire solution to prevent secrets from entering
the codebase. Only proper developer education can truly do that. This pre-commit
hook merely implements several heuristics to try and prevent obvious cases of
committing secrets.


### Things that won't be prevented

* Multi-line secrets
* Default passwords that don't trigger the `KeywordDetector` (e.g. `login = "hunter2"`)


### Plugin Configuration

One method that this package uses to find secrets is by searching for high
Expand Down
146 changes: 93 additions & 53 deletions detect_secrets/core/audit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,16 @@
from copy import deepcopy
from functools import lru_cache

from ..plugins.common import initialize
from ..plugins.common.util import get_mapping_from_secret_type_to_class_name
from ..util import get_git_remotes
from ..util import get_git_sha
from .baseline import merge_results
from .bidirectional_iterator import BidirectionalIterator
from .code_snippet import CodeSnippetHighlighter
from .color import AnsiColor
from .color import colorize
from .common import write_baseline_to_file
from detect_secrets.core.baseline import merge_results
from detect_secrets.core.bidirectional_iterator import BidirectionalIterator
from detect_secrets.core.code_snippet import CodeSnippetHighlighter
from detect_secrets.core.color import AnsiColor
from detect_secrets.core.color import colorize
from detect_secrets.core.common import write_baseline_to_file
from detect_secrets.plugins.common import initialize
from detect_secrets.plugins.common.util import get_mapping_from_secret_type_to_class_name
from detect_secrets.util import get_git_remotes
from detect_secrets.util import get_git_sha


class SecretNotFoundOnSpecifiedLineError(Exception):
Expand Down Expand Up @@ -48,7 +48,7 @@ class RedundantComparisonError(Exception):
'config': {},
}
EMPTY_STATS_RESULT = {
'signal': 0,
'signal': '0.00%',
'true-positives': {
'count': 0,
'files': defaultdict(int),
Expand Down Expand Up @@ -87,11 +87,12 @@ def audit_baseline(baseline_filename):

try:
_print_context(
filename,
secret,
current_secret_index,
total_choices,
original_baseline['plugins_used'],
filename=filename,
secret=secret,
count=current_secret_index,
total=total_choices,
plugins_used=original_baseline['plugins_used'],
custom_plugin_paths=original_baseline['custom_plugin_paths'],
)
decision = _get_user_decision(can_step_back=secret_iterator.can_step_back())
except SecretNotFoundOnSpecifiedLineError:
Expand Down Expand Up @@ -144,7 +145,7 @@ def compare_baselines(old_baseline_filename, new_baseline_filename):
This means that we won't have cases where secrets are moved around;
only added or removed.

NOTE F438 : We don't want to do a version check, because we want to be able to
Note: We don't want to do a version check, because we want to be able to
use this functionality across versions (to see how the new version fares
compared to the old one).
"""
Expand All @@ -170,6 +171,7 @@ def compare_baselines(old_baseline_filename, new_baseline_filename):
header = '{} {}'
if is_removed:
plugins_used = old_baseline['plugins_used']
custom_plugin_paths = old_baseline['custom_plugin_paths']
header = header.format(
colorize('Status:', AnsiColor.BOLD),
'>> {} <<'.format(
Expand All @@ -178,6 +180,7 @@ def compare_baselines(old_baseline_filename, new_baseline_filename):
)
else:
plugins_used = new_baseline['plugins_used']
custom_plugin_paths = new_baseline['custom_plugin_paths']
header = header.format(
colorize('Status:', AnsiColor.BOLD),
'>> {} <<'.format(
Expand All @@ -187,13 +190,14 @@ def compare_baselines(old_baseline_filename, new_baseline_filename):

try:
_print_context(
filename,
secret,
current_index,
total_reviews,
plugins_used,
filename=filename,
secret=secret,
count=current_index,
total=total_reviews,
plugins_used=plugins_used,
custom_plugin_paths=custom_plugin_paths,
additional_header_lines=header,
force=is_removed,
force_line_printing=is_removed,
)
decision = _get_user_decision(
can_step_back=secret_iterator.can_step_back(),
Expand Down Expand Up @@ -258,7 +262,9 @@ def determine_audit_results(baseline, baseline_path):
'stats': deepcopy(EMPTY_STATS_RESULT),
}

secret_type_to_plugin_name = get_mapping_from_secret_type_to_class_name()
secret_type_to_plugin_name = get_mapping_from_secret_type_to_class_name(
custom_plugin_paths=baseline['custom_plugin_paths'],
)

total = 0
for filename, secret in all_secrets:
Expand All @@ -269,7 +275,8 @@ def determine_audit_results(baseline, baseline_path):
try:
secret_info['plaintext'] = get_raw_secret_value(
secret=secret,
plugin_settings=baseline['plugins_used'],
plugins_used=baseline['plugins_used'],
custom_plugin_paths=baseline['custom_plugin_paths'],
file_handle=io.StringIO(file_contents),
filename=filename,
)
Expand All @@ -283,13 +290,13 @@ def determine_audit_results(baseline, baseline_path):
audit_results['stats'][audit_result]['count'] += 1
audit_results['stats'][audit_result]['files'][filename] += 1
total += 1
audit_results['stats']['signal'] = str(
audit_results['stats']['signal'] = '{:.2f}'.format(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The % go into this format string

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch!! I no longer need the [:4] so that makes a lot of sense 🎊

(
float(audit_results['stats']['true-positives']['count'])
/
total
) * 100,
)[:4] + '%'
) + '%'

for plugin_config in baseline['plugins_used']:
plugin_name = plugin_config['name']
Expand Down Expand Up @@ -473,9 +480,10 @@ def _print_context( # pragma: no cover
secret,
count,
total,
plugin_settings,
plugins_used,
custom_plugin_paths,
additional_header_lines=None,
force=False,
force_line_printing=False,
):
"""
:type filename: str
Expand All @@ -490,15 +498,24 @@ def _print_context( # pragma: no cover
:type total: int
:param total: total number of secrets in baseline

:type plugin_settings: list
:param plugin_settings: plugins used to create baseline.
:type plugins_used: list
:param plugins_used: output of "plugins_used" in baseline. e.g.
>>> [
... {
... 'name': 'Base64HighEntropyString',
... 'base64_limit': 4.5,
... },
... ]

:type custom_plugin_paths: List[str]
:param custom_plugin_paths: possibly empty list of paths that have custom plugins.

:type additional_header_lines: str
:param additional_header_lines: any additional lines to add to the
header of the interactive audit display.

:type force: bool
:param force: if True, will print the lines of code even if it doesn't
:type force_line_printing: bool
:param force_line_printing: if True, will print the lines of code even if it doesn't
find the secret expected

:raises: SecretNotFoundOnSpecifiedLineError
Expand All @@ -523,10 +540,11 @@ def _print_context( # pragma: no cover
error_obj = None
try:
secret_with_context = _get_secret_with_context(
filename,
secret,
plugin_settings,
force=force,
filename=filename,
secret=secret,
plugins_used=plugins_used,
custom_plugin_paths=custom_plugin_paths,
force_line_printing=force_line_printing,
)
print(secret_with_context)
except SecretNotFoundOnSpecifiedLineError as e:
Expand Down Expand Up @@ -608,9 +626,10 @@ def _get_file_line(filename, line_number):
def _get_secret_with_context(
filename,
secret,
plugin_settings,
plugins_used,
custom_plugin_paths,
lines_of_context=5,
force=False,
force_line_printing=False,
):
"""
Displays the secret, with surrounding lines of code for better context.
Expand All @@ -621,15 +640,24 @@ def _get_secret_with_context(
:type secret: dict, PotentialSecret.json() format
:param secret: the secret listed in baseline

:type plugin_settings: list
:param plugin_settings: plugins used to create baseline.
:type plugins_used: list
:param plugins_used: output of "plugins_used" in baseline. e.g.
>>> [
... {
... 'name': 'Base64HighEntropyString',
... 'base64_limit': 4.5,
... },
... ]

:type custom_plugin_paths: List[str]
:param custom_plugin_paths: possibly empty list of paths that have custom plugins.

:type lines_of_context: int
:param lines_of_context: number of lines displayed before and after
secret.

:type force: bool
:param force: if True, will print the lines of code even if it doesn't
:type force_line_printing: bool
:param force_line_printing: if True, will print the lines of code even if it doesn't
find the secret expected

:raises: SecretNotFoundOnSpecifiedLineError
Expand All @@ -649,18 +677,19 @@ def _get_secret_with_context(
)

raw_secret_value = get_raw_secret_value(
secret,
plugin_settings,
io.StringIO(file_content),
filename,
secret=secret,
plugins_used=plugins_used,
custom_plugin_paths=custom_plugin_paths,
file_handle=io.StringIO(file_content),
filename=filename,
)

try:
snippet.highlight_line(raw_secret_value)
except ValueError:
raise SecretNotFoundOnSpecifiedLineError(secret['line_number'])
except SecretNotFoundOnSpecifiedLineError:
if not force:
if not force_line_printing:
raise

snippet.target_line = colorize(
Expand All @@ -673,16 +702,26 @@ def _get_secret_with_context(

def get_raw_secret_value(
secret,
plugin_settings,
plugins_used,
custom_plugin_paths,
file_handle,
filename,
):
"""
:type secret: dict
:param secret: see caller's docstring

:type plugin_settings: list
:param plugin_settings: see caller's docstring
:type plugins_used: list
:param plugins_used: output of "plugins_used" in baseline. e.g.
>>> [
... {
... 'name': 'Base64HighEntropyString',
... 'base64_limit': 4.5,
... },
... ]

:type custom_plugin_paths: List[str]
:param custom_plugin_paths: possibly empty list of paths that have custom plugins.

:type file_handle: file object
:param file_handle: Open handle to file where the secret is
Expand All @@ -692,8 +731,9 @@ def get_raw_secret_value(
as a means of comparing whether two secrets are equal.
"""
plugin = initialize.from_secret_type(
secret['type'],
plugin_settings,
secret_type=secret['type'],
plugins_used=plugins_used,
custom_plugin_paths=custom_plugin_paths,
)

plugin_secrets = plugin.analyze(file_handle, filename)
Expand Down
Loading
0