Merge branch 'Yelp:master' into feature/support-multi-os-path

.github/issue_template.md

@@ -0,0 +1,20 @@
+* **I'm submitting a ...**
+  - [ ] bug report
+  - [ ] feature request
+
+* **What is the current behavior?**
+
+* **If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem**
+
+* **What is the expected behavior?**
+
+* **What is the motivation / use case for changing the behavior?**
+
+* **Please tell us about your environment:**
+  - detect-secrets Version:
+  - Python Version:
+  - OS Version:
+  - File type (if applicable):
+
+* **Other information**
+<!-- (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. Stack Overflow, gitter etc.). If you have or can produce a baseline file, please provide it as part of a separate comment. -->

.github/pull_request_template.md

@@ -0,0 +1,19 @@
+* **Please check if the PR fulfills these requirements**
+- [ ] Tests for the changes have been added
+<!-- (for bug fixes / features) -->
+- [ ] Docs have been added / updated
+<!-- (for bug fixes / features) -->
+- [ ] All CI checks are green
+
+* **What kind of change does this PR introduce?**
+<!-- (Bug fix, feature, docs update, ...) -->
+
+* **What is the current behavior?**
+<!-- (You can also link to an open issue here) -->
+
+* **What is the new behavior (if this is a feature change)?**
+
+* **Does this PR introduce a breaking change?**
+<!-- (What changes might users need to make in their application due to this PR?) -->
+
+* **Other information**:

.github/workflows/ci.yml

@@ -8,14 +8,16 @@ on:
     branches: [ master ]
   pull_request:
   workflow_dispatch:
+  schedule:
+    - cron: '0 0 1 * *'
 
 jobs:
   main:
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
-        python: ['3.6', '3.7', '3.8', '3.9']
+        python: ['3.8', '3.9', '3.10', '3.11']
     steps:
     - uses: actions/checkout@v2
     - uses: actions/setup-python@v2

.github/workflows/pypi.yml

@@ -15,7 +15,7 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest, macos-latest]
-        python: ['3.6', '3.7', '3.8', '3.9']
+        python: ['3.8', '3.9', '3.10', '3.11']
     steps:
     - uses: actions/checkout@v2
     - uses: actions/setup-python@v2

.secrets.baseline

@@ -1,5 +1,5 @@
 {
-  "version": "1.3.0",
+  "version": "1.4.0",
   "plugins_used": [
     {
       "name": "ArtifactoryDetector"

CHANGELOG.md

@@ -29,6 +29,43 @@ If you love `detect-secrets`, please star our project on GitHub to show your sup
 ### Unreleased
 -->
 
+### v1.4.0
+##### October 4th, 2022
+
+#### :newspaper: News
+- We're dropping support for Python 3.6 starting v1.5.0! Python 3.6 reached EOL on December 23, 2021 and, therefore, is currently unsupported. We hope this announcement gives you plenty of time to upgrade your project, if needed.
+
+#### :mega: Release Highlights
+- Improved filtering by excluding secrets that have already been detected by a regex-based detector ([#612])
+#### :tada: New Features
+- Added a detector for Discord bot tokens ([#614])
+
+#### :sparkles: Usability
+- Improved the audit report to make it easier to parse programmatically ([#619])
+#### :telescope: Accuracy
+- Improve ArtifactoryDetector plugin to reduce false positives ([#499])
+
+#### :bug: Bugfixes
+- Fixed the verify flow in audit report by adding the code snippet of the verified secret ([#620])
+- Fixed deploy process to be environment configuration independent ([#625])
+
+#### :snake: Miscellaneous
+- Added support for .NET packages.lock.json files in the heuristic filter ([#593])
+- Multiple dependency updates
+
+[#499]: https://github.com/Yelp/detect-secrets/pull/499
+[#556]: https://github.com/Yelp/detect-secrets/pull/556
+[#589]: https://github.com/Yelp/detect-secrets/pull/589
+[#593]: https://github.com/Yelp/detect-secrets/pull/593
+[#598]: https://github.com/Yelp/detect-secrets/pull/598
+[#612]: https://github.com/Yelp/detect-secrets/pull/612
+[#614]: https://github.com/Yelp/detect-secrets/pull/614
+[#615]: https://github.com/Yelp/detect-secrets/pull/615
+[#616]: https://github.com/Yelp/detect-secrets/pull/616
+[#619]: https://github.com/Yelp/detect-secrets/pull/619
+[#620]: https://github.com/Yelp/detect-secrets/pull/620
+[#625]: https://github.com/Yelp/detect-secrets/pull/625
+
 ### v1.3.0
 ##### July 22nd, 2022
 

CONTRIBUTING.md

@@ -70,10 +70,10 @@ sys     0m2.486s
 
 ### Running the Entire Test Suite
 
-You can run the test suite in the interpreter of your choice (in this example, `py36`) by doing:
+You can run the test suite in the interpreter of your choice (in this example, `py37`) by doing:
 
 ```bash
-tox -e py36
+tox -e py37
 ```
 
 This will also run the code through our series of coverage tests, `mypy` rules and other linting

README.md

@@ -96,6 +96,8 @@ AWSKeyDetector
 AzureStorageKeyDetector
 BasicAuthDetector
 CloudantDetector
+DiscordBotTokenDetector
+GitHubTokenDetector
 Base64HighEntropyString
 HexHighEntropyString
 IbmCloudIamDetector
@@ -105,8 +107,10 @@ KeywordDetector
 MailchimpDetector
 NpmDetector
 PrivateKeyDetector
+SendGridDetector
 SlackDetector
 SoftlayerDetector
+SquareOAuthDetector
 StripeDetector
 TwilioKeyDetector
 ```
@@ -388,7 +392,7 @@ We recommend setting this up as a pre-commit hook. One way to do this is by usin
 # .pre-commit-config.yaml
 repos:
 -   repo: https://github.com/Yelp/detect-secrets
-    rev: v1.3.0
+    rev: v1.4.0
     hooks:
     -   id: detect-secrets
         args: ['--baseline', '.secrets.baseline']

detect_secrets/__version__.py

@@ -1 +1 @@
-VERSION = '1.3.0'
+VERSION = '1.4.0'

detect_secrets/audit/analytics.py

@@ -58,7 +58,14 @@ def _get_plugin_counter(self, secret_type: str) -> 'StatisticsCounter':
         return cast(StatisticsCounter, self.data[secret_type]['stats'])
 
     def __str__(self) -> str:
-        raise NotImplementedError
+        output = ''
+
+        for secret_type, framework in self.data.items():
+            output += f'Plugin: {get_mapping_from_secret_type_to_class()[secret_type].__name__}\n'
+            for value in framework.values():
+                output += f'Statistics: {value}\n\n'
+
+        return output
 
     def json(self) -> Dict[str, Any]:
         output = {}
@@ -77,19 +84,36 @@ def __init__(self) -> None:
         self.incorrect: int = 0
         self.unknown: int = 0
 
-    def __repr__(self) -> str:
+    def __str__(self) -> str:
         return (
-            f'{self.__class__.__name__}(correct={self.correct}, '
-            'incorrect={self.incorrect}, unknown={self.unknown},)'
+            f'True Positives: {self.correct}, False Positives: {self.incorrect}, '
+            f'Unknown: {self.unknown}, Precision: {self.calculate_precision()}, '
+            f'Recall: {self.calculate_recall()}'
         )
 
     def json(self) -> Dict[str, Any]:
+        return {
+            'raw': {
+                'true-positives': self.correct,
+                'false-positives': self.incorrect,
+                'unknown': self.unknown,
+            },
+            'score': {
+                'precision': self.calculate_precision(),
+                'recall': self.calculate_recall(),
+            },
+        }
+
+    def calculate_precision(self) -> float:
         precision = (
             round(float(self.correct) / (self.correct + self.incorrect), 4)
             if (self.correct and self.incorrect)
             else 0.0
         )
 
+        return precision
+
+    def calculate_recall(self) -> float:
         # NOTE(2020-11-08|domanchi): This isn't the formal definition of `recall`, however,
         # this is the definition that we're going to attribute to it.
         #
@@ -124,14 +148,4 @@ def json(self) -> Dict[str, Any]:
             else 0.0
         )
 
-        return {
-            'raw': {
-                'true-positives': self.correct,
-                'false-positives': self.incorrect,
-                'unknown': self.unknown,
-            },
-            'score': {
-                'precision': precision,
-                'recall': recall,
-            },
-        }
+        return recall

detect_secrets/audit/common.py

@@ -19,6 +19,7 @@
 from ..transformers import get_transformed_file
 from ..types import NamedIO
 from ..util.inject import call_function_with_arguments
+from detect_secrets.util.code_snippet import get_code_snippet
 
 
 def get_baseline_from_file(filename: str) -> SecretsCollection:
@@ -91,6 +92,7 @@ def get_raw_secrets_from_file(
             line_numbers = list(range(len(lines_to_scan)))
 
         for line_number, line in zip(line_numbers, lines_to_scan):
+            context = get_code_snippet(lines=line_getter.lines, line_number=line_number + 1)
             identified_secrets = call_function_with_arguments(
                 plugin.analyze_line,
                 filename=secret.filename,
@@ -100,6 +102,7 @@ def get_raw_secrets_from_file(
                 # We enable eager search, because we *know* there's a secret here -- the baseline
                 # flagged it after all.
                 enable_eager_search=bool(secret.line_number),
+                context=context,
             )
 
             for identified_secret in (identified_secrets or []):

detect_secrets/audit/report.py

@@ -28,7 +28,7 @@ def generate_report(
     baseline_file: str,
     class_to_print: SecretClassToPrint = None,
     line_getter_factory: Callable[[str], 'LineGetter'] = open_file,
-) -> List[Dict[str, Any]]:
+) -> Dict[str, List[Dict[str, Any]]]:
 
     secrets: Dict[Tuple[str, str], Any] = {}
     for filename, secret in get_baseline_from_file(baseline_file):
@@ -63,8 +63,9 @@ def generate_report(
                     ],
                     'category': verified_result.name,
                 }
-
-    return list(secrets.values())
+    return {
+        'results': list(secrets.values()),
+    }
 
 
 def get_prioritized_verified_result(

detect_secrets/core/baseline.py

@@ -128,7 +128,7 @@ def upgrade(baseline: Dict[str, Any]) -> Dict[str, Any]:
 
     new_baseline = {**baseline}
     for module in modules:
-        module.upgrade(new_baseline)    # type: ignore
+        module.upgrade(new_baseline)
 
     new_baseline['version'] = VERSION
     return new_baseline

detect_secrets/core/plugins/initialize.py

@@ -21,7 +21,7 @@ def from_secret_type(secret_type: str) -> Plugin:
         raise TypeError
 
     try:
-        return plugin_type(**_get_config(plugin_type.__name__))     # type: ignore
+        return plugin_type(**_get_config(plugin_type.__name__))
     except TypeError:
         log.error('Unable to initialize plugin!')
         raise
@@ -44,7 +44,7 @@ def from_plugin_classname(classname: str) -> Plugin:
         raise TypeError
 
     try:
-        return plugin_type(**_get_config(classname))        # type: ignore
+        return plugin_type(**_get_config(classname))
     except TypeError:
         log.error('Unable to initialize plugin!')
         raise

detect_secrets/core/scan.py

@@ -101,7 +101,7 @@ def get_files_to_scan(
 
                 if (
                     valid_paths is True
-                    or relative_path in cast(Set[str], valid_paths)
+                    or relative_path in valid_paths
                 ):
                     yield relative_path
 

detect_secrets/core/usage/audit.py

@@ -1,4 +1,5 @@
 import argparse
+from typing import cast
 
 
 def add_audit_action(parent: argparse._SubParsersAction) -> argparse.ArgumentParser:
@@ -23,7 +24,7 @@ def add_audit_action(parent: argparse._SubParsersAction) -> argparse.ArgumentPar
     _add_mode_parser(parser)
     _add_report_module(parser)
     _add_statistics_module(parser)
-    return parser
+    return cast(argparse.ArgumentParser, parser)
 
 
 def _add_mode_parser(parser: argparse.ArgumentParser) -> None:

detect_secrets/core/usage/scan.py

@@ -20,7 +20,7 @@ def add_scan_action(parent: argparse._SubParsersAction) -> argparse.ArgumentPars
     _add_pragma_scanning(parser)
     _add_initialize_baseline_options(parser)
 
-    return parser
+    return cast(argparse.ArgumentParser, parser)
 
 
 def _add_adhoc_scanning(parser: argparse.ArgumentParser) -> None:

detect_secrets/filters/gibberish/__init__.py

@@ -32,7 +32,7 @@ def initialize(model_path: Optional[str] = None, limit: float = 3.7) -> None:
     """
     path = model_path
     if not path:
-        path = os.path.join(__path__[0], 'rfc.model')   # type: ignore  # mypy issue #1422
+        path = os.path.join(__path__[0], 'rfc.model')
 
     model = get_model()
 

detect_secrets/filters/heuristic.py

@@ -2,8 +2,12 @@
 import re
 import string
 from functools import lru_cache
+from typing import Optional
 from typing import Pattern
 
+from detect_secrets.plugins.base import BasePlugin
+from detect_secrets.plugins.base import RegexBasedDetector
+
 
 def is_sequential_string(secret: str) -> bool:
     sequences = (
@@ -57,13 +61,14 @@ def _get_uuid_regex() -> Pattern:
     )
 
 
-def is_likely_id_string(secret: str, line: str) -> bool:
+def is_likely_id_string(secret: str, line: str, plugin: Optional[BasePlugin] = None) -> bool:
     try:
         index = line.index(secret)
     except ValueError:
         return False
 
-    return bool(_get_id_detector_regex().search(line, pos=0, endpos=index))
+    return (not plugin or not isinstance(plugin, RegexBasedDetector)) \
+        and bool(_get_id_detector_regex().search(line, pos=0, endpos=index))
 
 
 @lru_cache(maxsize=1)
@@ -159,7 +164,7 @@ def is_prefixed_with_dollar_sign(secret: str) -> bool:
     # false negatives than `is_templated_secret` (e.g. secrets that actually start with a $).
     # This is best used with files that actually use this as a means of referencing variables.
     # TODO: More intelligent filetype handling?
-    return secret[0] == '$'
+    return bool(secret) and secret[0] == '$'
 
 
 def is_indirect_reference(line: str) -> bool:
@@ -208,6 +213,7 @@ def is_lock_file(filename: str) -> bool:
         'Pipfile.lock',
         'poetry.lock',
         'Cargo.lock',
+        'packages.lock.json',
     }