truffleHog-2.2.1/0000755000076500000000000000000014007162567014245 5ustar flowerwheel00000000000000truffleHog-2.2.1/PKG-INFO0000644000076500000000000000046414007162567015346 0ustar flowerwheel00000000000000Metadata-Version: 1.0 Name: truffleHog Version: 2.2.1 Summary: Searches through git repositories for high entropy strings, digging deep into commit history. Home-page: https://github.com/dxa4481/truffleHog Author: Dylan Ayrey Author-email: dxa4481@rit.edu License: GNU Description: UNKNOWN Platform: UNKNOWN truffleHog-2.2.1/README.md0000644000076500000000000001467414007162460015530 0ustar flowerwheel00000000000000 # truffleHog [![Build Status](https://travis-ci.org/dxa4481/truffleHog.svg?branch=master)](https://travis-ci.org/dxa4481/truffleHog) [![codecov](https://codecov.io/gh/dxa4481/truffleHog/branch/master/graph/badge.svg)](https://codecov.io/gh/dxa4481/truffleHog) Searches through git repositories for secrets, digging deep into commit history and branches. This is effective at finding secrets accidentally committed. ## NEW truffleHog previously functioned by running entropy checks on git diffs. This functionality still exists, but high signal regex checks have been added, and the ability to suppress entropy checking has also been added. ``` truffleHog --regex --entropy=False https://github.com/dxa4481/truffleHog.git ``` or ``` truffleHog file:///user/dxa4481/codeprojects/truffleHog/ ``` With the `--include_paths` and `--exclude_paths` options, it is also possible to limit scanning to a subset of objects in the Git history by defining regular expressions (one per line) in a file to match the targeted object paths. To illustrate, see the example include and exclude files below: _include-patterns.txt:_ ```ini src/ # lines beginning with "#" are treated as comments and are ignored gradle/ # regexes must match the entire path, but can use python's regex syntax for # case-insensitive matching and other advanced options (?i).*\.(properties|conf|ini|txt|y(a)?ml)$ (.*/)?id_[rd]sa$ ``` _exclude-patterns.txt:_ ```ini (.*/)?\.classpath$ .*\.jmx$ (.*/)?test/(.*/)?resources/ ``` These filter files could then be applied by: ```bash trufflehog --include_paths include-patterns.txt --exclude_paths exclude-patterns.txt file://path/to/my/repo.git ``` With these filters, issues found in files in the root-level `src` directory would be reported, unless they had the `.classpath` or `.jmx` extension, or if they were found in the `src/test/dev/resources/` directory, for example. Additional usage information is provided when calling `trufflehog` with the `-h` or `--help` options. These features help cut down on noise, and makes the tool easier to shove into a devops pipeline. ![Example](https://i.imgur.com/YAXndLD.png) ## Install ``` pip install truffleHog ``` ## Customizing Custom regexes can be added with the following flag `--rules /path/to/rules`. This should be a json file of the following format: ``` { "RSA private key": "-----BEGIN EC PRIVATE KEY-----" } ``` Things like subdomain enumeration, s3 bucket detection, and other useful regexes highly custom to the situation can be added. Feel free to also contribute high signal regexes upstream that you think will benefit the community. Things like Azure keys, Twilio keys, Google Compute keys, are welcome, provided a high signal regex can be constructed. trufflehog's base rule set sources from https://github.com/dxa4481/truffleHogRegexes/blob/master/truffleHogRegexes/regexes.json To explicitly allow particular secrets (e.g. self-signed keys used only for local testing) you can provide an allow list `--allow /path/to/allow` in the following format: ``` { "local self signed test key": "-----BEGIN EC PRIVATE KEY-----\nfoobar123\n-----END EC PRIVATE KEY-----", "git cherry pick SHAs": "regex:Cherry picked from .*", } ``` Note that values beginning with `regex:` will be used as regular expressions. Values without this will be literal, with some automatic conversions (e.g. flexible newlines). ## How it works This module will go through the entire commit history of each branch, and check each diff from each commit, and check for secrets. This is both by regex and by entropy. For entropy checks, truffleHog will evaluate the shannon entropy for both the base64 char set and hexidecimal char set for every blob of text greater than 20 characters comprised of those character sets in each diff. If at any point a high entropy string >20 characters is detected, it will print to the screen. ## Help ``` usage: trufflehog [-h] [--json] [--regex] [--rules RULES] [--allow ALLOW] [--entropy DO_ENTROPY] [--since_commit SINCE_COMMIT] [--max_depth MAX_DEPTH] git_url Find secrets hidden in the depths of git. positional arguments: git_url URL for secret searching optional arguments: -h, --help show this help message and exit --json Output in JSON --regex Enable high signal regex checks --rules RULES Ignore default regexes and source from json list file --allow ALLOW Explicitly allow regexes from json list file --entropy DO_ENTROPY Enable entropy checks --since_commit SINCE_COMMIT Only scan from a given commit hash --branch BRANCH Scans only the selected branch --max_depth MAX_DEPTH The max commit depth to go back when searching for secrets -i INCLUDE_PATHS_FILE, --include_paths INCLUDE_PATHS_FILE File with regular expressions (one per line), at least one of which must match a Git object path in order for it to be scanned; lines starting with "#" are treated as comments and are ignored. If empty or not provided (default), all Git object paths are included unless otherwise excluded via the --exclude_paths option. -x EXCLUDE_PATHS_FILE, --exclude_paths EXCLUDE_PATHS_FILE File with regular expressions (one per line), none of which may match a Git object path in order for it to be scanned; lines starting with "#" are treated as comments and are ignored. If empty or not provided (default), no Git object paths are excluded unless effectively excluded via the --include_paths option. ``` ## Running with Docker First, enter the directory containing the git repository ``` cd /path/to/git ``` To launch the trufflehog with the docker image, run the following" ``` docker run --rm -v "$(pwd):/proj" dxa4481/trufflehog file:///proj ``` `-v` mounts the current working dir (`pwd`) to the `/proj` dir in the Docker container `file:///proj` references that very same `/proj` dir in the container (which is also set as the default working dir in the Dockerfile) ## Wishlist - ~~A way to detect and not scan binary diffs~~ - ~~Don't rescan diffs if already looked at in another branch~~ - ~~A since commit X feature~~ - ~~Print the file affected~~ truffleHog-2.2.1/setup.cfg0000644000076500000000000000010314007162567016060 0ustar flowerwheel00000000000000[bdist_wheel] universal = 1 [egg_info] tag_build = tag_date = 0 truffleHog-2.2.1/setup.py0000644000076500000000000000106714007162513015752 0ustar flowerwheel00000000000000from setuptools import setup, find_packages setup( name='truffleHog', version='2.2.1', description='Searches through git repositories for high entropy strings, digging deep into commit history.', url='https://github.com/dxa4481/truffleHog', author='Dylan Ayrey', author_email='dxa4481@rit.edu', license='GNU', packages = ['truffleHog'], install_requires=[ 'GitPython == 3.0.6', 'truffleHogRegexes == 0.0.7' ], entry_points = { 'console_scripts': ['trufflehog = truffleHog.truffleHog:main'], }, ) truffleHog-2.2.1/truffleHog/0000755000076500000000000000000014007162567016352 5ustar flowerwheel00000000000000truffleHog-2.2.1/truffleHog/__init__.py0000644000076500000000000000000114005633171020442 0ustar flowerwheel00000000000000 truffleHog-2.2.1/truffleHog/truffleHog.py0000644000076500000000000004154614007162460021033 0ustar flowerwheel00000000000000#!/usr/bin/env python # -*- coding: utf-8 -*- from __future__ import absolute_import import shutil import sys import math import datetime import argparse import uuid import hashlib import tempfile import os import re import json import stat from git import Repo from git import NULL_TREE from truffleHogRegexes.regexChecks import regexes def main(): parser = argparse.ArgumentParser(description='Find secrets hidden in the depths of git.') parser.add_argument('--json', dest="output_json", action="store_true", help="Output in JSON") parser.add_argument("--regex", dest="do_regex", action="store_true", help="Enable high signal regex checks") parser.add_argument("--rules", dest="rules", help="Ignore default regexes and source from json file") parser.add_argument("--allow", dest="allow", help="Explicitly allow regexes from json list file") parser.add_argument("--entropy", dest="do_entropy", help="Enable entropy checks") parser.add_argument("--since_commit", dest="since_commit", help="Only scan from a given commit hash") parser.add_argument("--max_depth", dest="max_depth", help="The max commit depth to go back when searching for secrets") parser.add_argument("--branch", dest="branch", help="Name of the branch to be scanned") parser.add_argument('-i', '--include_paths', type=argparse.FileType('r'), metavar='INCLUDE_PATHS_FILE', help='File with regular expressions (one per line), at least one of which must match a Git ' 'object path in order for it to be scanned; lines starting with "#" are treated as ' 'comments and are ignored. If empty or not provided (default), all Git object paths are ' 'included unless otherwise excluded via the --exclude_paths option.') parser.add_argument('-x', '--exclude_paths', type=argparse.FileType('r'), metavar='EXCLUDE_PATHS_FILE', help='File with regular expressions (one per line), none of which may match a Git object path ' 'in order for it to be scanned; lines starting with "#" are treated as comments and are ' 'ignored. If empty or not provided (default), no Git object paths are excluded unless ' 'effectively excluded via the --include_paths option.') parser.add_argument("--repo_path", type=str, dest="repo_path", help="Path to the cloned repo. If provided, git_url will not be used") parser.add_argument("--cleanup", dest="cleanup", action="store_true", help="Clean up all temporary result files") parser.add_argument('git_url', type=str, help='URL for secret searching') parser.set_defaults(regex=False) parser.set_defaults(rules={}) parser.set_defaults(allow={}) parser.set_defaults(max_depth=1000000) parser.set_defaults(since_commit=None) parser.set_defaults(entropy=True) parser.set_defaults(branch=None) parser.set_defaults(repo_path=None) parser.set_defaults(cleanup=False) args = parser.parse_args() rules = {} if args.rules: try: with open(args.rules, "r") as ruleFile: rules = json.loads(ruleFile.read()) for rule in rules: rules[rule] = re.compile(rules[rule]) except (IOError, ValueError) as e: raise("Error reading rules file") for regex in dict(regexes): del regexes[regex] for regex in rules: regexes[regex] = rules[regex] allow = {} if args.allow: try: with open(args.allow, "r") as allowFile: allow = json.loads(allowFile.read()) for rule in allow: allow[rule] = read_pattern(allow[rule]) except (IOError, ValueError) as e: raise("Error reading allow file") do_entropy = str2bool(args.do_entropy) # read & compile path inclusion/exclusion patterns path_inclusions = [] path_exclusions = [] if args.include_paths: for pattern in set(l[:-1].lstrip() for l in args.include_paths): if pattern and not pattern.startswith('#'): path_inclusions.append(re.compile(pattern)) if args.exclude_paths: for pattern in set(l[:-1].lstrip() for l in args.exclude_paths): if pattern and not pattern.startswith('#'): path_exclusions.append(re.compile(pattern)) output = find_strings(args.git_url, args.since_commit, args.max_depth, args.output_json, args.do_regex, do_entropy, surpress_output=False, branch=args.branch, repo_path=args.repo_path, path_inclusions=path_inclusions, path_exclusions=path_exclusions, allow=allow) project_path = output["project_path"] if args.cleanup: clean_up(output) if output["foundIssues"]: sys.exit(1) else: sys.exit(0) def read_pattern(r): if r.startswith("regex:"): return re.compile(r[6:]) converted = re.escape(r) converted = re.sub(r"((\\*\r)?\\*\n|(\\+r)?\\+n)+", r"( |\\t|(\\r|\\n|\\\\+[rn])[-+]?)*", converted) return re.compile(converted) def str2bool(v): if v == None: return True if v.lower() in ('yes', 'true', 't', 'y', '1'): return True elif v.lower() in ('no', 'false', 'f', 'n', '0'): return False else: raise argparse.ArgumentTypeError('Boolean value expected.') BASE64_CHARS = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=" HEX_CHARS = "1234567890abcdefABCDEF" def del_rw(action, name, exc): os.chmod(name, stat.S_IWRITE) os.remove(name) def shannon_entropy(data, iterator): """ Borrowed from http://blog.dkbza.org/2007/05/scanning-data-for-entropy-anomalies.html """ if not data: return 0 entropy = 0 for x in iterator: p_x = float(data.count(x))/len(data) if p_x > 0: entropy += - p_x*math.log(p_x, 2) return entropy def get_strings_of_set(word, char_set, threshold=20): count = 0 letters = "" strings = [] for char in word: if char in char_set: letters += char count += 1 else: if count > threshold: strings.append(letters) letters = "" count = 0 if count > threshold: strings.append(letters) return strings class bcolors: HEADER = '\033[95m' OKBLUE = '\033[94m' OKGREEN = '\033[92m' WARNING = '\033[93m' FAIL = '\033[91m' ENDC = '\033[0m' BOLD = '\033[1m' UNDERLINE = '\033[4m' def clone_git_repo(git_url): project_path = tempfile.mkdtemp() Repo.clone_from(git_url, project_path) return project_path def print_results(printJson, issue): commit_time = issue['date'] branch_name = issue['branch'] prev_commit = issue['commit'] printableDiff = issue['printDiff'] commitHash = issue['commitHash'] reason = issue['reason'] path = issue['path'] if printJson: print(json.dumps(issue, sort_keys=True)) else: print("~~~~~~~~~~~~~~~~~~~~~") reason = "{}Reason: {}{}".format(bcolors.OKGREEN, reason, bcolors.ENDC) print(reason) dateStr = "{}Date: {}{}".format(bcolors.OKGREEN, commit_time, bcolors.ENDC) print(dateStr) hashStr = "{}Hash: {}{}".format(bcolors.OKGREEN, commitHash, bcolors.ENDC) print(hashStr) filePath = "{}Filepath: {}{}".format(bcolors.OKGREEN, path, bcolors.ENDC) print(filePath) if sys.version_info >= (3, 0): branchStr = "{}Branch: {}{}".format(bcolors.OKGREEN, branch_name, bcolors.ENDC) print(branchStr) commitStr = "{}Commit: {}{}".format(bcolors.OKGREEN, prev_commit, bcolors.ENDC) print(commitStr) print(printableDiff) else: branchStr = "{}Branch: {}{}".format(bcolors.OKGREEN, branch_name.encode('utf-8'), bcolors.ENDC) print(branchStr) commitStr = "{}Commit: {}{}".format(bcolors.OKGREEN, prev_commit.encode('utf-8'), bcolors.ENDC) print(commitStr) print(printableDiff.encode('utf-8')) print("~~~~~~~~~~~~~~~~~~~~~") def find_entropy(printableDiff, commit_time, branch_name, prev_commit, blob, commitHash): stringsFound = [] lines = printableDiff.split("\n") for line in lines: for word in line.split(): base64_strings = get_strings_of_set(word, BASE64_CHARS) hex_strings = get_strings_of_set(word, HEX_CHARS) for string in base64_strings: b64Entropy = shannon_entropy(string, BASE64_CHARS) if b64Entropy > 4.5: stringsFound.append(string) printableDiff = printableDiff.replace(string, bcolors.WARNING + string + bcolors.ENDC) for string in hex_strings: hexEntropy = shannon_entropy(string, HEX_CHARS) if hexEntropy > 3: stringsFound.append(string) printableDiff = printableDiff.replace(string, bcolors.WARNING + string + bcolors.ENDC) entropicDiff = None if len(stringsFound) > 0: entropicDiff = {} entropicDiff['date'] = commit_time entropicDiff['path'] = blob.b_path if blob.b_path else blob.a_path entropicDiff['branch'] = branch_name entropicDiff['commit'] = prev_commit.message entropicDiff['diff'] = blob.diff.decode('utf-8', errors='replace') entropicDiff['stringsFound'] = stringsFound entropicDiff['printDiff'] = printableDiff entropicDiff['commitHash'] = prev_commit.hexsha entropicDiff['reason'] = "High Entropy" return entropicDiff def regex_check(printableDiff, commit_time, branch_name, prev_commit, blob, commitHash, custom_regexes={}): if custom_regexes: secret_regexes = custom_regexes else: secret_regexes = regexes regex_matches = [] for key in secret_regexes: found_strings = secret_regexes[key].findall(printableDiff) for found_string in found_strings: found_diff = printableDiff.replace(printableDiff, bcolors.WARNING + found_string + bcolors.ENDC) if found_strings: foundRegex = {} foundRegex['date'] = commit_time foundRegex['path'] = blob.b_path if blob.b_path else blob.a_path foundRegex['branch'] = branch_name foundRegex['commit'] = prev_commit.message foundRegex['diff'] = blob.diff.decode('utf-8', errors='replace') foundRegex['stringsFound'] = found_strings foundRegex['printDiff'] = found_diff foundRegex['reason'] = key foundRegex['commitHash'] = prev_commit.hexsha regex_matches.append(foundRegex) return regex_matches def diff_worker(diff, curr_commit, prev_commit, branch_name, commitHash, custom_regexes, do_entropy, do_regex, printJson, surpress_output, path_inclusions, path_exclusions, allow): issues = [] for blob in diff: printableDiff = blob.diff.decode('utf-8', errors='replace') if printableDiff.startswith("Binary files"): continue if not path_included(blob, path_inclusions, path_exclusions): continue for key in allow: printableDiff = allow[key].sub('', printableDiff) commit_time = datetime.datetime.fromtimestamp(prev_commit.committed_date).strftime('%Y-%m-%d %H:%M:%S') foundIssues = [] if do_entropy: entropicDiff = find_entropy(printableDiff, commit_time, branch_name, prev_commit, blob, commitHash) if entropicDiff: foundIssues.append(entropicDiff) if do_regex: found_regexes = regex_check(printableDiff, commit_time, branch_name, prev_commit, blob, commitHash, custom_regexes) foundIssues += found_regexes if not surpress_output: for foundIssue in foundIssues: print_results(printJson, foundIssue) issues += foundIssues return issues def handle_results(output, output_dir, foundIssues): for foundIssue in foundIssues: result_path = os.path.join(output_dir, str(uuid.uuid4())) with open(result_path, "w+") as result_file: result_file.write(json.dumps(foundIssue)) output["foundIssues"].append(result_path) return output def path_included(blob, include_patterns=None, exclude_patterns=None): """Check if the diff blob object should included in analysis. If defined and non-empty, `include_patterns` has precedence over `exclude_patterns`, such that a blob that is not matched by any of the defined `include_patterns` will be excluded, even when it is not matched by any of the defined `exclude_patterns`. If either `include_patterns` or `exclude_patterns` are undefined or empty, they will have no effect, respectively. All blobs are included by this function when called with default arguments. :param blob: a Git diff blob object :param include_patterns: iterable of compiled regular expression objects; when non-empty, at least one pattern must match the blob object for it to be included; if empty or None, all blobs are included, unless excluded via `exclude_patterns` :param exclude_patterns: iterable of compiled regular expression objects; when non-empty, _none_ of the patterns may match the blob object for it to be included; if empty or None, no blobs are excluded if not otherwise excluded via `include_patterns` :return: False if the blob is _not_ matched by `include_patterns` (when provided) or if it is matched by `exclude_patterns` (when provided), otherwise returns True """ path = blob.b_path if blob.b_path else blob.a_path if include_patterns and not any(p.match(path) for p in include_patterns): return False if exclude_patterns and any(p.match(path) for p in exclude_patterns): return False return True def find_strings(git_url, since_commit=None, max_depth=1000000, printJson=False, do_regex=False, do_entropy=True, surpress_output=True, custom_regexes={}, branch=None, repo_path=None, path_inclusions=None, path_exclusions=None, allow={}): output = {"foundIssues": []} if repo_path: project_path = repo_path else: project_path = clone_git_repo(git_url) repo = Repo(project_path) already_searched = set() output_dir = tempfile.mkdtemp() if branch: branches = repo.remotes.origin.fetch(branch) else: branches = repo.remotes.origin.fetch() for remote_branch in branches: since_commit_reached = False branch_name = remote_branch.name prev_commit = None for curr_commit in repo.iter_commits(branch_name, max_count=max_depth): commitHash = curr_commit.hexsha if commitHash == since_commit: since_commit_reached = True break # if not prev_commit, then curr_commit is the newest commit. And we have nothing to diff with. # But we will diff the first commit with NULL_TREE here to check the oldest code. # In this way, no commit will be missed. diff_hash = hashlib.md5((str(prev_commit) + str(curr_commit)).encode('utf-8')).digest() if not prev_commit: prev_commit = curr_commit continue elif diff_hash in already_searched: prev_commit = curr_commit continue else: diff = prev_commit.diff(curr_commit, create_patch=True) # avoid searching the same diffs already_searched.add(diff_hash) foundIssues = diff_worker(diff, curr_commit, prev_commit, branch_name, commitHash, custom_regexes, do_entropy, do_regex, printJson, surpress_output, path_inclusions, path_exclusions, allow) output = handle_results(output, output_dir, foundIssues) prev_commit = curr_commit # Check if since_commit was used to check which diff should be grabbed if since_commit_reached: # Handle when there's no prev_commit (used since_commit on the most recent commit) if prev_commit is None: continue diff = prev_commit.diff(curr_commit, create_patch=True) else: diff = curr_commit.diff(NULL_TREE, create_patch=True) foundIssues = diff_worker(diff, curr_commit, prev_commit, branch_name, commitHash, custom_regexes, do_entropy, do_regex, printJson, surpress_output, path_inclusions, path_exclusions, allow) output = handle_results(output, output_dir, foundIssues) output["project_path"] = project_path output["clone_uri"] = git_url output["issues_path"] = output_dir if not repo_path: shutil.rmtree(project_path, onerror=del_rw) return output def clean_up(output): issues_path = output.get("issues_path", None) if issues_path and os.path.isdir(issues_path): shutil.rmtree(output["issues_path"]) if __name__ == "__main__": main() truffleHog-2.2.1/truffleHog.egg-info/0000755000076500000000000000000014007162567020044 5ustar flowerwheel00000000000000truffleHog-2.2.1/truffleHog.egg-info/PKG-INFO0000644000076500000000000000046414007162567021145 0ustar flowerwheel00000000000000Metadata-Version: 1.0 Name: truffleHog Version: 2.2.1 Summary: Searches through git repositories for high entropy strings, digging deep into commit history. Home-page: https://github.com/dxa4481/truffleHog Author: Dylan Ayrey Author-email: dxa4481@rit.edu License: GNU Description: UNKNOWN Platform: UNKNOWN truffleHog-2.2.1/truffleHog.egg-info/SOURCES.txt0000644000076500000000000000043214007162567021727 0ustar flowerwheel00000000000000README.md setup.cfg setup.py truffleHog/__init__.py truffleHog/truffleHog.py truffleHog.egg-info/PKG-INFO truffleHog.egg-info/SOURCES.txt truffleHog.egg-info/dependency_links.txt truffleHog.egg-info/entry_points.txt truffleHog.egg-info/requires.txt truffleHog.egg-info/top_level.txttruffleHog-2.2.1/truffleHog.egg-info/dependency_links.txt0000644000076500000000000000000114007162567024112 0ustar flowerwheel00000000000000 truffleHog-2.2.1/truffleHog.egg-info/entry_points.txt0000644000076500000000000000007314007162567023342 0ustar flowerwheel00000000000000[console_scripts] trufflehog = truffleHog.truffleHog:main truffleHog-2.2.1/truffleHog.egg-info/requires.txt0000644000076500000000000000005214007162567022441 0ustar flowerwheel00000000000000GitPython==3.0.6 truffleHogRegexes==0.0.7 truffleHog-2.2.1/truffleHog.egg-info/top_level.txt0000644000076500000000000000001314007162567022570 0ustar flowerwheel00000000000000truffleHog