](https://www.blackhillsinfosec.com/)
[
](https://volkis.com.au)
[
](https://qomplx.com/blog/cyber/)
[
](https://ledgerops.com)
[
](https://www.guidepointsecurity.com/)
[
](https://lostrabbitlabs.com/)
## Table of Contents
- [WitnessMe](#witnessme)
* [Motivation](#motivation)
* [Official Discord Channel](#official-discord-channel)
* [Installation](#Installation)
+ [Docker](#docker)
+ [Python Package](#python-package)
+ [Development Install](#development-install)
* [Quick starts](#quick-starts)
+ [Finding F5 Load Balancers Vulnerable to CVE-2020-5902](#finding-f5-load-balancers-vulnerable-to-cve-2020-5902)
+ [Scraping Javascript Heavy Webpages](#scraping-javascript-heavy-webpages)
* [RESTful API](#restful-api)
* [Deploying to the Cloud](#deploying-to-the-cloud-)
+ [GCP Cloud Run](#gcp-cloud-run)
+ [AWS ElasticBeanstalk](#aws-elasticbeanstalk)
* [Usage and Examples](#usage-and-examples)
+ [Modes of Operation](#modes-of-operation)
* [Screenshot Mode](#screenshot-mode)
* [Grab Mode](#grab-mode)
+ [Interacting with the Scan Database](#interacting-with-the-scan-database)
+ [Generating Reports](#generating-reports)
+ [Proxying](#Proxying)
+ [Previewing Screenshots Directly in the Terminal](#preview-screenshots-directly-in-the-terminal)
* [Creating Signatures](#call-for-signatures)
## Motivation
Are there are a bunch of other tools that do this? Absolutely. See the following projects for alternatives (I'm sure there are more, these are the ones I've personally tried):
- [Eyewitness](https://github.com/FortyNorthSecurity/EyeWitness)
- [GoWitness](https://github.com/sensepost/gowitness)
- [Aquatone](https://github.com/michenriksen/aquatone)
The reason why I wrote WitnessMe was that none of these projects had all of the features I wanted/needed in order for them to work well within my workflow. Additionally, some of them are prone to a decent amount of installation/dependency hell.
Here are some of the main features that make WitnessMe "stand out":
- Written in Python 3.7+
- Ability to parse extremely large Nessus and NMap XML files
- Docker compatible
- No installation/dependency hell
- Full test suite! Everything is less prone to bugs
- CSV & HTML reporting
- HTTP Proxy Support
- Provides a RESTful API! Scan stuff remotely!
- CLI interface to view and search scan results without having to view the reports.
- Signature scanning (Signatures use YAML files)
- Preview screenshots directly in the terminal (On MacOSX/ITerm2 and some Nix terminals)
- Extensibly written, allowing you to add functionality that can take advantage of headless chromium.
- Built to be deployed to the Clouds (e.g. GCP Cloud Run , AWS ElasticBeanstalk etc...)
## Official Discord Channel
Come hang out on Discord!
[](https://discord.gg/AKrqt6J)
## Installation
### Docker
Running WitnessMe from a Docker container is fully supported and is the easiest/recommended way of using the tool.
**Note: it is highly recommended to give the Docker container at least 4GB of RAM during large scans as Chromium can be a resource hog. If you keep running into "Page Crash" errors, it's because your container does not have enough memory. On Mac/Windows you can change this by clicking the Docker Task Bar Icon -> Preferences -> Resources. For Linux, refer to Docker's documentation**
Pull the image from Docker Hub:
```console
docker pull byt3bl33d3r/witnessme
```
You can then spin up a docker container, run it like the main `witnessme` script and pass it the same arguments:
```console
docker run --rm -ti $IMAGE_ID screenshot https://google.com 192.168.0.1/24
```
Alternatively, you can drop into a shell within the container and run the tools that way. This also allows you to execute the `wmdb` and `wmapi` scripts.
```console
docker run --rm -ti --entrypoint=/bin/sh $IMAGE_ID
```
### Python Package
WitnessMe is also available as a Python package (Python 3.7 or above is required). If you do install it this way it is extremely recommended to use [pipx](https://github.com/pipxproject/pipx) as it takes care of installing everything in isolated environments for you in a seamless manner.
Run the following commands:
```console
python3 -m pip install --user pipx
pipx install witnessme
```
All of the WitnessMe scripts should now be in your PATH and ready to go.
### Development Install
You really should only install WitnessMe this way if you intend to hack on the source code. You're going to Python 3.7+ and [Poetry](https://python-poetry.org/): please refer to the Poetry installation documentation in order to install it.
```console
git clone https://github.com/byt3bl33d3r/WitnessMe && cd WitnessMe
poetry install
```
## Quick Starts
### Finding F5 Load Balancers Vulnerable to CVE-2020-5902
Install WitnessMe using Docker:
```console
docker pull byt3bl33d3r/witnessme
```
Get the `$IMAGE_ID` from the `docker images` command output, then run the following command to drop into a shell inside the container. Additionally, specify the `-v` flag to mount the current directory inside the container at the path `/transfer` in order to copy the scan results back to your host machine (if so desired):
```console
docker run -it --entrypoint=/bin/sh -v $(pwd):/transfer $IMAGE_ID
```
Scan your network using WitnessMe, it can accept multiple .Nessus files, Nmap XMLs, IP ranges/CIDRs. Example:
```console
witnessme screenshot 10.0.1.0/24 192.168.0.1-20 ~/my_nessus_scan.nessus ~/my_nmap_scan.xml
```
After the scan is finished, a folder will have been created in the current directory with the results. Access the results using the `wmdb` command line utility:
```console
wmdb scan_2020_$TIME/
```
To quickly identify F5 load balancers, first perform a signature scan using the `scan` command. Then search for "BIG-IP" or "F5" using the `servers` command (this will search for the "BIG-IP" and "F5" string in the signature name, page title and server header):
Additionally, you can generate an HTML or CSV report using the following commands:
```console
WMDB ≫ generate_report html
```
```console
WMDB ≫ generate_report csv
```
You can then copy the entire scan folder which will contain all of the reports and results to your host machine by copying it to the `/transfer` folder.
### Scraping Javascript Heavy Webpages
As of v1.5.0, WitnessMe has a `grab` command which allows you to quickly scrape Javascript heavy webpages by rendering the page first with Headless Chromium and then parsing the resulting HTML using the specified XPath (see [here](https://devhints.io/xpath) for an XPath cheatsheet).
Below are a few examples to get your started.
This grabs a list of all advertised domains on the `144.161.160.0/23` subnet from [Hurricane Electric's BGP Toolkit](https://bgp.he.net/):
```console
witnessme -d grab -x '//div[@id="dns"]/table//tr/td[2]/a/text()' https://bgp.he.net/net/144.161.160.0/23#_dns
```
## RESTful API
As of version 1.0, WitnessMe has a RESTful API which allows you to interact with the tool remotely.
**Note: Currently, the API does not implement any authentication mechanisms. Make sure to allow/deny access at the transport level**
To start the RESTful API for testing/development purposes run :
```console
wmapi
```
The API documentation will then be available at http://127.0.0.1:8000/docs
[Uvicorn](https://www.uvicorn.org/) should be used to enable SSL and run the API in production. See [this dockerfile](https://github.com/byt3bl33d3r/WitnessMe/blob/master/dockerfiles/Dockerfile.selfhosted) for an example.
## Deploying to the Cloud (™)
Since WitnessMe has a RESTful API now, you can deploy it to the magical cloud and perform scanning from there. This would have a number of benefits, including giving you a fresh external IP on every scan (More OPSEC safe when assessing attack surface on Red Teams).
There are a number of ways of doing this, you can obviously do it the traditional way (e.g. spin up a machine, install docker etc..).
Recently cloud service providers started offering ways of running Docker containers directly in a fully managed environment. Think of it as serverless functions (e.g. AWS Lambdas) only with Docker containers.
This would technically allow you to really quickly deploy and run WitnessMe (or really anything in a Docker container) without having to worry about underlying infrastructure and removes a lot of the security concerns that come with that.
Below are some of the ones I've tried along with the steps necessary to get it going and any issues I encountered.
### GCP Cloud Run
**Unfortunately, it seems like Cloud Run doesn't allow outbound internet access to containers, if anybody knows of a way to get around this please get in touch**
Cloud Run is by far the easiest of these services to work with.
This repository includes the `cloudbuild.yaml` file necessary to get this setup and running.
From the repositories root folder (after you authenticated and setup a project), these two commands will automatically build the Docker image, publish it to the Gcloud Container Registry and deploy a working container to Cloud Run:
```bash
gcloud builds submit --config cloudbuild.yaml
gcloud run deploy --image gcr.io/$PROJECT_ID/witnessme --platform managed
```
The output will give you a HTTPS url to invoke the WitnessMe RESTful API from :)
When you're done:
```bash
gcloud run services delete witnessme
gcloud container images delete gcr.io/$PROJECT_ID/witnessme
```
### AWS ElasticBeanstalk
TO DO
## Usage
There are 3 main utilities:
- `witnessme`: is the main CLI interface.
- `wmdb`: allows you to browse the database (created on each scan) to view results and generate reports.
- `wmapi`: provides a RESTful API to schedule, start, stop and monitor scans.
### Modes of Operations
As of v1.5.0 there are two main modes (commands) that the `witnessme` utility Supports:
- The `screenshot` command, you guessed it, screenshots webpages. This is the main functionality.
- The `grab` command allows you to scrape pages and quickly grab server headers.
```
usage: witnessme [-h] [--threads THREADS] [--timeout TIMEOUT] [-d] [-v] {screenshot,grab} ...
WitnessMe!
positional arguments:
{screenshot,grab}
optional arguments:
-h, --help show this help message and exit
--threads THREADS Number of concurrent browser tab(s) to open
[WARNING: This can cause huge RAM consumption if set to high values] (default: 15)
--timeout TIMEOUT Timeout for each connection attempt in seconds (default: 15)
-d, --debug Enable debug output (default: False)
-v, --version show program's version number and exit
```
#### Screenshot Mode
```console
$ witnessme screenshot --help
usage: witnessme screenshot [-h] [-p PORTS [PORTS ...]] target [target ...]
positional arguments:
target The target IP(s), range(s), CIDR(s) or hostname(s), NMap XML file(s), .Nessus file(s)
optional arguments:
-h, --help show this help message and exit
-p PORTS [PORTS ...], --ports PORTS [PORTS ...]
Ports to scan if IP Range/CIDR is provided
```
Can accept a mix of .Nessus file(s), Nmap XML file(s), files containing URLs and/or IPs, IP addresses/ranges/CIDRs and URLs or alternatively read from stdin.
*Note: WitnessMe detects .Nessus and NMap files by their extension so make sure Nessus files have a `.nessus` extension and NMap scans have a `.xml` extension*
Long story short, should be able to handle anything you throw at it:
```console
witnessme screenshot 192.168.1.0/24 192.168.1.10-20 https://bing.com ~/my_nessus_scan.nessus ~/my_nmap_scan.xml ~/myfilewithURLSandIPs
```
```console
$ cat my_domain_list.txt | witnessme screenshot -
```
If an IP address/range/CIDR is specified as a target, WitnessMe will attempt to screenshot HTTP & HTTPS pages on ports 80, 8080, 443, 8443 by default. This is customizable with the `--port` argument.
Once a scan is completed, a folder with all the screenshots and a database will be in the current directory, point `wmdb` to the folder in order to see the results.
```console
wmdb scan_2019_11_05_021237/
```
#### Grab Mode
```console
$ witnessme grab --help
usage: witnessme grab [-h] [-x XPATH | -l] target [target ...]
positional arguments:
target The target IP(s), range(s), CIDR(s) or hostname(s), NMap XML file(s), .Nessus file(s)
optional arguments:
-h, --help show this help message and exit
-x XPATH, --xpath XPATH
XPath to use
-l, --links Get all links
```
The `grab` subcommand allows you to render Javascript heavy webpages and scrape their content using XPaths. See this [section](#scraping-javascript-heavy-webpages) for some examples.
### Interacting with the Scan Database
Once a scan is completed (using the `screenshot` mode), a folder with all the screenshots and a database will be in the current directory, point `wmdb` to the folder in order to see the results.
```console
wmdb scan_2019_11_05_021237/
```
This will drop you into the WMDB CLI menu.
Pressing tab will show you the available commands and a help menu:
ADManager Plus Authentication
\n\rManageEngine\n\r
\r\r\r
"""
fake_service = (0, "http://fakeurl:port/", service_html)
matches, _ = await sig_eng_l.find_match(fake_service)
print(matches)
assert len(matches) == 1
assert matches[0]["name"] == "ADManager"
WitnessMe-master/tests/test_target_parsing.py 0000664 0000000 0000000 00000002426 13745620352 0022064 0 ustar 00root root 0000000 0000000 import pytest
import io
from witnessme.parsers import AutomaticTargetGenerator
"""
TO DO: add tests for parsing .nessus files
"""
def test_ip_network_target_parsing():
with AutomaticTargetGenerator(
["192.168.0.1/28", "172.16.0.1-10", "http-simple-new://google.com:443/"]
) as gen:
urls = list(gen)
print(urls)
assert len(urls) == 209
assert (
all(
map(lambda u: u.startswith("http://") or u.startswith("https://"), urls)
)
== True
)
assert "http://google.com:443/" in urls
def test_stdin_parsing(monkeypatch):
monkeypatch.setattr(
"sys.stdin",
io.StringIO("192.168.0.1/28\n172.16.0.1-10\nhttp-simple-new://google.com:443/"),
)
with AutomaticTargetGenerator(["-"]) as gen:
urls = list(gen)
print(urls)
assert len(urls) == 209
assert (
all(
map(lambda u: u.startswith("http://") or u.startswith("https://"), urls)
)
== True
)
assert "http://google.com:443/" in urls
def test_nmap_xml_target_parsing():
with AutomaticTargetGenerator(["tests/nmap_example.xml"]) as gen:
urls = list(gen)
print(urls)
assert len(urls) == 13
WitnessMe-master/witnessme/ 0000775 0000000 0000000 00000000000 13745620352 0016312 5 ustar 00root root 0000000 0000000 WitnessMe-master/witnessme/__init__.py 0000664 0000000 0000000 00000000206 13745620352 0020421 0 ustar 00root root 0000000 0000000 import logging
__version__ = "1.5.0dev"
__codename__ = "ToeCutter"
logging.getLogger("witnessme").addHandler(logging.NullHandler())
WitnessMe-master/witnessme/api/ 0000775 0000000 0000000 00000000000 13745620352 0017063 5 ustar 00root root 0000000 0000000 WitnessMe-master/witnessme/api/__init__.py 0000664 0000000 0000000 00000000000 13745620352 0021162 0 ustar 00root root 0000000 0000000 WitnessMe-master/witnessme/api/models.py 0000664 0000000 0000000 00000001256 13745620352 0020724 0 ustar 00root root 0000000 0000000 import uuid
from witnessme.commands.screenshot import ScanState
from typing import Optional, List, Union
from pydantic import BaseModel, AnyUrl
class BrowserTasks(BaseModel):
inputs: int
execs: int
pending: int
class Config:
orm_mode = True
class ScanConfig(BaseModel):
target: List[Union[AnyUrl, str]]
ports: Optional[List[int]] = [80, 8080, 443, 8443]
threads: Optional[int] = 25
timeout: Optional[int] = 35
class Scan(BaseModel):
id: uuid.UUID
target: List[str]
ports: List[int]
threads: int
timeout: int
stats: BrowserTasks
state: ScanState
report_folder: str
class Config:
orm_mode = True
WitnessMe-master/witnessme/api/routers/ 0000775 0000000 0000000 00000000000 13745620352 0020566 5 ustar 00root root 0000000 0000000 WitnessMe-master/witnessme/api/routers/__init__.py 0000664 0000000 0000000 00000000000 13745620352 0022665 0 ustar 00root root 0000000 0000000 WitnessMe-master/witnessme/api/routers/scan.py 0000664 0000000 0000000 00000005203 13745620352 0022064 0 ustar 00root root 0000000 0000000 import asyncio
import logging
import uuid
import functools
from witnessme.commands.screenshot import ScreenShot, ScanState
from witnessme.api.models import Scan, ScanConfig
from witnessme.utils import patch_pyppeteer, zip_scan_folder
from fastapi import APIRouter, Request, Response, status, UploadFile, File
from fastapi.responses import JSONResponse, FileResponse
log = logging.getLogger("witnessme.api")
patch_pyppeteer()
router = APIRouter()
@router.post("/")
async def create_scan(scan_config: ScanConfig, request: Request):
conf = scan_config.dict()
scan = ScreenShot(**conf)
request.app.state.SCANS.add(scan)
return Scan.from_orm(scan)
@router.get("/")
async def get_scans(request: Request):
return {scan.id: Scan.from_orm(scan) for scan in request.app.state.SCANS.scans}
@router.get("/{scan_id}")
async def get_scan_by_id(scan_id: uuid.UUID, request: Request):
scan = request.app.state.SCANS.get(scan_id)
return Scan.from_orm(scan)
@router.get("/{scan_id}/start")
async def start_scan(scan_id: uuid.UUID, request: Request):
scan = request.app.state.SCANS.get(scan_id)
if scan.state in [ScanState.CONFIGURED, ScanState.STOPPED]:
asyncio.create_task(scan.start())
return Response(status_code=status.HTTP_200_OK)
return JSONResponse(
{"error": "finished scans cannot be started"},
status_code=status.HTTP_400_BAD_REQUEST,
)
@router.get("/{scan_id}/stop")
async def stop_scan(scan_id: uuid.UUID, request: Request):
scan = request.app.state.SCANS.get(scan_id)
await scan.stop()
return Response(status_code=status.HTTP_200_OK)
@router.get("/{scan_id}/result")
async def get_scan_result(scan_id: uuid.UUID, request: Request):
scan = request.app.state.SCANS.get(scan_id)
if not scan.stats.done:
return JSONResponse(
{"error": "scan has not finished"}, status_code=status.HTTP_400_BAD_REQUEST
)
loop = asyncio.get_running_loop()
zip_file = await loop.run_in_executor(
None, functools.partial(zip_scan_folder, scan.report_folder)
)
return FileResponse(zip_file, media_type="application/zip")
@router.post("/{scan_id}/upload/{file_id}")
async def upload_scan_target_file(
scan_id: uuid.UUID, file_id: str, request: Request, file: UploadFile = File(...)
):
scan = request.app.state.SCANS.get(scan_id)
for i, t in enumerate(scan.target):
if t.startswith("file:"):
_, f_name, f_id = t.split(":")
if f_id == file_id:
scan.target[i] = file.path
return Response(status_code=status.HTTP_200_OK)
return Response(status_code=status.HTTP_400_BAD_REQUEST)
WitnessMe-master/witnessme/commands/ 0000775 0000000 0000000 00000000000 13745620352 0020113 5 ustar 00root root 0000000 0000000 WitnessMe-master/witnessme/commands/__init__.py 0000664 0000000 0000000 00000000072 13745620352 0022223 0 ustar 00root root 0000000 0000000 from .grab import Grab
from .screenshot import ScreenShot
WitnessMe-master/witnessme/commands/grab.py 0000664 0000000 0000000 00000005115 13745620352 0021402 0 ustar 00root root 0000000 0000000 import logging
import asyncio
import lxml.html
from urllib.parse import urlparse
from typing import List
from witnessme.utils import beautify_json
from witnessme.headlessbrowser import HeadlessChromium
log = logging.getLogger("witnessme.grab")
class Grab:
def __init__(
self,
target: List[str],
threads: int = 25,
timeout: int = 35,
xpath=None,
links=False,
) -> None:
self.target = target
self.threads = threads
self.timeout = timeout
self.xpath = xpath
self.links = links
async def grab_page(self, browser, url, page):
url = urlparse(url)
"""
await page.goto(url.geturl())
response = await page.waitForNavigation(
options={"waitUntil": "domcontentloaded"}
)
"""
_, response = await asyncio.gather(
*[
page.goto(url.geturl()),
page.waitForNavigation(options={"waitUntil": "networkidle0"}),
]
)
if not url.port:
url = url._replace(netloc=f"{url.hostname}:{response.remotePort}")
return {
"url": url.geturl(),
"ip": response.remoteIPAddress,
"port": url.port,
"scheme": url.scheme,
"title": await page.title(), # await page.evaluate('document.title')
"server": response.headers.get("server"),
"headers": response.headers,
"body": await response.text(),
}
async def parse_html(self, browser, results):
html = lxml.html.fromstring(results["body"])
results["body"] = f"{len(results['body'])} bytes"
log.info(f"Grabbed page:" + beautify_json(results))
if self.xpath:
for match in html.xpath(self.xpath):
if isinstance(match, str):
print(match.strip(" ").strip())
else:
print(match)
elif self.links:
for match in html.xpath("//@href"):
if len(match) > 0 and match != "#":
print(match.strip(" ").strip())
async def start(self):
if self.xpath:
try:
lxml.etree.XPath(self.xpath)
except lxml.etree.XPathSyntaxError:
log.error("Bad xpath! Exiting...")
return
async with HeadlessChromium(
threads=self.threads,
timeout=self.timeout,
on_new_tab=self.grab_page,
on_finished=self.parse_html,
) as browser:
await browser.run(self.target)
WitnessMe-master/witnessme/commands/screenshot.py 0000664 0000000 0000000 00000011070 13745620352 0022641 0 ustar 00root root 0000000 0000000 import logging
import asyncio
import pathlib
import uuid
import shutil
import os
from enum import Enum
from typing import List
from datetime import datetime
from urllib.parse import urlparse
from witnessme.utils import is_ipaddress, agethostbyaddr
from witnessme.headlessbrowser import HeadlessChromium
from witnessme.database import ScanDatabase
log = logging.getLogger("witnessme.screenshot")
class NoResponseReceived(Exception):
pass
class ScanState(str, Enum):
STARTED = "started"
STOPPED = "stopped"
# CANCELLED = 'cancelled'
DONE = "done"
# PAUSED = 'paused'
CONFIGURED = "configured"
class ScreenShot:
def __init__(
self,
target: List[str],
ports: List[int] = [80, 8080, 443, 8443],
threads: int = 25,
timeout: int = 35,
) -> None:
self.id = uuid.uuid4()
self.target = target
self.ports = ports
self.threads = threads
self.timeout = timeout
self.state = ScanState.CONFIGURED
self._browser = HeadlessChromium(
threads=self.threads,
timeout=self.timeout,
on_new_tab=self.screenshot,
on_finished=self.add_to_database,
)
self.stats = self._browser.stats
self._scan_task = None
self._report_folder = None
@property
def report_folder(self):
if self._report_folder:
return self._report_folder
time = datetime.now().strftime("%Y_%m_%d_%H%M%S")
self._report_folder = f"scan_{time}"
return self._report_folder
async def screenshot(self, browser, url, page):
"""
The page.goto() options might need to be tweaked depending on testing in real environments.
https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagewaitfornavigationoptions
load - consider navigation to be finished when the load event is fired.
domcontentloaded - consider navigation to be finished when the DOMContentLoaded event is fired.
networkidle0 - consider navigation to be finished when there are no more than 0 network connections for at least 500 ms.
networkidle2 - consider navigation to be finished when there are no more than 2 network connections for at least 500 ms.
"""
url = urlparse(url)
dns_resolution_task = asyncio.create_task(agethostbyaddr(url.hostname))
response = await page.goto(url.geturl(), options={"waitUntil": "networkidle0"})
if not response:
raise NoResponseReceived(f"Empty response was received")
if not url.port:
url = url._replace(netloc=f"{url.hostname}:{response.remotePort}")
screenshot = f"{url.scheme}_{url.hostname}_{url.port}.png"
screenshot_path = str(
pathlib.Path(f"./{self.report_folder}/{screenshot}").absolute()
)
await page.screenshot({"path": screenshot_path, "fullPage": True})
hostname = await dns_resolution_task
return {
"ip": response.remoteIPAddress,
"hostname": hostname,
"url": url.geturl(),
"screenshot": screenshot,
"port": url.port,
"scheme": url.scheme,
"title": await page.title(), # await page.evaluate('document.title')
"server": response.headers.get("server"),
"headers": response.headers,
"body": await response.text(),
}
async def add_to_database(self, browser, results):
log.info(f"Took screenshot of {results['url']}")
async with ScanDatabase(self.report_folder) as db:
await db.add_host_and_service(**results)
async def setup_and_run(self):
try:
os.mkdir(self.report_folder)
except FileExistsError:
if self.state == ScanState.STOPPED:
shutil.rmtree(self.report_folder, ignore_errors=True)
os.mkdir(self.report_folder)
await ScanDatabase.create_db_and_schema(self.report_folder)
await self._browser.run(self.target, self.ports)
async def start(self):
self.state = ScanState.STARTED
log.info(f"Starting scan {self.id}")
self._scan_task = asyncio.create_task(self.setup_and_run())
await self._scan_task
log.info(f"Saved scan to {self.report_folder}/")
if not self._scan_task.cancelled():
self.state = ScanState.DONE
async def stop(self):
if self._scan_task:
log.info(f"Stopping scan {self.id}")
self._scan_task.cancel()
self.state = ScanState.STOPPED
WitnessMe-master/witnessme/console/ 0000775 0000000 0000000 00000000000 13745620352 0017754 5 ustar 00root root 0000000 0000000 WitnessMe-master/witnessme/console/__init__.py 0000664 0000000 0000000 00000000000 13745620352 0022053 0 ustar 00root root 0000000 0000000 WitnessMe-master/witnessme/console/witnessme.py 0000775 0000000 0000000 00000006052 13745620352 0022352 0 ustar 00root root 0000000 0000000 #! /usr/bin/env python3
import argparse
import logging
from witnessme import __version__, __codename__
from witnessme.utils import patch_pyppeteer, start_event_loop, WitnessMeArgFormatter
from witnessme.commands import ScreenShot, Grab
@start_event_loop
async def screenshot(args):
cmd = ScreenShot(
target=args.target,
ports=args.ports,
timeout=args.timeout,
threads=args.threads,
)
await cmd.start()
@start_event_loop
async def grab(args):
cmd = Grab(
target=args.target,
timeout=args.timeout,
threads=args.threads,
xpath=args.xpath,
links=args.links,
)
await cmd.start()
def run():
# logging.Formatter("%(asctime)s [%(levelname)s] - %(filename)s: %(funcName)s - %(message)s")
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("[%(name)s] %(levelname)s - %(message)s"))
log = logging.getLogger("witnessme")
log.setLevel(logging.INFO)
log.addHandler(handler)
parser = argparse.ArgumentParser(
description="WitnessMe!", formatter_class=WitnessMeArgFormatter
)
parser.add_argument(
"--threads",
default=15,
type=int,
help="Number of concurrent browser tab(s) to open\n[WARNING: This can cause huge RAM consumption if set to high values]",
)
parser.add_argument(
"--timeout",
default=15,
type=int,
help="Timeout for each connection attempt in seconds",
)
parser.add_argument(
"-d", "--debug", default=False, action="store_true", help="Enable debug output"
)
parser.add_argument(
"-v", "--version", action="version", version=f"{__version__} - '{__codename__}'"
)
subparsers = parser.add_subparsers(dest="command")
screenshot_parser = subparsers.add_parser("screenshot")
screenshot_parser.add_argument(
"target",
nargs="+",
type=str,
help="The target IP(s), range(s), CIDR(s) or hostname(s), NMap XML file(s), .Nessus file(s)",
)
screenshot_parser.add_argument(
"-p",
"--ports",
nargs="+",
default=[80, 8080, 443, 8443],
help="Ports to scan if IP Range/CIDR is provided",
)
screenshot_parser.set_defaults(func=screenshot)
grab_parser = subparsers.add_parser("grab")
grab_parser.add_argument(
"target",
nargs="+",
type=str,
help="The target IP(s), range(s), CIDR(s) or hostname(s), NMap XML file(s), .Nessus file(s)",
)
exclusive_group = grab_parser.add_mutually_exclusive_group()
exclusive_group.add_argument("-x", "--xpath", type=str, help="XPath to use")
exclusive_group.add_argument(
"-l", "--links", action="store_true", help="Get all links"
)
grab_parser.set_defaults(func=grab)
args = parser.parse_args()
if not args.command:
parser.print_help()
parser.exit(1)
if args.debug:
log.setLevel(logging.DEBUG)
patch_pyppeteer()
log.debug(vars(args))
args.func(args)
if __name__ == "__main__":
run()
WitnessMe-master/witnessme/console/wmapi.py 0000664 0000000 0000000 00000004123 13745620352 0021443 0 ustar 00root root 0000000 0000000 import logging
import uvicorn
import argparse
from witnessme.api.routers import scan
from argparse import ArgumentDefaultsHelpFormatter
from fastapi import FastAPI, Request, status
from fastapi.encoders import jsonable_encoder
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
class ScanNotFoundError(Exception):
pass
class ActiveScans:
def __init__(self):
self.scans = []
def add(self, scan):
self.scans.append(scan)
def get(self, scan_id):
try:
return list(filter(lambda s: s.id == scan_id, self.scans))[0]
except IndexError:
raise ScanNotFoundError
app = FastAPI(title="WitnessMe API")
app.state.SCANS = ActiveScans()
app.include_router(scan.router, prefix="/screenshot", tags=["scan"])
@app.exception_handler(RequestValidationError)
async def validation_exception_handler(request: Request, exc: RequestValidationError):
return JSONResponse(
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
content=jsonable_encoder({"detail": exc.errors(), "body": exc.body}),
)
@app.exception_handler(ScanNotFoundError)
async def scan_not_found_exception_handler(request: Request, exc: ScanNotFoundError):
return JSONResponse(
status_code=status.HTTP_404_NOT_FOUND,
content={"error": "specified scan id does not exist"},
)
def run():
handler = logging.StreamHandler()
handler.setFormatter(
logging.Formatter(
"%(asctime)s [%(levelname)s] %(name)s - %(filename)s: %(funcName)s - %(message)s"
)
)
log = logging.getLogger("witnessme")
log.setLevel(logging.DEBUG)
log.addHandler(handler)
parser = argparse.ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter)
parser.add_argument(
"host", type=str, nargs="?", help="IP to bind to", default="127.0.0.1"
)
parser.add_argument(
"port", type=int, nargs="?", help="port to bind to", default=8000
)
args = parser.parse_args()
uvicorn.run(app, host=args.host, port=args.port)
if __name__ == "__main__":
run()
WitnessMe-master/witnessme/console/wmdb.py 0000775 0000000 0000000 00000026460 13745620352 0021272 0 ustar 00root root 0000000 0000000 #! /usr/bin/env python3
import asyncio
import argparse
import shlex
import logging
import sys
import pathlib
import aiosqlite
import webbrowser
from imgcat import imgcat
from time import time, gmtime, strftime
from argparse import ArgumentDefaultsHelpFormatter
from terminaltables import AsciiTable
from witnessme.database import ScanDatabase
from witnessme.signatures import Signatures
from witnessme.reporting import generate_html_report, generate_csv_report
from prompt_toolkit import PromptSession
from prompt_toolkit.completion import Completer, Completion
# from prompt_toolkit.patch_stdout import patch_stdout
# from prompt_toolkit.history import InMemoryHistory
from prompt_toolkit.auto_suggest import AutoSuggestFromHistory
from prompt_toolkit.formatted_text import HTML
# from prompt_toolkit.application import run_in_terminal
from prompt_toolkit.styles import Style
# from prompt_toolkit.document import Document
# logging.Formatter("%(asctime)s [%(levelname)s] - %(filename)s: %(funcName)s - %(message)s")
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("[%(name)s] %(levelname)s - %(message)s"))
log = logging.getLogger("witnessme")
log.setLevel(logging.DEBUG)
log.addHandler(handler)
class WMCompleter(Completer):
def __init__(self, cli_menu):
self.cli_menu = cli_menu
def get_completions(self, document, complete_event):
word_before_cursor = document.get_word_before_cursor()
try:
list(map(lambda s: s.lower(), shlex.split(document.current_line)))
except ValueError:
pass
else:
for cmd in [
"exit",
"show",
"open",
"hosts",
"servers",
"scan",
"generate_report",
"open_html_report",
]:
if cmd.startswith(word_before_cursor):
yield Completion(
cmd,
-len(word_before_cursor),
display_meta=getattr(self.cli_menu, cmd).__doc__.strip(),
)
class WMDBShell:
def __init__(self, scan_folder_path):
self.scan_folder_path = scan_folder_path
self.db_path = scan_folder_path / "witnessme.db"
self.completer = WMCompleter(self)
self.signatures = Signatures()
self.prompt_session = PromptSession(
HTML("WMDB ≫ "),
# bottom_toolbar=functools.partial(bottom_toolbar, ts=self.teamservers),
completer=self.completer,
complete_in_thread=True,
complete_while_typing=True,
auto_suggest=AutoSuggestFromHistory(),
# rprompt=get_rprompt(False),
# style=example_style,
search_ignore_case=True,
)
async def _print_services(self, services, table_title=None):
table_data = [["Id", "URL", "Title", "Server", "Matched Signature(s)"]]
for entry in services:
service_id, url, _, _, _, title, server, _, _, matched_sigs, _ = entry
table_data.append([service_id, url, title, server, matched_sigs])
table = AsciiTable(table_data)
table.inner_row_border = True
table.title = table_title
print(table.table)
async def _print_hosts(self, hosts, table_title=None):
table_data = []
async with ScanDatabase(connection=self.db) as db:
for entry in hosts:
host_id, hostname, ip = entry
service_count = await db.get_service_count_on_host(host_id)
matched_sigs = map(
lambda x: x[0].split(","),
filter(
lambda x: x[0] is not None,
await db.get_matched_sigs_on_host(host_id),
),
)
table_data.append(
[
host_id,
ip,
hostname,
service_count,
",".join(
set(
sig_name
for result in matched_sigs
for sig_name in result
)
),
]
)
table_data = list(sorted(table_data, key=lambda i: (i[3], i[4]), reverse=True))
table_data.insert(
0, ["Id", "IP", "Hostname", "Discovered Services", "Matched Signature(s)"]
)
table = AsciiTable(table_data)
table.inner_row_border = True
table.title = table_title
print(table.table)
async def exit(self):
"""
Guess what this does
"""
print("Ciao!")
async def show(self, args):
"""
Preview a screenshot in the terminal
"""
try:
server_id = int(args[0])
except IndexError:
print("No server id given")
except ValueError:
print("Invalid server id")
else:
async with ScanDatabase(connection=self.db) as db:
entry = await db.get_service_by_id(server_id)
with open((self.scan_folder_path / entry[2]).absolute(), "rb") as image:
imgcat(image)
async def open(self, args):
"""
Open a screenshot in your default browser/previewer
"""
try:
server_id = int(args[0])
except IndexError:
print("No server id given")
except ValueError:
print("Invalid server id")
else:
async with ScanDatabase(connection=self.db) as db:
entry = await db.get_service_by_id(server_id)
screenshot_path = self.scan_folder_path / entry[2]
webbrowser.open(screenshot_path.absolute().as_uri())
async def hosts(self, args):
"""
Show hosts
"""
async with ScanDatabase(connection=self.db) as db:
try:
filter_term = args[0]
except IndexError:
hosts = await db.get_hosts()
await self._print_hosts(hosts)
else:
try:
host = await db.get_host_by_id(int(filter_term))
if not host:
raise ValueError(f"No host found with id: {filter_term}")
except ValueError:
query_results = await db.search_hosts(filter_term)
await self._print_hosts(query_results)
else:
await self._print_hosts([host])
services = await db.get_services_on_host(host[0])
await self._print_services(services)
async def servers(self, args):
"""
Show discovered servers
"""
async with ScanDatabase(connection=self.db) as db:
if len(args):
query_results = await db.search_services(args[0])
else:
query_results = await db.get_services()
await self._print_services(query_results)
async def generate_report(self, args):
"""
Generate a report
"""
await self.scan()
if not args or args[0] == "html":
await generate_html_report(self.scan_folder_path, self.db)
elif args[0] == "csv":
await generate_csv_report(self.scan_folder_path, self.db)
elif args[0] == "json":
raise NotImplementedError
else:
print("Valid report formats are: csv,html,json")
async def open_html_report(self):
"""
Open a generated HTML report in the default browser
"""
report_path = self.scan_folder_path / "witnessme_report.html"
if not report_path.exists():
await self.generate_report(["html"])
webbrowser.open(report_path.absolute().as_uri())
async def scan(self):
"""
Peform a signature scan on all discovered servers
"""
self.signatures.load()
log.debug("Starting signature scan...")
start_time = time()
async with ScanDatabase(connection=self.db) as db:
tasks = [
self.signatures.find_match(service)
for service in await db.get_services()
]
matches = list(
filter(lambda x: len(x[0]) > 0, await asyncio.gather(*tasks))
)
for match in matches:
await db.add_matched_sigs_to_service(
match[1][0], ",".join([sig["name"] for sig in match[0]])
)
completed_time = strftime("%Mm%Ss", gmtime(time() - start_time))
log.debug(
f"Signature scan completed, identified {len(matches)} service(s) in {completed_time}"
)
async def cmdloop(self):
self.db = await aiosqlite.connect(self.db_path)
try:
while True:
# with patch_stdout():
text = await self.prompt_session.prompt_async()
command = shlex.split(text)
if len(command):
# Apperently you can't call await on a coroutine retrieved via getattr() ??
# So this sucks now but thankfully we don't have a lot of commands
try:
if command[0] == "exit":
await self.exit()
break
elif command[0] == "show":
await self.show(command[1:])
elif command[0] == "generate_report":
await self.generate_report(command[1:])
elif command[0] == "open_html_report":
await self.open_html_report()
elif command[0] == "open":
await self.open(command[1:])
elif command[0] == "hosts":
await self.hosts(command[1:])
elif command[0] == "servers":
await self.servers(command[1:])
elif command[0] == "scan":
await self.scan()
else:
print("Command does not exist")
except Exception as e:
import traceback
traceback.print_exc()
print(f"Error calling command '{command[0]}': {e}")
finally:
await self.db.close()
def run():
parser = argparse.ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter)
parser.add_argument("scan_folder", type=str, help="WitnessMe scan folder")
args = parser.parse_args()
scan_folder = pathlib.Path(args.scan_folder)
# For backwards compatibility, handle pointing directly to the database
if scan_folder.name == "witnessme.db":
scan_folder = scan_folder.parent
db_path = scan_folder / "witnessme.db"
if not db_path.exists():
print("Unable to find db, not a WitnessMe scan folder or invalid path?")
sys.exit(1)
print("[!] Press tab for autocompletion and available commands")
dbcli = WMDBShell(scan_folder.expanduser())
asyncio.run(dbcli.cmdloop())
if __name__ == "__main__":
run()
WitnessMe-master/witnessme/database.py 0000664 0000000 0000000 00000014065 13745620352 0020436 0 ustar 00root root 0000000 0000000 import aiosqlite
import json
import logging
log = logging.getLogger("witnessme.database")
class ScanDatabase:
def __init__(self, report_folder=None, connection=None):
self.report_folder = report_folder
self.connection = connection
@staticmethod
async def create_db_and_schema(report_folder):
async with aiosqlite.connect(f"{report_folder}/witnessme.db") as db:
await db.execute(
"""CREATE TABLE "hosts" (
"id" integer PRIMARY KEY,
"hostname" text,
"ip" text,
UNIQUE(hostname, ip)
)"""
)
await db.execute(
"""CREATE TABLE "services" (
"id" integer PRIMARY KEY,
"url" text,
"screenshot" text,
"port" integer,
"scheme" text,
"title" text,
"server" text,
"headers" text,
"host_id" integer,
"matched_sigs" text,
"body" text,
FOREIGN KEY(host_id) REFERENCES hosts(id),
UNIQUE(port, host_id, scheme)
)"""
)
await db.commit()
async def add_host(self, ip, hostname):
return await self.db.execute(
"INSERT OR IGNORE INTO hosts (ip, hostname) VALUES (?,?)", [ip, hostname]
)
async def add_service(
self, url, screenshot, port, scheme, title, server, headers, body, host_id
):
return await self.db.execute(
"INSERT OR IGNORE INTO services (url, screenshot, port, scheme, title, server, headers, body, host_id) VALUES (?,?,?,?,?,?,?,?,?)",
[url, screenshot, port, scheme, title, server, headers, body, host_id],
)
async def add_host_and_service(
self, ip, hostname, url, screenshot, port, scheme, title, server, headers, body
):
cursor = await self.add_host(ip, hostname)
host_id = cursor.lastrowid
if host_id == 0:
async with self.db.execute(
"SELECT id FROM hosts WHERE ip=(?) AND hostname=(?)", [ip, hostname]
) as cursor:
row = await cursor.fetchone()
host_id = row[0]
await self.add_service(
url,
screenshot,
port,
scheme,
title,
server,
json.dumps(headers),
body,
host_id,
)
async def add_matched_sigs_to_service(self, service_id, matches):
if await self.get_service_by_id(service_id):
await self.db.execute(
"UPDATE services SET matched_sigs=(?) WHERE id=(?)",
[matches, service_id],
)
async def get_matched_sigs_on_host(self, host_id: int):
async with self.db.execute(
"SELECT matched_sigs FROM services WHERE host_id=(?)", [host_id]
) as cursor:
return await cursor.fetchall()
async def get_service_count(self):
async with self.db.execute("SELECT count(*) FROM services") as cursor:
result = await cursor.fetchone()
return result[0]
async def get_host_count(self):
async with self.db.execute("SELECT count(*) FROM hosts") as cursor:
result = await cursor.fetchone()
return result[0]
async def get_service_count_on_host(self, host_id: int):
async with self.db.execute(
"SELECT count(*) FROM services WHERE host_id=(?)", [host_id]
) as cursor:
result = await cursor.fetchone()
return result[0]
async def get_services_on_host(self, host_id: int):
async with self.db.execute(
"SELECT * FROM services WHERE host_id=(?)", [host_id]
) as cursor:
result = await cursor.fetchall()
return result
async def get_service_by_id(self, service_id: int):
async with self.db.execute(
"SELECT * FROM services WHERE id=(?)", [service_id]
) as cursor:
return await cursor.fetchone()
async def get_host_by_id(self, host_id: int):
async with self.db.execute(
"SELECT * FROM hosts WHERE id=(?)", [host_id]
) as cursor:
return await cursor.fetchone()
async def get_hosts(self, limit=-1, offset=-1):
async with self.db.execute(
"SELECT * FROM hosts LIMIT (?) OFFSET (?)", [limit, offset]
) as cursor:
return await cursor.fetchall()
async def get_services(self, limit=-1, offset=-1):
async with self.db.execute(
"SELECT * FROM services LIMIT (?) OFFSET (?)", [limit, offset]
) as cursor:
return await cursor.fetchall()
async def get_services_with_host(self, limit=-1, offset=-1):
services_with_hosts = []
async with self.db.execute(
"SELECT * FROM services LIMIT (?) OFFSET (?)", [limit, offset]
) as cursor:
services = await cursor.fetchall()
for service in services:
_, hostname, ip = await self.get_host_by_id(service[8])
services_with_hosts.append(service + (hostname, ip))
return services_with_hosts
async def search_hosts(self, search: str):
async with self.db.execute(
"SELECT * FROM hosts WHERE ip LIKE (?) OR hostname LIKE (?)",
[f"%{search}%"] * 2,
) as cursor:
return await cursor.fetchall()
async def search_services(self, search: str):
async with self.db.execute(
"SELECT * FROM services WHERE title LIKE (?) OR server LIKE (?)",
[f"%{search}%"] * 2,
) as cursor:
return await cursor.fetchall()
async def __aenter__(self):
if not self.connection:
self.db = await aiosqlite.connect(f"{self.report_folder}/witnessme.db")
else:
self.db = self.connection
return self
async def __aexit__(self, exec_type, exc, tb):
await self.db.commit()
if not self.connection:
await self.db.close()
WitnessMe-master/witnessme/headlessbrowser.py 0000664 0000000 0000000 00000014310 13745620352 0022057 0 ustar 00root root 0000000 0000000 import logging
import asyncio
import os
import pyppeteer
from typing import List
from witnessme.parsers import AutomaticTargetGenerator
from pyppeteer.errors import PageError
log = logging.getLogger("witnessme.headlessbrowser")
class BrowserStats:
inputs: int = 0
execs: int = 0
@property
def pending(self) -> int:
return self.inputs - self.execs
async def navigate_to_page(browser, url, page):
log.debug(f"Navigating to {url}, replace me with something useful")
async def finished_navigation(browser, results):
log.debug("Finished navigation, replace me with something useful")
class HeadlessChromium:
def __init__(
self, threads: int = 15, timeout: int = 15, on_new_tab=None, on_finished=None
) -> None:
self.threads = threads
self.timeout = timeout
self.on_new_tab = on_new_tab or navigate_to_page
self.on_finished = on_finished or finished_navigation
self.stats = BrowserStats()
self._queue = asyncio.Queue()
self._browser_stop_event = asyncio.Event()
async def _on_request(self, request):
pass
# log.info(f"on_request() called: url: {request.url}")
async def _on_response(self, response):
pass
# log.info(f"on_response() called, url: {response.url}")
async def _on_requestfinished(self, request):
pass
# log.info(f"on_requestfinished() called, url: {request.url}")
async def task_watch(self):
while not self._browser_stop_event.is_set():
try:
await asyncio.sleep(5)
log.info(
f"total: {self.stats.inputs}, done: {self.stats.execs}, pending: {self.stats.pending}"
)
except asyncio.CancelledError:
break
async def target_producer(self, targets, ports):
with AutomaticTargetGenerator(targets, ports) as generated_targets:
for url in generated_targets:
self.stats.inputs += 1
await self._queue.put(url)
async def open_browser_tab(self, context):
url = await self._queue.get()
# page.setDefaultNavigationTimeout() accepts milliseconds
page = await context.newPage()
page.setDefaultNavigationTimeout(self.timeout * 1000)
# page.on('request', lambda req: asyncio.create_task(self._on_request(req)))
# page.on('requestfinished', lambda req: asyncio.create_task(self._on_requestfinished(req)))
# page.on('response', lambda resp: asyncio.create_task(self._on_response(resp)))
try:
results = await asyncio.wait_for(
self.on_new_tab(self, url, page), timeout=self.timeout
)
await self.on_finished(self, results)
log.debug(f"Navigated to {url}")
except asyncio.TimeoutError:
log.error(f"Navigation to url {url} timed out")
except Exception as e:
# if not any(err in str(e) for err in ['ERR_ADDRESS_UNREACHABLE', 'ERR_CONNECTION_REFUSED', 'ERR_CONNECTION_TIMED_OUT']):
log.error(f"Error navigating to url {url}: {e}")
finally:
self.stats.execs += 1
try:
await page.close()
except:
pass
self._queue.task_done()
async def start_browser(self, n_urls: int):
log.info("Starting headless browser")
# --no-sandbox is required to make Chrome/Chromium run under root
browser_args = ["--disable-gpu", "--no-sandbox"]
if os.getuid() == 0:
log.warning("Running under root privileges, browser will start with --no-sandbox")
browser_args.append("--no-sandbox")
proxy = os.environ.get("HTTP_PROXY")
if proxy:
browser_args.append(f"--proxy-server={proxy}")
log.info(f"Proxy set to {proxy}")
browser = await pyppeteer.launch(
headless=True,
ignoreHTTPSErrors=True,
autoClose=True,
args=browser_args,
executablePath=os.environ.get("CHROMIUM_EXECUTABLE_PATH"),
)
context = await browser.createIncognitoBrowserContext()
try:
worker_threads = [
self.open_browser_tab(context)
for _ in range(n_urls)
]
log.info(f"Using {len(worker_threads)} browser tab(s)/thread(s)")
done, pending = await asyncio.wait(worker_threads, timeout=(self.timeout * 2))
if len(pending):
log.warning(f"{len(pending)} tasks did not finish, continuing to avoid stalling")
for task in pending:
task.cancel()
except asyncio.CancelledError:
log.info(f"Cancelling tab(s)/thread(s)")
finally:
try:
await context.close()
#log.error("Page crashed, ignoring...")
log.info("Killing headless browser")
await browser.close()
await browser.disconnect()
except:
pass
async def run(
self, targets: List[str], ports: List[int] = [80, 8080, 443, 8443]
) -> None:
asyncio.create_task(self.target_producer(targets, ports))
log.debug("Waiting for queue to populate...")
while self._queue.qsize() == 0:
await asyncio.sleep(0.1)
task_watcher = asyncio.create_task(self.task_watch())
try:
while self._queue.qsize() > 0 and not self._browser_stop_event.is_set():
try:
await asyncio.wait_for(
self.start_browser(
n_urls=self.threads
if self._queue.qsize() > self.threads
else self._queue.qsize(),
),
timeout=(self.timeout *2) + 10
)
except asyncio.TimeoutError:
log.error("Browser task timed out")
finally:
task_watcher.cancel()
async def stop(self):
self._browser_stop_event.set()
async def __aenter__(self):
return self
async def __aexit__(self, exc_type, exc_value, traceback):
await self._queue.join()
WitnessMe-master/witnessme/parsers.py 0000664 0000000 0000000 00000020743 13745620352 0020351 0 ustar 00root root 0000000 0000000 import sys
import xmltodict
import pathlib
import logging
from collections import OrderedDict
from ipaddress import ip_address, ip_network, summarize_address_range
from contextlib import ContextDecorator
from xml.parsers.expat import ExpatError
log = logging.getLogger("witnessme.parsers")
class TargetGenerator(ContextDecorator):
def __init__(self, target, ports=[80, 8080, 443, 8443]):
self.target = target
self.ports = ports
def expand_ip_cidr_or_range(self, target):
try:
if "-" in target:
start_ip, end_ip = target.split("-")
try:
end_ip = ip_address(end_ip)
except ValueError:
first_three_octets = start_ip.split(".")[:-1]
first_three_octets.append(end_ip)
end_ip = ip_address(".".join(first_three_octets))
for ip_range in summarize_address_range(ip_address(start_ip), end_ip):
for ip in ip_range:
yield str(ip)
else:
for ip in ip_network(target, strict=False):
yield str(ip)
except ValueError:
yield str(target)
def __enter__(self):
if self.target.startswith("http://") or self.target.startswith("https://"):
yield self.target
elif self.target.startswith("http-simple-new://"):
yield self.target.replace("http-simple-new://", "http://")
elif self.target.startswith("https-simple-new://"):
yield self.target.replace("https-simple-new://", "https://")
else:
for host in self.expand_ip_cidr_or_range(self.target):
for port in self.ports:
for scheme in ["http", "https"]:
yield f"{scheme}://{host}:{port}"
def __exit__(self, *exc):
pass
class GenericFileParser(ContextDecorator):
def __init__(self, file_path, ports=[80, 8080, 443, 8443]):
self.file_path = file_path
self.ports = ports
def __enter__(self):
with open(self.file_path) as target_file:
for target in target_file:
with TargetGenerator(
target.strip(), ports=self.ports
) as target_generator:
for url in target_generator:
yield url
def __exit__(self, *exc):
pass
class XmlParser(ContextDecorator):
def __init__(self, file_path):
self.xml_file_path = file_path
self.item_depth = 4
self.https_ports = [443, 8443]
self.http_ports = [80, 8080]
self.urls = set()
def parser_callback(self, path, item):
return True
def __enter__(self):
with open(self.xml_file_path, "rb") as xml_file_path:
try:
xmltodict.parse(
xml_file_path,
item_depth=self.item_depth,
item_callback=self.parser_callback,
process_namespaces=True,
)
except ExpatError as e:
log.error(e)
for url in self.urls:
yield url
def __exit__(self, *exc):
self.urls = set()
class NmapParser(XmlParser):
def __init__(self, file_path):
super().__init__(file_path)
self.item_depth = 2
def parser_callback(self, path, item):
if isinstance(item, OrderedDict):
if "address" in item.keys() and "ports" in item.keys():
try:
address = item["address"]["@addr"]
except TypeError:
address = list(filter(
lambda x: x["@addrtype"] == "ipv4",
item["address"])
)[0]["@addr"]
ports = item["ports"].get("port")
if not ports:
return True
# If there's only a single port discovered, ports will be an OrderedDict
if isinstance(ports, OrderedDict):
ports = [ports]
for port in ports:
if port["@protocol"] == "tcp" and port["state"]["@state"] == "open" and port.get("service"):
service = port["service"].get("@name")
tunnel = port["service"].get("@tunnel")
port_number = port["@portid"]
if service in ["ssl", "https"]:
self.urls.add(f"https://{address}:{port_number}")
# See https://github.com/byt3bl33d3r/WitnessMe/issues/19
elif tunnel == "ssl" and service in ["http", "http-proxy"]:
self.urls.add(f"https://{address}:{port_number}")
elif service in ["http-alt", "http-proxy", "http"]:
self.urls.add(f"http://{address}:{port_number}")
return True
class NessusParser(XmlParser):
def parser_callback(self, path, item):
"""
Apperently, Nessus's plugins are far from being consistent when trying to detect http/https pages
https://github.com/FortyNorthSecurity/EyeWitness/blob/master/modules/helpers.py#L100-L106
https://github.com/FortyNorthSecurity/EyeWitness/blob/master/modules/helpers.py#L225-L230
"""
entry = dict(path)
if entry.get("ReportItem"):
report_item = entry["ReportItem"]
if (
report_item.get("port")
and report_item.get("svc_name")
and report_item.get("pluginName")
):
if (
report_item["svc_name"] == "https?"
or int(report_item["port"]) in self.https_ports
):
self.urls.add(
f"https://{entry['ReportHost']['name']}:{report_item['port']}"
)
elif (
report_item["pluginID"] == "22964"
and report_item["svc_name"] == "www"
):
plugin_output = item.get("plugin_output")
if plugin_output.lower() in [
"a web server is running on this port.",
"a web server is running on the remote host.",
]:
self.urls.add(
f"http://{entry['ReportHost']['name']}:{report_item['port']}"
)
elif plugin_output.lower().startswith(
"a web server is running on this port through"
):
self.urls.add(
f"https://{entry['ReportHost']['name']}:{report_item['port']}"
)
elif report_item["svc_name"] in ["www", "http?"]:
self.urls.add(
f"http://{entry['ReportHost']['name']}:{report_item['port']}"
)
return True
class AutomaticTargetGenerator(ContextDecorator):
def __init__(self, targets: list, ports=[80, 8080, 443, 8443]):
self.targets = targets
self.ports = ports
def generate(self, targets):
for target in targets:
target = target.strip()
if pathlib.Path(target).exists():
target = str(pathlib.Path(target).expanduser())
if target.lower().endswith(".nessus"):
log.debug("Detected .nessus file as a target")
file_parser = NessusParser(target)
elif target.lower().endswith(".xml"):
log.debug("Detected NMap XML file as a target")
file_parser = NmapParser(target)
else:
log.debug("Detected file as a target")
file_parser = GenericFileParser(target, ports=self.ports)
with file_parser as generated_urls:
for url in generated_urls:
yield url
else:
log.debug("Detected IP Address/Range/CIDR, hostname or URL as a target")
with TargetGenerator(target, ports=self.ports) as generated_urls:
for url in generated_urls:
yield url
def __enter__(self):
if "-" in self.targets:
log.debug("Reading targets from stdin")
return self.generate(sys.stdin)
return self.generate(self.targets)
def __exit__(self, *exc):
pass
WitnessMe-master/witnessme/reporting.py 0000664 0000000 0000000 00000006076 13745620352 0020706 0 ustar 00root root 0000000 0000000 import csv
import logging
import pkg_resources
from jinja2 import Template
from witnessme.database import ScanDatabase
log = logging.getLogger("witnessme.reporting")
async def generate_html_report(scan_folder, db):
results_per_page = 100
template_path = pkg_resources.resource_filename(__name__, "templates/template.html")
log.info("Generating HTML report, please wait...")
async with ScanDatabase(connection=db) as db:
service_count = await db.get_service_count()
total_pages = (
service_count // results_per_page
if service_count % results_per_page == 0
else (service_count // results_per_page) + 1
)
current_page = 1
offset = 0
while True:
services = await db.get_services_with_host(
limit=results_per_page, offset=offset
)
if not services:
break
report_file = scan_folder / f"report_page_{current_page}.html"
if current_page == 1:
report_file = scan_folder / "witnessme_report.html"
with open(report_file, "w") as report:
with open(template_path) as file_:
template = Template(file_.read())
report.write(
template.render(
name="WitnessMe Report",
current_page=current_page,
total_pages=total_pages,
services=services,
)
)
current_page += 1
offset += results_per_page
log.info("Done")
async def generate_csv_report(scan_folder, db):
result_limit = 100
csv_path = scan_folder / "witnessme_report.csv"
with open(csv_path, "w", newline="") as csvfile:
writer = csv.writer(csvfile)
writer.writerow(
[
"url",
"ip",
"hostname",
"port",
"scheme",
"title",
"server",
"screenshot",
"matched_sigs",
]
)
log.info("Generating CSV report, please wait...")
async with ScanDatabase(connection=db) as db:
offset = 0
while True:
services = await db.get_services_with_host(
limit=result_limit, offset=offset
)
if not services:
break
for service in services:
writer.writerow(
[
service[1],
service[12],
service[11],
service[3],
service[4],
service[5],
service[6],
service[2],
service[9],
]
)
offset += result_limit
log.info("Done")
WitnessMe-master/witnessme/signatures.py 0000664 0000000 0000000 00000002465 13745620352 0021057 0 ustar 00root root 0000000 0000000 import os
import pkg_resources
import logging
import yaml
import pathlib
try:
from yaml import CLoader as Loader
except ImportError:
from yaml import Loader
log = logging.getLogger("witnessme.signatures")
class Signatures:
def __init__(self, sig_folder=None):
self.signatures = []
self.sig_folder = sig_folder
if not sig_folder:
self.sig_folder = pathlib.Path(
pkg_resources.resource_filename(__name__, "signatures")
)
def load(self):
self.signatures = []
for sig_file in os.listdir(self.sig_folder):
sig_file_path = self.sig_folder / sig_file
with open(sig_file_path.absolute()) as sig:
self.signatures.append(yaml.load(sig, Loader=Loader))
log.debug(f"Loaded {len(self.signatures)} signature(s)")
def get_sig(self, name: str):
return next(filter(lambda s: s["name"] == name, self.signatures), None)
async def find_match(self, service):
matches = []
for sig in self.signatures:
if all(
i > 0 for i in map(lambda s: service[-1].find(s), sig["signatures"])
):
log.debug(f"Matched {service[1]} for signature '{sig['name']}'")
matches.append(sig)
return matches, service
WitnessMe-master/witnessme/signatures/ 0000775 0000000 0000000 00000000000 13745620352 0020476 5 ustar 00root root 0000000 0000000 WitnessMe-master/witnessme/signatures/ace_4710_device_manager.yaml 0000664 0000000 0000000 00000000245 13745620352 0025577 0 ustar 00root root 0000000 0000000 credentials:
- password: admin
username: admin
name: ACE 4710 Device Manager
signatures:
- ACE 4710 Device Manager
- Cisco
# (ACE 4710 Device Manager) admin/admin
WitnessMe-master/witnessme/signatures/admanager.yaml 0000664 0000000 0000000 00000000427 13745620352 0023304 0 ustar 00root root 0000000 0000000 credentials:
- password: admin
username: admin
name: ADManager
signatures:
- ADManager Plus Authentication
- sel.options[i].text == 'ADManager Plus Authentication'
- ManageEngine
- js/framework/Hashtable.js
# ADManager - admin/admin (ADManager Plus Authentication in dropdown)
WitnessMe-master/witnessme/signatures/airos.yaml 0000664 0000000 0000000 00000000355 13745620352 0022502 0 ustar 00root root 0000000 0000000 credentials:
- password: ubnt
username: ubnt
name: AirOS
signatures:
- airos_logo.png
- form enctype="multipart/form-data" id="loginform" method="post"
- align="center" class="loginsubtable"
- function onLangChange()
# AirOS ubnt/ubnt
WitnessMe-master/witnessme/signatures/alienvault_usm_if_on_aws.yaml 0000664 0000000 0000000 00000000540 13745620352 0026435 0 ustar 00root root 0000000 0000000 credentials:
- password:
# Altova License Server /default
WitnessMe-master/witnessme/signatures/apache_lenya.yaml 0000664 0000000 0000000 00000000224 13745620352 0023771 0 ustar 00root root 0000000 0000000 credentials:
- password: levi
username: admin
name: Apache Lenya
signatures:
- Apache Lenya
- lenya.action=introspect
# (Apache Lenya) admin/levi
WitnessMe-master/witnessme/signatures/apache_tomcat.yaml 0000664 0000000 0000000 00000000424 13745620352 0024152 0 ustar 00root root 0000000 0000000 credentials:
- password: tomcat
username: tomcat
- password: admin
username: admin
name: Apache Tomcat
signatures:
- Apache Tomcat
- Manager Application HOW-TO
- Manager App
- For security, access to the
# Apache Tomcat tomcat/tomcat admin/admin etc.
WitnessMe-master/witnessme/signatures/apache_tomcat_default_page.yaml 0000664 0000000 0000000 00000000221 13745620352 0026645 0 ustar 00root root 0000000 0000000 name: Apache Tomcat Default Page
signatures:
-