Reviewing Automated HTTP Screenshot Tools
In our Screenshot Tool blog series, Cyber Advisors is reviewing some of the top HTTP screenshot tools that are currently available for penetration testers or bug bounty hunters. Each of these blog posts covers the available features within the tool, how easy it is to use, and any problems that may arise while performing simple scans. At the end of this series, we’ll have all the tools put to the test to compare their effectiveness against a set of metrics to find the most useful tool.
If you’ve read part one, feel free to jump over the criteria section down to the WitnessMe tool section.
Screenshot Tools
We use HTTP screenshotting tools on external, internal, and red team engagements to quickly analyze the footprint of web servers and services exposed across the target environment.
The following items represent the criteria for how I define ‘usefulness’ for a given HTTP screenshot tool. These criteria are skewed towards attack surfacing reconnaissance and penetration testing techniques.
Note: this series was written in Feb. 2022.
SOURCES & CUSTOMIZATION
- Are common network scanner output formats supported as input sources by the tool? (Nmap/Masscan/Nessus)
- Can we append a second scan to the results of the first scan within the Database?
- Can you set a custom User-Agent?
- Can you set custom request headers and/or cookies?
PERFORMANCE
- How accurate is the tool? How many errors does it generate? Does it successfully capture screenshots (no white or blank screenshots)?
- How quickly does the tool perform the task? Can we speed the scan up without degrading its accuracy of the scan?
- Can we easily proxy the network traffic to http and socks proxies?
- Can we provide a navigation timeout against slow assets?
REPORTING
- Does the tool group or categorize similar hosts based on their screenshots?
- Does the tool provide useful output? Can it export to PDF/HTML/JSON/CSV?
- Is the UI/UX functional and enjoyable to use?
GENERAL
- Is the tool actively maintained with updates?
- Any Cross-Platform compatibility?
- How easy is it to build and install the project?
WitnessMe
Continuing down the path of tools written in Python, the next tool we will be reviewing is WitnessMe. The project’s description is as follows: “WitnessMe is primarily a Web Inventory tool inspired by EyeWitness, it’s also written to be extensible allowing you to create custom functionality that can take advantage of the headless browser it drives in the back-end.” WitnessMe was created by byt3bl33d3r, who is known for making awesome tools for the infosec community (such as CrackMapExec).
WitnessMe Notable Features
WitnessMe was created with several motivations – the most notable reasons that stand out are the efficient installation/dependency management and the CLI interface. Byt3bl33der’s Python tools are known for these two points and WitnessMe continues that trend, so users can integrate the tool into their workflow more efficiently. The documentation that the repository provides is excellent, covering a wide variety of installation instructions and usability instructions.
A snapshot of the CLI arguments can be seen below which provides a window into the tool’s feature set.
WitnessMe Sources & Customization
WitnessMe checks a couple of the boxes when it comes to the feature that I’m expecting, but it does have a couple of limitations that will need to be added for efficiency. WitnessMe does have the ability to timeout requests and HTTP/SOCKS proxy support, as well as parses Nessus, Nmap, and flat files. However, there is no built-in capability for custom user agents, custom headers, or screenshot navigation time delays.
WitnessMe Reporting
One of the strengths of WitnessMe is in its reporting capabilities as far as UI/UX is concerned. The built-in WitnessMe database (wmdb) provides one of the best experiences compared to any other screenshot tool. Its handy auto-completion for commands provides a ton of useful information quickly, such as exporting the hosts scanned, servers discovered, and the capability to export the report. It also has the ability to preview a screenshot in the terminal, however, this did not work for me on my Debian system, but I really like this idea and will have to troubleshoot my errors in the future.
The HTML report is clean, shows a good amount of detail, and provides pagination for quick navigation. The built-in search functionality and signature matching provide a nice touch of polish as well. Compared to EyeWitness, this is a significant leap forward in maturity.
However, there is no built-in argument when capturing screenshots to customize the viewport size, resulting in gigantic full-page screenshots. A lack of screenshot similarity categorization results in mixed screenshots and makes identifying similar assets extremely tedious.
WitnessMe General Setup
WitnessMe’s setup is a breeze, with a ton of supporting documentation if you want to install a docker image or as a python package. The python package uses pipx with virtual environments to cut down on dependency conflicts and this tool performs its job perfectly. I had no issues installing pipx and subsequently using WitnessMe in a short amount of time.
Under the hood, WitnessMe is using the “pypetteer” library to perform its HTTP automation.
WitnessMe does not provide any specific Windows installation instructions, but I imagine if pipx works on Windows you should have no issues installing this utility there.
WitnessMe appears to have a decent amount of support, though it has not had a new update since October 2020. I hope that this project continues to receive support, as the experience using WitnessMe from a CLI point of view is the best experience across all the tools that I reviewed.
Out-Of-The-Box Tests
Now that we’ve covered some of the high-level features of WitnessMe, it’s time to actually try the utility out. As previously discussed, I did not run into any issues installing WitnessMe on Debian and the pipx virtual environment prevented any other package conflicts within my host OS.
At first, I gave WitnessMe the task of processing 100 screenshots across 10 threads. Similar to my experiences with EyeWitness, when the tool encountered an error things started to break. During my tests, WitnessMe hung during execution when it encountered an error and did not proceed with the rest of the scan unless the execution flow was manually interrupted.
The good news is, unlike EyeWitness, the report will actually still generate even with a bunch of broken assets/failed captures.
WitnessMe Performance Improvements
In order to create an equal comparison environment, I built a development environment using Poetry, following the steps within WitnessMe’s documentation. This process was a breeze and I actually noticed that launching WitnessMe from Poetry actually resolved the issues I had with the process hanging. I’m not entirely sure what the root cause of this issue was, but at least I could perform a full-scale test with WitnessMe.
Knowing some of the limitations with a default puppeteer configuration, I built in new functionality for headers and user-agent support as well as tweaked the default chromium arguments. In the future, I may open a pull request for these features once I’m sure they don’t introduce any additional errors.
These changes slightly increased the overall accuracy during the performance test.
WitnessMe Best Performance Configuration
The following is the best possible screenshot capture configuration based on WitnessMe’s built-in arguments and new additional features that allowed for the highest success percentage of captures during my analysis:
- Navigation Timeout (–timeout 20)
- Custom User-Agent
- Custom Headers
- Additional Chrome Arguments
- Threads (–threads 10)
Conclusion To Part 2 – WitnessMe
Overall, I think the WitnessMe project is close to being one of the best HTTP screenshotting tools. It definitely needs quite a bit more support and bug patching to ensure that the processes do not hang (and actually are killed as well) to ensure effective screenshot capturing. The CLI and WitnessMe database provide the best experience across any tool from a user experience point of view.
If you use WitnessMe, please support and contribute to this Open-Source project.
Author’s Note: This blog series is my personal take on the state of screenshot tools. My network environment and physical setup may differ from yours when using these tools and you may notice your tool is more or less successful. I’ve done the best I can to be platform agnostic to provide the best environment for success. If you run into any errors, remember to check the individual project’s issues page for support.
Our Screenshot Tool Series
Here is the Part 1 – EyeWitness post.
Part 3 – Snapback is now available too!
MORE FROM OUR TECHNICAL BLOG
Cyber Advisors specializes in providing fully customizable cyber security solutions & services. Our knowledgeable, highly skilled, talented security experts are here to help design, deliver, implement, manage, monitor, put your defenses to the test, & strengthen your systems - so you don’t have to.
