GoWitness is one of our favorite tools for web service reconnaissance, particularly for its ability to group the screenshots that it takes by visual appearance. But when used to cover large scopes, this can result in pages of blank screenshots flooding the gallery.
Worse than that, the gallery view performs a cropping of the screenshots, obscuring the top of the image. Screenshots may appear to be blank, but actually contain a small text response that is hidden from view. If a service includes a small amount of sensitive data in a non-HTML response, these edge cases can be missed by the analyst:
In addition to potentially obscuring these needle-in-haystack disclosures, these services also introduce a huge bottleneck for discovery scans. Each URL tested must have the browser pause and load for 10-15 seconds prior to screenshot, which increases scan cadence significantly when targeting thousands of URLs.
To reduce the overhead of scans and improve the ability to detect interesting responses, we have published the HTTPX Analyzer python application. Within our improved scanning process, target URLs are preprocessed using ProjectDiscovery's massively parallel HTTPX client to retrieve request data, filtered using HTTPX Analyzer, and then scanned with GoWitness after the uninteresting responses have been removed. This was found to reduce scan times of large target scopes by over a factor of 10.
Candidate URLs may be scanned using HTTPX's –fr (follow redirection), -title (title grabbing), -oa (output all file types), and –irr (include response in output) arguments to save these details to the report output. HTTPX Analyzer can then be used to analyze the results based on frequency of similar responses.
Analysts sort data in two Rounds: first, they sort requests which did not contain any HTML title tag, which are grouped based on byte size of the response. For example, nearly all scans will include some hosts which had an empty response as seen below. Analysts can use the Spacebar to set these Groups of as Marked (marked "Not Interesting"):
The Page Up and Down keys may be used to browse through the currently displayed response. The Right and Left arrow keys allow the user to view the other requests within the currently viewed Group. They can also hold these keys down to rapidly visually inspect thousands of similar responses in a short period of time. The Up and Down arrow keys are used to navigate between the Groups.
When they are ready for Round Two of analysis, the "n" key shifts to the next Round to inspect responses which contain Titles. Here, the responses are grouped byd the value of the Title tag. Some generic titles (e.g., Login) may have a wide variety of different responses, so you may wish to screenshot them all. For common errors and landing pages (e.g., AWS S3 URLs seen below), this junk can be Marked using Spacebar again. When the user presses "q" to exist, HTTPx Analyzer writes its output (non-Marked target URLs) to standard out, along with any other output options set.
HTTPX Analyzer also allows use of the --findUnique command flag, which inverts its typical behavior. Rather than showing the most frequent ("junkiest") responses first, HTTPX Analyzer will show rare requests first, in order of increasing response size. This feature has less applicability to GoWitness workflows, but can identify edge case responses (e.g., stored credentials, information disclosure within odd errors).
Just like Nmap, the –oA flag can be used to output all types at once. HTTPX Analyzer supports output of the following filetypes:
.jq file containing jq executable queries, which can be used to perform the same filtering against HTTPX json files without needing to run HTTPX Analyzer again (i.e., automated workflows).
.state file used by HTTPX analyzer to determine which Groups were Marked by the user. This state file can be used between different input files so that the analyst does not need to manually browse and mark Groups again.
.targets file containing all non-Marked (i.e., interesting) requests. HTTPX Analyzer additionally includes one sample URL from every Marked Group in this output.
.junk file containing 15 samples from every Marked group to validate that no sensitive URLs were accidentally omitted (i.e. low priority GoWitness candidates).