Skip to content

Commit ada4124

Browse files
bandrelclaude
andcommitted
feat: add greppable output (spoonmap_output.gnmap)
Closes #36. Originally proposed in PR #11 (Feb 2021), closed as unmergeable. Each port's banner pass now writes nmap_results/portN.gnmap via -oG, and those are merged into spoonmap_output.gnmap after the workers join. Why not #11's approach: it pointed every port at one shared -oG file with --append-output. That was safe when nmap ran serially, but nmap_scan() now runs five concurrent workers, so five processes would interleave their appends into a plausible-looking but corrupt host list — the worst outcome for a format whose whole purpose is machine consumption. One file per port gives each a single writer. Merged, not concatenated: a host open on three ports is scanned by three nmap runs and so appears in three files, but grepable output is one line per host. Concatenating would make anything counting lines over-count hosts and would disagree with spoonmap_output.xml/.json, which merge the same hosts. Hosts are ordered by _ip_sort_key() and port entries numerically, so the file is stable across runs and diffable between scans. Two consistency details: - _quarantine_failed_output() now renames the sibling .gnmap too. Otherwise a port excluded from spoonmap_output.xml would still contribute its partial hosts to spoonmap_output.gnmap, leaving the greppable artifact the more optimistic of the two about what was actually scanned. - Ports holding an .xml with no .gnmap — what a resume from before this feature looks like — are named in a warning instead of being silently absent. An artifact that quietly covers less than the run did is a coverage claim the scan never made. port{N}_sql.xml is excluded from that check, since _scan_extra_sql_ports() writes it with no -oG and it is not a gap. Adding .gnmap to nmap_results/ is safe for existing readers: all three that scan the directory filter on an .xml extension, and _parse_result_xml() returns None for anything else. spoonmap_output.gnmap is registered in _RESULT_FILES so --cleanup removes it. The parser was verified against real nmap 7.99 -oG output, not just a fixture: tab-separated fields, the 'Status: Up' line emitted alongside the ports line, the trailing 'Ignored State' field, and the comment header/footer. 939 tests pass (32 new), 100% coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 9675406 commit ada4124

3 files changed

Lines changed: 411 additions & 2 deletions

File tree

‎README.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -281,15 +281,32 @@ Inter-scan wait: 29s (target ~256 hosts)
281281
masscan_results/portN.xml # raw masscan XML per port (masscan port discovery path)
282282
live_hosts/portN.txt # deduplicated IPs per port
283283
nmap_results/portN.xml # nmap banner (-sV) XML per port
284+
nmap_results/portN.gnmap # nmap greppable (-oG) output per port
284285
nse_results/portN.xml # nmap NSE script XML per port (script_scan only)
285286
all_live_hosts.txt # union of all live IPs
286287
spoonmap_output.xml # merged nmap XML (or masscan if no banner scan)
287288
spoonmap_output.json # same data as JSON — list of host objects by IP
289+
spoonmap_output.gnmap # merged greppable output, one line per host (banner scan only)
288290
findings.txt # severity-sorted findings report (script_scan only)
289291
findings.md # same report in Markdown table format
290292
findings.json # same report as a JSON array (script_scan only)
291293
```
292294

295+
### Greppable output
296+
297+
`spoonmap_output.gnmap` is nmap's grepable format — one line per host — merged across every port scanned in the banner pass:
298+
299+
```
300+
# SpooNMAP aggregated greppable output
301+
Host: 10.0.0.5 (web01) Ports: 22/open/tcp//ssh//OpenSSH 8.9p1/, 80/open/tcp//http//nginx 1.18/
302+
Host: 10.0.0.200 () Ports: 445/open/tcp//microsoft-ds///
303+
# SpooNMAP done: 2 hosts
304+
```
305+
306+
Because each port is scanned by its own nmap invocation, nmap writes one `nmap_results/portN.gnmap` per port and SpooNMAP merges them, so a host open on three ports is one line with three port entries rather than three lines — matching how `spoonmap_output.xml`/`.json` merge the same hosts, and keeping line counts usable as host counts. Hosts are in numeric IP order and port entries in numeric port order, so the file is stable across runs and diffable between scans.
307+
308+
Written for the banner scan only; a masscan-only run (`banner_scan` off) produces no grepable output. If a port has nmap XML but no `.gnmap` — a scan resumed from before this output existed — those ports are named in a warning rather than silently dropped from the aggregate.
309+
293310
`spoonmap_output.json` consolidates hosts across all per-port files, merging ports for the same IP:
294311

295312
```json

‎spoonmap.py‎

Lines changed: 148 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1155,7 +1155,7 @@ def _fname_port(stem: str) -> str:
11551155
# Result dirs and files written during a scan run.
11561156
_RESULT_DIRS = ('discovery', 'nmap_results', 'nse_results', 'live_hosts')
11571157
_RESULT_FILES = ('all_live_hosts.txt', 'spoonmap_output.xml',
1158-
'spoonmap_output.json',
1158+
'spoonmap_output.json', 'spoonmap_output.gnmap',
11591159
'findings.txt', 'findings.md', 'findings.json')
11601160

11611161

@@ -1979,10 +1979,21 @@ def _build_nmap_cmd(dest_port, input_file, output_file, source_port,
19791979
if source_port and not (script_scan and dest_port in _MULTI_SCRIPT_PORTS):
19801980
cmd += ['--source-port', source_port]
19811981

1982-
cmd += ['-iL', input_file, '-oX', output_file]
1982+
# Greppable output alongside the XML, one file per port so that each has a
1983+
# single writer. nmap_scan() runs several workers concurrently, so a shared
1984+
# -oG target with --append-output would interleave their writes into a
1985+
# plausible-looking but corrupt host list. Banner pass only: the script pass
1986+
# writes to its own directory and its results are surfaced as findings.
1987+
cmd += ['-iL', input_file, '-oX', output_file,
1988+
'-oG', _gnmap_path(output_file)]
19831989
return cmd
19841990

19851991

1992+
def _gnmap_path(xml_path):
1993+
"""Return the ``.gnmap`` sibling of an nmap ``.xml`` output path."""
1994+
return os.path.splitext(xml_path)[0] + '.gnmap'
1995+
1996+
19861997
def _quarantine_failed_output(output_file):
19871998
"""Rename a failed pass's XML to ``<name>.xml.failed``; return the new path.
19881999
@@ -2001,6 +2012,12 @@ def _quarantine_failed_output(output_file):
20012012
Returns None when there was nothing to rename or the rename failed — a
20022013
missing output file already fails the gate, so either way the port is
20032014
re-scanned.
2015+
2016+
The sibling '.gnmap' is quarantined alongside it, for the same reason and to
2017+
the same '.failed' suffix. Leaving it in place would let a port excluded
2018+
from spoonmap_output.xml still contribute its partial hosts to
2019+
spoonmap_output.gnmap, so the two artifacts would disagree about what was
2020+
scanned — and the greppable one would be the optimistic of the two.
20042021
"""
20052022
if not os.path.exists(output_file):
20062023
return None
@@ -2009,6 +2026,10 @@ def _quarantine_failed_output(output_file):
20092026
os.replace(output_file, failed_path)
20102027
except OSError:
20112028
return None
2029+
gnmap = _gnmap_path(output_file)
2030+
if os.path.exists(gnmap):
2031+
with contextlib.suppress(OSError):
2032+
os.replace(gnmap, gnmap + '.failed')
20122033
return failed_path
20132034

20142035

@@ -4560,6 +4581,126 @@ def _write_combined_results(output_path, hosts_json, xml_hosts):
45604581
print(_COLOR_RESULT + f'\nResults written to {output_path}/spoonmap_output.xml / .json' + _COLOR_RESET)
45614582

45624583

4584+
def _parse_gnmap_line(line):
4585+
"""Split one grepable ``Host:`` line into ``(ip, host_field, port_entries)``.
4586+
4587+
Returns None for anything that is not a host line carrying ports: comment
4588+
lines, and the ``Status: Up`` form a host-discovery pass emits. Fields are
4589+
tab-separated, which is what makes the format greppable in the first place.
4590+
"""
4591+
if not line.startswith('Host: '):
4592+
return None
4593+
fields = line.split('\t')
4594+
host_field = fields[0][len('Host: '):].strip()
4595+
if not host_field:
4596+
return None
4597+
ip = host_field.split()[0]
4598+
for field in fields[1:]:
4599+
if field.startswith('Ports: '):
4600+
entries = [e.strip() for e in field[len('Ports: '):].split(',')]
4601+
entries = [e for e in entries if e]
4602+
if entries:
4603+
return ip, host_field, entries
4604+
return None
4605+
return None
4606+
4607+
4608+
def _gnmap_port_sort_key(entry):
4609+
"""Order a grepable port entry (``22/open/tcp//ssh///``) by port number."""
4610+
try:
4611+
return (0, int(entry.split('/', 1)[0]))
4612+
except ValueError:
4613+
return (1, 0)
4614+
4615+
4616+
def _aggregate_gnmap(result_dir):
4617+
"""Merge ``*.gnmap`` in *result_dir* into one host line per host.
4618+
4619+
Each port is scanned by its own nmap run, so a host open on three ports
4620+
appears in three files. Grepable output is one line per host, so the
4621+
per-port lines are merged rather than concatenated — otherwise anything
4622+
counting lines would over-count hosts, and the file would disagree with
4623+
spoonmap_output.xml/.json, which merge the same hosts.
4624+
4625+
Returns ``(lines, scanned_ports)``: the merged host lines in IP order, and
4626+
the set of port stems whose .gnmap was present.
4627+
"""
4628+
merged = {} # ip -> [host_field, {entry: None} used as ordered set]
4629+
scanned_ports = set()
4630+
try:
4631+
names = sorted(os.listdir(result_dir))
4632+
except OSError:
4633+
return [], scanned_ports
4634+
for fname in names:
4635+
if not fname.endswith('.gnmap'):
4636+
continue
4637+
scanned_ports.add(fname[:-len('.gnmap')])
4638+
try:
4639+
with open(os.path.join(result_dir, fname)) as fh:
4640+
content = fh.read()
4641+
except OSError as exc:
4642+
# One unreadable file must not cost the other ports' hosts.
4643+
print(_COLOR_ERROR + f'Warning: could not read {fname}: {exc}'
4644+
+ _COLOR_RESET)
4645+
continue
4646+
for line in content.splitlines():
4647+
parsed = _parse_gnmap_line(line.strip())
4648+
if parsed is None:
4649+
continue
4650+
ip, host_field, entries = parsed
4651+
if ip not in merged:
4652+
merged[ip] = [host_field, {}]
4653+
elif len(host_field) > len(merged[ip][0]):
4654+
# Prefer the form carrying a hostname over a bare address.
4655+
merged[ip][0] = host_field
4656+
for entry in entries:
4657+
merged[ip][1][entry] = None
4658+
lines = []
4659+
for ip in sorted(merged, key=_ip_sort_key):
4660+
host_field, entries = merged[ip]
4661+
ordered = sorted(entries, key=_gnmap_port_sort_key)
4662+
lines.append(f'Host: {host_field}\tPorts: {", ".join(ordered)}')
4663+
return lines, scanned_ports
4664+
4665+
4666+
def _write_gnmap_result(output_path, result_dir):
4667+
"""Write spoonmap_output.gnmap from the per-port .gnmap files.
4668+
4669+
Ports whose .xml is present but whose .gnmap is missing are named rather
4670+
than silently omitted: that is what a scan completed before greppable
4671+
output existed looks like on resume, and an artifact that quietly covers
4672+
fewer ports than the run did is a coverage claim the scan never made.
4673+
"""
4674+
lines, scanned = _aggregate_gnmap(result_dir)
4675+
try:
4676+
# Only banner-pass outputs have a .gnmap sibling to be missing.
4677+
# _scan_extra_sql_ports() writes port{N}_sql.xml into this same
4678+
# directory with no -oG, so counting it here would warn about a gap
4679+
# that does not exist.
4680+
xml_stems = {f[:-len('.xml')] for f in os.listdir(result_dir)
4681+
if f.startswith('port') and f.endswith('.xml')
4682+
and not f.endswith('_sql.xml')}
4683+
except OSError:
4684+
xml_stems = set()
4685+
missing = sorted(xml_stems - scanned)
4686+
if not lines and not missing:
4687+
return
4688+
body = ('# SpooNMAP aggregated greppable output\n'
4689+
+ ''.join(f'{line}\n' for line in lines)
4690+
+ f'# SpooNMAP done: {len(lines)} hosts\n')
4691+
if _write_artifact(f'{output_path}/spoonmap_output.gnmap', body):
4692+
print(_COLOR_RESULT + f'\nGreppable output written to '
4693+
f'{output_path}/spoonmap_output.gnmap' + _COLOR_RESET)
4694+
if missing:
4695+
print(_COLOR_ERROR
4696+
+ f'Warning: {len(missing)} port(s) have nmap XML but no greppable '
4697+
+ 'output and are absent from spoonmap_output.gnmap: '
4698+
+ ', '.join(missing[:5])
4699+
+ (' ...' if len(missing) > 5 else '')
4700+
+ '. Re-run those ports, or use spoonmap_output.xml, which covers them.'
4701+
+ _COLOR_RESET)
4702+
4703+
45634704
def _read_config_file(path):
45644705
"""Parse *path* as a JSON object, or print a clear diagnostic and exit.
45654706
@@ -5702,6 +5843,11 @@ def main(): # pragma: no cover -- interactive CLI entry point; orchestrates
57025843
hosts_json, xml_hosts = _aggregate_result_dir(result_dir, ip_to_hostname)
57035844
_write_combined_results(output_path, hosts_json, xml_hosts)
57045845

5846+
# Greppable output only exists for the nmap banner pass; a
5847+
# masscan-only run has no -oG files to merge.
5848+
if banner_scan or script_scan:
5849+
_write_gnmap_result(output_path, result_dir)
5850+
57055851
_combine_live_hosts(disc, output_path)
57065852

57075853
if script_scan:

0 commit comments

Comments
 (0)