Refactor analysis views and enhance task handling

[doomedraven]

Performance Optimization & Stability Summary

This update addresses critical performance bottlenecks and infrastructure hangs in the CAPEv2 web interface, specifically targeting uWSGI HARAKIRI timeouts and database N+1 query problems.

---

1. Core Performance Improvements

• Resolved N+1 Query Patterns:
  • Analysis Index: Updated all task listing logic to use SQL JOINs (include_hashes=True), fetching task and sample data in a single round-trip instead of 100+ individual queries.
  • Dashboard: Replaced multiple individual task counts with a single efficient SQL GROUP BY query.
  • APIv2: Applied similar pre-fetching optimizations to tasks_list and tasks_search endpoints.
• Bulk MongoDB Fetching: Implemented a bulk metadata fetch on the main analysis page, consolidating up to 100 individual MongoDB lookups into one single query.
• High-Speed Report Loading:
  • Optimized the individual report view to use an extremely narrow MongoDB projection for the initial load, deferring heavy data (Sigma, Sysmon, Suricata) to AJAX-loaded tabs.
  • Implemented a 10-second hard timeout (max_time_ms) for MongoDB queries to prevent workers from hanging on slow lookups.

2. Stability & Memory Safety

• Fixed Critical Infinite Loop: Resolved a generator bug in the denormalize_files MongoDB hook that caused uWSGI workers to loop indefinitely when loading certain reports.
• Resource Capping:
  • Noisy Signatures: Capped signature processing at 1,000 matches per signature to prevent CPU exhaustion.
  • Massive Logs: Implemented a 1MB safety cap on process.log reading to prevent memory-related crashes.
  • Network & BinGraphs: Capped DNS/domain lookups at 1,000 entries and binary graphs at 10 files (max 512KB each).
• Hardened Database Layer:
  • Implemented Lazy Initialization for MongoDB connections to ensure fork-safety within uWSGI.
  • Added strict connection (5s) and socket (30s) timeouts to the MongoDB client.

3. Infrastructure Enhancements

• SQL Lock Prevention: Removed/Optimized DBTransactionMiddleware interactions to ensure that slow MongoDB queries do not leave "zombie" SQL locks that hang the entire web UI.
• Improved Logging: Hardened the internal logging mechanisms to ensure critical errors are reliably captured in uWSGI logs via sys.stderr.

---

Impact

• Database Round-trips: Reduced by ~90% on the main index and report pages.
• Page Load Speed: Significantly faster initial "Time to First Byte" (TTFB) for reports.
• System Reliability: Eliminates the primary causes of signal 9 kills and 502 Bad Gateway errors in large-scale installations.

[gemini-code-assist]

Code Review

This pull request optimizes the index view by batch-fetching MongoDB analysis data for multiple tasks and refactoring get_analysis_info to utilize pre-fetched data and existing task relationships. It also adds a safety check in filtered_chunk to handle missing call records. Feedback identifies a redundant hasattr check in the sample retrieval logic and a potential IndexError in the Elasticsearch query handling that should be addressed to prevent runtime crashes.