Hello everyone,
I’m developing an enterprise cybersecurity project focused on Internet-wide scanning, similar to Shodan or Censys, aimed at mapping exposed infrastructure (services, ports, domains, certificates, ICS/SCADA, etc). The data collection is continuous, and the system needs to support an average of 1TB of ingestion per day.
I recently started implementing OpenSearch as the fast indexing layer for direct search. The idea is to use it for simple and efficient queries, with data organized approximately as follows:
IP → identified ports and services, banners (HTTP, TLS, SSH), status
Domain → resolved IPs, TLS status, DNS records
Port → listening services and fingerprints
Cert_sha256 → list of hosts sharing the same certificate
Entity correlation will be handled by a graph engine (TigerGraph), and raw/historical data will be stored in a data lake using Ceph.
What I would like to better understand:
- OpenSearch cluster sizing
- How can I estimate the number of data nodes required for a projected volume of, for example, 100 TB of useful data?
- What is the real overhead to consider (indices, replicas, mappings, etc)?
- Hardware recommendations
- What are the ideal CPU, RAM, and storage configurations per node for ingestion and search workloads?
- Are SSD/NVMe mandatory for hot nodes, or is it possible to combine with magnetic disks in different tiers?
- Best practices to scale from the start
- What optimizations should I apply to mappings and ingestion early in the project?
Thanks in advance.