Log management ELK stack centralized logging

🎭 Role

Act as a Senior Site Reliability Engineer (SRE) and Infrastructure Architect with deep expertise in observability, distributed systems, and the Elastic Stack (ELK). Your goal is to design a robust, scalable, and secure centralized logging architecture that adheres to industry best practices for high-availability environments.

🌐 Context

We are implementing a centralized logging strategy for a distributed [ENVIRONMENT_TYPE, e.g., Kubernetes/Cloud-Native] architecture to streamline troubleshooting, security auditing, and performance monitoring. The solution must support high-volume data ingestion while maintaining strict retention and security compliance.

🛠️ Task Instruction

Design a comprehensive technical blueprint for the ELK Stack implementation by addressing the following modules:

1. Architecture Design: Propose a [CLUSTER_SIZE]-node Elasticsearch cluster topology with high availability. Define the role of Beats for data shipping and the pipeline configuration for Logstash.
2. Log Ingestion Strategy: Define a schema for application logs (JSON-based with correlation IDs) and system/infrastructure logs. Outline the ingestion pipeline filters for parsing and enrichment.
3. Index Lifecycle Management (ILM): Create an ILM strategy including rollover conditions (e.g., [SIZE_LIMIT] or [TIME_LIMIT]) and a tiered storage strategy:
   • Hot: [DURATION]
   • Warm: [DURATION]
   • Cold: [DURATION]
   • Retention: Delete after [MAX_RETENTION_PERIOD].
4. Security & Compliance: Detail the implementation of X-Pack security features, specifically Role-Based Access Control (RBAC), TLS encryption for data in transit (mTLS), and at-rest encryption.
5. Observability & Alerting: Define key performance indicators (KPIs) for the cluster (indexing rate, latency) and configure Watcher-based alerts for critical thresholds, integrated with [NOTIFICATION_CHANNELS, e.g., Slack/PagerDuty].

⚖️ Constraints & Tone

• Tone: Professional, technical, authoritative, and concise.
• Length: Keep explanations focused; prioritize actionable technical implementation details over high-level concepts.
• Avoid: Do not include fluff, generic marketing language, or redundant descriptions of basic ELK functionality. Focus on configuration, architectural trade-offs, and best practices.

📝 Output Format

Structure the response using the following hierarchy:

1. Executive Summary: A brief overview of the proposed architecture.
2. Component Configuration: Bulleted breakdown of Elasticsearch, Logstash, and Beats requirements.
3. Data Lifecycle Matrix: A table outlining the ILM phases and retention strategy.
4. Security Hardening Checklist: Step-by-step security implementation requirements.
5. Monitoring & Incident Response Plan: Defined metrics and specific alert triggers for the SRE team.

🧩 Variables

• [ENVIRONMENT_TYPE]:
• [CLUSTER_SIZE]:
• [SIZE_LIMIT]:
• [TIME_LIMIT]:
• [DURATION]:
• [MAX_RETENTION_PERIOD]:
• [NOTIFICATION_CHANNELS]: