Prompt: Enterprise-Grade Real-Time Data Pipeline Architecture

🎭 Role

You are a Senior Distributed Systems Architect with extensive experience in high-throughput event-driven architectures. You specialize in designing resilient, scalable, and observable data pipelines using the Apache Kafka ecosystem.

🌐 Context

[SCENARIO: e.g., We are building a high-traffic e-commerce clickstream analytics platform that processes millions of events per hour.] The system must ensure data integrity, low-latency processing, and operational observability. You are tasked with providing a technical blueprint for a robust data pipeline that handles events from ingestion to long-term storage in [STORAGE_SYSTEM: e.g., Elasticsearch].

🛠️ Task Instruction

Design a comprehensive architecture for a real-time pipeline consisting of the following modules:

1. Producer Implementation: Define the structure of the JSON clickstream events and best practices for asynchronous production.
2. Kafka Topology: Design the topic configuration with a partition strategy for [NUMBER_OF_PARTITIONS] partitions to ensure horizontal scalability.
3. Consumer Logic: Outline a consumer group strategy that guarantees parallel processing without data duplication.
4. Stream Processing: Provide a conceptual Kafka Streams logic for performing [AGGREGATION_TYPE: e.g., rolling window counts or sessionization].
5. Data Sink: Describe the integration of the Kafka Connect framework to reliably flush data into [STORAGE_SYSTEM].

Technical Constraints & Requirements

• Reliability: Implement "Exactly-Once Semantics" (EOS) across the pipeline.
• Error Handling: Outline a Dead Letter Queue (DLQ) strategy for handling malformed events or processing failures.
• Observability: Detail the key Kafka lag metrics (e.g., consumer lag, partition offset) that must be monitored for health alerts.
• Performance: Optimize for high throughput while maintaining sub-second latency.
• Tone: Professional, technical, and prescriptive. Avoid fluff; focus on architectural best practices.

📝 Output Format

Please structure your response using the following headers:

• Architectural Overview: A high-level summary of the data flow.**

• Component Configuration: Technical specifications for the Producers, Kafka Cluster, and Consumers.**

• Stream Processing & Logic: A high-level pseudocode or logical flow for Kafka Streams.**

• Reliability & Error Handling: Strategies for DLQs and Exactly-Once delivery.**

• Monitoring & Observability: Critical metrics to track in Prometheus/Grafana or your preferred monitoring stack.**

• Security Considerations: A brief note on encryption and authentication (SASL/SCRAM or mTLS).**

Constraints

• Do not write generic introductory text.
• Limit the response to 1500 words.
• If a trade-off is required (e.g., latency vs. consistency), clearly state the compromise.

How to use this:

2. Paste it into your AI assistant.
3. Replace the bracketed variables (e.g., [SCENARIO], [STORAGE_SYSTEM]) with your specific project details before hitting send.