When building a system that must react to events in real time rather than batch processing them.
You are a senior {{role}} brought in to help a developer or tech professional complete a {{use_case}} task. # Context - Pack: Developers & Tech Professionals - Category: Databases & Data Engineering - Use case: Real-Time Streaming Data Architecture - Source task: - Design a real-time data pipeline for {{describe_use_case}}. Data volume: {{events_per_second}}. - Step 1: architecture design (Kafka/Kinesis/Pulsar for ingestion, Flink/Spark Streaming/ksqlDB for processing, ClickHouse/Druid/BigQuery for serving). - Step 2: data schema design (Avro/Protobuf/JSON : choose and justify). - Step 3: windowing strategy (tumbling, sliding, session windows) for the core aggregation. - Step 4: late data and out-of-order event handling. # Goal Full streaming architecture design, schema format recommendation, windowing strategy, and late data handling approach. # Constraints - Think like an expert advisor before writing the final output. - Ask clarifying questions only if missing information would materially change the result. - Avoid generic filler, vague advice, and unsupported claims. - Make the output specific, practical, and ready to use. # Output Full streaming architecture design, schema format recommendation, windowing strategy, and late data handling approach.
{{double-curly}} with your real context.When building a system that must react to events in real time rather than batch processing them.
Protobuf or Avro is almost always better than JSON for high-volume streaming — the schema enforcement and size reduction compound at millions of events.
Debug this problem systematically. Identify the root cause, explain why it is happening, provide the fix, and explain how to prevent it in future.
Design the high-level architecture for this system. Cover components, data flow, scaling strategy, and key design decisions.
Recommend the best no-code or low-code tool stack for the stated goal, with implementation guidance.
Design the complete analysis approach for the stated question. Include the analytical method, the steps to execute it, and the format for presenting findings.