When automating data movement between systems that currently relies on manual exports or scripts with no error handling.
You are a senior {{role}} brought in to help a developer or tech professional complete a {{use_case}} task. # Context - Pack: Developers & Tech Professionals - Category: Databases & Data Engineering - Use case: ETL Pipeline Builder - Source task: - Build an ETL pipeline that: extracts data from {{source_database_csv_api_s3}}, transforms it by {{describe_transformations_cleaning_aggregation_joins_type_casting}}, and loads it into {{destination_data_warehouse_database_file}}. - Step 1: pipeline architecture and tool choice (Python/Pandas, Apache Spark, dbt, Airflow : recommend for the data volume). - Step 2: complete Python or SQL code for the extract, transform, and load stages. - Step 3: error handling, logging, and idempotency design : how to safely re-run without duplicating data. # Goal Architecture recommendation, full ETL code, error handling strategy, and idempotency design to prevent duplicate data on re-runs. # Constraints - Think like an expert advisor before writing the final output. - Ask clarifying questions only if missing information would materially change the result. - Avoid generic filler, vague advice, and unsupported claims. - Make the output specific, practical, and ready to use. # Output Architecture recommendation, full ETL code, error handling strategy, and idempotency design to prevent duplicate data on re-runs.
{{double-curly}} with your real context.When automating data movement between systems that currently relies on manual exports or scripts with no error handling.
Design every ETL job to be idempotent — it should produce the same result whether run once or ten times.
Debug this problem systematically. Identify the root cause, explain why it is happening, provide the fix, and explain how to prevent it in future.
Design the high-level architecture for this system. Cover components, data flow, scaling strategy, and key design decisions.
Recommend the best no-code or low-code tool stack for the stated goal, with implementation guidance.
Design the complete analysis approach for the stated question. Include the analytical method, the steps to execute it, and the format for presenting findings.