ETL Developer

 Responsibilities:

• Design, develop, and implement ETL workflows, data models and pipelines using PySpark to meet business requirements
• Extract data from multiple sources, transform it for consistency, and load it into data lake/repository.
• Optimize PySpark scripts for performance, scalability, and efficiency, ensuring minimal resource consumption.
• Ensure data accuracy and reliability through quality checks, error handling, and validations within ETL pipelines.
• Monitor, troubleshoot, and enhance ETL pipeline performance.
• Automate ETL processes and integrate with scheduling tools like Apache Airflow, AWS Eventbridge, or similar.
• Manage and optimize data storage, compute resources, and security configurations on cloud platforms.
• Develop and maintain technical documentation for ETL processes, workflows, and data definitions.

Key Skills:

• ETL tools.
• Data warehouse.
• ETL processes.
• Scripting language.
• SQL.
• Data modeling.
• Data quality.
• BI.
• Problem-solving abilities.