Azure Data Engineer

 Responsibilities:

• Expert in Azure ADF, Databricks pipelines, functions, ADLS(Gen2) and Azure storage explorer.
• Strong experience working on Pyspark
• Ability to write data frames to implement business logic in notebook in Databricks
• Experience working on financial reporting
• Performance tuning of Spark notebooks
• Designing and implementing scalable and secure data processing pipelines using Azure Data Factory, Azure Databricks, and other Azure services. Managing and optimizing data storage using Azure Data Lake Storage, Azure SQL Data Warehouse, and Azure Cosmos DB.

Requirement:

• 5-6 years of experience in data warehousing systems including both traditional data warehouse and data lake is required.
• 3-4 years of implementing ETL in Azure cloud using Data factory and Databricks is a must. Ability to design and implement ETL pipeline using Databricks, Azure ADF ETL pipeline is required.
• Implementation experience of Pyspark programming and performance tuning is required.

Key Skills:

• Data warehouse.
• ETL.
• Data modeling.
• Big data.
• Cloud computing.
• NoSQL.
• Database.
• Machine learning.
• SQL.