Azure Data Engineer

Responsibilities:

• Employ batch and streaming data processing methodologies, leveraging Kafka and Azure Function technologies.
• Design and manage databases on Azure including but not limited to Azure SQL Database, Postgres, MongoDB, MySQL, and MariaDB.
• Create end-to-end data pipelines in databricks and ADF, integrating data from source systems to target data repositories like Snowflake and Azure Databases.
• Contribute to the evolution and optimization of our data model.
• Collaborate with the data team to design and implement data models that serve business requirements, ensuring data integrity and availability.
• Leverage Azure Functions and App Service for serverless data operations and tasks.
• Utilize Azure API Management (APIM) to facilitate data interaction between various services and applications.
• Utilize DevOps principles for deployment and automation using Azure DevOps or Bitbucket.
• Create and maintain CI/CD pipelines using BitBucket or Azure DevOps for automated deployment and data transformations.
• Ensure data quality, consistency, and security throughout the data pipelines.
• Implement monitoring, logging, and alerting tools such as New Relic to ensure the robustness and performance of data operations.

Requirements:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• Minimum of 5 years of experience working with Azure Cloud services.
• In-depth experience with Azure Data Lake Storage Azure Data Factory and ETL/ELT processes.
• Extensive experience with Azure Cloud services including Azure Databricks, Azure Data Factory , Azure Synapse, Azure Data Lake Storage , Azure Analysis Services, HDInsight, Power BI, SSIS, and Event Hub.
• Familiarity with real-time data processing and streaming technologies such as Kafka.
• Strong experience with Azure Functions, Azure App Service, and Azure API Management.

Key Skills:

• Data Modeling.
• Programming skills. 
• Data Warehousing.
• ETL. 
• Cloud Computing. 
• Azure Services. 
• Data Analytics and Visualization tools.