Azure Data Engineer

Responsibilities:

Employ batch and streaming data processing methodologies, leveraging Kafka and Azure Function technologies.
Design and manage databases on Azure including but not limited to Azure SQL Database, Postgres, MongoDB, MySQL, and MariaDB.
Create end-to-end data pipelines in databricks and ADF, integrating data from source systems to target data repositories like Snowflake and Azure Databases.
Contribute to the evolution and optimization of our data model.
Collaborate with the data team to design and implement data models that serve business requirements, ensuring data integrity and availability.
Leverage Azure Functions and App Service for serverless data operations and tasks.
Utilize Azure API Management (APIM) to facilitate data interaction between various services and applications.
Utilize DevOps principles for deployment and automation using Azure DevOps or Bitbucket.
Create and maintain CI/CD pipelines using BitBucket or Azure DevOps for automated deployment and data transformations.
Ensure data quality, consistency, and security throughout the data pipelines.
Implement monitoring, logging, and alerting tools such as New Relic to ensure the robustness and performance of data operations.

Requirements:

Bachelors or Masters degree in Computer Science, Engineering, or a related field.
Minimum of 5 years of experience working with Azure Cloud services.
In-depth experience with Azure Data Lake Storage Azure Data Factory and ETL/ELT processes.
Extensive experience with Azure Cloud services including Azure Databricks, Azure Data Factory , Azure Synapse, Azure Data Lake Storage , Azure Analysis Services, HDInsight, Power BI, SSIS, and Event Hub.
Familiarity with real-time data processing and streaming technologies such as Kafka.
Strong experience with Azure Functions, Azure App Service, and Azure API Management.

Key Skills: