Big Data Engineer

Responsibilities:

• Develop solutions related to Big Data, and Data Sciences from end-to-end.
• Develop and maintain scalable data pipelines that will ingest, transform, and distribute data streams and/or batches within the AWS, Snowflake and Microsoft Platforms.
• Identify, design, and implement process improvements for automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability.
• Self-motivated, self-directed with strong problem-solving skills.
• Support and decommission legacy platforms.
• Enable business strategy, lean processes, increased data velocity, and insights.
• Embody a culture of continuous innovation and learning.
• Adhere to programming/development standards and governance framework.
• Collaborate with business, analytical teams, and data scientist to improve efficiency, increase the applicability of predictive models, and help translate ad-hoc analyses into scalable data delivery solutions.
• Consulting on data ingestion, data modeling, security, and capabilities.
• Manage the innovation cycle of conducting analyses, generating insights.
• Assist with the selection and management of consultants and vendors.
• Assist with the recruitment and development of talent.
• Collaborate with DevOps team to integrate innovations and algorithms into a production system.
• Support business decisions with ad hoc analysis as needed.
• Work with the DevOps team to create and manage deployment workflows for all scripts and code using Microsoft Azure.

Key Skills:

• Databases and SQL.
• ETL and data warehousing.
• Talend, IBM DataStage, Pentaho, and Informatica.
• Operating system knowledge for Unix, Linux, Windows, and Solaris.
• Hadoop.
• Apache Spark.
• Data mining and modeling.