Senior Data Engineer

Responsibilities:

• Plan, create, and maintain data architectures, ensuring alignment with business requirements.
• Obtain data, formulate dataset processes, and store optimized data.
• Identify problems and inefficiencies and apply solutions.
• Determine tasks where manual participation can be eliminated with automation.
• Identify and optimize data bottlenecks, leveraging automation where possible.
• Create, maintain, and manage ETL/ELT pipelines.
• Create, maintain, and manage data transformations.
• Create, maintain, and manage data pipeline schedules.
• Create, maintain, and manage data quality gates  to ensure high data quality.
• Support AI/ML teams with optimizing feature engineering code
• Create, maintain, and manage Spark Structured Steaming jobs, including using the newer Delta Live Tables and/or DBT.
• Research existing data in the data lake to determine best sources for data.

Key Skills:

• Strong command of relational databases and SQL.
• .Transform, and Load  data into a relational database.
• Proficiency with Python or R,
• Data manipulation and analysis,