Hadoop Developer

Key responsibilities :

• Designing and developing scalable, efficient, and reliable data processing pipelines using Hadoop and related technologies such as MapReduce and Hive.
• Writing and executing MapReduce jobs to process large datasets stored in Hadoop Distributed File System (HDFS).
• Collaborating with stakeholders to understand their data processing requirements and develop solutions that meet their needs.
• Integrating Hadoop with other data storage and processing technologies, such as NoSQL databases and data warehouses.
• Developing and maintaining data processing workflows using Apache Oozie or similar workflow management tools.
• Debugging and optimizing Hadoop applications to improve performance and scalability.
• Ensuring data quality and security through proper data validation, cleansing, and encryption.
• Writing and maintaining technical documentation for Hadoop solutions.
• Staying up-to-date with the latest developments in big data processing and related technologies.
• Performing other tasks as assigned by management.Qualifications:
• Strong programming skills in Java and/or Python.
• Experience with MapReduce programming, Hive and Pig scripting, and Hadoop Distributed File System (HDFS).
• Knowledge of NoSQL databases, such as HBase, Cassandra, and MongoDB.
• Experience with data warehousing, ETL, and data integration technologies.
• Strong analytical and problem-solving skills, with the ability to debug and optimize complex data processing pipelines.
• Good communication and collaboration skills, with the ability to work effectively with cross-functional teams.
• Familiarity with Agile software development methodologies.
• Experience with cloud computing platforms, such as Amazon Web Services (AWS) and Microsoft Azure, is an advantage.

Key skills:

• hive
• spark
• Hadoop
• SQL