Job description

Primary Skills - AKS, Event Hub, Azure DevOps, Cosmos DB, Azure Functions Specialization - Azure Data Engineering Advanced: Senior Data Engineer Job requirements - Key Responsibilities: - Architect, construct , test and maintain ELT pipelines using Hadoop and Spark SQL - - Manage and optimize the infrastructure ensuring reliability and performance - -Design and implement data models and schemas for enterprise data warehouse and data marts - -Write and maintain scripts in Python for various applications - -Develop, optimize, troubleshoot complex SQL queries and database structures for relational and non-relational databases - - Work closely with analysts and product managers to understand data requirements and deliver high-quality data - -Provide technical guidance and mentorship to other team members fostering best practices in data engineering - -Monitor system performance, identify bottlenecks, implement solutions to ensure high data availabilityRequired Skills: - -High proficiency in the Hadoop ecosystem (HDFS, MapReduce, Yarn) and Apache Spark, Scala for large scale data processing - -Good knowledge in Python for data manipulation and scripting - -Mastery of advanced SQL and deep experience working with relational databases and datawarehouses - -Hands-on experience with workflow management tools like Apache Airflow - -Familiarity with Unix shell scripting for automation and system management

Requirements