Job description

Job description Responsibilities • Lead the architecture design and implementation of advanced analytics solutions using Azure Databricks Fabric The ideal candidate will have a deep understanding of big data technologies data engineering and cloud computing with a strong focus on Azure Databricks along with Strong SQL • Work closely with business stakeholders and other IT teams to understand requirements and deliver effective solutions • Oversee the endtoend implementation of data solutions ensuring alignment with business requirements and best practices • Lead the development of data pipelines and ETL processes using Azure Databricks PySpark and other relevant tools • Integrate Azure Databricks with other Azure services eg Azure Data Lake Azure Synapse Azure Data Factory and onpremise systems • Provide technical leadership and mentorship to the data engineering team fostering a culture of continuous learning and improvement • Ensure proper documentation of architecture processes and data flows while ensuring compliance with security and governance standards • Ensure best practices are followed in terms of code quality data security and scalability • Stay updated with the latest developments in Databricks and associated technologies to drive innovation Essential Skills • Strong experience with Azure Databricks including cluster management notebook development and Delta Lake • Proficiency in big data technologies eg Hadoop Spark and data processing frameworks eg PySpark • Deep understanding of Azure services like Azure Data Lake Azure Synapse and Azure Data Factory • Experience with ETLELT processes data warehousing and building data lakes • Strong SQL skills and familiarity with NoSQL databases • Experience with CICD pipelines and version control systems like Git • Knowledge of cloud security best practices Soft Skills • Excellent communication skills with the ability to explain complex technical concepts to nontechnical stakeholders • Strong problemsolving skills and a proactive approach to identifying and resolving issues • Leadership skills with the ability to manage and mentor a team of data engineers Nice to have Skills • Power BI for dashboarding and reporting • Microsoft Fabric for analytics and integration tasks • Spark Streaming for processing realtime data streams • Familiarity with Azure Resource Manager ARM templates for infrastructure as code IaC practices Experience • Demonstrated expertise of 8 years in developing data ingestion and transformation pipelines using DatabricksSynapse notebooks and Azure Data Factory • Solid understanding and handson experience with Delta tables Delta Lake and Azure Data Lake Storage Gen2 • Experience in efficiently using Auto Loader and Delta Live tables for seamless data ingestion and transformation • Proficiency in building and optimizing query layers using Databricks SQL • Demonstrated experience integrating Databricks with Azure Synapse ADLS Gen2 and Power BI for endtoend analytics solutions • Prior experience in developing optimizing and deploying Power BI reports • Familiarity with modern CICD practices especially in the context of Databricks and cloudnative solutions Skills Mandatory Skills : Azure Cloud Architecture, Cloud Architecture, Cloud Solution Architecture, Enterprise Cloud Architecture

Requirements