Job description

Key Responsibilities : • Azure Databricks Architecture & Design : • Design, build, and optimize scalable, high-performance data pipelines and data lakes using Azure Databricks. • Architect and implement end-to-end analytics solutions leveraging Databricks and other Azure services (e.g., Azure Data Lake, Azure SQL, Azure Blob Storage). • Lead the design of cloud-based architectures using Azure Databricks for data processing, transformation, and reporting. • Data Engineering & Integration : • Design and implement ETL/ELT processes to ingest, process, and transform data across various sources (structured and unstructured). • Collaborate with data scientists, analysts, and other stakeholders to understand business requirements and develop data solutions. • Manage data integration workflows between Databricks and other platforms like Power BI, Azure SQL, Synapse, etc. • Optimization & Performance Tuning : • Identify performance bottlenecks in Databricks environments and optimize clusters, queries, and code. • Continuously improve and scale data pipelines to accommodate growing data volumes and business needs. • Collaboration & Leadership : • Mentor and guide junior engineers and team members in best practices related to Azure Databricks and data engineering. • Work closely with cross-functional teams (data science, analytics, business intelligence) to deliver integrated solutions. • Provide technical leadership in cloud data architecture and data engineering. • Monitoring & Security : • Implement monitoring and alerting systems to ensure data pipelines and workflows are running smoothly. • Ensure compliance with data security and governance standards across the data ecosystem. • Continuous Improvement & Innovation : • Stay up to date with the latest advancements in Azure Databricks, cloud technologies, and data engineering trends. • Continuously evaluate and introduce new technologies and methodologies to enhance the data platform. Required Skills and Qualifications : • Experience : • Minimum 10 years of hands-on experience in data engineering and cloud-based data platforms. • At least 5 years of experience working with Azure Databricks, building data pipelines, and performing data engineering tasks. • Strong experience with Azure services such as Azure Data Lake, Azure Synapse Analytics, Azure Blob Storage, Azure SQL Database, etc. • Technical Skills : • Proficiency in Spark, PySpark, Scala, or SQL for large-scale data processing. • Expertise in Databricks notebooks, clusters, job scheduling, and libraries. • Deep understanding of cloud-native data engineering practices and architecture on Azure. • Familiarity with data modeling, data lakes, and data warehouse concepts. • Expertise in Azure services: Data Factory, Data Lake, Synapse Analytics, Event Hubs, Cosmos DB. • Data Pipeline and ETL/ELT Development : • Experience building and optimizing ETL/ELT workflows on Databricks. • Knowledge of data orchestration tools such as Azure Data Factory, Apache Airflow, or similar. • Big Data Technologies : • Proficiency in handling large-scale distributed data processing with tools like Apache Spark. • Experience with technologies such as Kafka, Delta Lake, and Databricks Runtime. • Strong understanding of Delta Lake and Lakehouse architecture . • Programming Languages : • Strong programming skills in Python, Scala, or Java. • Experience with SQL-based querying and optimizations. • Cloud & DevOps : • Strong understanding of Azure cloud services and architecture. • Familiarity with DevOps principles, CI/CD pipelines, and automation tools. Terraform required. • Knowledge of Git, Azure DevOps, or similar version control and deployment systems. • Collaboration & Leadership : • Excellent communication skills to work with both technical and non-technical stakeholders. • Proven track record of mentoring and leading teams of data engineers. • Experience managing complex projects and working with cross-functional teams. Preferred Qualifications : • Certifications : • Microsoft Certified: Azure Data Engineer Associate or equivalent certification. • Databricks Certified Associate Developer for Apache Spark. • Education : • Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.

Requirements