Job description

Microsoft Azure~Digital 6-9 yrs Data Engineer - Databricks - Must have at least 8+ years of extensive Data Engineering Experience in Databricks, ETL/ELT using Data pipelines, SQL/Procedure experience. - Must have 8+ Years of hands-on development proficiency in implementing Databricks solutions using Scala and Spark with Data frames and Notebooks/SQL. - Must have 8+ Years of hands-on development and performance tuning/enhancement experience in Scala/PySpark and SQL. - Must have 8+ Years of hands-on development experience in working on Azure Cloud. - Must have strong experience with Databricks, including developing and optimizing Spark jobs, data transformations, and data processing workflows. - Must have experience and a good understanding on the below topics -- Databricks Delta Lake storage -- Unity Catalog -- Autoloader -- Lakehouse Architecture -- Medallion Architecture -- Databricks Serverless Options. -- Dataset and Data frame -- Databricks Clusters Configuration and Sizing -- Databricks Job Orchestration -- Spark Structured Streaming - Must have experience with complex and large volume data streaming and data transformations using Azure Databricks. - Must have extensive experience in Datawarehouse/Data Lakehouse implementations. - Must have experience with monitoring, debugging, and resolving issues in live Databricks jobs. - Must have strong hands-on expertise in troubleshooting DevOps pipelines, Azure, and AKS services - The candidate should be able to provide Data Ingestion, Streaming and Transformations solutions and do development work. - Experience in Apache Big Data implementations with programming proficiency like PySpark, R or Java prior to Databricks is advantageous. - Good to have experience with containerization technologies such as Docker and Kubernetes. - Strong critical thinking, communication, and problem-solving skills is essential.

Requirements