Job description

Job Title: Azure Databricks Lead Location: Erlanger, Kentucky – Onsite Job Type: Contract Mandatory Skills : Java,Python,Scala,Snowflake,Azure BLOB,Azure Data Factory,Azure Functions,Azure SQL,Azure Synapse Analytics,AZURE DATA LAKE,ANSI-SQL,Databricks,HDInsight Key Responsibilities • Python Programming Lead the development of robust scalable and efficient Python code for data processing transformation and analysis within the Databricks environment Ensure adherence to coding standards quality and performance best practices • Azure Databricks Administration Oversee Databricks platform configuration resource management cluster optimization and monitoring to ensure high availability and performance Implement best practices for managing Databricks workspaces libraries and notebooks • CICD for Databricks Artifacts Design and implement CICD pipelines for automating the deployment of Databricks artifacts such as notebooks libraries jobs and Delta tables Use tools such as Azure DevOps GitHub Actions or Jenkins to streamline the deployment process • Delta Live Tables Design and manage Delta Live Tables pipelines for real time data processing ensuring data quality reliability and optimal performance in the processing layers • Auto Loader Implement and optimize Auto Loader for scalable fault tolerant ingestion of data from cloud storage sources into Databricks • Unity Catalog Manage and configure Unity Catalog for centralized governance of data across workspaces ensuring proper access controls data lineage and compliance • Databricks Asset Bundles Leverage Databricks Asset Bundles to manage and share reusable components including notebooks libraries and ML models across teams or projects Required Qualifications: Experience in Python Programming: • Minimum of 5 years of experience in Python programming with a focus on data engineering and data science workflows • Expertise in designing and building scalable Python based ETL pipelines using libraries such as Pandas NumPy PySpark and requests • Proficiency in debugging optimizing and maintaining Python code to ensure high performance and reliability in a distributed environment Extensive Experience with Azure Databricks: • Minimum of 3 years of hands on experience working with Azure Databricks in a production environment • Strong knowledge of Databricks clusters workspaces and the Databricks runtime • Ability to optimize and troubleshoot Spark based jobs and notebooks ensuring performance and costefficiency in cloud environments CICD for Databricks Artifacts : • Proven experience in setting up and managing Continuous Integration Continuous Deployment CICD pipelines for Databricks artifacts notebooks libraries and jobs • Familiarity with version control systems Git and experience with tools such as Azure DevOps GitHub Actions or Jenkins for automated deployment • Experience with integrating Databricks with external version control repositories eg GitHub Azure Repos for collaboration and artifact management Delta Live Tables : • Strong experience in designing managing and optimizing Delta Live Tables DLT pipelines to support realtime reliable and scalable data processing • Ability to handle complex data transformations ensure data quality and monitor pipeline performance within the Delta Lake architecture • Expertise in implementing automated data quality checks and validation within Delta Live Tables pipelines Auto Loader • Practical experience with the Auto Loader feature in Databricks for realtime faulttolerant and scalable data ingestion from cloud storage eg Azure Blob Storage ADLS Gen2 • Knowledge of best practices for optimizing Auto Loader performance and ensuring minimal latency in largescale data ingestion tasks • Unity Catalog for Data Governance • Indepth experience in implementing and managing Unity Catalog for central data governance in Databrick

Requirements