Job description

Job Description At VDart Inc, we are seeking a seasoned Azure Data Engineer to join our team. We offer a competitive salary and a comprehensive benefits package. About the Role: The ideal candidate will have extensive experience in designing, building, and maintaining large-scale data pipelines using Databricks. You will work closely with cross-functional teams to deliver tailored data solutions that meet business requirements. Key Responsibilities: • Data Pipeline Development: • Develop and maintain scalable ETL/ELT pipelines using Databricks. • Leverage PySpark/Spark and SQL to transform and process large datasets. • Integrate data from multiple sources including Azure Blob Storage, ADLS, and other relational/non-relational systems. • Collaboration & Analysis: • Collaborate with multiple teams to prepare data for dashboard and BI tools. • Work with cross-functional teams to understand business requirements and deliver tailored data solutions. • Performance & Optimization: • Optimize Databricks workloads for cost efficiency and performance. • Monitor and troubleshoot data pipelines to ensure reliability and accuracy. • Governance & Security: • Implement and manage data security, access controls, and governance standards using Unity Catalog. • Ensure compliance with organizational and regulatory data policies. • Deployment: • Leverage Databricks Asset Bundles for seamless deployment of Databricks jobs, notebooks, and configurations across environments. • Manage version control for Databricks artifacts and collaborate with the team to maintain development best practices.

Requirements