Job description

Note: The job is a remote job and is open to candidates in USA. Dice is the leading career destination for tech experts at every stage of their careers. They are seeking a highly skilled Data Engineer with deep expertise in Databricks and Azure Cloud to join a decision analytics and data engineering team within one of their global consulting clients. Responsibilities • Design, build, and enhance ETL/ELT data pipelines using Azure Data Factory, Databricks (PySpark, SQL, Python), and related services. • Develop and manage Delta Live Tables, Autoloader, and Unity Catalog within the Databricks ecosystem for structured, incremental data processing. • Implement data ingestion, transformation, and validation frameworks that ensure high performance, scalability, and reliability. • Monitor data pipelines, troubleshoot issues, and ensure optimal system performance and SLA adherence. • Collaborate with business analysts and reporting teams to define logical and physical data models supporting analytical and operational needs. • Implement data warehousing and lakehouse solutions using Azure Data Lake and Delta Lake. • Optimize data structures for query performance, cost efficiency, and reusability. • Design and implement robust data quality checks and validation mechanisms to maintain integrity across sources and transformations. • Automate repetitive processes using scripts, parameterized pipelines, and reusable frameworks. • Conduct periodic audits and compliance checks aligned with governance policies. • Contribute to metadata management, documentation, and lineage tracking. Skills • 7 12 years of experience in Data Engineering with proven expertise in Databricks and Azure Cloud ecosystems. • Strong hands-on experience in PySpark, Python, and SQL for data transformation, validation, and performance tuning. • Solid understanding of Delta Lake architecture, ETL/ELT frameworks, and data warehousing principles. • Proficiency with Azure services including Data Factory (ADF), Data Lake (ADLS), and Databricks Notebooks. • Experience with Delta Live Tables, Unity Catalog, and Autoloader for batch and streaming data processing. • Strong background in data modeling, performance optimization, and automation scripting. • Familiarity with Agile methodologies and DevOps-based deployment practices (Git, CI/CD preferred). • Strong analytical, communication, and problem-solving skills to collaborate effectively across diverse teams. • Exposure to insurance, healthcare, or financial services data ecosystems. • Experience in data migration projects (on-prem to cloud or multi-cloud). • Familiarity with Delta Sharing, Databricks SQL Warehouses, or MLflow for advanced use cases. • Experience with data cataloging, lineage, or quality frameworks such as Purview, Collibra, or Great Expectations. • Exposure to BI/reporting tools like Power BI or Tableau for end-to-end integration understanding. Company Overview • Welcome to Jobs via Dice, the go-to destination for discovering the tech jobs you want. It was founded in undefined, and is headquartered in , with a workforce of 0-1 employees. Its website is https://www.dice.com.

Requirements