Job description

Azure Databricks Engineer (PySpark) job at Sira Consulting, an Inc 5000 company. Branchville, NJ. Job Role: Azure Databricks Engineer (PySpark) Location: Branchville, NJ Job Type: Full-time Job Summary: We are looking for an experienced Azure Databricks Engineer with expertise in PySpark to design, develop, and optimize big data solutions on Azure cloud. The ideal candidate will work with large-scale data processing, ETL pipelines, and performance optimization using Databricks, Spark, and Azure services. Key Responsibilities: • Design, develop, and implement data pipelines and ETL processes using Azure Databricks and PySpark . • Optimize Spark jobs for better performance and cost efficiency. • Work with structured, semi-structured, and unstructured data to create scalable data solutions. • Collaborate with Data Engineers, Data Scientists, and Business Analysts to meet business requirements. • Implement best practices for data security, governance, and compliance on Azure Cloud . • Work with Azure services such as Azure Data Factory (ADF), Azure Data Lake (ADLS), Azure Synapse, and Azure SQL . • Troubleshoot performance bottlenecks and optimize query execution plans . • Develop and maintain CI/CD pipelines for data pipeline automation. • Write clean, reusable, and efficient PySpark code following industry best practices. Required Skills & Qualifications: • 10+ years of experience in Azure Databricks and PySpark . • Strong expertise in Apache Spark (RDDs, DataFrames, Datasets). • Experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), and Azure Synapse Analytics . • Strong understanding of SQL and data modeling techniques. • Experience with version control tools (Git, GitHub, or Bitbucket) . • Hands-on experience with CI/CD pipelines using Azure DevOps . • Knowledge of Scala or Python for big data processing. • Strong analytical and problem-solving skills. • Experience working in Agile/Scrum environments . Education: • Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or a related field .

Requirements