Job description

Role: Senior Pyspark & ADF Location: Charlotte, NC -Hybrid Duration: 12+ Months Job Summary: We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. The ideal candidate will have expertise in Azure Data Factory (ADF), Databricks, PySpark, Python, and Azure services. This role involves designing, implementing, and optimizing data pipelines and ETL processes to support our data analytics and business intelligence efforts. Key Responsibilities: • Design and Implement Data Pipelines: Develop and manage end-to-end data pipelines using Azure Data Factory (ADF) and Databricks, ensuring data is processed efficiently and accurately. • ETL Process Management: Create and optimize ETL processes using PySpark and Python to handle large volumes of data, ensuring high performance and scalability. • Data Integration and Transformation: Utilize Databricks to integrate and transform data from various sources, applying best practices in data engineering to enhance data quality and usability. • Medallion Architecture Implementation: Implement and maintain Medallion Architecture for data lakes, ensuring structured data flow from Bronze to Silver to Gold layers. • Collaborate with Teams: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and provide solutions that support analytical and reporting needs. • Performance Tuning: Monitor and optimize data processing jobs and pipelines for performance and cost-efficiency in an Azure environment. • Documentation and Best Practices: Document data engineering processes, pipeline configurations, and data flows. Adhere to best practices and contribute to the development of data engineering standards and guidelines. Required Qualifications: • Education: Bachelor's degree in computer science, Engineering, Data Science, or a related field. Advanced degrees or certifications are a plus. • Experience: Experience in data engineering with hands-on experience in Azure Data Factory (ADF), Databricks, PySpark, and Python. • Technical Skills: • Proficiency in Azure Data Factory (ADF) for orchestrating data workflows. • Experience with Databricks for data processing and analytics. • Strong programming skills in PySpark and Python. • Knowledge of Medallion Architecture and its implementation in data lakes. • Familiarity with Azure services including Azure Data Lake Storage, Azure SQL Database, and Azure Synapse Analytics. • Soft Skills: • Excellent problem-solving abilities and analytical skills. • Strong communication skills, with the ability to convey complex technical concepts to non-technical stakeholders. • Ability to work independently and as part of a team in a fast-paced environment.

Requirements