Job description

Job Summary We are seeking a skilled and experienced Data Engineer to join our team, focusing on the Microsoft data platform. The ideal candidate will have extensive hands-on experience in designing, building, and maintaining robust and scalable data pipelines using Microsoft Fabric, Azure Synapse Analytics, and Python. You will be responsible for orchestrating the flow of data from diverse sources into our analytical platform, enabling data-driven decision-making across the organization. Key Responsibilities • Pipeline Development: Design, develop, and maintain efficient ETL/ELT data pipelines using Microsoft Fabric and its components, including Fabric Pipelines and Spark jobs. • Notebook-based Transformation: Write, test, and optimize Python notebooks (PySpark) within the Fabric/Synapse environment to perform complex data ingestion, cleansing, and transformation tasks. • Data Warehouse and Lakehouse Management: Implement and manage data storage solutions, leveraging Azure Data Lake and Fabric's OneLake for a modern lakehouse architecture. • Data Integration: Integrate data from a wide variety of sources, including transactional databases, APIs, and streaming platforms, into the centralized data platform. • Performance Tuning: Optimize data pipelines and Spark workloads for performance and cost efficiency, especially when dealing with large-scale datasets in Synapse Analytics. • Data Quality and Governance: Establish and enforce data quality and governance standards to ensure the accuracy, consistency, and reliability of data. Implement policies like row-level security and data classification. • Collaboration: Work closely with data analysts, data scientists, and business stakeholders to understand data requirements and deliver high-quality data assets. • Automation and CI/CD: Ensure monitoring, logging, and implement CI/CD automation for data workflows using tools like Azure DevOps. Required Skills and Qualifications • Proven experience as a Data Engineer with a focus on the Microsoft Azure ecosystem. • Strong hands-on experience with Microsoft Fabric, specifically its data engineering workloads (Notebooks, Pipelines, and Lakehouses). • Demonstrable expertise in Azure Synapse Analytics, including Synapse Notebooks (PySpark) and SQL pool resources. • Proficiency in Python for data manipulation, transformation, and automation. • Experience in building and maintaining ETL/ELT pipelines using Azure Data Factory and Fabric Pipelines. • Solid understanding of modern data architecture concepts, including data lakehouses and medallion architecture. • Strong SQL skills for querying and optimizing data workloads. • Familiarity with data governance tools and concepts, such as Microsoft Purview. • Experience with version control systems, such as Git. • Excellent problem-solving, analytical, and communication skills. Preferred Qualifications • Experience with real-time or streaming data processing. • Knowledge of Power BI for building datasets and reports. • Relevant Microsoft certifications (e.g., Microsoft Certified: Fabric Data Engineer Associate).

Requirements