Job description

Senior Azure Data Engineer - Downtown Minneapolis - W2 Position job at smart folks inc. Minneapolis, MN. Job Description Job Description Role : Senior Azure Data Engineer Location : Downtown Minneapolis (Day 1 Onsite) Contract W2 . Position Overview We are seeking a Lead Developer with advanced skills in Python, Apache Spark, Azure Synapse, and Azure Data Engineering services to develop and manage ETL pipelines and data processing solutions that support AI/ML initiatives. The ideal candidate will have strong expertise in Azure Cloud, including SQL, Data Factory, SQL Pools, Spark Pools, and Data Warehousing. A background in CI/CD processes, cross-functional collaboration, and methodologies is essential. In addition, knowledge of Data Science, Java, Kubernetes, Azure Data Lake Storage (ADLS) Gen2, and API development is required. Key Responsibilities 1. ETL and Data Pipeline Development - Design, develop, and optimize scalable ETL processes using Python, Apache Spark, and Azure Synapse. - Build and manage Azure Data Factory pipelines to orchestrate complex data workflows. - Use SQL Pools and Spark Pools within Synapse to manage and process large datasets efficiently. - Implement Data Warehousing solutions using Azure Synapse Analytics to provide structured and queryable data layers. - Ensure the data platform supports real-time and batch AI/ML data requirements. 2. Azure Cloud Development & CI/CD Deployment - Build, configure, and manage CI/CD pipelines on Azure DevOps for ETL and data processing tasks. - Automate infrastructure provisioning, testing, and deployment using Infrastructure-as-Code (IaC) tools like ARM templates or Terraform. - Optimize Azure Data Lake Storage (ADLS Gen2) to store and manage raw and processed data efficiently, ensuring proper access control and data security. 3. Cross-Functional Collaboration - Collaborate with Data Scientists, Data Engineers, ML Engineers, and Business Analysts to translate business requirements into data solutions. - Work with the DevOps and Security teams to ensure smooth and secure deployment of applications and pipelines. - Act as the technical lead in designing, developing, and implementing data solutions, mentoring junior team members. 4. Data Engineering and API Development - Develop and integrate with external and internal APIs for data ingestion and data exchange. - Build, test, and deploy RESTful APIs for secure data access. - Use Kubernetes for containerizing and deploying data processing applications. - Manage data storage and transformation to support advanced Data Science and AI/ML models. 5. Agile Project Management - Participate in and lead Agile ceremonies, such as sprint planning, daily stand-ups, and retrospectives. - Collaborate with cross-functional teams in iterative development to ensure high-quality and timely feature delivery. - Adapt to changing project priorities and business needs in an Agile environment. Thanks & Regards Kumar Sri Bharath

Requirements