Job description

About the Company VivSoft is an emerging technology company specializing in leveraging modern technologies to solve our clients' most complex mission challenges. We focus on Cloud, Enterprise DevSecOps, Artificial Intelligence (AI), and Digital Customer Experience to drive mission-enabling digital transformation. Our passion lies in building mission-focused, open, and scalable solutions. We are a diverse team of strategists, engineers, designers, and innovators with deep experience in developing high-performance software and AI factory accelerators through automation and agile practices. Position Overview: We are seeking a highly skilled Azure Databricks Engineer to design, build, and optimize large-scale data pipelines and lakehouse solutions within the Azure ecosystem. The ideal candidate will have deep technical expertise in Apache Spark, Delta Lake, and Azure Data Services, along with a strong understanding of data modeling, ETL/ELT processes, and performance optimization. Key Responsibilities: • Design, build, and maintain ETL/ELT pipelines using Azure Databricks, Delta Lake, and Apache Spark for structured and unstructured data. • Implement data ingestion and transformation processes integrating multiple internal and external data sources. • Architect and optimize data lakehouse solutions, ensuring efficient data storage, retrieval, and partitioning strategies. • Develop and manage batch and real-time streaming data pipelines using Structured Streaming and other Azure services. • Configure and manage Databricks workspaces, clusters, and compute resources for optimal performance and cost efficiency. • Implement data security, access control, and governance measures within the Azure environment. • Collaborate with data scientists to deploy and operationalize machine learning models. • Utilize MLflow for model tracking, versioning, and automated deployment pipelines. • Monitor pipeline performance, troubleshoot issues, and optimize Spark jobs for improved efficiency and reduced processing costs. • Implement data quality checks and ensure reliability across all data processing workflows. • Work closely with stakeholders to translate business needs into scalable data solutions. • Demonstrate strong knowledge of ETL/ELT frameworks, data modeling, and big data technologies. • Hands-on experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), and Azure Synapse Analytics. • Proficient in Apache Spark, Delta Lake, Python, SQL, and Scala. • Minimum 5+ years of experience as a Data Engineer, with 3+ years specifically in Azure Databricks and related Azure services. Skills Required: • Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related field. • Strong understanding of data architecture, performance tuning, and cost optimization in Azure. • Excellent analytical, problem-solving, and communication skills. • Experience working in Agile environments and implementing data governance frameworks. • Preferred certifications include Microsoft Certified: Azure Data Engineer Associate (DP-203) or Databricks Certified Data Engineer. • Passion for innovation, continuous improvement, and driving efficiency in large-scale data environments. • Opportunity to work with cutting-edge Azure data and AI technologies within a collaborative, high-impact environment.

Requirements