Job description

Lead Data Engineer job at VC5 Consulting. Houston, TX. Position: Lead Data Engineer We are in search of a highly proficient Lead Data Engineer to join our dynamic team. As the Data Engineering Technical Lead, you will spearhead a team of skilled data engineers, overseeing the design, construction, and maintenance of resilient data pipelines and processing systems. This pivotal role demands expertise in an array of technologies, including Azure, Spark SQL, SQL Server, Azure Data Factory, Azure Synapse, Azure Databricks, Dimensional Modeling, Agile, Azure DevOps, and DataOps. Your responsibilities will encompass providing technical leadership, championing best practices, and ensuring the successful execution of data engineering projects. Must-Haves: • 5-7+ years of hands-on experience in data engineering, data integration, and data warehousing using Azure services such as SQL Server, Azure Data Factory, Azure Synapse, and Azure Databricks. • Proficiency (5-7+ years) in SQL programming with expertise in working with database management systems like SQL Server. • In-depth comprehension of dimensional modeling principles and adeptness in designing scalable and high-performance data structures. • Practical experience with Spark SQL and data processing using Apache Spark. • Familiarity with cloud-based architecture and services, especially Microsoft Azure. • Knowledge of data governance, data quality, and data security principles. • Experience working in an Agile development environment with a strong understanding of Agile methodologies. • Proficiency in Azure DevOps practices and tools for version control, CI/CD, and automation. • Knowledge and experience with DataOps principles, including data pipeline testing, deployment automation, and monitoring. Responsibilities: • Technical Leadership: • Lead and inspire a team of data engineers, offering technical guidance, mentoring, and coaching. • Advocate for and establish best practices, standards, and methodologies for data engineering projects. • Design and Architecture: • Collaborate with stakeholders to comprehend data requirements and translate them into scalable and high-performing data engineering and data product solutions. • Design data pipelines, data models, and system architectures utilizing Azure services, including Azure Data Factory, Azure Synapse, and Azure Databricks. • Data Pipeline Development: • Oversee the development of data pipelines, ELT processes, and data integration workflows using Azure Databricks, Azure Synapse Pipelines, Spark SQL, PySpark, and other pertinent technologies. • Ensure the availability, reliability, and scalability of data processing systems. • Dimensional Modeling: • Apply dimensional modeling techniques to devise and implement data models in SQL Server, adhering to best practices for analytical reporting and business intelligence. • Design star schemas, fact tables, and dimension tables to support data analysis and reporting requirements. • Performance Optimization: • Optimize data processing workflows, SQL queries, and transformations to enhance system performance, reduce latency, and improve scalability. • Data Quality and Governance: • Establish data quality standards, implement data validation rules, and develop data cleansing processes. • Ensure adherence to data governance policies and practices to maintain the accuracy, integrity, and security of data. • Agile Project Management: • Collaborate within an Agile team, participating in sprint planning, backlog refinement, and other Agile ceremonies. • Azure DevOps and DataOps: • Implement and oversee Azure DevOps practices for version control, CI/CD, and automation of data engineering artifacts. • Drive the adoption of DataOps principles, including testing, deployment automation, and monitoring of data pipelines. • Collaboration and Communication: • Collaborate effectively with stakeholders, including data analysts and business users, to understand their data requirements and align data engineering solutions accordingly. • Communicate technical concepts and project progress to non-technical stakeholders. • Continuous Learning and Innovation: • Stay current with the latest advancements in data engineering technologies and tools. • Maintain and achieve relevant certifications. • Evaluate, establish, and recommend methodologies and frameworks that can enhance the data engineering capabilities of the team. Join our team and contribute to the forefront of data engineering excellence. Apply now to be part of our innovative and collaborative environment dedicated to pushing the boundaries of technology. #9790

Requirements