Techyard is excited to partner with a dynamic consultancy specializing in Databricks to find a Lead Databricks Solution Architect. In this influential role, you will spearhead the design and implementation of advanced data and AI solutions on the Databricks Lakehouse Platform. We are looking for a seasoned expert who can act as both a hands-on architect and an invaluable advisor, adept at linking business initiatives with technical prowess. Your contributions will be crucial in fostering innovation, establishing best practices, and enhancing the consultancy's data engineering capabilities as the organization expands its team and client portfolio. Key Responsibilities: • Architect and deliver comprehensive, scalable data and AI solutions using the Databricks Lakehouse, including Delta Lake, Unity Catalog, and MLflow. • Design and lead the creation of modular, high-performance data pipelines with Apache Spark and PySpark. • Advocate for the Lakehouse architecture (bronze/silver/gold layers) to facilitate scalable, governed data platforms. • Collaborate with various stakeholders, including analysts and data scientists, to convert business requirements into clean, production-ready datasets. • Promote best practices in CI/CD, DevOps, and data reliability engineering (DRE) across Databricks environments. • Integrate and orchestrate workflows with cloud-native services using tools like dbt, Airflow, and Databricks Workflows. • Optimize performance, manage costs, and monitor data workloads effectively. • Mentor engineering teams and guide architectural decisions as a recognized expert in Databricks. Essential Skills & Experience: • Proven expertise with Databricks and Apache Spark in production settings. • Strong proficiency in PySpark, SQL, and experience with one or more cloud platforms (Azure, AWS, or GCP). • Thorough understanding of Lakehouse concepts, medallion architecture, and contemporary data warehousing practices. • Experience with version control, testing frameworks, and automated deployment pipelines (e.g., GitHub Actions, Azure DevOps). • Solid knowledge of data governance, security, and compliance, including the use of Unity Catalog. • Outstanding communication, leadership, and problem-solving capabilities. Desirable: • Databricks certifications (e.g., Data Engineer Associate/Professional or Solutions Architect). • Familiarity with MLflow, dbt, and BI tools like Power BI or Tableau. • Experience with MLOps practices and deploying ML models within Databricks. • Experience in Agile and DevOps-oriented delivery environments.
Job Type
Fulltime role
Skills required
No particular skills mentioned.
Location
Location not specified
Salary
No salary information was found.
Date Posted
July 1, 2025
TechYard is seeking a Lead Databricks Solution Architect to design and implement advanced data and AI solutions on the Databricks Lakehouse Platform. This role requires a seasoned expert to bridge business initiatives with technical expertise while mentoring engineering teams.