Job description

Note: The job is a remote job and is open to candidates in USA. QData Inc is seeking a Senior Azure Databricks Platform Architect to design, implement, and manage their Azure Databricks platform. The successful candidate will work closely with data engineering and data science teams to ensure seamless integration of data pipelines and Lakehouse architecture while maintaining high availability, scalability, and security. Responsibilities • Design, implement, and manage our Azure Databricks platform, ensuring high availability, scalability, and security. • Configure and optimize Databricks clusters, jobs, and notebooks for performance and cost-effectiveness. • Maintain up to date architecture environment. • Works with Azure and Databricks to resolved platform issues/tickets. • Optimize cost / performance • Collaborate with data engineering teams to design and implement/migrate data pipelines using using industry best practice. • Automate manual admin tasks. • Develop and maintain data Lakehouse architecture, ensuring data quality, integrity, and security. Work with Azure IAM and Engineering team to come up with RBAC for Databricks tables. • Design and manage Lakehouse and sandbox environments for data science teams, ensuring data governance and security. • Collaborate with data science teams to integrate data pipelines and Lakehouse architecture. • Ensure Azure Databricks platform security and compliance with organizational security policies and regulatory requirements (e.g., HIPAA,NIST,GDPR). • Implement data encryption, access controls, and auditing. • Develop and maintain technical documentation for Azure Databricks platform, data pipelines, and Lakehouse architecture. • Share knowledge and best practices with data engineering and data science teams. Skills • Strong background in Azure Databricks • Experience in data engineering • Experience in data architecture • Ability to design, implement, and manage Azure Databricks platform • Experience in configuring and optimizing Databricks clusters, jobs, and notebooks • Ability to maintain up to date architecture environment • Experience in resolving platform issues/tickets with Azure and Databricks • Ability to optimize cost/performance • Experience collaborating with data engineering teams to design and implement/migrate data pipelines • Ability to automate manual admin tasks • Experience developing and maintaining data Lakehouse architecture • Knowledge of Azure IAM and RBAC for Databricks tables • Ability to design and manage Lakehouse and sandbox environments for data science teams • Experience ensuring data governance and security • Experience collaborating with data science teams to integrate data pipelines and Lakehouse architecture • Knowledge of Azure Databricks platform security and compliance with organizational security policies and regulatory requirements (e.g., HIPAA, NIST, GDPR) • Experience implementing data encryption, access controls, and auditing • Ability to develop and maintain technical documentation for Azure Databricks platform, data pipelines, and Lakehouse architecture • Ability to share knowledge and best practices with data engineering and data science teams Company Overview • Q Data Incorporation is one of the most progressive organizations in the country providing Information Technology (IT) services in staffing and consulting through out the USA. It was founded in 2008, and is headquartered in Little Elm, Texas, USA, with a workforce of 11-50 employees. Its website is http://q-datainc.com.

Requirements