Job description

Note: The job is a remote job and is open to candidates in USA. QData Inc is seeking a Senior Azure Databricks Platform Architect to design, implement, and manage their Azure Databricks platform. The role involves collaborating with data engineering and data science teams to ensure seamless integration of data pipelines and Lakehouse architecture while maintaining high availability, scalability, and security. Responsibilities • Design, implement, and manage our Azure Databricks platform, ensuring high availability, scalability, and security. • Configure and optimize Databricks clusters, jobs, and notebooks for performance and cost-effectiveness. • Maintain up to date architecture environment. • Works with Azure and Databricks to resolved platform issues/tickets. • Optimize cost / performance • Collaborate with data engineering teams to design and implement/migrate data pipelines using using industry best practice. • Automate manual admin tasks. • Develop and maintain data Lakehouse architecture, ensuring data quality, integrity, and security. Work with Azure IAM and Engineering team to come up with RBAC for Databricks tables. • Design and manage Lakehouse and sandbox environments for data science teams, ensuring data governance and security. • Collaborate with data science teams to integrate data pipelines and Lakehouse architecture. • Ensure Azure Databricks platform security and compliance with organizational security policies and regulatory requirements (e.g., HIPAA,NIST,GDPR). • Implement data encryption, access controls, and auditing. • Develop and maintain technical documentation for Azure Databricks platform, data pipelines, and Lakehouse architecture. • Share knowledge and best practices with data engineering and data science teams. Skills • Strong background in Azure Databricks • Experience in data engineering • Experience in data architecture • Ability to design, implement, and manage Azure Databricks platform • Experience in ensuring high availability, scalability, and security of platforms • Ability to configure and optimize Databricks clusters, jobs, and notebooks for performance and cost-effectiveness • Experience in maintaining up to date architecture environment • Ability to work with Azure and Databricks to resolve platform issues/tickets • Experience in optimizing cost/performance • Ability to collaborate with data engineering teams to design and implement/migrate data pipelines using industry best practices • Experience in automating manual admin tasks • Ability to develop and maintain data Lakehouse architecture, ensuring data quality, integrity, and security • Experience working with Azure IAM and Engineering team to implement RBAC for Databricks tables • Ability to design and manage Lakehouse and sandbox environments for data science teams, ensuring data governance and security • Ability to collaborate with data science teams to integrate data pipelines and Lakehouse architecture • Experience in ensuring Azure Databricks platform security and compliance with organizational security policies and regulatory requirements (e.g., HIPAA, NIST, GDPR) • Experience in implementing data encryption, access controls, and auditing • Ability to develop and maintain technical documentation for Azure Databricks platform, data pipelines, and Lakehouse architecture • Ability to share knowledge and best practices with data engineering and data science teams Company Overview • Q Data Incorporation is one of the most progressive organizations in the country providing Information Technology (IT) services in staffing and consulting through out the USA. It was founded in 2008, and is headquartered in Little Elm, Texas, USA, with a workforce of 11-50 employees. Its website is http://q-datainc.com.

Requirements