Job description

Job Title: Senior Azure Databricks Platform Engineer Location: Little Elm, Tx (onsite 2 days a week) Note: Looking for a Visa-Independent consultant; H1B transfers are also accepted Job Description: Responsible for designing, implementing, and managing our Azure Databricks platform. The ideal candidate will have a strong background in Azure Databricks, data engineering, and data architecture, with a focus on building scalable and secure data platforms. The successful candidate will work closely with our data engineering and data science teams to ensure seamless integration of data pipelines and Lakehouse architecture. This role should ensure coding standards and coding migration to higher environments and CI/CD. Azure Databricks Platform Management Design, implement, and manage our Azure Databricks platform, ensuring high availability, scalability, and security. Configure and optimize Databricks clusters, jobs, and notebooks for performance and cost-effectiveness. Maintain an up to date architecture environment. Works with Azure and Databricks to resolve platform issues/tickets. Optimize cost / performance Data Engineering and Architecture: Collaborate with data engineering teams to design and implement/migrate data pipelines using industry best practice. Automate manual admin tasks. Develop and maintain data lake house architecture, ensuring data quality, integrity, and security. Work with the Azure IAM and Engineering team to come up with RBAC for Databricks tables. Lakehouse and Sandbox Management: Design and manage lakehouse and sandbox environments for data science teams, ensuring data governance and security. Collaborate with data science teams to integrate data pipelines and lakehouse architecture. Security and Compliance: Ensure Azure Databricks platform security and compliance with organizational security policies and regulatory requirements (e.g., HIPAA,NIST,GDPR). Implement data encryption, access controls, and auditing.

Requirements