Job description

Job Title - Databricks Platform Administrator / Engineer Location - Hybrid Duration - 12+ Months Job Summary: We are seeking a skilled Databricks Platform Administrator / Engineer with proven experience supporting modern data platforms in a Property & Casualty (P&C) insurance setting. The ideal candidate will be responsible for managing, configuring, optimizing, and maintaining the Databricks environment to support data engineering, analytics, and machine learning workloads in a secure and compliant manner. Education: Bachelor s or Master s degree in Computer Science, Engineering, Information Systems, or related field. Required Qualifications: • 5+ years of experience in data platform engineering or system administration. • 2+ years of hands-on experience with Databricks administration on Azure, AWS, or Google Cloud Platform. • Solid understanding of Apache Spark, Delta Lake, and Lakehouse architecture. • Proficiency in scripting and automation using Python, Bash, PowerShell, or Terraform. • Familiarity with data security, audit, and compliance frameworks relevant to P&C insurance (e.g., GDPR, NYDFS, SOC 2). • Experience with Azure Data Services, ADLS, Key Vault, and Azure DevOps (or equivalent on AWS/Google Cloud Platform). • Strong understanding of the P&C insurance data ecosystem (policy, claims, underwriting, actuarial, etc.). Preferred Qualifications: • Databricks Certified Associate / Professional (Administrator / Data Engineer) certification. • Experience integrating Databricks with core systems like Guidewire, Duck Creek, or custom P&C platforms. • Knowledge of industry standards like ACORD data models and ISO data feeds. • Exposure to tools like Unity Catalog, Purview, Collibra, Informatica, or similar. Soft Skills: • Strong problem-solving and analytical skills. • Ability to work cross-functionally in an agile team environment. • Excellent communication and stakeholder management skills. • Experience working in regulated enterprise environments with strong data governance standards. Key Responsibilities: • Platform Administration & Management: • Manage and administer Databricks workspaces, clusters, jobs, libraries, and notebooks. • Implement access controls using Unity Catalog, SCIM, and workspace-level RBAC. • Configure and monitor autoscaling, cluster policies, and compute usage to optimize cost and performance. • Support platform upgrades, patching, and operational automation. • Security, Compliance & Governance: • Enforce security policies aligned with P&C insurance industry regulations. • Integrate with enterprise identity providers (Azure AD, Okta) and manage SSO. • Support data governance using tools like Unity Catalog, Immuta, or Collibra. • DevOps & Automation: • Build CI/CD pipelines for notebooks, jobs, and ML workflows using tools like Azure DevOps, GitHub Actions, or Jenkins. • Automate platform provisioning and configurations using Infrastructure-as-Code (Terraform, ARM, etc.). • Monitoring & Troubleshooting: • Set up logging, monitoring, and alerting using tools like Azure Monitor, Datadog, or Splunk. • Troubleshoot performance issues, job failures, and cluster errors. • Collaboration & Enablement: • Work closely with Data Engineers, Data Scientists, and Business Analysts to ensure optimal platform usage. • Develop platform usage guidelines, onboarding documentation, and best practices. • Provide technical support and training to users.

Requirements