Job description

About the job Databricks Architect • Title: Databricks Architect • Key Requirements: Hands on experience in Databricks, SQL, PySpark. Must have Insurance domain experience. • Location: Jersey City, NJ • Job Type: Hybrid (2 days work from office) • Duration: 12+ Months • Start Date: Immediate • Number of roles: 1 Role & Responsibilities Overview: • Develop and optimize ETL pipelines from various data sources using Databricks on cloud (AWS, Azure, etc.) • Experienced in implementing standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD, Terraform for infrastructure as code, and Splunk for monitoring • Continuously improve systems through performance enhancements and cost reductions in compute and storage • Data Processing and API Integration: Utilize Spark Structured Streaming for real-time data processing and integrate data outputs with REST APIs • Lead Data Engineering Projects to manage and implement data-driven communication systems • Experienced with Scrum and Agile Methodologies to coordinate global delivery teams, run scrum ceremonies, manage backlog items, and handle escalations • Integrate data across different systems and platforms • Strong verbal and written communication skills to manage client discussions Candidate Profile: • 8+ years experience in developing and implementing ETL pipelines from various data sources using Databricks on cloud • Experience in insurance domain/ data is must • Programming Languages SQL, Python • Technologies - IaaS (AWS or Azure or GCP), Databricks platform, Delta Lake storage, Spark (PySpark, Spark SQL). • Good to have - Airflow, Splunk, Kubernetes, Power BI, Git, Azure DevOps • Project Management using Agile, Scrum • B.S. Degree in a data-centric field (Mathematics, Economics, Computer Science, Engineering or other science field), Information Systems, Information Processing or engineering. • Excellent communication & leadership skills, with the ability to lead and motivate team members • Ability to work independently with some level of ambiguity and juggle multiple demands Inizio Partners

Requirements