Job description

About Role: Lead Databricks Engineer with Pyspark & Python Exp Location: NYC, NY OR Iselin, NJ (Hybrid - 3 days' work from office) Job Description Inperson interview is must, Expenses will be paid for in-person if not local (Financial background is must) Major Responsibilities • Work on Finance data related to Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo • Design, develop, and deploy Databricks jobs to process and analyze large volumes of data. • Collaborate with data engineers and data scientists to understand data requirements and implement appropriate data processing pipelines. • Optimize Databricks jobs for performance and scalability to handle big data workloads. • Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks. • Implement best practices for data management, security, and governance within the Databricks environment. Experience designing and developing Enterprise Data Warehouse solutions. • Demonstrated proficiency with Data Analytics, Data Insights • Proficient writing SQL queries and programming including stored procedures and reverse engineering existing process • Azure Synapse/Bigquery/Redshift is good to have. • Perform code reviews to ensure fit to requirements, optimal execution patterns and adherence to established standards. Skills • 5+ years' - Strong experience in Finance / Banking industry - Capital markets, investment banking - Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo • 10+ years - Enterprise Data Management • 10+ years - SQL Server based development of large datasets • 5+ years with Data Warehouse Architecture, hands-on experience with Databricks platform. Extensive experience in PySpark coding. • Azure Synapse/BigqExperience with Cloud based data architectures, messaging, and analytics • Superior communication skills • Cloud certification(s) • Any experience with Regulatory Reporting is a Plus Education • Minimally a BA degree within an engineering and/or computer science discipline • Master's degree strongly preferred Nice-to-have skills • PySpark • Python • SQL • Redshift • New York, New York Work experience • Data Engineer • Data Infrastructure • Data Analyst Languages • English

Requirements