Job description

Job Title: Databricks Lead Engineer Location: Metropark, NJ or NYC, NY - Onsite Tax Term (W2, C2C): W2/C2C both Job Type (Permanent/Contract) : Contract Duration: 6+ Months Pay Range: $70/hr on C2C Job Description: Responsibilities • Design, develop, and maintain data pipelines using Databricks and Spark, and other cloud technologies as needed • Optimize data pipelines for performance, scalability, and reliability • Ensure data quality and integrity throughout the data lifecycle • Collaborate with data scientists, analysts, and other stakeholders to understand and meet their data needs Qualifications • 5+ years of experience in data engineering with Databricks and Spark • Proficient in SQL and Python and Pyspark • Experience with Azure Databricks Medallion Architecture with DLT, Iceberg • Financial/Corporate Banking context would be a plus • Experience with data integration and ETL tools, such as Azure Data Factory • Experience with Azure cloud platform and services Requirements: • Strong knowledge on data bricks architecture and tools. • Have experience of task and wf jobs creations in data bricks. • Deep understanding of distributed computing and how to use spark for data processing. • SQL and Pyspark - strong command over querying databases and proficiency in Pyspark. • Cloud platform: Preferred Azure for data bricks deployment.

Requirements