Job description

Hi Everyone, Greetings !!!! Please find the job description below and Let me know if you are interested. Role: Sr. Data Engineer (Databricks data engineer professional) Location: New York, NY 10045 .(Onsite) Duration: Long-term Only: USC/GC Certification : Databricks Job Description: We are seeking a highly skilled Senior Data Engineer to join our team and drive the design, development, and optimization of our data pipelines and platforms. The ideal candidate will have extensive experience in modern data engineering frameworks, cloud platforms, and distributed processing systems, with a strong focus on building scalable and reliable data solutions. Key Responsibilities: • Design, develop, and maintain scalable data pipelines and ETL/ELT processes using Python, PySpark, and SQL. • Build and optimize data solutions on Databricks and integrate with AWS or Azure cloud ecosystems. • Develop and optimize Spark SQL queries for large-scale data processing. • Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality, production-ready data sets. • Ensure data quality, reliability, and performance across all pipelines. • Implement best practices in data modeling, governance, and security. • Troubleshoot and optimize pipeline performance and support production workflows. • Mentor junior engineers and contribute to engineering standards and best practices. Required Skills & Qualifications • 7+ years of experience in Data Engineering or related field. • Strong proficiency in Python and SQL. • Hands-on experience with PySpark and Spark SQL for large-scale data processing. • Expertise in Databricks platform for data engineering and analytics. • Proficiency in AWS or Azure cloud services (e.g., S3, Redshift, Glue, EMR, Delta Lake, ADLS, Synapse, Data Factory). • Strong understanding of data warehousing, ETL, and data modeling concepts. • Experience with performance tuning for Spark jobs and SQL queries. • Excellent problem-solving and communication skills. Nice-to-Have • Experience with Delta Lake, Lakehouse architectures, and modern data governance frameworks (e.g., Unity Catalog). • Exposure to CI/CD for data pipelines and DevOps practices. • Knowledge of streaming data frameworks (Kafka, Spark Structured Streaming, etc.).

Requirements