Job description

Databricks, PySpark, Scala - Developer in NY or DC Must Have Technical/Functional Skills • 8–10 years of total experience in data engineering, with at least 3–4 years in Databricks and PySpark. • Strong programming skills in Scala and Python (PySpark). • Deep experience with Databricks platform, including job orchestration, Delta Lake, and cluster management. • Proficient in distributed data processing, performance tuning, and handling large datasets. • Experience with cloud platforms (e.g., Azure, AWS, or GCP) and cloud-native data engineering tools. • Strong knowledge of data modeling, data warehousing, and data integration techniques. • Hands-on experience with version control (Git), CI/CD tools, and Agile development practices. • Excellent problem-solving, communication, and collaboration skills. Roles & Responsibilities • Design, develop, and optimize scalable ETL/ELT pipelines using Databricks and Apache Spark (PySpark and Scala). • Build data ingestion workflows from various sources (structured, semi-structured, and unstructured). • Develop reusable components and frameworks for efficient data processing. • Implement best practices for data quality, validation, and governance. • Collaborate with data architects, analysts, and business stakeholders to understand data requirements. • Tune Spark jobs for performance and scalability in a cloud-based environment. • Maintain robust data lake or lakehouse architectures. • Ensure high availability, security, and integrity of data pipelines and platforms. • Support troubleshooting, debugging, and performance optimization in production workloads. • Create and maintain technical documentation for pipelines and data models. Generic Managerial Skills, If any Working with multiple stakeholders and architecture review Education Bachelors

Requirements