Job description

Position Title: Databricks Data Engineer Location: Austin, TX (Remote Eligible) Clearance Requirements: None Position Status: 9-Month Contract (Likely to Extend) Position Description: Our public sector client in Austin, TX is seeking an experienced Databricks Data Engineer to join their data engineering and analytics team. This role focuses on designing, building, and optimizing scalable data pipelines and ETL/ELT workflows on the Databricks Unified Analytics Platform, integrated with Azure Data Factory. The ideal candidate is an expert in Apache Spark, Azure Data Lake, and Python, with deep experience in data governance, security, and automation using CI/CD. You will work closely with data scientists, analysts, and DevOps teams to deliver high-quality, production-ready data solutions that power analytics and machine learning initiatives. Key Responsibilities: • Design and develop scalable data pipelines using Apache Spark on Databricks. • Implement and maintain ETL/ELT workflows for both structured and unstructured data. • Optimize Spark jobs for performance, scalability, and cost-efficiency. • Integrate Databricks solutions with Azure Data Factory and other Azure cloud services. • Design and manage data models, schemas, and database structures supporting analytics and operational workloads. • Implement robust data validation, quality checks, and contribute to metadata management, data lineage, and cataloging. • Apply data security and compliance best practices, including encryption, access control, and auditing. • Automate deployment pipelines with CI/CD tools in collaboration with DevOps. • Partner with data scientists, analysts, and stakeholders to align technical solutions with business goals. • Troubleshoot and resolve performance bottlenecks and data integrity issues to ensure system reliability. Required Skills & Experience: • 8+ years of experience in data engineering, including ETL/ELT development for structured and unstructured data. • 5+ years of experience designing and optimizing data pipelines on Apache Spark / Databricks. • Proven experience with Azure Data Factory, Azure Data Lake, and related cloud data services. • Expertise in Python, R, and SQL for data processing and analysis. • Hands-on experience with CI/CD automation, DevOps tools, and version control systems (e.g., Git). • Strong background in data modeling, data governance, and data quality assurance. • Experience with Unity Catalog or Delta Lake for data governance and security. • Deep understanding of Apache Spark architecture, including RDDs, DataFrames, and Spark SQL. • Skilled in troubleshooting complex data systems and improving pipeline performance. • Experience working in Agile, multicultural environments. Preferred Qualifications: • Experience with machine learning tools such as MLflow, Scikit-learn, or TensorFlow. • Databricks Certified Associate Developer for Apache Spark. • Microsoft Azure Data Engineer Associate certification. About Seneca Resources At Seneca Resources, we are more than just a staffing and consulting firm, we are a trusted career partner. With offices across the U.S. and clients ranging from Fortune 500 companies to government organizations, we provide opportunities that help professionals grow their careers while making an impact. When you work with Seneca, you're choosing a company that invests in your success, celebrates your achievements, and connects you to meaningful work with leading organizations nationwide. We take the time to understand your goals and match you with roles that align with your skills and career path. Our consultants and contractors enjoy competitive pay, comprehensive health, dental, and vision coverage, 401(k) retirement plans, and the support of a dedicated team who will advocate for you every step of the way. Seneca Resources is proud to be an Equal Opportunity Employer, committed to fostering a diverse and inclusive workplace where all qualified individuals are encouraged to apply.

Requirements