Job description

Databricks Data Engineer Requirements Public Trust- Can Start with it pending REMOTE (Quarterly Travel to Gaithersburg) Client is seeking a Databricks Data Engineer to develop/support new/existing data pipelines, and data analytics environments in Azure cloud-based data lake. As a data engineer, you will translate business requirements to data engineering solutions to support an enterprise scale Microsoft Azure based data analytics platform. You will support continued maintenance of ETL operations and development of new pipelines ensuring data quality and data management. The ideal candidate will bring deep expertise in Databricks, a solid foundation in advanced AI technologies, and applies critical thinking to create innovative functions and solve technical issues by cross-functional team collaboration. Candidate must be a U.S. citizen. This role is remote for candidates located within the United States; however, the ideal candidate will be based in or willing to relocate to the Washington, DC or Indianapolis, IN area to support periodic on-site activities.? In this role, you will: • Design, build, and optimize scalable data solutions using Databricks and Medallion Architecture. • Manage ingestion routines for processing multi-terabyte datasets efficiently for multiple projects simultaneously, where each project may have multiple Databricks workspaces. • Integrate data from various structured and unstructured sources to enable high-quality business insights. Proficiency in data analysis techniques for deriving insights from large datasets. • Implement effective data management strategies to ensure data integrity, availability, and accessibility. Identify opportunities for cost optimization in data storage, processing, and analytics operations. • Monitor and support user requests, addressing platform or performance issues, cluster stability, Spark optimization, and configuration management. • Collaborate with the team to enable advanced AI-driven analytics and data science workflows. • Integrate with various Azure services including Azure Functions, Storage Services, Data Factory, Log Analytics, and User Management for seamless data workflows. • Experience with the above Azure services is a plus. • Provision and manage infrastructure using Infrastructure-as-Code (IaC) • Apply best practices for data security, data governance, and compliance, ensuring support for federal regulations and public trust standards. • Proactively collaborate with technical and non-technical teams to gather requirements and translate business needs into data solutions. For this position, you must possess: • BS degree in Computer Science or related field and 3 years or Master's degree with 2 years of experience • 3 years of experience developing and designing Ingestion flows (structured, streaming, and unstructured data) using cloud platform services with data quality • Databricks Data Engineer certification and 2 years of experience maintaining Databricks platform and development in Spark • Ability to work directly with clients and act as front line support for requests coming in from clients. Clearly document and express the solution in form of architecture and interface diagrams. • Proficient at Python, Spark and R are essential. .NET based development is a plus. • Knowledge and experience with data governance, including metadata management, enterprise data catalog, design standards, data quality governance, and data security. • Experience with Agile process methodology, CI/CD automation, and cloud-based developments (Azure, AWS). • Not required, but additional education, certifications, and/or experience are a plus: Certifications in Azure cloud, Knowledge of FinOps principles and cost management #DICE #CJ About Equiliem Equiliem believes in empowering success. It's our job to cultivate relationships that connect people and employers in a way that is inclusive, intelligent, and allows both to thrive. Across the U.S., leading companies in healthcare, government, engineering, manufacturing, professional services, and energy rely on us for their workforce solutions. Our recruiting and HR services include contract and direct hire staffing, Payrolling/EOR, Independent Contractor Compliance, and Managed Services. For almost 30 years, we've helped shape our industry. Today, we continue to research, ask questions, and continuously enhance the candidate journey and client experience. Benefits We offer a comprehensive benefits package to our employees, which includes: * Medical Insurance * Vision & Dental Insurance * Life Insurance * 401K * Commuter Benefits * Employee Discounts & Rewards * Payroll Payment Options EEO Employer Equiliem is an equal opportunity employer. We do not discriminate or allow discrimination based on race, color, religion, creed, sex, age, sexual orientation, gender identity, national origin, ancestry, citizenship, genetic information, registered domestic partner status, marital status, disability, status as a crime victim, protected veteran status, political affiliation, union membership, or any other characteristic protected by law. Equiliem will consider qualified applicants with criminal histories in a manner consistent with the requirements of applicable law. If you have visited our website in search of information on employment opportunities or to apply for a position, and you require an accommodation in using our website for a search or application, please contact our Employee Services Department at [email protected] or (732) 238-6050 .

Requirements