Azure Databricks Engineer

Role: Data EngineerCompany: Crox Consulting Inc

Job description

Role: Azure Databricks Engineer Location: Jersey City, NJ - Hybrid (2 days Onsite) NOTE :: Must be available for face to face for screening and final round of interview Responsibilities: 1. Design and implement a scalable data warehouse on Azure Databricks using data and dimensional modeling techniques to support analytical and reporting requirements. 2. Develop and optimize ETL/ELT pipelines using Python, Azure Databricks and PySpark for large-scale data processing, ensuring data quality, consistency, and integrity. 3. Establish and implement best practices for data ingestion, transformation, and storage using the medallion architecture (Bronze, Silver, Gold). 4. Architect and develop highly scalable data applications using Azure Databricks and distributed computing. 5. Optimize Databricks clusters and ETL/ELT workflows for performance and scalability. 6. Manage data storage solutions using Azure Data Lake Storage (ADLS) and Delta Lake while leveraging Unity Catalog for data governance, security, and access control. 7. Develop and schedule Databricks notebooks and jobs for automated daily execution, implementing monitoring, alerting, and automated recovery processes for job failures. 8. Identify and resolve bottlenecks in existing code and follow best coding practices to improve performance and maintainability. 9. Use GitHub as version control tool to manage code and collaborate effectively with other developers; build and maintain CI/CD pipelines for deployment and testing using Azure DevOps and GitHub. 10. Create comprehensive documentation for data architecture, ETL processes, and business logic. 11. Work closely with business stakeholders to understand project goals and architect scalable and efficient solutions. 12. Knowledge of user authentication on Unity Catalog and authorization between multiple systems, servers and environments. 13. Ensure that programs are written to the highest standards (e.g., Unit Tests) and technical specifications. 14. Ability to collaborate on projects and work independently when required. Qualifications: 1. 10+ years of prior experience as a developer in the required technologies (Azure Databricks, Python, PySpark, Datawarehouse Designing) 2. Solid organizational skills, ability to multi-task across different projects 3. Experience with Agile methodologies 4. Skilled at independently researching topics using all means available to discover relevant information. 5. Ability to work in a team environment. 6. Excellent verbal and written communication skills 7. Self-starter with ability to multi-task and to maintain momentum.

Requirements

• Ensure that programs are written to the highest standards (e.g., Unit Tests) and technical specifications

• 10+ years of prior experience as a developer in the required technologies (Azure Databricks, Python, PySpark, Datawarehouse Designing)

• Solid organizational skills, ability to multi-task across different projects

• Experience with Agile methodologies

• Skilled at independently researching topics using all means available to discover relevant information

• Ability to work in a team environment

• Excellent verbal and written communication skills

• Self-starter with ability to multi-task and to maintain momentum

SHARE THIS OPENING

Similar jobs

Senior Application Consultant

May 20, 2025

Hexagonusfederal

Washington, D.C.

The Senior Application Consultant at Hexagon US Federal will manage and enhance the OnCall Records & Analytics systems, focusing on system administration, troubleshooting, and user support. This role requires strong problem-solving skills and expertise in Power BI and SQL databases.

Lead Data Analyst

Invalid date

Nike

Hillsboro, OR

Python

GitHub

Power BI

Title: Lead Data Analyst Location: Beaverton, OR Duration: 6 Month Contract Become a Part of the NIKE, Inc. Team NIKE, Inc. does more than outfit the world's best athletes. It is a place to explore potential, obliterate boundaries and push out the edges of what can be. The company looks for people who can grow, think, dream and create. Its culture thrives by embracing diversity and rewarding imagination. The brand seeks achievers, leaders and visionaries. At Nike, it’s about each person bringing skills and passion to a challenging and constantly evolving game. What You Will Do: Seeking a Lead Data Analyst to join our data team. In this role, you will lead the generation of reports, datasets, and data visualizations to provide actionable insights to stakeholders across the business. • Develop and generate recurring reports, dashboards, and ad-hoc analyses using SQL, Python, R, Tableau, etc. to surface insights. • Mine, cleanse, and curate data from multiple sources (databases, data warehouses, etc.). • Create and document SQL queries, scripts, visualizations, and data models. • Identify and implement opportunities to improve data quality, efficiency, and analytics. • Clearly communicate insights through presentations, reports, and visualization best practices. • Manage technical projects and lead a team of junior analysts. What You Will Need: • 7+ years proven work experience in Data Analyst or Business Data Analyst role. • Strong Experience in SQL and a scripting language such as Python or R. • Strong problem solving and analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy. • Experience with data warehouses/RDBMS like Snowflake & Teradata. • Experience working with BI tools like Tableau, Power BI or DOMO. • Strong understanding of data modeling and analytical algorithms (including forecasting and modeling). • Experience working in AWS environment primarily EMR, S3, Redshift, Athena, etc preferred. • Good understanding of Hadoop and Big Data processing frameworks. • Experience with source control tools such as GitHub. • Ability to cleanse, curate, and manipulate data; as well as visualize and story-telling the results of an analysis. • Proven experience and ability to deliver results on multiple projects in a fast-paced agile environment. • Self-starter, highly organized and result oriented; and should be able to work with minimal direction.

Senior Azure Data Engineer

May 19, 2025

VirtualVocations

Alexandria, Virginia

VirtualVocations is seeking a Senior Azure Data Engineer to lead a Cloud Data Modernization project in Alexandria, Virginia. The role involves migrating ETLs to Azure Cloud and Snowflake while optimizing data pipelines.

Save job

Apply now

Job Type

Contractor role

Skills required

Python, Agile

Location

Edison, New Jersey

Salary

No salary information was found.

Date Posted

May 14, 2025

Save job Apply now