Job description

Roles & Responsibilities 1. Data Engineering & Pipeline Development Automation Framework Development,Azure Data Bricks,Devops,PySpark,Terraform • Design, develop, and maintain scalable ETL/ELT pipelines using Apache Spark (PySpark/Scala) on Databricks. • Implement Delta Lake for data reliability, ACID transactions, and versioning. • Integrate data from various sources including cloud storage (e.g., GCS, S3), relational databases, APIs, and streaming platforms. 2. Cloud Integration (GCP Focus) • Connect Databricks with GCP services like BigQuery, Cloud Storage, Pub/Sub, and Vertex AI. • Optimize data movement between Databricks and GCP using connectors and APIs. • Ensure seamless authentication and authorization using GCP IAM roles and Databricks workspace configurations. 3. Performance Optimization • Tune Spark jobs for performance and cost-efficiency. • Implement best practices for partitioning, caching, and job scheduling. • Monitor cluster performance and resource utilization. 4. Automation & CI/CD • Automate deployment of notebooks, jobs, and clusters using Terraform, GitHub Actions, or Azure DevOps. • Implement CI/CD pipelines for version control and continuous integration of Databricks assets. 5. Security & Governance • Implement role-based access control (RBAC) and manage secrets securely. • Ensure data encryption at rest and in transit. • Collaborate with security teams to meet compliance and governance standards. 6. Cross-Functional Collaboration • Work closely with data scientists, analysts, and business stakeholders to understand data requirements. • Translate business needs into technical solutions using Databricks and GCP. 7. Mentorship & Leadership • Mentor junior developers and conduct code reviews. • Lead technical discussions and contribute to architectural decisions. • Promote best practices in coding, testing, and documentation. 8. Innovation & Continuous Improvement • Stay updated with the latest features in Databricks and GCP. • Evaluate new tools and frameworks for potential adoption. • Continuously improve existing pipelines and workflows for efficiency and scalability. Regards, Deepak Kumar Infinite | Exciting times...infinite possibilities… Tel: +1- 301-355-7756 Email: deepak.kumar@infinite.com

Requirements