Job description

PySpark Developer job at VSG Business Solutions LLC. Iselin, NJ. Responsibilities: • Develop, maintain, and optimize data processing pipelines using PySpark on Azure Cloud. • Collaborate with cross-functional teams to understand data requirements and translate business needs into technical specifications. • Design and implement scalable data models and data architecture to support analytical and reporting needs. • Perform data cleansing, transformation, and enrichment processes to enhance data quality and usability. • Monitor and troubleshoot data processing jobs, ensuring high availability and performance. • Implement best practices for data governance and security within Azure Cloud. • Stay up-to-date with emerging technologies in big data, cloud computing, and the financial services industry. Requirements: • Minimum of 9 years of experience in PySpark development. • Proven experience working with Azure Cloud services (Azure Data Factory, Azure Databricks, etc.). • Prior experience in the banking, investment banking, or financial services sector is mandatory. • Proficient in PySpark for data processing and ETL tasks. • Experience with SQL and NoSQL databases. • Familiarity with data warehousing concepts and tools. • Knowledge of Azure services related to big data and analytics. Preferred, but not required: • Experience with containerization technologies (e.g., Docker, Kubernetes). • Familiarity with machine learning frameworks (e.g., MLlib, TensorFlow).

Requirements