Job description

Centurion Consulting Group is seeking an Azure Data Factory Engineer to collaborate with business analysts and stakeholders to ensure data integrity throughout the migration lifecycle. The role involves data profiling, cleansing, and validation, as well as writing SQL queries and configuring data pipelines in Azure Data Factory. Responsibilities • Collaborate with business analysts, data architects, and stakeholders to gather requirements and ensure data integrity throughout the migration lifecycle. • Ability to create Data mapping documents (Source to Target) by understanding business rules. • Perform data profiling, cleansing, and quality checks to ensure accuracy and completeness of migrated data. • Analyze and map source Oracle database schemas to target Dataverse tables/entities, defining transformation and cleansing logic as required. • Write and optimize SQL queries, stored procedures, and scripts for Oracle databases to support data migration and validation. • Implement data transformation processes to ensure data compatibility with Dataverse data types, relationships, and constraints. • Experience with Dataverse data types and CRUD operations. • Experience configuring data pipelines in ADF (to extract, transform, load). Experience using .csv files as sources of data for the ADF pipelines. • Experience using Oracle CDCs as sources of data for the ADF pipelines. Experience logging errors (audit framework) from an ADF pipeline to a Dataverse table. • Experience pushing data to Dataverse table(s) including necessary transformations to match target field types (i.e. Dataverse lookups, choices, date/time, etc.). • Experience configuring setting for pipelines to improve concurrency/performance (i.e., ADF cores, memory, etc.). • Experience with additional Azure services (e.g., Logic Apps, Data Lake, Azure Functions, Azure SQL). • Experience/exposure in python/shell scripting is a plus. • Ability to generate a reconcile report of the migrated data using SQL or scripts. • Must be familiar with Git or source code repository & CI/CD process. Skills • Collaborate with business analysts, data architects, and stakeholders to gather requirements and ensure data integrity throughout the migration lifecycle. • Ability to create Data mapping documents (Source to Target) by understanding business rules. • Perform data profiling, cleansing, and quality checks to ensure accuracy and completeness of migrated data. • Analyze and map source Oracle database schemas to target Dataverse tables/entities, defining transformation and cleansing logic as required. • Write and optimize SQL queries, stored procedures, and scripts for Oracle databases to support data migration and validation. • Implement data transformation processes to ensure data compatibility with Dataverse data types, relationships, and constraints. • Experience with Dataverse data types and CRUD operations • Experience configuring data pipelines in ADF (to extract, transform, load). Experience using .csv files as sources of data for the ADF pipelines • Experience using Oracle CDCs as sources of data for the ADF pipelines. Experience logging errors (audit framework) from an ADF pipeline to a Dataverse table • Experience pushing data to Dataverse table(s) including necessary transformations to match target field types (i.e. Dataverse lookups, choices, date/time, etc.) • Experience configuring setting for pipelines to improve concurrency/performance (i.e., ADF cores, memory, etc.) • Experience with additional Azure services (e.g., Logic Apps, Data Lake, Azure Functions, Azure SQL). • Ability to generate a reconcile report of the migrated data using SQL or scripts • Must be familiar with Git or source code repository & CI/CD process. • Experience/exposure in python/shell scripting is a plus Company Overview • Centurion Consulting Group provides expert marketing communication, market research, strategy and marketing, data analysis services, etc. It was founded in 1995, and is headquartered in Los Angeles, California, USA, with a workforce of 2-10 employees. Its website is http://www.centurionconsulting.com.

Requirements