Job description

Project Overview We are building a prototype Data Agent that automates: Data migration Pipeline creation Monitoring Validation Self-healing …using modular AI agents. The system will integrate with Azure Data Factory and Databricks workflows, feature a Conversational UI, and include a Lightweight Web Dashboard. Our objective: replace repetitive manual data engineering tasks—especially around data migrations and pipeline failure handling—with an AI-driven, modular, and extensible solution. Core Capabilities Required 1. Conversational UI • Natural language interface for collecting migration/pipeline setup requirements. • Guided Q&A to gather: • Source/target system details • Connection parameters • Tables to migrate • Frequency (one-time, scheduled, real-time) • Historical data migration preference • Validation/reporting preferences • Retry and notification settings 2. Lightweight Web Dashboard • Live job status tracking • Grouped summaries by database/schema • Data validation reports (basic row count) • Manual retry for failed jobs • User-friendly for technical & business users 3. Modular AI Agents • The solution will use specialized agents for each task: • Business Analyst Agent – Gathers and interprets requirements. • Connector Agent – Establishes secure connections to source/target. • Schema Reader Agent – Reads and understands source schema. • Data Profiling Agent – Performs basic profiling. • Transformation Agent – Applies datatype & structure changes. • Pipeline Builder Agent – Creates pipelines in ADF / Databricks. • Validation Agent – Performs basic reconciliation checks. • Monitoring Agent – Monitors job health. • Self-Heal & Retry Agent – Detects failures, applies fixes, retries automatically. Prototype Workflow 1.User Interaction • user is welcomed in Conversational UI • Key migration details collected 2.Pipeline Creation • Agents collaborate to connect to source/target, read schema, build target tables, and create the pipeline • Option to create pipelines without migrating historical data 3.Execution & Monitoring • Pipelines execute as per schedule • Validation agent performs row count reconciliation • Self-heal agent retries failures automatically • Manual retry available Initial Prototype Scope • Cloud: Azure • Pipeline Tools: Azure Data Factory & Databricks workflows • Use Case 1: Migrate Azure SQL tables to Databricks and optionally decommission Azure SQL • Use Case 2: Automate pipeline/table failure recovery with self-healing Technical Requirements • Backend: Python (Cython for performance parts) • AI Orchestration: LangChain / CrewAI / LangGraph • Cloud Integration: Azure SDKs, Azure Data Factory APIs, *Databricks REST APIs • Database Connectivity: pyodbc, SQLAlchemy, JDBC • UI: React.js / Next.js for dashboard; chatbot framework for conversational UI • Deployment: Azure Functions / Azure Kubernetes Service • Security: Azure Key Vault for credential storage Engagement Model (Mandatory Conditions) Phase 0 – Unpaid Discovery Discussion • Short video call to assess experience, approach, and initial ideas • Discuss high-level scope, feasibility, technical considerations Phase 1 – Paid Architecture & Design Phase • Deliverables: • Finalized system architecture diagram • Technology stack selection • Detailed module definitions (agents, UI, backend) • Development plan with milestones • Fixed fee mutually agreed before starting Phase 2 – Development & Delivery Phase • Build prototype as per Phase 1 architecture • Deliver MVP with documentation, deployment instructions, and code repository access • Milestone-based payments Notes: • Only proceed to Phase 1 after satisfactory Phase 0 discussion • Only proceed to Phase 2 after successful Phase 1 delivery Developer Profile Requirements • 5+ years backend development (Python essential) • Strong Azure expertise (ADF, Databricks, Key Vault) • Hands-on AI agentic solutions (LangChain, CrewAI, LangGraph) • Database migration & pipeline automation experience • API integration & cloud deployment skills • Experience building dashboards & conversational UIs Project Timeline • Phase 0: 1–2 days • Phase 1: 1 week • Phase 2: 4–6 weeks How to Apply • Share examples of similar AI/data automation solutions you’ve built • Confirm Azure Data Factory & Databricks experience • Indicate familiarity with LangChain, CrewAI, LangGraph • Provide hourly rate + fixed price estimates for Phase 1 & *Phase 2 separately If you have proven expertise in AI-driven data engineering automation and can work in a structured, milestone-driven engagement, we’d like to hear from you.

Requirements