Azure Data Engineer

Azure Data Engineer (Senior) with skills Data Engineering, AI/ML Architecture, Data Engineering, databricks, Azure Data Factory, GenAI Fundamentals for location Any Infogain Base Location (Noida, Gurugram, Bangalore, Mumbai, Pune) We are seeking an experienced Azure Data Engineer with 6–8 years of hands‑on experience in designing, developing, and optimizing large‑scale data platforms on Microsoft Azure. The ideal candidate will possess deep expertise in Azure Databricks, Azure Data Factory, Delta Lake, and modern data engineering practices. This role involves building scalable data pipelines, enabling advanced analytics, and ensuring efficient data availability for business and AI‑driven use cases. Key Responsibilities Design, build, and maintain end‑to‑end data pipelines using Azure Databricks (PySpark/Scala), ADF, and Delta Lake. Implement ETL/ELT workflows with robust logging, monitoring, orchestration, and error‑handling capabilities. Develop high‑performance data ingestion frameworks for structured, semi‑structured, and unstructured data. Design and maintain data models, including Bronze‑Silver‑Gold architectures on the Lakehouse. Contribute to data lake and data warehouse architecture leveraging Azure Synapse, Delta Lake, and Databricks Unity Catalog. Optimize data storage and compute costs using best practices in partitioning, caching, and workload management. Databricks Expertise Write scalable and optimized PySpark/Scala code for transformations and batch/streaming workloads. Work with MLflow, Databricks job clusters, workflows, and CI/CD integration. Integrate data pipelines with Azure Data Factory, Azure Functions, Event Hub, Service Bus, and other Azure services. Manage environments through Infrastructure as Code (IaC) using Terraform, ARM, or Bicep. Configure and maintain data security and governance using Azure AD, Key Vault, Purview, and role‑based access. Performance Optimization & Quality Ensure high‑performance processing and troubleshoot issues related to cluster performance, cost, and scalability. Implement data quality frameworks, unit testing, and observability through tools like DQ Frameworks, Databricks expectations, or Great Expectations. Collaboration & Stakeholder Interaction Work closely with data architects, analysts, and business stakeholders to understand requirements and deliver data solutions. Participate in Agile ceremonies, sprint planning, and documentation. Provide thought leadership by recommending modern tools, best practices, and optimization strategies. Required Skills & Qualifications Technical Skills 6–8 years of experience in data engineering, with at least 3+ years in Azure Databricks. Proficiency in PySpark/Scala, SQL, Delta Lake, and Databricks workflows. Strong experience with Azure Data Factory for orchestration and pipeline management. Hands‑on experience with Azure Data Lake Storage Gen2, Azure Synapse (SQL Pools or Serverless), Event Hub, and Key Vault. Solid understanding of data warehousing, distributed computing, and Lakehouse principles. Experience building and maintaining CI/CD pipelines using Azure DevOps or GitHub Actions. Knowledge of cloud security, encryption, access management, and compliance standards. Experience 6-8 Years Skills Primary Skill: Data Engineering Additional Skill(s): AI/ML Architecture, Data Engineering, databricks, Azure Data Factory, GenAI Fundamentals About the Company Infogain is a human‑centered digital platform and software engineering company based out of Silicon Valley. We engineer business outcomes for Fortune 500 companies and digital natives in the technology, healthcare, insurance, travel, telecom, and retail & CPG industries using technologies such as cloud, microservices, automation, IoT, and artificial intelligence. We accelerate experience‑led transformation in the delivery of digital platforms. Infogain is also a Microsoft (NASDAQ: MSFT) Gold Partner and Azure Expert Managed Services Provider (MSP). Infogain, an Apax Funds portfolio company, has offices in California, Washington, Texas, the UK, the UAE, and Singapore, with delivery centers in Seattle, Houston, Austin, Kraków, Noida, Gurgaon, Mumbai, Pune, and Bengaluru. #J-18808-Ljbffr