Sr.AzureDataEngineer

Job Title: Senior Azure Data EngineerLocation: Frisco, TX/Atlanta-GADuration: Long term Job Responsibilities:Design, develop, and implement robust and scalable data solutions utilizing Azure technologies, including Azure Data Factory, Azure SQL Data Warehouse, Azure Databricks, and/or Azure Synapse Analytics.Proficiency in programming languages such as Spark (with either Python or Scala) and SQL.Experience with various databases including SQL Server, Teradata, Snowflake, and Synapse.Solid understanding of data engineering principles, data modeling, data warehousing, and ETL/ELT processes, encompassing data testing, validation, and reconciliation procedures.Hands-on experience with data integration and transformation frameworks, tools, and methodologies.Familiarity with version control systems like Git, GitHub, etc.Collaborate with cross-functional and business teams to grasp business requirements and translate them into technical designs and solutions.Develop and maintain data pipelines, integrations, and transformations to facilitate efficient data processing, storage, and retrieval.Optimize data infrastructure and solutions for performance, scalability, and cost-efficiency, ensuring high availability and reliability.Conduct data profiling, validation, and cleansing activities to maintain data integrity and accuracy.Provide mentorship and technical guidance to junior data engineers, interns, and s, promoting knowledge sharing and skill development within the team. Good to have:Experience with version control systems, CI/CD pipelines, and automated testing frameworks. 1. **Snowflake Database Management:** - Can you explain the process of creating and managing database schemas in Snowflake? - What strategies do you use for optimizing Snowflake databases and queries to ensure efficient data processing? 2. **Snowflake Modeling:** - Describe a specific instance where you implemented Slowly Changing Dimension (SCD) in Snowflake. What challenges did you face, and how did you overcome them? - How do you handle surrogate key assignment and change data capture in Snowflake modeling? 3. **ELT/ETL Pipelines with Snowflake:** - Can you provide an example of a complex ELT/ETL pipeline you developed using Python and Snowflake Snow SQL? - How do you ensure the reliability and scalability of data movements in your ELT/ETL pipelines? 4. **Snowflake Integration with Other Tools:** - Explain the process of integrating Snowflake with Tableau for creating interactive dashboards and reports. - How do you collaborate with cross-functional teams, including data scientists and analysts, to deliver data solutions? 5. **Continuous Integration and Deployment (CI/CD):** - How do you use Azure DevOps for continuous integration and deployment of data pipelines related to Snowflake? - Can you describe a scenario where CI/CD improved the efficiency of your Snowflake-related projects? 6. **Data Archiving and Retention:** - Discuss the strategies you employed for data archiving using Azure Blob Storage in conjunction with Snowflake's Time Travel feature. - How do you determine the appropriate data retention policies for different types of data in Snowflake? 7. **Snowflake Staging:** - Explain the role of Snowflake stages in loading data efficiently into Snowflake tables. - Can you provide an example of a challenging data loading scenario and how you optimized it using Snowflake stages? 8. **Azure Data Factory:** - How do you handle end-to-end operations of ETL data pipelines in Azure Data Factory? - Share an experience where you had to troubleshoot and resolve an issue in Azure Data Factory. 9. **Azure Databricks:** - Discuss the advantages of integrating Azure Databricks with Azure Data Lake Storage for scalable big data processing. - Provide an example of a complex Spark job you designed in Azure Databricks for enhanced performance.
Sr.AzureDataEngineer

---