Senior Data Engineer

4 months ago

Dallas, United States Anblicks Full time
Job Title: Senior Data Engineer

JOB DUTIES:

Participate in daily agile and scrum processes to understand changing business requirements, including examining system configurations and operating procedures, as well as functional requirements gathering. Ingest and prepare business-ready data in the cloud using Azure Data Factory (ADF) to build ELT(Extract Load and Transform)/ETL (Extract Transform and Load) data pipelines, then move the data into a data warehouse (Dedicated SQL Pool) and create data lake zones for data analytics and visualization. Work with a combination of Azure Data Factory and Azure Databricks, extract, load, and transform data from cloud sources and on-premises databases such as Oracle, SAP, and SQL Server to Data Lake Storage and Azure Synapse Analytics. Create an Azure Data Factory Pipeline Template to migrate an on-premises data platform to the Azure Cloud using batch processing methods by incremental or full load, and an ADF config driven framework to pull data from multiple sources with different table structures with less manual work and less resource usage. Analyze, design, and build modern data solutions using Azure PaaS services to support data visualization. Understand the current production state of the application and the impact of new implementation on existing business processes. Enable private end point, firewall setting and Azure Key Vault for robust data security. Analyze the existing SSIS packages and integrate it with Azure Data Factory, using SSIS transformations like Lookup, Derived column, Data conversion, Aggregate, Conditional split, SQL task, Script task, and Send Mail task. Create a JSON structure for data storage in Azure Cosmos DB (SQL API), write stored procedures and functions and work with the API team to create Cosmos DB queries that use fewer request units. Data Model in Snowflake and ELT using Snowflake SQL, implementing complex stored procedures and best practices with data warehouse and ETL concepts. Design and customize dimension data models for data warehouse supporting data using Azure Synapse Analytics, and select the appropriate distribution method in dimension and fact tables to load the data in an optimized manner, as well as implement complex stored procedures and best practices with data warehouse. Build a distributed in-memory application using spark applications and perform analytics efficiently on large datasets using python and Spark SQL, also use Spark SQL to implement transformation logic in Databricks and mount/unmount Azure Blob Storage. Read the data from different file format parquet, avro, csv and json using pySpark (Python API) in Azure Databricks and perform data extraction, transformation to uncover insights into customer usage patterns and insert curated data into a data warehouse. Create data visualization reports and dashboards in Power BI using data from the data warehouse, flat files, and Azure SQL. Responsible for fixing problems and conducting investigations into SQL queries, Stored Procedures related to long running jobs and Azure service performance. Utilize Azure Monitor and Alert services, create monitors, alerts, and notifications for Data Factory, Synapse Analytics, and Data Lake Storage. Perform the required daily GIT support for various projects. Be in charge of maintaining GIT repositories and access control procedures. Create CI&CD using azure dev ops pipeline to deploy Azure Services (Storage, Data factory, Key vault & Logic App) using ARM Templates.

JOB REQUIREMENT:

Master's degree in Computer Science, Computer Information Systems, or Engineering related or Technical related fields plus 2 years of experience. In lieu of the above, we will also accept a Bachelor's degree in Computer Science, Computer Information Systems, or Engineering related or Technical related fields plus 5 years of progressively responsible post-baccalaureate experience. Foreign degree equivalent is acceptable. We will also accept any suitable combination of education, training and/or experience. Experience to include working on Azure Data Factory (ADF), Oracle, SQL server, Azure Databricks, Azure Synapse Analytics, Data Lake Storage, Azure PaaS services, SSIS packages, Azure Cosmos DB (SQL API), Python, Spark applications, Azure SQL, SQL queries, Azure Monitor and Alert services, GIT Support, ARM Templates.

HOURS: M-F, 8:00 a.m. - 5:00 p.m.

JOB LOCATION: Dallas, Texas. Travel is not required, but candidates must be willing to relocate to unanticipated work locations across the country per contract demand.

CONTACT: Email resume referencing job code# SDE01302023ANB to Maruthi Technologies INC. DBA Anblicks at [email protected]

  • Data Engineer

    1 month ago

    Dallas, Texas, United States NTT DATA Full time

    Job SummaryNTT DATA Services is seeking a highly skilled Data Engineer to join our team in Dallas, Texas. As a Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data systems to support our business operations.Key Responsibilities:Develop and maintain data pipelines to extract, transform, and load data from diverse...

  • Data Engineer

    5 months ago

    Dallas, United States NTT DATA Full time

    Job Description NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer (Onsite) to join our team in Dallas, Texas (US-TX), United States (US).Job Responsibilities...

  • Senior Data Engineer

    6 days ago

    Dallas, United States InfoVision Inc. Full time

    Senior Data EngineerDallas, TXLong term contractJob Description:We are looking for a Data Engineer to join our Advanced Analytics team under the Wholesale organization. This engineer will design and deliver data solutions for the analytics and data science teams in the consumer organization. The engineer will be able to work with all types of interesting...

  • Senior Linux Engineer

    3 weeks ago

    Dallas, United States NTT DATA Full time

    Req ID: 299982 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Senior Linux Engineer - Irving, Texas to join our team in Irving, Texas (US-TX), United States (US).NTT Data Americas is...

  • Senior Data Engineer

    4 months ago

    Dallas, United States Saviance Full time

    Job Title-Senior Data Engineer Location: Remote Duration: 6 Months Contract to Hire (Quarterly Onsite for a few days in Dallas TX) About BigRio: BigR.io is a technology consulting firm empowering data to drive innovation and advanced analytics. We specialize in cutting-edge Big Data, Machine Learning, and Custom Software strategy, analysis,...

  • Senior Data Engineer

    1 month ago

    dallas, United States TopShelf Talent Group Full time

    Our client is a well-known investment management firm in the Dallas area and they're seeking a Senior Data Engineer to join their team.They need a Senior Data Engineer with 8+ years of hands-on expertise in data engineering, with a focus on creating reliable, efficient, and scalable systems. This role is pivotal in shaping the data architecture and pipelines...

  • Senior Data Engineer

    2 months ago

    Dallas, United States TopShelf Talent Group Full time

    Our client is a well-known investment management firm in the Dallas area and they're seeking a Senior Data Engineer to join their team.They need a Senior Data Engineer with 8+ years of hands-on expertise in data engineering, with a focus on creating reliable, efficient, and scalable systems. This role is pivotal in shaping the data architecture and pipelines...

  • Senior Data Engineer

    2 months ago

    dallas, United States TopShelf Talent Group Full time

    Our client is a well-known investment management firm in the Dallas area and they're seeking a Senior Data Engineer to join their team.They need a Senior Data Engineer with 8+ years of hands-on expertise in data engineering, with a focus on creating reliable, efficient, and scalable systems. This role is pivotal in shaping the data architecture and pipelines...

  • Senior Data Systems Engineer

    1 month ago

    Dallas, Texas, United States Shtudy Full time

    Job Title: Senior Data Systems EngineerWe are seeking a highly skilled Senior Data Systems Engineer to join our dynamic team at Shtudy. This role requires a strong background in data engineering, cloud technologies, and quality assurance.Key Responsibilities:Automate ETL pipelines using Python to streamline data integration, transformation, and migration...

  • Senior Data Engineering Architect

    4 months ago

    Dallas, United States Anblicks Full time

    Senior Data Engineering Architect Provide enterprise-level architectural solutions for applications using MS Stack technologies/concepts. Assess business requirements and provide design leadership to address impact analysis to process flows, systems, and applications. Technically lead and mentor Data Architects & Engineers building end to end Enterprise Data...

  • Senior Scrum Master

    4 weeks ago

    dallas, United States Tiva's Client Full time

    Senior Scrum Master (Data Analytics and Data Engineering Teams)We are looking for a Senior Scrum Master who has driven projects/programs that are data heavy projects around Data Analytics, Data Engineering, Data Curation, Data Ingestion, Cloud Data etc because they are supporting Data Engineering and Data Analytics Team.ResponsibilitiesFacilitating daily...

  • Senior Scrum Master

    4 weeks ago

    Dallas, United States Tiva's Client Full time

    Senior Scrum Master (Data Analytics and Data Engineering Teams)We are looking for a Senior Scrum Master who has driven projects/programs that are data heavy projects around Data Analytics, Data Engineering, Data Curation, Data Ingestion, Cloud Data etc because they are supporting Data Engineering and Data Analytics Team.ResponsibilitiesFacilitating daily...

  • Senior Scrum Master

    3 weeks ago

    dallas, United States Tiva's Client Full time

    Senior Scrum Master (Data Analytics and Data Engineering Teams)We are looking for a Senior Scrum Master who has driven projects/programs that are data heavy projects around Data Analytics, Data Engineering, Data Curation, Data Ingestion, Cloud Data etc because they are supporting Data Engineering and Data Analytics Team.ResponsibilitiesFacilitating daily...

  • Senior Data Engineer

    4 weeks ago

    dallas, United States Talent Groups Full time

    Direct Hire - the client is unable to offer sponsorship at this time. Hybrid: Mostly remote role in Dallas, TX. Must be able to interview onsite and work onsite 1x a month. Client is located near 75/635. Our client is seeking a Senior Data Engineer from a larger-scale healthcare production environment of 500 KB - up to 5 TB....

  • Senior Data Engineer

    4 weeks ago

    dallas, United States Talent Groups Full time

    Direct Hire - the client is unable to offer sponsorship at this time. Hybrid: Mostly remote role in Dallas, TX. Must be able to interview onsite and work onsite 1x a month. Client is located near 75/635. Our client is seeking a Senior Data Engineer from a larger-scale healthcare production environment of 500 KB - up to 5 TB....

  • Senior Data Engineer

    4 weeks ago

    Dallas, United States Talent Groups Full time

    Direct Hire - the client is unable to offer sponsorship at this time. Hybrid: Mostly remote role in Dallas, TX. Must be able to interview onsite and work onsite 1x a month. Client is located near 75/635. Our client is seeking a Senior Data Engineer from a larger-scale healthcare production environment of 500 KB - up to 5 TB....

  • Senior Data Engineer

    1 month ago

    dallas, United States Murphy USA Full time

    DATA ENGINEER VMurphy USA is in the midst of a technological transformation, investing heavily in our team and technologies to drive business impact. We are looking for a talented Data Engineer to build and maintain our data infrastructure, ensuring our analytical capabilities stay ahead of the curve.GENERAL DESCRIPTION OF POSITIONThe Data Engineer will be...

  • Senior Data Engineer

    4 weeks ago

    Dallas, United States Murphy USA Full time

    DATA ENGINEER VMurphy USA is in the midst of a technological transformation, investing heavily in our team and technologies to drive business impact. We are looking for a talented Data Engineer to build and maintain our data infrastructure, ensuring our analytical capabilities stay ahead of the curve.GENERAL DESCRIPTION OF POSITIONThe Data Engineer will be...

  • Senior Data Engineer

    4 weeks ago

    dallas, United States Murphy USA Full time

    DATA ENGINEER VMurphy USA is in the midst of a technological transformation, investing heavily in our team and technologies to drive business impact. We are looking for a talented Data Engineer to build and maintain our data infrastructure, ensuring our analytical capabilities stay ahead of the curve.GENERAL DESCRIPTION OF POSITIONThe Data Engineer will be...

  • Senior Data Engineer

    4 weeks ago

    dallas, United States Murphy USA Full time

    DATA ENGINEER VMurphy USA is in the midst of a technological transformation, investing heavily in our team and technologies to drive business impact. We are looking for a talented Data Engineer to build and maintain our data infrastructure, ensuring our analytical capabilities stay ahead of the curve.GENERAL DESCRIPTION OF POSITIONThe Data Engineer will be...