Azure Data Engineer

1 week ago

Chicago, IL, United States 1872 Consulting Full time
zure Data Engineer
NYC, NY (Midtown) - hybrid WFH

Summary
s a crucial member of our Data and AI team, this role is pivotal in driving our data strategy and achieving our analytical goals. The individual will play an essential part in transforming raw data into actionable insights, thereby empowering the team to make data-driven decisions that enhance our financial services. With a focus on innovation and precision, the role ensures the integrity and availability of data, which is fundamental to our reporting, analytics, and machine learning initiatives. By developing and maintaining robust data pipelines and implementing efficient ETL processes, the role supports the team's mission to deliver unparalleled financial solutions and services to our clients.

What you'll be doing

General
  • Interact with business stakeholders at all levels to understand data needs.
  • Collaborate with in-house resources and consultants acting as data scientists, technical analysts, business report writers, and other SMEs to deliver high-quality data solutions.
  • Coordinate and communicate with members of the IT team regularly to ensure smooth IT operations.
  • Stay up to date with the latest trends and best practices in data engineering and related technologies.

Data Ingestion
  • Design, develop, deploy, and maintain scalable data pipelines and ETL processes using Microsoft Fabric.
  • Ingest data from various sources including, but not limited to, REST APIs, flat files, SQL databases, SharePoint libraries and lists, and Azure storage.
  • Optimize data ingestion method for each source considering factors such as data availability from vendor, frequency of data updates, resource usage, cost, etc.
  • Implement data integration solutions to consolidate data from various sources into a unified data platform.

Data Transformation
  • Perform data transformations at all layers of a medallion architecture to support business reporting, analytics, and machine learning needs.
  • Create and maintain notebooks, stored procedures, and other data transformation methods, leveraging version control.
  • utomate data transformations through scheduling and data pipelines
  • Implement data security using row-level and column-level security, and other available methods.

Platform Maintenance
  • Optimize and tune data workflows for performance and reliability.
  • Ensure data quality, integrity, and security across all data processes.
  • Monitor platform performance and provide utilization analysis and recommendations on required resources.
  • Develop and maintain documentation for data engineering processes and solutions.
Skills we're seeking
  • 2+ years of experience with Azure Data Engineering
  • Proficiency in Microsoft Fabric and related technologies (e.g., Azure Data Factory, Azure Synapse Analytics, etc.)
  • Strong SQL experience
  • Experience with Python programming
  • Experience with Spark

Nice to haves (in no particular order)
  • T-SQL experience
  • Experience with Data Modeling
  • Experience with Python libraries, specifically Pandas and MLlib
  • Experience with Data Science/Machine Learning
  • Experience with AzureDevOps
  • Scala or other Functional Programming experience
  • zure certifications
  • Experience working in financial services
  • Bachelor's or Master's Degree in Computer Science or another IT/Data related field

  • Azure Data Engineer

    2 weeks ago

    Chicago, IL, United States CData Software Full time

    As a Microsoft Azure Data Engineer, you will be responsible for designing, implementing, and managing data pipelines and architectures on the Azure Synapse platform. You will work with a variety of data sources and ensure data is processed and stored efficiently using the Medallion Architecture (Bronze, Silver, Gold layers). Your expertise in PySpark,...

  • Azure Data Engineer

    2 days ago

    Chicago, IL, United States CData Software Full time

    As a Microsoft Azure Data Engineer, you will be responsible for designing, implementing, and managing data pipelines and architectures on the Azure Synapse platform. You will work with a variety of data sources and ensure data is processed and stored efficiently using the Medallion Architecture (Bronze, Silver, Gold layers). Your expertise in PySpark,...

  • Azure Data Engineer

    6 days ago

    Chicago, IL, United States CData Software Full time

    As a Microsoft Azure Data Engineer, you will be responsible for designing, implementing, and managing data pipelines and architectures on the Azure Synapse platform. You will work with a variety of data sources and ensure data is processed and stored efficiently using the Medallion Architecture (Bronze, Silver, Gold layers). Your expertise in PySpark,...

  • Azure Data Engineer- Snowflake

    6 days ago

    Chicago, IL, United States Spruce Infotech Full time

    Overview: Looking for an experienced Azure Data Engineer with strong Snowflake expertise to build and optimize enterprise data solutions for Northern Trust's financial services environment. Key Responsibilities: Design, develop, and maintain scalable Snowflake data warehouses and Azure-based data pipelines. Implement ETL/ELT processes using Azure Data...

  • Azure Data Engineer- Snowflake

    7 days ago

    Chicago, IL, United States Spruce Infotech Full time

    Overview: Looking for an experienced Azure Data Engineer with strong Snowflake expertise to build and optimize enterprise data solutions for Northern Trust's financial services environment. Key Responsibilities: Design, develop, and maintain scalable Snowflake data warehouses and Azure-based data pipelines. Implement ETL/ELT processes using Azure Data...

  • Azure Data Engineer

    2 weeks ago

    Chicago, IL, United States RIT Solutions, Inc. Full time

    Job Title - Azure Data Engineer Location - Chicago, IL - Remote Duration - 4+ months Interview - Video Job Description - Responsibilities: Designs and develops data pipelines that extract data from various sources, transform it into the desired format, and load it into the appropriate data storage systems Collaborates with architects, data...

  • Azure Data Engineer

    7 days ago

    Chicago, IL, United States RIT Solutions, Inc. Full time

    Job Title - Azure Data Engineer Location - Chicago, IL - Remote Duration - 4+ months Interview - Video Job Description - Responsibilities: Designs and develops data pipelines that extract data from various sources, transform it into the desired format, and load it into the appropriate data storage systems Collaborates with architects, data...

  • Azure Data Engineer

    6 days ago

    Chicago, IL, United States RIT Solutions, Inc. Full time

    Job Title - Azure Data Engineer Location - Chicago, IL - Remote Duration - 4+ months Interview - Video Job Description - Responsibilities: Designs and develops data pipelines that extract data from various sources, transform it into the desired format, and load it into the appropriate data storage systems Collaborates with architects, data...

  • Azure Data Engineer

    1 week ago

    Chicago, IL, United States Capgemini Full time

    Azure Data Engineer Choosing Capgemini means choosing a company where you will be empowered to shape your career in the way you'd like, where you'll be supported and inspired by?a collaborative community of colleagues around the world, and where you'll be able to reimagine what's possible. Join us and help the world's leading organizations?unlock the value...

  • Azure Data Engineer

    2 weeks ago

    Chicago, IL, United States Capgemini Full time

    Azure Data Engineer Choosing Capgemini means choosing a company where you will be empowered to shape your career in the way you'd like, where you'll be supported and inspired by?a collaborative community of colleagues around the world, and where you'll be able to reimagine what's possible. Join us and help the world's leading organizations?unlock the value...