Lead Data Engineer

2 weeks ago


Chicago, United States Abacus Full time

Requisition Number
OMF-REQ-0001409

Number of Resources
3

Start Date
7/24/2023

End Date
1/23/2024

Work Location
IL Remote Location (IL Remote Location, Virtual, IL 00000)

Type of Service

Contingent Workforce - Technology

Job Title

Lead Data Engineer

Job Description
The Lead Data Engineer will report to the Marketing Technology (MarTech) team and will support the development of the customer centric strategy to increase automation and the use of data and analytics throughout the customer journeys. The candidate will be responsible for identifying relevant data and utilizing data tools, technologies and processes to develop continuous, data driven and automated customer communications across marketing and servicing towards omni channel personalized customer experience vision and outcomes.
CORE RESPONSIBILITIES
• Create and manage cloud resources in AWS
• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies
• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform
• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations
• Develop an infrastructure to collect, transform, combine and publish/distribute customer data.
• Define process improvement opportunities to optimize data collection, insights and displays.
• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible
• Identify and interpret trends and patterns from complex data sets
• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders.
• Key participant in regular Scrum ceremonies with the agile teams
• Proficient at developing queries, writing reports and presenting findings
• Mentor junior members and bring best industry practices
QUALIFICATIONS
• 7+ years' experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales)
• Strong background in math, statistics, computer science, data science or related discipline
• Advanced knowledge one of language: Java, Scala, Python, C#
• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake
• Proficient with:
• Data mining/programming tools (e.g. SAS, SQL, R, Python)
• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)
• Data visualization (e.g. Tableau, Looker, MicroStrategy)
• Comfortable learning about and deploying new technologies and tools.
• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines.
• Good written and oral communication skills and ability to present results to non-technical audiences
• Knowledge of business intelligence and analytical tools, technologies and techniques.
• Familiarity and experience in the following is a plus:
• AWS certification
• Spark Streaming
• Kafka Streaming / Kafka Connect
• ELK Stack
• Cassandra / MongoDB
• CI/CD: Jenkins, GitLab, Jira, Confluence other related tools

Skill/Experience/Education
Mandatory Skills dvanced knowledge one of language: Java, Scala, Python, C#? Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow,?Amazon Web Services (AWS),'Docker / Kubernetes,'Snowflake?? Proficient with Data mining/programming tools (e.g.'SAS, SQL, R, Python) Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)

Desired Skills --
Skill Matrix

No records found

dditional Requirements
No records found

Estimated Number of Regular Hours/Week
40.00

Shift
First Shift


  • Data Engineer

    2 weeks ago


    Chicago, United States HexawareAvance Consulting Full time

    The data engineer considers themselves an expert in data management with cloud technologies. The data engineer leads the definition and implementation of data pipelines, data storage, and data access/security for a complex Azure hosted application environment. They are responsible for curating a sustainable and scalable data environment, servicing a mix of...

  • Systems Engineer

    2 weeks ago


    Chicago, United States Data Theorem Full time

    Job DescriptionJob DescriptionData Theorem is an exciting company focused on creating a more secure world for data. Rooted in a strong engineer first culture, every employee has an impact on product and direction. We are searching for exceptional talent pursuing an opportunity to grow and take ownership of the projects that resonate most with them.Our...


  • Chicago, United States Abode Techzone LLC Full time

    Starburst AWS Lead Data Engineer 12 Months Contract Chicago, IL Onsite from Day 1 HealthCare Domain experience is a MUST LinkedIn ID is must NO Sub vending Consultant has to be local or Nearby location Qualifications • Bachelor's degree in Computer Science, Information Technology, or related field • Proven experience as a Starburst in...

  • Data Engineer

    2 weeks ago


    Chicago, United States BAI Full time

    Data Engineer     Reports To: Director, Product Development & Operations        BAI is the financial services industry’s leading provider of powerful tools, relevant content, and meaningful connections.  Our mission is to give financial services leaders the confidence to make smart business decisions because a strong financial services industry...

  • Data Engineer

    2 weeks ago


    Chicago, United States The LaSalle Group Full time

    Want to grow your career with one of the fastest growing industries? LaSalle Network is partnered with a top Chicago-based client looking to hire a Data Engineer their team. The Data Engineer will be responsible for delivering strategic and tangible solutions through data. We are looking for self-starters that have the ability to work cross-functionally...

  • Data Engineer

    2 weeks ago


    Chicago, United States Ci Software Full time

    *Data Engineer* *Location: Chicago, IL (Hybrid- 3 days Onsite) * *PR: $80-85/hr on W2**JOB DESCRIPTION* * Develops and troubleshoots data integration solutions with complex data transformations and provides guidance to other team members * Influences other team members to achieve commitments per guidance from Chapter Leads and actively contributes to agile...

  • Data Engineer

    1 week ago


    Chicago, Illinois, United States McDonald's Corporation Full time

    Job Description We are seeking a highly motivated individual for the Data Engineering Manager for the Marketer Customer Data Platform team on Global Data and Architecture.  This role will manage, develop, and lead the execution and implementation cloud-based data delivery systems for the MCD Customer Data Platform. This role will specialize in leading and...


  • Chicago, United States HirexHire Full time

    Our client is looking for an experienced Engineering Team Lead who can lead the development of an early stage data platform from inception to launch (0-1) and through the entire product lifecycle. You will partner with a Product Manager and your team Team Lead, Engineer, Data Engineer, Product Manager, Healthcare, Business Services, Staffing, Lead

  • Data Engineer

    1 month ago


    Chicago, Illinois, United States Trajektory Full time

    Based in downtown Chicago, IL, Trajektory is an innovative data insights and valuation technology start-up with a unique take on how to leverage and value data throughout the sports, media, and entertainment industries. Trajektory was founded by a group of former industry executives from throughout the sports, media, and entertainment industries - drawing...


  • Chicago, United States CME Group Full time

    Senior Data Reliability Engineer (Data SRE) CME Group is the world's leading and most diverse derivatives marketplace. But who we are goes deeper than that, here you can impact markets worldwide, transform industries and build a career shaping tomorrow. We invest in your success and you own it, all the while working alongside a team of leading experts who...

  • Lead Data Scientist

    4 weeks ago


    Chicago, United States Intelletec Full time

    This hands-on role is responsible for working closely with business partners to identify opportunities where the application of innovative data science techniques and methods can dramatically advance our business objectives. You will transform these priority opportunities into incremental business value through the design, development, and deployment of...

  • Lead Data Scientist

    4 weeks ago


    Chicago, United States Intelletec Full time

    This hands-on role is responsible for working closely with business partners to identify opportunities where the application of innovative data science techniques and methods can dramatically advance our business objectives. You will transform these priority opportunities into incremental business value through the design, development, and deployment of...


  • Chicago, United States Chicago Mercantile Exchange Inc. Full time

    Description Position Overview: Data System Reliability Engineer (dSRE) CME Group: Where Futures Are Made CME Group is the world's leading and most diverse derivatives marketplace. But who we are goes deeper than that, here you can impact markets worldwide, transform industries and build a career shaping tomorrow. We invest in your success and you own it, all...


  • Chicago, United States CME Group Full time

    Description Position Overview: Data System Reliability Engineer (dSRE) CME Group: Where Futures Are Made CME Group is the world's leading and most diverse derivatives marketplace. But who we are goes deeper than that, here you can impact markets worldwide, transform industries and build a career shaping tomorrow. We invest in your success and you own...


  • Chicago, United States CME Group Full time

    Description This role is hybrid requires to be 2 days on site in our Chicago office. This role does not allow to work outside of Illinois state. Position Overview: Data System Reliability Engineer (dSRE) CME Group: Where Futures Are Made CME Group is the world's leading and most diverse derivatives marketplace. But who we are goes deeper than...


  • Chicago, United States CME Group Full time

    Description Position Overview: Data System Reliability Engineer (dSRE) CME Group: Where Futures Are Made CME Group is the world's leading and most diverse derivatives marketplace. But who we are goes deeper than that, here you can impact markets worldwide, transform industries and build a career shaping tomorrow. We invest in your success and you own...

  • Staff Data Engineer

    4 weeks ago


    Chicago, Illinois, United States Motion Recruitment Full time

    This leading supplier of maintenance, repair, and operating products for businesses and a wide range of industries across the globe has a focus on sustainability, social responsibility, and community engagement, serving more than 4.5 million customers worldwide.The ideal candidate will have a strong background in data engineering, with a focus on Databricks,...


  • Chicago, United States PatientIQ Full time

    Are you interested in building the future of healthcare and transforming the patient experience? Are you hopeful about what data and medical research can do to improve medicine? We’re looking for a Sr. Manager, Data Engineering to ensure PatientIQ remains on the forefront of using data to drive positive healthcare outcomes. As the manager for our Data...


  • Chicago, Illinois, United States Caterpillar Full time

    Career Area: Business Technologies, Digital and Data Job Description: Your Work Shapes the World at Caterpillar Inc. When you join Caterpillar, you're joining a global team who cares not just about the work we do - but also about each other. We are the makers, problem solvers, and future world builders who are creating stronger, more sustainable...


  • Chicago, United States Cooperate Marketing, LLC Full time

    Job DescriptionJob Description                                                                                              Senior Data Engineer                                                                               Job Description & CompetenciesAbout usCooperate...