Lead Data Engineer

2 weeks ago

Newark, United States Hire Talent Full time
Lead Data Engineer VMS0033161 Prudential
Level 3 (potentially level 4)
Right to hire (conversion is possible)
Long term requirement (Initial couple months will want to join them FT)
Narayana Murthy is HM
Interview process: about 3 rounds of interviews

Notes:
  • Chief Data Office
  • Provided data to entire CEO offices.
  • EDP corporate team
  • AWS platform, Pyspark, Hadoop
  • Understanding technical requirements and give guidelines to the team.
  • Help implement the development structure for the team.
  • Acts as a bridge for development.
  • 2-3 members he will be leading.
  • AWS Solution architect certified is helpful. (nice to have but very useful)
  • Candidates need to understand why the solution is built and how does it solve the problem.
  • Sourcing the problem, creating a solution, looking to the market to create a solution (industry standard/new tech), implement the solution and look at.
  • Need to build end to end solutions with Pyspark, AWS glue, API Calls (Fullstack dev comes into play here)
  • They already have a pre-built data platform.
  • No coding at all, needs to know what needs to be done. Will work with the business and work as a bridge.
  • Bridge for EDP corporate team
  • Need to be able to explain the how they build pipeline and why they used such technology
Struggles
  • Candidates do not understand the business problem, what is the solution.
  • Candidates have never done solutions themselves.


9p Manager (Data Engineering Lead, Data Team) - Engineering lead to help the design/build of the CDO enterprise data projects.
Responsibilities
Implement and support
  • end-to-end data lake/warehousing/mart/business intelligence/ analytics/ services solutions (ingest, storage, integration, processing, services, access) in AWS
  • data lake data intake/request/onboarding services and service documentation
  • data lake ingestion services for batch/real time data ingest and service documentation
  • data processing services (ETL/ELT) for batch/real time (Glue/Kinesis/EMR) and service documentation
  • data storage services for data lake (S3)/ data warehouses (RDS/Redshift)/ data marts and service documentation
  • data services layer including Athena, Redshift, RDS, microservices and APIs
  • pipeline orchestration services including lambda, step functions, MWAA (optional)
  • data security services (IAM/KMS/SM/encryption/anonymization/RBAC) and service documentation
  • data access provisioning services (Accounts, IAM Roles RBAC), processes, documentation and education
  • data provisioning services for data consumption patterns including microservices, APIs and extracts
  • metadata capture and catalog services for data lake(S3/Athena), data warehouses (RDS/Redshift), Microservices/APIs
  • metadata capture and catalog services for pipeline/log data for monitoring /support
  • Implement CI/CD pipelines
  • Prepare documentation for data projects utilizing AWS based enterprise data platform
  • Implement high velocity streaming solutions using Amazon Kinesis, SQS, and SMS
  • Migrate data from traditional relational database systems to AWS relational databases such as Amazon RDS, Aurora, and Redshift
  • Migrate data from traditional file systems and NAS shares to AWS data lake (S3) and relational databases such as Amazon RDS, Aurora, and Redshift
  • Migrate data from APIs to AWS data lake (S3) and relational databases such as Amazon RDS, Aurora, and Redshift
  • Provide cost/spend monitoring reporting for AWS based data platform data initiatives
  • Provide governance/audit reporting for access of AWS based data platform
  • Lead the implementation of a data lake strategy to enable LOBs and Corporate Functions with a robust, holistic view of data - driven decision making
  • Serve as delivery lead for EDP data initiatives product owner
  • Partner with immediate engineering team, product owner, IT, partners on EDP agenda
  • Provide technology thought leadership, consulting, and coaching/mentoring
  • Establish development, qa, stage and production migration/support processes
  • Establish best practices for development and support teams
  • Deliver end-end data initiatives from ingest-consume via microservices/ apis, jdbc/ odbc, file extracts etc.
  • Work with scrum master to develop and own backlog, stories, epics, sprints
Qualifications
  • Bachelor's degree in Computer Science, Software Engineering, MIS or equivalent combination of education and experience
  • Experience implementing, supporting data lakes, data warehouses and data applications on AWS for large enterprises
  • Programming experience with Java, Python/ Scala, Shell scripting
  • Solid experience of AWS services such as CloudFormation, S3, Glue, EMR/ Spark, RDS, Redshift, DynamoDB, Lambda, Step Functions, IAM, KMS, SM etc.
  • Solid experience implementing solutions on AWS based data lakes
  • Experience implementing metadata solutions leveraging AWS non-relational data solutions such as ElastiCache and DynamoDB
  • AWS Solutions Architect or AWS Big Data Certification preferred
  • Experience in AWS data lake/data warehouse/business analytics
  • Experience and understanding of various core AWS services such as IAM, Cloud Formation, EC2, S3, EMR/Spark, Glue, Datasync, CloudHealth, CloudWatch, Lambda, Athena, and Redshift
  • Experience in system analysis, design, development, and implementation of data ingestion pipeline in AWS
  • Experience with DevOps and Continuous Integration/ Delivery (CI/ CD) concepts and tools
  • Experience with business intelligence tools such as Tableau, Power BI or equivalent
  • Knowledge of ETL/ ELT
  • Experience in production support from Level 1 to Level 3
  • Awareness of Data Management & Governance tools

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender, identity, national origin, disability, or protected veteran status.

  • Lead Data Engineer

    1 month ago

    Newark, United States MDMS Recruiting LLC Full time

    Job DescriptionJob Description***W2 or Self Incorporated consultants Only*** No C2C/No Corp to Corp***Lead Data Engineer for AWS data platform (to engineer applications integrating with enterprise data platform using AWS)Responsibilities:Designing, building and maintaining efficient, reusable, and reliable architecture and codeEnsure the best possible...

  • Lead Data Engineer

    4 weeks ago

    Newark, United States MDMS Recruiting LLC Full time

    Job DescriptionJob Description***W2 or Self Incorporated consultants Only*** No C2C/No Corp to Corp***Lead Data Engineer for AWS data platform (to engineer applications integrating with enterprise data platform using AWS)Responsibilities:Designing, building and maintaining efficient, reusable, and reliable architecture and codeEnsure the best possible...

  • Data Engineer Lead/Architect

    2 weeks ago

    Newark, New Jersey, United States Spectraforce Technologies Full time

    Job Title: Data Engineer Lead/ArchitectLocation: Newark, NJ - Hybrid (2/3 days onsite per week) Duration: 6+ Months (Temp to Perm) Pay rate starts from 70/hr.W2 onlyResponsibilities:Implement and support: end-to-end data lake/warehousing/mart/business intelligence/ analytics/ services solutions (ingest, storage, integration, processing, services, access)...

  • Data Engineer

    4 days ago

    Newark, United States Mars Incorporated Full time

    Job Description: Data and Analytics is foundation and core to our Analytics growth strategy We have data assets in all of our segments (Corporate, Petcare, MWC, Food, Multi Segment) across functions and divisions in R&D, that we will connect and enhance to build a rich understanding to answer specific questions which we as a business have. To deliver on...

  • Lead AWS Data Engineer

    2 weeks ago

    Newark, New Jersey, United States VSG Business Solutions LLC Full time

    Job DescriptionJob DescriptionLevel 3 (5 to < 10 Year)At-a-Glance:Are you ready to build your career by joining a global financial company? If so, our client is hiring a Lead AWS Data EngineerWhat You'll Do:Help lead the team in designing, building and maintaining efficient, reusable, and reliable architecture and code in AWS.Meet with the business to...

  • Data Engineer

    6 days ago

    Newark, United States Mars Full time

    Job Description:Data and Analytics is foundation and core to our Analytics growth strategy  We have data assets in all of our segments (Corporate, Petcare, MWC, Food, Multi Segment) across functions and divisions in R&D, that we will connect and enhance to build a rich understanding to answer specific questions which we as a business have.  To deliver on...

  • AWS Data Lead

    6 days ago

    Newark, United States Iris Software Inc. Full time

    Iris's client, a large insurance company, is currently searching for a strong AWS Data Engineer. Our client has offices worldwide and has the global reach, advisory services, and distribution power to meet the needs of issuers and investors worldwide. Job title: AWS Data Lead Location: Hybrid 3 days onsite (Newark, NJ)Duration: 24 monthsMust Skills: AWS...

  • AWS Data Lead

    1 week ago

    Newark, United States Iris Software Inc. Full time

    Iris's client, a large insurance company, is currently searching for a strong AWS Data Engineer. Our client has offices worldwide and has the global reach, advisory services, and distribution power to meet the needs of issuers and investors worldwide. Job title: AWS Data Lead Location: Hybrid 3 days onsite (Newark, NJ)Duration: 24 monthsMust Skills: AWS...

  • Data Engineer

    3 weeks ago

    Newark, United States Seven Seven Software Full time

    Key Job Responsibilities: Analyze data needs and objectives within the broader journey. Source, analyze and organize raw data, prepare data for transformation and consumption. Develop analytical tools and content. Identify ways to improve data governance, reliability, efficiency, and quality. Build applications ensuring that the code follows latest coding...

  • Data Engineer

    2 weeks ago

    Newark, United States JSR Tech Consulting Full time

    Data Tech team is looking for a Data Engineer to join a diverse team dedicated to providing best in class data services to our customers, stakeholders and partners. As a part of our organization, you will work with our Client Engineers, Data Scientist and various BUs to define solutions for operationalizing data-driven decision making in a cost effective...

  • Data Engineer

    2 weeks ago

    Newark, New Jersey, United States Phaxis Full time

    Junior Data Engineer (3+ years): Salary is $110k-$140k + bonusSenior Data Engineer (5+ years): Salary is $135k-$155k + bonus3 days remote and 2 days onsite in Newark, NJAll candidates will need to complete a Python assessmentMust Have Qualifications:3+ experience in building data and analytics platform focused on Azure data and analytics solutions.Strong...

  • Data Engineer

    2 months ago

    Newark, United States CognITek Group Full time

    We have a challenging opportunity for a Data Engineer for a high-growth and highly acquisitive $1B+ residential services company in Newark DE (this position can work remotely with periodic visits to corporate).The Data Engineer will design and maintain our client’s data Infrastructure and ETL processes and will collaborate with an amazing Data Analytics...

  • Data Engineer

    2 months ago

    Newark, United States CognITek Group Full time

    We have a challenging opportunity for a Data Engineer for a high-growth and highly acquisitive $1B+ residential services company in Newark DE (this position can work remotely with periodic visits to corporate).The Data Engineer will design and maintain our client’s data Infrastructure and ETL processes and will collaborate with an amazing Data Analytics...

  • Data Engineer

    4 weeks ago

    Newark, United States CognITek Group Full time

    We have a challenging opportunity for a Data Engineer for a high-growth and highly acquisitive $1B+ residential services company in Newark DE (this position can work remotely with periodic visits to corporate).The Data Engineer will design and maintain our client’s data Infrastructure and ETL processes and will collaborate with an amazing Data Analytics...

  • Data Engineer

    4 weeks ago

    Newark, United States CognITek Group Full time

    We have a challenging opportunity for a Data Engineer for a high-growth and highly acquisitive $1B+ residential services company in Newark DE (this position can work remotely with periodic visits to corporate).The Data Engineer will design and maintain our client’s data Infrastructure and ETL processes and will collaborate with an amazing Data Analytics...

  • Sr. Data Engineer

    4 weeks ago

    Newark, United States Summit Tech Partners Full time

    Job Description : 8p - Engineering lead to help the design/build of the CDO enterprise data platforms Design, Develop and Implement End-to-end data solutions (ingest, storage, integration, processing, access) on AWS. Data intake/ request/ onboarding services and service documentation. Data ingestion services for batch/ real time data ingest and service...

  • Data Engineer

    2 weeks ago

    Newark, United States Hire Talent Full time

    High level requirements for the role: Data ingestion from various on-prem and cloud sources Ability to build ETL pipelines to transform raw data into format suitable for analysis (hands on experience working with ETL tools like AWS Glue, Informatica etc.) and orchestrate transformation (using Airflow, other DAG tools) Experience working with big data...

  • Data Engineer

    7 days ago

    Newark, United States Spectraforce Technologies Full time

    Job Title: Data Engineer Location: Newark, NJ (Hybrid) Duration: 6+ Months Job Description: 10+ years of experience in building out Data pipelines in Java/Scala 5 + years of experience working in AWS Cloud especially services like S3, EMR, Lambda, AWS Glue and Step Functions. 5+ years of experience with Spark Any experience with Microsoft Fabric is a...

  • Lead Software Engineer

    1 month ago

    Newark, United States Dice Full time

    Dice is the leading career destination for tech experts at every stage of their careers. Our client, MDMS Recruiting, is seeking the following. Apply via Dice today! Lead Software Engineer Location: Newark, NJ (hybrid), Local Candidates only, w2 or Self Inc, no 3rd party Duration: 12+ (Right to hire) As a Lead Software Engineer in the Technology...

  • Lead Software Engineer

    4 weeks ago

    Newark, United States Dice Full time

    Dice is the leading career destination for tech experts at every stage of their careers. Our client, MDMS Recruiting, is seeking the following. Apply via Dice today! Lead Software Engineer Location: Newark, NJ (hybrid), Local Candidates only, w2 or Self Inc, no 3rd party Duration: 12+ (Right to hire) As a Lead Software Engineer in the Technology...