AWS Data Engineer with Big Data Processing

4 weeks ago


Piscataway, New Jersey, United States Accord Technologies Inc. Full time
Job Description

Data Engineering Role

A highly skilled AWS Data Engineer is required to design and implement robust data pipelines for big data processing. The ideal candidate will have expertise in AWS services such as EMR, S3, RDS, and Redshift, as well as experience with container orchestration using Kubernetes.

Key Responsibilities

  • Data Pipeline Development: Design and implement ETL processes to extract, transform, and load data from various sources into data lakes and warehouses.
  • AWS EMR Clusters: Configure, manage, and optimize Amazon EMR clusters for big data processing using Apache Spark, Hive, or Presto.
  • Container Orchestration: Utilize Kubernetes for deploying, scaling, and managing containerized applications and services.
  • CI/CD Implementation: Develop and maintain CI/CD pipelines for automated deployment of data applications and services using tools like Jenkins, GitLab CI, or AWS CodePipeline.
  • SQL Development: Write complex SQL queries for data manipulation and retrieval, ensuring high performance and scalability.
  • Data Quality and Governance: Implement data quality checks, monitoring, and logging mechanisms to ensure data reliability and compliance.
  • Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
  • Documentation: Maintain comprehensive documentation of data architecture, processes, and workflows.

Qualifications

  • Education: Bachelor's degree in Computer Science, Data Science, Information Technology, or a related field.
  • Experience: 3+ years of experience in data engineering or a related role, with a focus on AWS technologies.
  • AWS Expertise: Proficient in AWS services such as EMR, S3, RDS, Redshift, Lambda, and CloudFormation.
  • Kubernetes Knowledge: Experience with Kubernetes for container orchestration and microservices architecture.
  • CI/CD Tools: Familiarity with CI/CD tools and practices, including version control using Git.
  • SQL Proficiency: Strong knowledge of SQL, with experience in relational databases (e.g., MySQL, PostgreSQL) and data warehousing solutions.
  • Programming Skills: Proficiency in programming languages such as Python, Java, or Scala for data processing tasks.
  • Problem-Solving Skills: Ability to troubleshoot complex data issues and optimize performance.
  • Communication: Excellent communication skills with the ability to work collaboratively in a team environment.


  • Piscataway, New Jersey, United States Lorven Technologies Full time

    Hi, Lorven Technologies is seeking a highly skilled Data Engineer with expertise in cloud services to join our team. This is a long-term contract position that offers a unique opportunity to work with cutting-edge technology and contribute to the success of our clients. As a Data Engineer with Lorven Technologies, you will be responsible for designing,...

  • Data Engineer

    4 weeks ago


    Piscataway, New Jersey, United States Colgate-Palmolive Full time

    Job SummaryWe are seeking a highly skilled Data Engineer to join our team at Colgate-Palmolive. As a Data Engineer, you will be responsible for developing digital infrastructure and data pipelines to accelerate product innovation, product design, and product development. You will work closely with cross-functional teams to gather, wrangle, cleanse,...


  • Piscataway, New Jersey, United States Colgate-Palmolive Full time

    We are seeking a highly skilled Data Engineering Manager to lead our team in building and maintaining data pipelines that support our analytics strategy.The ideal candidate will have a strong background in data engineering tools and methods, with expertise in code and platform deployment. They will be responsible for enforcing data engineering standards,...


  • Piscataway, New Jersey, United States Colgate-Palmolive Full time

    Job Title: Data Engineering ManagerAbout the Role:We are seeking a highly skilled Data Engineering Manager to lead our team in building and maintaining scalable data pipelines. As a Data Engineering Manager, you will be responsible for leading a technical team to design, develop, and deploy data engineering solutions that meet the needs of our business.Key...


  • Piscataway, New Jersey, United States Synechron Full time

    Job Title:AWS EMR Security SpecialistJob Description:We are seeking an experienced AWS EMR Security Specialist to join our team in the healthcare informatics sector. The successful candidate will be responsible for identifying and resolving security vulnerabilities within our AWS EMR environment to ensure the protection of sensitive healthcare data.Key...


  • Piscataway, New Jersey, United States Colgate-Palmolive Full time

    About the RoleWe are seeking a skilled Data Engineering Platform Owner to join our team at Colgate-Palmolive. As a key member of our Data Engineering team, you will be responsible for enhancing our platforms, implementing governance, and maximizing value.Key ResponsibilitiesLead Agile Sprints, utilizing the expertise of platform engineers to drive platform...


  • Piscataway, New Jersey, United States Synechron Full time

    Job Title: AWS Cloud Engineering LeadJob Summary:We are seeking an experienced AWS Cloud Engineering Lead to join our team at Synechron. The successful candidate will be responsible for designing, implementing, and maintaining cloud services for our developers.Key Responsibilities:Develop and maintain cloud infrastructure using Amazon Web Services (AWS)...

  • Product Engineer

    4 weeks ago


    Piscataway, New Jersey, United States Tiger Analytics Full time

    Tiger Analytics is a leading advanced analytics consulting firm, trusted by Fortune 100 companies to generate business value from data. Our consultants bring deep expertise in Data Science, Machine Learning, and AI, recognized by market research firms like Forrester and Gartner.We are seeking a skilled Product Engineer to build enterprise-grade scalable...


  • Piscataway, New Jersey, United States First Tek Full time

    As a Cloud Data Engineer at First Tek, Inc., you will be responsible for developing custom programs and applications using technical skills in Java, J2EE, JSP, JavaScript, Data Engineering, and Data Analytics. You will have expertise in Python, SQL, BigData, and AWS services like S3, Redshift, EMR, Kinesis, and Apache Spark. You will work in a Datawarehouse...


  • Piscataway, New Jersey, United States Amazon Full time

    About the RoleWe are seeking a highly skilled Data Center Operations Technician to join our team at Amazon. As a Data Center Operations Technician, you will be responsible for the day-to-day operations and maintenance of mechanical and electrical equipment in our data centers.Key ResponsibilitiesOversee the day-to-day operations and maintenance of mechanical...


  • Piscataway, New Jersey, United States SHI GmbH Full time

    Job SummaryThe Data Analytics Manager at SHI GmbH plays a crucial role in guiding the collection, analysis, and interpretation of data to aid strategic decision-making within the organization. This position entails leading a team of analysts, ensuring the accuracy and reliability of data, and creating detailed dashboards that offer actionable insights. The...


  • Piscataway, New Jersey, United States Quality Technology Services, LLC Full time

    Job SummaryThe Data Center Operations Specialist is responsible for the installation, repair, and maintenance of IT, electrical, mechanical, plumbing/piping equipment, machinery, and controls located in the interior and exterior of the building.This role monitors operations and conducts a routine and ongoing assessment of the building systems, operations,...


  • Piscataway, New Jersey, United States Marsh LLC Full time

    Job SummaryMarsh McLennan Agency is seeking a highly skilled Senior Data Security Architect to join our team. As a key member of our Data team, you will be responsible for establishing consistent data standards, reference architectures, patterns, and practices across the organization.Key ResponsibilitiesDefine reference data architecture and work with agile...

  • Data Science Manager

    4 weeks ago


    Piscataway, New Jersey, United States Jobs for Humanity Full time

    Job DescriptionJobs for Humanity is collaborating with Upwardly Global and Rutgers Ag to build an inclusive and just employment ecosystem. We support individuals coming from all walks of life.Job Title: Professor - Data Science - TenuredRutgers, The State University of New Jersey (New Brunswick, NJ, USA) invites applications for a distinguished data...


  • Piscataway, New Jersey, United States SAAS Technologies, Inc. Full time

    Job SummarySAAS Technologies, Inc. is seeking a highly skilled SAS Data Analytics Specialist to join our team. The ideal candidate will have a strong background in data modeling and analytics, with expertise in SAS, MS SQL, Tableau, and Python. The successful candidate will be responsible for designing and implementing custom application data models,...


  • Piscataway, New Jersey, United States QTS Realty Trust , Inc. Full time

    Data Center Operations SpecialistAt QTS Realty Trust, Inc., we are seeking a highly skilled Data Center Operations Specialist to join our team. As a key member of our operations team, you will be responsible for the installation, repair, and maintenance of IT, electrical, and mechanical equipment in our data centers. Your expertise will ensure the smooth...


  • Piscataway, New Jersey, United States Softek International Inc Full time

    Job DescriptionAt Softek International Inc, we are seeking a highly skilled Senior Splunk Engineer to join our team. The ideal candidate will have a strong background in Splunk architecture, design, and implementation, as well as experience in leading engineering efforts and mentoring team members.Key Responsibilities:Design, implement, and maintain Splunk...

  • Project Manager

    4 weeks ago


    Piscataway, New Jersey, United States Sunbird Software Inc. Full time

    Project Manager Role at Sunbird Software Inc.Sunbird, a leading provider of Data Center Infrastructure Management solutions, is seeking a skilled Project Manager to support our engineering team. Our award-winning web-based solutions help clients understand and manage energy consumption, environmental conditions, and assets in their facilities.The successful...


  • Piscataway, New Jersey, United States National Manufacturing Full time

    Job DescriptionNational Manufacturing Co., Inc. is a leading supplier of deep and shallow draw metal forming services, providing critical components to sophisticated customers worldwide.With over 70 years of experience, the company has established itself as a specialist in deep draw technologies, producing precision deep drawn enclosures and shallow drawn...


  • Piscataway, New Jersey, United States Accord Technologies Inc. Full time

    Job DescriptionAs a seasoned Cloud Engineer, you will lead a team of four members in managing cloud infrastructure, ensuring seamless operations and scalability.Key responsibilities include:Managing cloud infrastructure, including AWS Cloud platforms and servicesLeading a team of cloud engineers in cloud infrastructure management, CI/CD pipeline, security...