Senior Data Engineer

4 weeks ago

New York, New York, United States Capgemini Full time

Role Summary

Capgemini is seeking a highly skilled Senior Data Engineer - Python/PySpark to join our dynamic data engineering team. The ideal candidate will have a strong background in building scalable distributed data processing systems using PySpark and working on large datasets.

Key Responsibilities

  • Collaborate with data scientists engineers and other stakeholders to design and implement efficient data pipelines.
  • Develop optimize and maintain ETL pipelines using PySpark to process large-scale datasets across distributed environments.
  • Design and implement complex data transformation logic using PySpark and other Big Data tools.
  • Work with various Big Data technologies such as Hadoop Hive HBase Kafka and Spark to build robust scalable data systems.
  • Collaborate with data engineers and data scientists to integrate data from multiple sources and create unified datasets.
  • Write efficient reusable and scalable Python code to handle both batch and real-time data processing tasks.
  • Ensure data quality consistency and reliability by implementing data validation monitoring and error handling.
  • Fine-tune and optimize PySpark jobs to improve performance in distributed environments.
  • Manage and maintain data flows in HDFS ensuring scalability and fault tolerance.
  • Perform data extraction aggregation and reporting using SQL and NoSQL databases.
  • Participate in system design discussions and provide recommendations for architecture and performance improvements.

What We Offer

  • Flexible work arrangements
  • Healthcare including dental vision mental health and well-being programs
  • Financial well-being programs such as 401(k) and Employee Share Ownership Plan
  • Paid time off and paid holidays
  • Family building benefits like adoption assistance surrogacy and cryopreservation
  • Social well-being benefits like subsidized back-up child/elder care and tutoring
  • Disaster Relief

Equal Opportunity Employer

Capgemini is an Equal Opportunity Employer encouraging diversity in the workplace. All qualified applicants will receive consideration for employment without regard to race national origin gender identity/expression age religion disability sexual orientation genetics veteran status marital status or any other characteristic protected by law.

  • Senior Azure Data Engineer

    4 weeks ago

    New York, New York, United States Hudson Data Full time

    Job Title: Senior Azure Data EngineerLocation: NYC, NYDuration: Long Term W2 RoleJob Description:The ideal candidate should have hands-on experience with various Azure services, including Azure Data Factory, Databricks, Data Lake, and Power BI. They should be able to design, build, and maintain ETL pipelines, manage data lakes, and create insightful reports...

  • Senior Data Engineer

    4 weeks ago

    New York, New York, United States TechnoGen Full time

    Job Title: Senior Data EngineerJob Summary:We are seeking a Senior Data Engineer to join our Data Infrastructure team at TechnoGen. As a Senior Data Engineer, you will be responsible for building and maintaining tools and infrastructure to write, test, and schedule batch data pipelines. Your work will enable other developers, data scientists, and analysts to...

  • Senior Data Engineer

    4 weeks ago

    New York, New York, United States Diverse Lynx Full time

    Job Title: Senior Data EngineerLocation: RemoteDuration: Long Term contractJob Description:We are seeking a highly skilled Senior Data Engineer to join our team at Diverse Lynx LLC. The ideal candidate will have 8 to 10 years of experience in Data Engineering space with AWS as primary cloud experience. Key responsibilities include conducting design reviews...

  • Senior Data Engineer

    4 weeks ago

    New York, New York, United States A-Line Staffing Solutions Full time

    Job Title: Senior Data EngineerJob Summary:A-Line Staffing Solutions is seeking a highly skilled Senior Data Engineer to join our team. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining complex data pipelines in cloud environments. You will work closely with our technical teams to ensure the smooth operation of our...

  • Senior Security Systems Engineer

    1 month ago

    New York, New York, United States Acquired Data Solutions Full time

    Job Title: Senior Security Systems EngineerWe are seeking a highly skilled Senior Security Systems Engineer to join our team at Acquired Data Solutions, Inc. in Egg Harbor, NJ.Job Summary:The successful candidate will be responsible for performing in-depth engineering analysis of security-related systems to determine their potential for further development,...

  • Data Engineering Specialist

    4 weeks ago

    New York, New York, United States The Data Sherpas Full time

    About The Data SherpasWe are a team of highly skilled and motivated engineers who help our clients at every phase of their cloud journey.Our team has extensive experience in data and analytics engineering, with a strong foundation in Snowflake, FiveTran, and DBT.We are looking for a self-motivated and data-driven engineer to join our team and collaborate...

  • Senior Cloud Data Engineer

    4 weeks ago

    New York, New York, United States Glow Networks Full time

    Job Title: Senior Cloud Data EngineerJob Summary: We are seeking a highly skilled Senior Cloud Data Engineer to join our team at Glow Networks. As a Senior Cloud Data Engineer, you will be responsible for designing, building, and maintaining large-scale data engineering systems in the cloud.Key Responsibilities:* Design and implement data engineering systems...

  • Senior Cloud Data Engineer

    1 month ago

    New York, New York, United States Tech Tammina Full time

    Job Description**Senior Cloud Data Engineer**We are seeking a highly skilled Senior Cloud Data Engineer to join our team at Tech Tammina. As a Senior Cloud Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines on Google Cloud Platform (GCP).The ideal candidate should have extensive experience in IT and data...

  • Senior Data Engineer

    4 weeks ago

    New York, New York, United States Justworks Full time

    Job DescriptionWe are seeking a highly skilled Senior Data Engineer to join our Tax Technology team at Justworks. As a Senior Data Engineer, you will be responsible for designing, building, and maintaining data pipelines and models that power tax computation, withholding, and reporting for our customers.Key Responsibilities:Design and build data pipelines...

  • Senior Data Engineer

    4 weeks ago

    New York, New York, United States Justworks Full time

    Job SummaryWe are seeking a highly skilled Senior Data Engineer to join our Tax Technology team at Justworks. As a Senior Data Engineer, you will be responsible for designing, building, and maintaining data pipelines and models that power tax computation, withholding, and reporting for our customers.Key ResponsibilitiesDesign and build reliable and resilient...

  • Senior Data Engineer

    4 weeks ago

    New York, New York, United States Capital One Full time

    About the Role:We are seeking a highly skilled Senior Data Engineer to join our team at Capital One. As a Senior Data Engineer, you will be responsible for designing, developing, and implementing technical solutions in full-stack development tools and technologies.Key Responsibilities:Collaborate with Agile teams to design, develop, test, implement, and...

  • Senior Data Engineer

    4 weeks ago

    New York, New York, United States Capital One Full time

    About the Role:We are seeking a Senior Data Engineer to join our team at Capital One. As a Senior Data Engineer, you will be responsible for designing, developing, and implementing technical solutions in full-stack development tools and technologies.Key Responsibilities:Collaborate with Agile teams to design, develop, test, implement, and support technical...

  • Senior Data Engineering Lead

    4 weeks ago

    New York, New York, United States Delphi-US Full time

    Job Title: Senior Data Engineer - MartechLocation : RemoteWe are seeking a Senior Data Engineer to collaborate with our team and contribute to the transition of our legacy Martech pipelines to a modern ELT architecture.You will work with BASH, Perl, dbt, Fivetran, Snowflake, GitHub, and modern CI/CD & IAC tools to design, develop, test, deploy, and support...

  • Senior Data Engineer

    4 weeks ago

    New York, New York, United States Multi Media, LLC Full time

    About Multi Media, LLC:We're a leader in digital innovation, focusing on creating modern products for the content creator community. Our main platform, Chaturbate, is a key player in the adult entertainment industry, bringing billions of people together worldwide.We aim to make Chaturbate the best place for users and creators to interact and connect,...

  • Senior Data Engineer

    4 weeks ago

    New York, New York, United States Yahoo Holdings Inc. Full time

    At Yahoo Holdings Inc., we're seeking a talented Senior Data Engineer to join our Data Services team. As a key member of our team, you'll be responsible for designing, defining, building, and delivering new data services for Yahoo News.We're a team that values collaboration, innovation, and customer-centricity. We believe in learning through rapid...

  • Senior ELT Data Engineer

    4 weeks ago

    New York, New York, United States Delphi-US Full time

    Job Title : Senior ELT Data Engineer (Contract) - Job#5213Location : RemoteAdditional Information : 12 month+ Contract, EST hoursWe are seeking a highly skilled Senior Data Engineer to design, develop, and deploy ELT data pipelines and models using dbt, Snowflake, Fivetran, and GitHub.You will collaborate with data analysts, data scientists, business...

  • Senior Data Engineer

    4 weeks ago

    New York, New York, United States VALiNTRY Full time

    Job SummaryVALiNTRY is seeking a highly skilled Senior Analytics Engineer to support our Business Intelligence lead in managing the enterprise's data pipeline and ecosystem from end to end. As a key member of our team, you will collaborate with cross-functional business partners to understand their needs and transform raw data into actionable insights.Key...

  • Senior Data Infrastructure Engineer

    4 weeks ago

    New York, New York, United States Vimeo Full time

    Job Title: Senior Data Infrastructure EngineerJob Summary:Vimeo is seeking a highly skilled Senior Data Infrastructure Engineer to manage and optimize our vector databases and data stores, including Elastic Search and AWS OpenSearch. The ideal candidate will have a strong background in cloud platforms and SRE principles, as well as experience leading data...

  • Senior Project Engineer

    4 weeks ago

    New York, New York, United States (EDO) Entertainment Data Oracle, Inc. Full time

    About the Role:We are seeking a highly skilled Senior Project Engineer to join our team at (EDO) Entertainment Data Oracle, Inc. in New York City. As a Senior Project Engineer, you will be responsible for supporting the planning, engineering, and execution of clean energy projects in metro New York City.Key Responsibilities:Assist in the development of...

  • Senior Data Engineer

    4 weeks ago

    New York, New York, United States Capital One Services, LLC Full time

    Job SummaryCapital One is seeking a Senior Data Engineer to join our team. As a Senior Data Engineer, you will be responsible for designing, developing, and implementing technical solutions in full-stack development tools and technologies. You will work with a team of developers with deep experience in machine learning, distributed microservices, and...