Senior Data Engineer

2 weeks ago

New York, New York, United States Capgemini Full time

Role Summary

Capgemini is seeking a highly skilled Senior Data Engineer - Python/PySpark to join our dynamic data engineering team. The ideal candidate will have a strong background in building scalable distributed data processing systems using PySpark and working on large datasets.

Key Responsibilities

  • Collaborate with data scientists engineers and other stakeholders to design and implement efficient data pipelines.
  • Develop optimize and maintain ETL pipelines using PySpark to process large-scale datasets across distributed environments.
  • Design and implement complex data transformation logic using PySpark and other Big Data tools.
  • Work with various Big Data technologies such as Hadoop Hive HBase Kafka and Spark to build robust scalable data systems.
  • Collaborate with data engineers and data scientists to integrate data from multiple sources and create unified datasets.
  • Write efficient reusable and scalable Python code to handle both batch and real-time data processing tasks.
  • Ensure data quality consistency and reliability by implementing data validation monitoring and error handling.
  • Fine-tune and optimize PySpark jobs to improve performance in distributed environments.
  • Manage and maintain data flows in HDFS ensuring scalability and fault tolerance.
  • Perform data extraction aggregation and reporting using SQL and NoSQL databases.
  • Participate in system design discussions and provide recommendations for architecture and performance improvements.

What We Offer

  • Flexible work arrangements
  • Healthcare including dental vision mental health and well-being programs
  • Financial well-being programs such as 401(k) and Employee Share Ownership Plan
  • Paid time off and paid holidays
  • Family building benefits like adoption assistance surrogacy and cryopreservation
  • Social well-being benefits like subsidized back-up child/elder care and tutoring
  • Disaster Relief

Equal Opportunity Employer

Capgemini is an Equal Opportunity Employer encouraging diversity in the workplace. All qualified applicants will receive consideration for employment without regard to race national origin gender identity/expression age religion disability sexual orientation genetics veteran status marital status or any other characteristic protected by law.

  • Senior Azure Data Engineer

    2 weeks ago

    New York, New York, United States Hudson Data Full time

    Job Title: Senior Azure Data EngineerLocation: NYC, NYDuration: Long Term W2 RoleJob Description:The ideal candidate should have hands-on experience with various Azure services, including Azure Data Factory, Databricks, Data Lake, and Power BI. They should be able to design, build, and maintain ETL pipelines, manage data lakes, and create insightful reports...

  • Senior Data Engineer

    2 weeks ago

    New York, New York, United States The Data Sherpas Full time

    Job SummaryWe are seeking a highly skilled Senior Data Engineer to lead our team in designing, developing, and deploying high-performance data analytics solutions on Google Cloud Platform (GCP). As a key member of our team, you will be responsible for architecting transformation and modernization of enterprise data solutions on GCP, integrating native GCP...

  • Senior Data Engineer

    3 weeks ago

    New York, New York, United States Capital One Full time

    Job Title: Senior Data EngineerCapital One is seeking a Senior Data Engineer to join our team. As a Senior Data Engineer, you will be responsible for designing, developing, and implementing data engineering solutions that drive business value.Key Responsibilities:Collaborate with cross-functional teams to design and develop data engineering solutionsDevelop...

  • Senior Data Engineer

    1 month ago

    New York, New York, United States Capital One Full time

    Job Title: Senior Data EngineerCapital One is seeking a Senior Data Engineer to join our team. As a Senior Data Engineer, you will be responsible for designing, developing, and implementing data engineering solutions that drive business value.Key Responsibilities:Collaborate with cross-functional teams to design and develop data engineering solutions that...

  • Senior Data Engineer

    3 weeks ago

    New York, New York, United States Capital One Full time

    Job Title: Senior Data EngineerCapital One is seeking a Senior Data Engineer to join our team. As a Senior Data Engineer, you will be responsible for designing, developing, and implementing data engineering solutions that drive business value.Key Responsibilities:Collaborate with cross-functional teams to design and develop data engineering solutionsDevelop...

  • Senior Data Engineer

    3 weeks ago

    New York, New York, United States Capital One Full time

    Job Title: Senior Data EngineerCapital One is seeking a Senior Data Engineer to join our team. As a Senior Data Engineer, you will be responsible for designing, developing, and implementing data pipelines and architectures to support our business needs.About the Role:Collaborate with cross-functional teams to design and develop data pipelines and...

  • Senior Data Engineer

    3 weeks ago

    New York, New York, United States Silver Search, Inc Full time

    Senior Data EngineerSilver Search, Inc is seeking a highly skilled Senior Data Engineer to join our data and analytics team. As a key member of our team, you will be responsible for designing and building data processing frameworks, working closely with data science and visualization teams, and ensuring the integrity and scalability of our data...

  • Senior Engineer

    3 weeks ago

    New York, New York, United States Acquired Data Solutions Full time

    Job SummaryWe are seeking a highly skilled Senior Level Engineer to join our Transportation Security Laboratory (TSL) Team in Egg Harbor, NJ as Key Personnel to support Development Test & Evaluation (DT&E) of existing and emerging technologies for airport security equipment.Key ResponsibilitiesPerform in-depth engineering analysis of security related systems...

  • Senior Data Engineer

    1 month ago

    New York, New York, United States Coinbase Full time

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at Coinbase. As a Senior Data Engineer, you will be responsible for designing, building, and maintaining robust data pipelines, trusted data sources, and analytics/data products that inject automation into data science processes.Key Responsibilities:Design...

  • Senior Data Engineer

    3 weeks ago

    New York, New York, United States MidStream PA Full time

    Join Our Team as a Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to lead the development of our core data processing platform. As a key member of our engineering team, you will be responsible for designing and implementing scalable, reliable, and high-performance data systems.Your expertise in data modeling, warehousing, and...

  • Senior Data Engineer

    3 weeks ago

    New York, New York, United States A Place For Mom Full time

    Job Title: Senior Data EngineerA Place for Mom is seeking a highly skilled Senior Data Engineer to join our team. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining our data warehouse on Databricks within AWS, ensuring optimal performance and reliability.Key Responsibilities:Design and develop ETL processes to...

  • Senior Data Engineer

    3 weeks ago

    New York, New York, United States Capital One Full time

    Job Title: Senior Data EngineerCapital One is seeking a Senior Data Engineer to join our team. As a Senior Data Engineer, you will be responsible for designing, developing, and implementing data engineering solutions that drive business value.Key Responsibilities:Collaborate with cross-functional teams to design and develop data engineering solutionsDevelop...

  • Senior Data Engineer

    2 weeks ago

    New York, New York, United States TechnoGen Full time

    Job Title: Senior Data EngineerJob Summary:We are seeking a Senior Data Engineer to join our Data Infrastructure team at TechnoGen. As a Senior Data Engineer, you will be responsible for building and maintaining tools and infrastructure to write, test, and schedule batch data pipelines. Your work will enable other developers, data scientists, and analysts to...

  • Senior Data Engineer

    3 weeks ago

    New York, New York, United States Capital One Full time

    Job Title: Senior Data EngineerCapital One is seeking a Senior Data Engineer to join our team. As a Senior Data Engineer, you will be responsible for designing, developing, and implementing technical solutions in full-stack development tools and technologies.You will work with a team of developers with deep experience in machine learning, distributed...

  • Senior Data Engineer

    4 weeks ago

    New York, New York, United States Citigroup Full time

    Job Title: Senior Data EngineerCitigroup is seeking a highly skilled Senior Data Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining data infrastructure, pipelines, and solutions that support our business operations.Key Responsibilities:Design and implement data...

  • Senior Data Engineer

    3 weeks ago

    New York, New York, United States Capital One Full time

    Job Title: Senior Data EngineerCapital One is seeking a Senior Data Engineer to join our team. As a Senior Data Engineer, you will be responsible for designing, developing, and implementing technical solutions in full-stack development tools and technologies.You will work with a team of developers with deep experience in machine learning, distributed...

  • Senior Data Engineer

    1 month ago

    New York, New York, United States Intersog Full time

    Unlock Business Potential with DataAt Intersog, we're on a mission to empower our clients with data-driven insights that drive business growth. As a Senior Data Engineer, you'll play a critical role in turning data into actionable information that informs strategic decisions.Key Responsibilities:Analyze existing data sources to identify trends and...

  • Senior Data Engineer

    3 weeks ago

    New York, New York, United States Capital One Full time

    Job Title: Senior Data EngineerCapital One is seeking a highly skilled Senior Data Engineer to join our team. As a Senior Data Engineer, you will be responsible for designing, developing, and implementing large-scale data systems and architectures.Key Responsibilities:Collaborate with cross-functional teams to design and develop data solutionsDevelop and...

  • Senior Data Engineer

    1 month ago

    New York, New York, United States American Express Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at American Express. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data systems and infrastructure. You will work closely with cross-functional teams to ensure the delivery of high-quality data solutions that meet...

  • Senior Data Engineer

    2 weeks ago

    New York, New York, United States Diverse Lynx Full time

    Job Title: Senior Data EngineerLocation: RemoteDuration: Long Term contractJob Description:We are seeking a highly skilled Senior Data Engineer to join our team at Diverse Lynx LLC. The ideal candidate will have 8 to 10 years of experience in Data Engineering space with AWS as primary cloud experience. Key responsibilities include conducting design reviews...