Data Reconciliation Engineer

2 weeks ago

jersey city, United States Photon Full time

Data Reconciliation Engineer

Jersey City, NJ

Fulltime with Photon

No. of position: 6


We're seeking a skilled PySpark Data Reconciliation Engineer to join our team and drive the development of robust data reconciliation solutions within our financial systems. You will be responsible for designing, implementing, and maintaining PySpark-based applications to perform complex data reconciliations, identify and resolve discrepancies, and automate data matching processes. The ideal candidate possesses strong PySpark development skills, experience with data reconciliation techniques, and the ability to integrate with diverse data sources and rules engines.

Key Responsibilities:

Data Reconciliation Development:

  • Design, develop, and test PySpark-based applications to automate data reconciliation processes across various financial data sources, including relational databases, NoSQL databases, batch files, and real-time data streams.
  • Implement efficient data transformation, matching algorithms (deterministic and heuristic) using PySpark and relevant big data frameworks.
  • Develop robust error handling and exception management mechanisms to ensure data integrity and system resilience within Spark jobs.

Data Analysis and Matching:

  • Collaborate with business analysts and data architects to understand data requirements and matching criteria.
  • Analyze and interpret data structures, formats, and relationships to implement effective data matching algorithms using PySpark.
  • Work with distributed datasets in Spark, ensuring optimal performance for large-scale data reconciliation.

Rules Engine Integration:

  • Integrate PySpark applications with rules engines (e.g., Drools) or equivalent to implement and execute complex data matching rules.
  • Develop PySpark code to interact with the rules engine, manage rule execution, and handle rule-based decision-making.

Problem Solving and Gap Analysis:

  • Collaborate with cross-functional teams to identify and analyze data gaps and inconsistencies between systems.
  • Design and develop PySpark-based solutions to address data integration challenges and ensure data quality.
  • Contribute to the development of data governance and quality frameworks within the organization.

Qualifications and Skills:

  • 5+ years of hands-on experience in big data development, preferably with exposure to data-intensive applications.
  • Strong understanding of data reconciliation principles, techniques, and best practices.
  • Proficiency in PySpark, Apache Spark, and related big data technologies for data processing and integration.
  • Experience with rules engine integration and development
  • Strong analytical and problem-solving skills, with the ability to translate business requirements into technical solutions.
  • Excellent communication and collaboration skills to work effectively with business analysts, data architects, and other team members.
  • Familiarity with data streaming platforms (e.g., Kafka, Kinesis) and big data technologies (e.g., Hadoop, Hive, HBase) is a plus.

  • Data Reconciliation Engineer

    2 weeks ago

    Jersey City, United States Photon Full time

    Data Reconciliation EngineerJersey City, NJFulltime with PhotonNo. of position: 6We're seeking a skilled PySpark Data Reconciliation Engineer to join our team and drive the development of robust data reconciliation solutions within our financial systems. You will be responsible for designing, implementing, and maintaining PySpark-based applications to...

  • PySpark Data Reconciliation Engineer

    3 weeks ago

    Jersey City, New Jersey, United States Photon Full time

    Job Title: PySpark Data Reconciliation EngineerWe are seeking a highly skilled PySpark Data Reconciliation Engineer to join our Digital Engineering team at Photon. As a key member of our team, you will be responsible for designing, developing, and implementing large-scale data reconciliation solutions using PySpark.Key Responsibilities:Design and develop...

  • SPARK Data Reconciliation Engineer

    2 weeks ago

    Jersey City, United States Caltriko Solutions Full time

    Hi,Greetings of the day! I have an urgent requirement below, please go through JD and let me know if you are comfortable or have any profile. Kindly revert me back with your updated resume as well. Job Title: SPARK Data Reconciliation Engineer Location: Jersey City, NJ (Work from Office)Duration: Long Term Contract Summary:We're seeking a skilled PySpark...

  • PySpark Data Reconciliation Engineer

    1 month ago

    Jersey City, United States Photon Full time

    Greetings everyone,We hope you are staying safe. We are hiring a PySpark Data Reconciliation Engineer to join our Digital Engineering team.Who are we?For the past 20 years, we have powered many Digital Experiences for the Fortune 500. Since 1999, we have grown from a few people to more than 4000 team members across the globe that are engaged in various...

  • Data Reconciliation Specialist

    2 weeks ago

    Jersey City, New Jersey, United States Photon Full time

    Join Photon as a Spark Data Reconciliation EngineerWe are Photon, a leading Digital Platform Engineering company, and we're looking for a talented Spark Data Reconciliation Engineer to join our team.As a Spark Data Reconciliation Engineer, you will play a crucial role in ensuring the accuracy and integrity of our data. You will be responsible for designing...

  • Data Reconciliation Specialist

    2 weeks ago

    Jersey City, New Jersey, United States Photon Full time

    Data Reconciliation EngineerWe're seeking a skilled Data Reconciliation Engineer to drive the development of robust data reconciliation solutions within our financial systems. As a key member of our team, you will be responsible for designing, implementing, and maintaining PySpark-based applications to perform complex data reconciliations, identify and...

  • Data Reconciliation Specialist

    3 weeks ago

    Jersey City, New Jersey, United States Photon Full time

    Join Photon as a Spark Data Reconciliation EngineerWe are a high-growth organization serving over 40% of Fortune 100 companies, and we truly believe that our people make the world of difference for our clients and our business.As a Spark Data Reconciliation Engineer at Photon, you will have the opportunity to learn and grow, build deep digital/technical...

  • Senior Java Data Reconciliation Engineer

    4 days ago

    Jersey City, New Jersey, United States Photon Full time

    Job Title:Data Reconciliation Engineer - Java ExpertJob Summary:We're seeking a skilled Data Reconciliation Engineer - Java Expert to drive the development of robust data reconciliation solutions within our financial systems. The ideal candidate possesses strong Java development skills, experience with data reconciliation techniques, and the ability to...

  • Data Reconciliation Specialist

    4 days ago

    Jersey City, New Jersey, United States Photon Full time

    PySpark Data Reconciliation EngineerWe are seeking a skilled PySpark Data Reconciliation Engineer to join our Digital Engineering team at Photon. The ideal candidate will have a strong understanding of data reconciliation principles, techniques, and best practices.Key Responsibilities:Design, develop, and test PySpark-based applications to automate data...

  • SPARK Data Reconciliation Engineer

    1 month ago

    Jersey City, United States Photon Full time

    Greetings Everyone,We are Photon, one of the world's largest Digital Platform Engineering companies providing a unique combination of Strategy Consulting, Creative Design and Technology Services to a wide range of customers.We are a high growth organization serving over 40% of Fortune 100 companies and truly believe that our people make the world of...

  • Data Reconciliation Specialist

    6 days ago

    Jersey City, New Jersey, United States Photon Full time

    About PhotonWe are Photon, a leading Digital Platform Engineering company that combines strategy consulting, creative design, and technology services to deliver innovative solutions to our clients.As a high-growth organization serving over 40% of Fortune 100 companies, we believe that our people are the key to making a difference for our clients and our...

  • PySpark Data Reconciliation Specialist

    1 month ago

    Jersey City, New Jersey, United States Photon Full time

    PySpark Data Reconciliation EngineerWe are seeking a highly skilled PySpark Data Reconciliation Engineer to join our Digital Engineering team at Photon. As a key member of our team, you will be responsible for designing, developing, and testing PySpark-based applications to automate data reconciliation processes across various financial data sources.Key...

  • PySpark Data Reconciliation Specialist

    4 weeks ago

    Jersey City, New Jersey, United States Photon Full time

    Job Title: PySpark Data Reconciliation EngineerWe are seeking a highly skilled PySpark Data Reconciliation Engineer to join our Digital Engineering team at Photon. As a key member of our team, you will be responsible for designing, developing, and testing PySpark-based applications to automate data reconciliation processes across various financial data...

  • Java Data Reconciliation Specialist

    6 days ago

    Jersey City, New Jersey, United States Photon Full time

    Job Title:Java Data Reconciliation EngineerJob Summary:We are seeking a skilled Java Data Reconciliation Engineer to drive the development of robust data reconciliation solutions within our financial systems.Key Responsibilities:Design, develop, and test Java-based applications to automate data reconciliation processes across various financial data...

  • PySpark Data Reconciliation Specialist

    4 weeks ago

    Jersey City, New Jersey, United States Photon Full time

    PySpark Data Reconciliation EngineerWe are seeking a highly skilled PySpark Data Reconciliation Engineer to join our Digital Engineering team at Photon. As a key member of our team, you will be responsible for designing, developing, and testing PySpark-based applications to automate data reconciliation processes across various financial data sources.Key...

  • PySpark Data Reconciliation Specialist

    2 weeks ago

    Jersey City, New Jersey, United States Photon Full time

    PySpark Data Reconciliation EngineerWe are seeking a highly skilled PySpark Data Reconciliation Engineer to join our Digital Engineering team at Photon. As a key member of our team, you will be responsible for designing, developing, and testing PySpark-based applications to automate data reconciliation processes across various financial data sources.Key...

  • PySpark Data Reconciliation Specialist

    1 month ago

    Jersey City, New Jersey, United States Photon Full time

    Job Title: PySpark Data Reconciliation EngineerWe're seeking a skilled PySpark Data Reconciliation Engineer to drive the development of robust data reconciliation solutions within our financial systems.Key Responsibilities:Design, develop, and test PySpark-based applications to automate data reconciliation processes across various financial data...

  • PySpark Data Reconciliation Specialist

    4 weeks ago

    Jersey City, New Jersey, United States Photon Full time

    PySpark Data Reconciliation EngineerWe are seeking a highly skilled PySpark Data Reconciliation Engineer to join our Digital Engineering team at Photon. As a key member of our team, you will be responsible for designing, developing, and testing PySpark-based applications to automate data reconciliation processes across various financial data sources.Key...

  • PySpark Data Reconciliation Specialist

    2 weeks ago

    Jersey City, New Jersey, United States Photon Full time

    PySpark Data Reconciliation EngineerWe are seeking a highly skilled PySpark Data Reconciliation Engineer to join our Digital Engineering team at Photon. With over 20 years of experience in powering Digital Experiences for the Fortune 500, we are now looking for a talented individual to help us drive innovation and growth.About UsPhoton has a rich history of...

  • PySpark Data Reconciliation Specialist

    3 weeks ago

    Jersey City, New Jersey, United States Photon Full time

    Job Title: PySpark Data Reconciliation EngineerWe are seeking a highly skilled PySpark Data Reconciliation Engineer to join our Digital Engineering team at Photon. As a key member of our team, you will be responsible for designing, developing, and testing PySpark-based applications to automate data reconciliation processes across various financial data...