Data Pipeline Testing

4 weeks ago

New York, United States Happiest Minds Technologies Limited Full time

We are seeking a highly skilled and motivated Data Pipeline Testing Lead Engineer to join our team. This role is crucial for ensuring the accuracy and reliability of our data solutions on Google Cloud. The candidate will be responsible for testing data pipelines that involve technologies like Big Query, Kafka, Hive, Parquet files, and Snowflake. This position requires a deep understanding of both batch and streaming data processes and a proven ability to automate tests using Python.

*Key Responsibilities:

  • Design and execute tests on data pipelines that integrate various technologies such as BigQuery, Kafka, Hive, Parquet files, and Snowflake.
  • Develop automated tests for batch and streaming data systems to validate the functionality and performance of data pipelines.
  • Implement Data Quality Testing frameworks to ensure the integrity and accuracy of data stored and processed.
  • Collaborate with development teams to understand business requirements and translate them into test scenarios.
  • Verify the correctness of metrics calculations based on predefined rules and ensure compliance with data governance standards.
  • Collaborate with onsite and offshore engineering teams and product managers to ensure seamless integration and alignment with project objectives.
  • Troubleshoot and resolve issues within the data pipelines and related infrastructure.
  • Continuously improve testing strategies and automation frameworks to enhance test coverage and efficiency.
  • Document test results and collaborate with engineering teams to refine data solutions based on feedback.

*Required Skills and Qualifications:

  • Bachelor s degree in Computer Science, Information Technology, or a related field.
  • Minimum of 10 years of experience in data pipeline testing, preferably in a cloud environment.
  • Strong experience with Google Cloud Platform services, especially BigQuery.
  • Expertise with test data modelling and Quality Assurance for ETL processes.
  • Proficient in working with Kafka, Hive, Parquet files, and Snowflake.
  • Expertise in Data Quality Testing and metrics calculations for both batch and streaming data.
  • Excellent programming skills in Python and experience with test automation.
  • Strong analytical and problem-solving abilities.
  • Excellent communication and teamwork skills.

*Preferred Skills:

  • Experience with CI/CD pipelines in a cloud environment.
  • Knowledge of additional programming languages such as Java or Scala.

  • Sr. Data Pipeline Engineer

    2 days ago

    New York, United States NR Consulting Full time

    BS or MS in Computer Science, a related field, or equivalent industry experience . 3 years of professional experience engineering complex, high-volume data pipelines using SQL, Python, and Airflow . 3 years of experience building cloud scalable and high-performance data lake / data warehouse solutions using AWS products - S3, Athena, Glue, and EMR ....

  • Vice President, Data Pipeline Solutions

    1 week ago

    New York, United States Demyst Full time

    Job DescriptionJob DescriptionAbout DemystDemyst is a data management company specialising in external data orchestration, helping leading global financial institutions support their business users with data access at scale within their centralized data platforms. In response to growing demand, we’re seeking new team members to help us scale. By joining...

  • Vice President, Data Pipeline Solutions

    2 weeks ago

    New York, United States Demyst Full time

    Job DescriptionJob DescriptionAbout DemystDemyst is a data management company specialising in external data orchestration, helping leading global financial institutions support their business users with data access at scale within their centralized data platforms. In response to growing demand, we’re seeking new team members to help us scale. By joining...

  • Vice President, Data Pipeline Solutions

    2 months ago

    New York, United States Demyst Full time

    Job DescriptionJob DescriptionAbout DemystDemyst is a data management company specialising in external data orchestration, helping leading global financial institutions support their business users with data access at scale within their centralized data platforms. In response to growing demand, we’re seeking new team members to help us scale. By joining...

  • Vice President, Data Pipeline Solutions

    1 month ago

    New York, United States Demyst Full time

    About DemystDemyst is a data management company specialising in external data orchestration, helping leading global financial institutions support their business users with data access at scale within their centralized data platforms. In response to growing demand, we're seeking new team members to help us scale. By joining our team, you will play a crucial...

  • Vice President, Data Pipeline Solutions

    1 week ago

    New York, United States Demyst Full time

    About DemystDemyst is a data management company specialising in external data orchestration, helping leading global financial institutions support their business users with data access at scale within their centralized data platforms. In response to growing demand, we're seeking new team members to help us scale. By joining our team, you will play a crucial...

  • Vice President, Data Pipeline Solutions

    2 months ago

    New York, United States Demyst Full time

    About DemystDemyst is a data management company specialising in external data orchestration, helping leading global financial institutions support their business users with data access at scale within their centralized data platforms. In response to growing demand, we're seeking new team members to help us scale. By joining our team, you will play a crucial...

  • Software Engineer

    1 month ago

    New York, United States Dripos Full time

    Dripos is a customer-obsessed company that uses a data-driven methodology to create an all-in-one solution for coffee shops. We take customers from relying on 5-10 different solutions to run their business to only needing Dripos for all their needs (ordering, scheduling, payroll, etc). We work closely with our partnered locations to build their dream product...

  • Senior Mechanical Engineer

    4 weeks ago

    New York, United States Ellaway Blues Consulting Full time

    We are seeking a Senior Mechanical Engineer to lead in the mechanical engineering and design aspects of projects, ensuring adherence to design criteria, material selection, and equipment sizing in accordance with established codes and standards. Collaborating closely with client project teams, you will ascertain work scope requirements, oversee project...

  • ETL Developer

    1 month ago

    New York, United States Creative Data Resources Full time

    Onsite/remote schedule - 2 days/week onsite•Migrate existing SSIS ETL scripts to Python; develop new ETL scripts •Support existing SSIS SQL Projects •Maintain ETL pipelines in and out of data warehouse using combination of Python and Snowflakes SnowSQL •Write SQL queries against Snowflake. •Understanding data pipelines and modern ways of automating...

  • Data QA Engineer

    1 month ago

    New York, United States InterEx Group Full time

    A key client of mine is seeking a highly skilled and experienced Data QA Engineer to join their team. In this role, you will be responsible for performing data quality analysis and validation, writing test plans, test cases, and test scripts, and validating solutions built on REST APIs, Snowflake, and data pipelines. The successful candidate will be able to...

  • Software Test Engineer

    2 weeks ago

    New York, United States Automatic Data Processing Full time

    Job DescriptionJob DescriptionTest Automation Engineer IIAs a ‘Test Automation Engineer II’ you will be an integral member of our client's software development team, testing best-in-class advancements to our products. This position is expected to analyze, requirements for client's suite of products to design and develop automation scripts and to...

  • Software Test Engineer

    2 months ago

    New York, United States Automatic Data Processing Full time

    Job DescriptionJob DescriptionTest Automation Engineer IIAs a ‘Test Automation Engineer II’ you will be an integral member of our client's software development team, testing best-in-class advancements to our products. This position is expected to analyze, requirements for client's suite of products to design and develop automation scripts and to...

  • Software Test Engineer

    3 days ago

    New York, United States Automatic Data Processing Full time

    Job DescriptionJob DescriptionTest Automation Engineer IIAs a ‘Test Automation Engineer II’ you will be an integral member of our client's software development team, testing best-in-class advancements to our products. This position is expected to analyze, requirements for client's suite of products to design and develop automation scripts and to...

  • Software Test Engineer

    1 week ago

    New York, United States Automatic Data Processing Full time

    Job DescriptionJob DescriptionTest Automation Engineer IIAs a ‘Test Automation Engineer II’ you will be an integral member of our client's software development team, testing best-in-class advancements to our products. This position is expected to analyze, requirements for client's suite of products to design and develop automation scripts and to...

  • ETL Developer

    4 weeks ago

    New York, United States Creative Data Resources Full time

    Onsite/remote schedule - 2 days/week onsite •Migrate existing SSIS ETL scripts to Python; develop new ETL scripts •Support existing SSIS SQL Projects •Maintain ETL pipelines in and out of data warehouse using combination of Python and Snowflakes SnowSQL •Write SQL queries against Snowflake. •Understanding data pipelines and modern ways of...

  • Senior Data Engineer

    1 month ago

    New York, United States Riva Scientific LLC Full time

    Job DescriptionJob DescriptionRole:Senior Data Engineer Location- NYC, NY (Hybrid)Experience: 10 yearsSummary :We are looking for Snowflake developer with a financial background typically who plays a crucial role in leveraging Snowflake, a cloud-based data warehousing platform, to manage and analyse financial data.Primary Responsibilities:Strong...

  • Senior Data Engineer

    1 week ago

    New York, United States Riva Scientific LLC Full time

    Job DescriptionJob DescriptionRole:Senior Data Engineer Location- NYC, NY (Hybrid)Experience: 10 yearsSummary :We are looking for Snowflake developer with a financial background typically who plays a crucial role in leveraging Snowflake, a cloud-based data warehousing platform, to manage and analyse financial data.Primary Responsibilities:Strong...

  • Data Engineer

    2 days ago

    New York, United States StartUs GmbH Full time

    We are looking for a data engineer that will build data-driven solutions to deliver podcast experiences to our 170+ million active users by analysing our on-platform usage data, understanding our data from an off-platform perspective and improving the accuracy and precision of our data and related recommendations. Above all, your work will impact the way the...

  • Data Engineer

    4 weeks ago

    New York, United States Open Systems Technologies Full time

    An international law firm is looking for a Data Engineer to join their team in NYC. Compensation: $115-160kThe Data Engineer is responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection. The Data Engineer is an experienced data pipeline builder who enjoys optimizing data systems and...