Data Pipeline Architect

5 days ago


New York, New York, United States Harnham Full time

DATA PIPELINE ENGINEER

GEN AI STARTUP

HYBRID REMOTE

$170,000 - $200,000

ROLE OVERVIEW

We are working with a generative AI startup that has recently raised its next round of funding and is scaling rapidly. This innovative startup within the construction industry is looking to hire a Data Pipeline Engineer to design, develop, and maintain pipeline infrastructure and workflows while working closely with the ML team.

KEY RESPONSIBILITIES

  • Design, develop, and maintain robust file processing pipeline infrastructure
  • Orchestrate the flow of data through various stages of processing
  • Ensure observability and monitoring of the pipeline's health
  • Integrate data from various sources including industry storage platforms, project management tools, and external APIs
  • Implement data quality checks and error handling mechanisms to ensure data integrity
  • Collaborate with the machine learning team to enhance pipeline functionality and efficiency

REQUIREMENTS

  • Experience in ETL development and data engineering
  • Expert coding proficiency in Python and database systems (SQL, noSQL)
  • Strong experience with pipeline orchestration tools (eg. Prefect), infrastructure-as-code (eg. Terraform), and observability and monitoring tools
  • Understanding of serverless architectures (eg. AWS Lambda)
  • Familiarity with ML workflows and requirements (to effectively collaborate with the ML team)
  • Knowledge of data modeling and data warehouse concepts
  • Startup experience

BENEFITS

This role offers a competitive salary range of $170,000- $200,000, depending on experience, along with equity in the company. Comprehensive health, dental, and vision benefits are also included.

HOW TO APPLY

Please register your interest by sending your resume to the hiring manager via the provided link.

KEYWORDS

Data Pipelines, Engineer, ETL, Data Engineering, Artificial Intelligence, Machine Learning, Python, Amazon Web Services, AWS, Data Warehouse, Terraform, Prefect, SQL, NoSQL, Data Warehouse



  • New York, New York, United States Harnham Full time

    Data Pipeline ArchitectWe are seeking a skilled Data Pipeline Architect to join our team at Harnham. As a Data Pipeline Architect, you will be responsible for designing and developing the infrastructure that supports our client's machine learning and data-driven projects. You will manage the orchestration of data flows from various sources, such as industry...


  • New York, New York, United States Harnham Full time

    Data Pipeline EngineerWe are seeking a highly skilled Data Pipeline Engineer to join our team at a generative AI startup in New York, NY. As a key member of our engineering team, you will design, develop, and maintain robust pipeline infrastructure and workflows while collaborating closely with our machine learning team.Key Responsibilities:Design and...


  • New York, New York, United States Trunk Tools, Inc. Full time

    Transforming Construction with DataAt Trunk Tools, Inc., we're revolutionizing the $13 trillion construction industry by automating workflows and enhancing productivity. Our team of experienced entrepreneurs and industry experts has successfully developed software for 140k+ users and worked on +$2 billion of built-environment projects.We're seeking a skilled...

  • Data Architect

    6 days ago


    New York, New York, United States Medidata Solutions Full time

    Job Title: Data ArchitectWe are seeking a highly skilled Data Architect to join our team at Medidata Solutions. As a Data Architect, you will be responsible for designing and implementing optimal data pipeline architecture to support our growing analytics capabilities.Key Responsibilities:* Design and implement data pipelines required in the data warehouse...

  • Data Architect Lead

    1 month ago


    New York, New York, United States CTI DATA Full time

    Join Our Team of Data and Analytics ExpertsWe are seeking a highly skilled and experienced Data Architect Lead to join our team at CTI Data. As a key member of our team, you will be responsible for crafting modern data and analytics solutions for our Global 2000 customers.Key ResponsibilitiesLead the development of cloud data strategies and implementation...


  • New York, New York, United States Financial Insight Technology, Inc. dba SmartAsset Full time

    Job SummaryAs a Senior Data Architect at Financial Insight Technology, Inc. dba SmartAsset, you will lead data engineering efforts by collaborating with cross-functional teams to harness the power of data and turn it into a strategic asset that drives business forward. This role requires a strong understanding of data engineering principles, including data...

  • Senior Data Architect

    2 months ago


    New York, New York, United States Silver Search, Inc Full time

    Job Title: Senior Data ArchitectCompany: Silver Search, IncJob Type: Full-timeLocation: RemoteAbout Us:Silver Search, Inc is a cutting-edge technology company that specializes in providing innovative solutions for data-driven businesses. We are seeking a highly skilled Senior Data Architect to join our team and lead the development of our data engineering...

  • Data Architect

    1 month ago


    New York, New York, United States Material Bank Full time

    Job DescriptionMaterial Bank is a fast-paced, high-growth technology company that has revolutionized the way architects and designers discover and sample materials. We are seeking a Principal Data Engineer/Architect to join our team and help us drive business between our members and brand partners.About the RoleWe are looking for a highly skilled data...

  • Data Architect

    2 weeks ago


    New York, New York, United States CCG Business Solutions, LLC Full time

    Data Engineer Job DescriptionCCG Business Solutions, LLC is seeking a highly skilled Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, building, and maintaining our data pipelines and architectures.The ideal candidate will have a strong background in data engineering, with experience in designing and implementing data...

  • Data Architect

    3 weeks ago


    New York, New York, United States Harnham Full time

    Data ArchitectHarnham is seeking an experienced Data Architect to design and implement scalable data solutions in Azure environments. The ideal candidate will have a strong background in data architecture, cloud computing, and a deep understanding of Azure services.Key ResponsibilitiesDesign and implement data architectures that support business requirements...

  • Data Architect

    2 weeks ago


    New York, New York, United States Inizio Partners Full time

    About the Role:Data Modeler & ArchitectJob Type:Fulltime, RemotePreferred Locations:TX / NY / DEWe are seeking a skilled Data Modeler and Architect to optimize the data architecture for our clients. This role is ideal for someone who thrives in understanding architecture, data lineage, and mapping to identify data gaps.Key Responsibilities:Evaluate and...

  • Data Architect

    2 days ago


    New York, New York, United States Material Bank Full time

    Job DescriptionMaterial Bank is a leading technology company in the Architecture and Design industry, providing a comprehensive material marketplace for design professionals. We are seeking a highly skilled Principal Data Engineer/Architect to join our team and contribute to the development of our internal catalog systems.The ideal candidate will have deep...

  • Data Architect

    2 weeks ago


    New York, New York, United States Tik Tok Full time

    Job Title: Data ArchitectAt TikTok, we're on a mission to inspire creativity and bring joy to our users. As a Data Architect, you'll play a critical role in shaping our data strategy and ensuring the integrity of our data pipelines.Responsibilities:* Design and implement scalable data architectures to support our fast-growing business* Collaborate with...

  • Senior Data Architect

    2 weeks ago


    New York, New York, United States Harnham Full time

    Job SummaryWe are seeking a highly skilled Senior Data Architect to join our team. As a key member of our data engineering team, you will be responsible for designing and implementing large-scale data architectures that drive business growth and innovation.About the RoleThis is a unique opportunity to work with a leading AdTech firm, leveraging your...

  • Data Engineer

    1 week ago


    New York, New York, United States CCG Business Solutions, LLC Full time

    Job OverviewCCG Business Solutions, LLC is seeking a highly skilled Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, building, and maintaining our data pipelines and architectures.Key ResponsibilitiesDesign and implement data pipelines and architectures to support business requirementsCollaborate with...

  • Azure Data Engineer

    1 week ago


    New York, New York, United States Hudson Data Full time

    Job Title: Azure Data Engineer - Design and Implement Data SolutionsLocation: NYC, NYDuration: Long Term W2 RoleJob Description:The ideal candidate will have hands-on experience with various Azure services, including Azure Data Factory, Databricks, Data Lake, and Power BI. They should be able to design, build, and maintain ETL pipelines, manage data lakes,...


  • New York, New York, United States Career Renew Full time

    Career Renew is seeking a seasoned data engineer to spearhead the design and scaling of our data infrastructure. This role requires a deep understanding of data pipelines and a proven track record of architecting and building large-scale data systems.The ideal candidate will have a strong mix of technical expertise, project management skills, and leadership...


  • New York, New York, United States Harnham Full time

    Job Opportunity:We are seeking a highly experienced Data Architect to join our team as a Principal Data Engineer, Azure. This role involves designing and developing Azure data warehouses and data pipelines for ML model execution, collaborating with a cross-functioning team, and serving as an Azure subject matter expert to lead the design and development...


  • New York, New York, United States Material Bank Full time

    Job DescriptionMaterial Bank is a fast-paced technology company that has revolutionized the architecture and design industry by creating the world's largest material marketplace. We provide a powerful platform for design professionals to discover and sample materials, connecting them to hundreds of manufacturers through brand discovery, rep engagement, and...

  • Senior Data Architect

    2 weeks ago


    New York, New York, United States Circa - IPG DXTRA Full time

    Job Title: Senior Data ArchitectWeber Shandwick is seeking a highly skilled Senior Data Architect to lead the development and maintenance of our data pipelines, databases, and processing systems. The ideal candidate will have a strong background in cloud computing, particularly in AWS, and experience with data engineering tools such as Apache Airflow and...