Staff Software Developer, Batch Data Pipeline

7 days ago

New York, New York, United States Tecton Corporation Full time
About Tecton Corporation

Tecton helps companies unlock the full potential of their data for AI applications. Our platform streamlines the complex process of preparing and delivering data to models, accelerating the development of smarter, more impactful AI applications.

Tecton is funded by Sequoia Capital, Andreessen Horowitz, and Kleiner Perkins, along with strategic investments from Snowflake and Databricks. We have a fast-growing team distributed around the world, with offices in San Francisco and New York City.

Our team has years of experience building and operating business-critical machine learning systems at leading tech companies like Uber, Google, Meta, Airbnb, Lyft, and Twitter.

About the Batch Compute Team

We are building Rift, a new fully managed compute environment that allows data scientists to construct powerful batch and streaming pipelines in Python. Our new environment leverages popular open-source technologies such as Ray, Arrow, and DuckDB, along with deep integrations with Spark platforms (Databricks, EMR, Dataproc) and data warehouses (e.g. Snowflake, BigQuery, RedShift).

This role is a critical part of our Batch Compute team, responsible for architecting, designing, and scaling the core compute engines and storage architecture used by every Tecton customer. You will contribute to the performance of our query optimizer, from parsing & optimization to plan selection.

Responsibilities
  • Own and lead large technical domains starting from the problem definition and technical requirements to implementation and maintenance
  • Lead multi-engineer projects of strategic importance to Tecton spanning cross-functional teams including product management and other engineering teams
  • Drive efforts to improve engineering practices, tooling, and processes along with mentorship for senior engineers
  • Develop a deep understanding of the fundamental problems our customers face in building ML systems
  • Be a generalist as needed, with versatility in engineering expertise
Requirements
  • Experience working in large Python, Java, Kotlin, or Go codebases and running cloud-native Spark systems (e.g. AWS EMR, Databricks, GCP Dataproc)
  • Experience in performance tuning of Spark, Ray, Maestro, or Airflow jobs
  • Knowledge of data formats such as Parquet, Avro, Arrow, Iceberg, or Delta Lake and object storage (e.g. S3, GCS)
  • Expertise with cloud-scale query performance, query optimization, query planning, heuristic query execution techniques, and cost-driven optimizations
  • Experience with internals of distributed systems, SQL/NoSQL databases, data lakes, or data warehouses
  • Strong communication skills and ability to write detailed technical specifications
  • Excitement about coaching and mentorship of junior engineers
  • BSc, MS, or PhD in Computer Science or related fields
  • 8+ years of experience in building product software systems
  • 5+ years of technical leadership experience for a group of engineers

Tecton values diversity and is an equal opportunity employer committed to creating an inclusive environment for all employees and applicants without regard to race, color, religion, national origin, gender, sexual orientation, age, marital status, veteran status, disability status, or other applicable legally protected characteristics.

  • Senior Software Engineer

    1 week ago

    New York, New York, United States thisiswhy Full time

    Job Title: Senior Software Engineer - Data Pipeline SpecialistOverviewThis role is crucial in designing, developing, and implementing high-quality software solutions using Python programming. As a key member of the team, you will contribute significantly to the organization's success by creating efficient and scalable applications.SalaryWe offer a...

  • Data Engineer for Large-Scale Data Pipelines

    6 days ago

    New York, New York, United States ASCENDING Full time

    Job DescriptionData Engineer for Large-Scale Data PipelinesWe're seeking a skilled data engineer with 3-7 years of experience to join our team at ASCENDING. This role centers on ensuring the robustness and reliability of large-scale data pipelines, from inception through deployment.Key Responsibilities:Conduct thorough testing of SQL-centric applications to...

  • Chief Data Solutions Engineer

    7 days ago

    New York, New York, United States Condé Nast Full time

    Condé Nast is a global media company renowned for producing high-quality content with a significant presence in over 32 territories. Our portfolio includes influential publications such as Vogue, Vanity Fair, and Wired, among others. About the Company We are seeking an experienced Data Solutions Engineer to join our team. The ideal candidate will have a...

  • Enterprise Software Engineer

    6 days ago

    New York, New York, United States United Software Group Full time

    About United Software GroupAt United Software Group, we're driven by a passion for innovation and a commitment to delivering exceptional software solutions. We're currently seeking an experienced Enterprise Software Engineer to join our team and contribute to the development of real-time data systems that make a meaningful impact.Salary and BenefitsWe offer...

  • Data Integration Specialist

    1 week ago

    New York, New York, United States Intelliswift Software Full time

    About the RoleWe are seeking an experienced Data Integration Specialist to join our team at Intelliswift Software. This is a unique opportunity to work on complex data integration projects and contribute to the development of cutting-edge solutions using Palantir.Job DescriptionThe successful candidate will have extensive experience in implementing solutions...

  • Senior Data Pipeline Architect

    7 days ago

    New York, New York, United States EDO Full time

    About EDOEDO is the leading platform for measuring Convergent TV advertising, combining real-time engagement signals with world-class decision science. Our mission is to maximize creative and media performance for modern marketers at leading brands, TV networks, entertainment studios, and ad agencies.The RoleWe are seeking a skilled Data Engineer to join our...

  • Principal AI Data Systems Architect

    7 days ago

    New York, New York, United States Shaped Full time

    Shaped is a pioneering company at the forefront of artificial intelligence discovery. We are seeking an exceptional Senior Data Engineer to spearhead the design, development, and optimization of our real-time and batch streaming infrastructure.We're building a state-of-the-art AI discovery engine that relies on the reliable ingestion of customer data through...

  • Senior Data Engineer

    7 days ago

    New York, New York, United States NYU Langone Health Full time

    Job Title:Senior Data Engineer - Data Integration and AnalyticsJob Summary:NYU Langone Health is seeking a Senior Data Engineer to join our team. As a Senior Data Engineer, you will play a critical role in designing and developing robust ETL pipelines to integrate diverse data sources.Key Responsibilities:Strategize, design, and develop automated cloud...

  • Artificial Intelligence Engineer

    2 weeks ago

    New York, New York, United States MongoDB Full time

    About MongoDBMongoDB empowers innovators to transform industries by unleashing the power of software and data. We enable organizations to build, scale, and run modern applications by helping them modernize legacy workloads, innovate with emerging technologies, and unlock AI-driven insights.We offer a globally distributed, multi-cloud database in over 115...

  • Senior Data Engineer

    7 days ago

    New York, New York, United States NBCUniversal Full time

    About NBCUniversalNBCUniversal owns and operates over 20 different businesses across the globe, including a portfolio of news and entertainment television networks, a premier motion picture company, significant television production operations, a leading television stations group, world-renowned theme parks, and a premium ad-supported streaming service.As a...

  • Senior Software Developer

    1 week ago

    New York, New York, United States Triunity Software Full time

    We are seeking a highly skilled and experienced Senior Software Developer to join our dynamic development team at Triunity Software.Job DescriptionThe ideal candidate will have a minimum of 10 years of hands-on experience in designing, developing, and maintaining complex web applications using the .NET framework. The successful candidate will play a key role...

  • Backend Data Systems Engineer

    5 days ago

    New York, New York, United States Flume Health Full time

    Job OverviewFlume Health is a cutting-edge healthcare software company that streamlines data integration and exchange between various stakeholders. As a pioneer in the industry, Flume's Relay platform offers a cloud-native integration solution that simplifies complex data workflows.We are seeking an experienced Backend Data Systems Engineer to join our team....

  • Software Development Engineer II

    7 days ago

    New York, New York, United States Yahoo Holdings Inc. Full time

    Job Description:At Yahoo Holdings Inc., we are seeking a highly skilled Software Development Engineer II to join our team. As a key member of our Data Engineering team, you will be responsible for designing, building, and launching efficient and reliable data pipelines to move and transform data at scale.Responsibilities:• Design and develop complex data...

  • AI Engineer for High-Performance Data Pipelines

    6 days ago

    New York, New York, United States Capital One Full time

    About the RoleAt Capital One, we're seeking an experienced AI Engineer to lead our team in designing and building high-performance data pipelines that drive business impact. As a key member of our Agile team, you'll collaborate with cross-functional teams to create and enhance software that enables state-of-the-art big data and AI applications.Key...

  • Distributed Systems Engineer

    3 days ago

    New York, New York, United States Alethea Full time

    Job OverviewAlethea is a pioneering tech company dedicated to detecting and mitigating disinformation and social media manipulation. We are seeking an exceptional Senior Distributed Systems Engineer to spearhead the design, development, and maintenance of our complex data acquisition and processing pipelines.Key Responsibilities & Skills:Design...

  • Software Development Architect

    7 days ago

    New York, New York, United States NTT DATA Group Corporation Full time

    About the RoleWe are seeking a seasoned Software Development Architect to join our team at NTT DATA Group Corporation. In this role, you will be responsible for leading the development of cloud-native applications that enhance healthcare services.Job DescriptionThe ideal candidate will have 10+ years of experience in software development, with a strong...

  • Highly Skilled Data Solutions Developer

    7 days ago

    New York, New York, United States Data Intellect Full time

    Data Intellect is a leading data and technology consultancy firm, specialized in financial and capital markets technology solutions. We pride ourselves on our ability to turn complex data problems into smart data solutions.As a highly skilled Junior kdb+ Developer, you will play a key role in delivering projects on time and on budget. Your primary...

  • Software Development Specialist

    3 weeks ago

    New York, New York, United States Litnerd Full time

    Litnerd is a leading provider of innovative educational solutions. As a Senior Software Engineer, you will play a pivotal role in developing and maintaining our cutting-edge software products.About the RoleDesign and build user-friendly software applications that empower students to fall in love with reading.Collaborate with cross-functional teams to drive...

  • Senior Cancer Research Software Developer

    7 days ago

    New York, New York, United States Memorial Sloan Kettering Full time

    Job DescriptionWe are seeking a skilled and dedicated software developer to join our team at Memorial Sloan Kettering in the role of Senior Cancer Research Software Developer. This position involves developing data processing pipelines for large-scale cancer genomics data, collaborating with scientists to design and implement data platforms, and ensuring...

  • Senior Data Engineer

    1 month ago

    New York, New York, United States TechnoGen Full time

    Job Title: Senior Data EngineerJob Summary:We are seeking a Senior Data Engineer to join our Data Infrastructure team at TechnoGen. As a Senior Data Engineer, you will be responsible for building and maintaining tools and infrastructure to write, test, and schedule batch data pipelines. Your work will enable other developers, data scientists, and analysts to...