Web Crawler Engineer

3 weeks ago


San Francisco, United States Acceler8 Talent Full time

Web Crawler Engineer


In the rapidly evolving landscape of artificial intelligence, the role of a Web Crawler Engineer is becoming increasingly critical. As technology progresses, the demand for high-quality data to fuel innovative models and applications grows significantly. This pivotal role supports the development of cutting-edge AI by ensuring a steady supply of well-curated data.


Our organization stands at the forefront of AI technology, backed by industry leaders and pioneering venture capital firms. With a mission to enhance human-computer interaction, we are committed to solving complex AI challenges through a synergy of technology and human creativity. Our team consists of dedicated professionals who bring their diverse expertise to tackle unique problems, driving forward the boundaries of what's possible with AI.


As a Web Crawler Engineer, you will play a fundamental role in sculpting the landscape of data that our AI models rely on. You'll be tasked with architecting and implementing robust web crawler systems that are scalable and efficient. This involves crafting high-performance data acquisition pipelines and ensuring the integrity and reliability of the data collected.


What we can offer you:

  • A collaborative environment with driven industry experts.
  • Opportunities to work on groundbreaking projects that shape the future of AI.
  • Competitive salary and comprehensive benefits.
  • Support for professional development and continuous learning.
  • Relocation assistance for qualified candidates.


Key responsibilities include:

  • Designing and developing large-scale distributed web crawler systems.
  • Implementing and maintaining web crawlers and scrapers for dynamic content handling.
  • Creating and managing data acquisition pipelines to process large data volumes efficiently.
  • Optimizing the performance and scalability of web crawlers.
  • Collaborating with teams across the organization to enhance data quality and utility.


This role is designed for innovators and problem solvers ready to make a significant impact in the AI field. If you are driven by challenging technical problems and have a passion for data and its capabilities in AI, we would love to hear from you.


Relevant technologies and skills for the role include proficiency in high-performance programming languages like Go, Rust, or C++, expertise in Docker/Kubernetes for orchestration, and experience with cloud services like GCP or AWS. A strong background in web crawling and a curiosity about the impact of data quality on large language models (LLMs) are also highly valued.


  • Web Crawler Engineer

    3 weeks ago


    San Francisco, United States Acceler8 Talent Full time

    Web Crawler EngineerIn the rapidly evolving landscape of artificial intelligence, the role of a Web Crawler Engineer is becoming increasingly critical. As technology progresses, the demand for high-quality data to fuel innovative models and applications grows significantly. This pivotal role supports the development of cutting-edge AI by ensuring a steady...

  • MTS: Data Acquisition

    1 month ago


    San Francisco, United States essential AI Full time

    Job DescriptionJob DescriptionEssential AI’s mission is to deepen the partnership between humans and computers, unlocking collaborative capabilities that far exceed what could be achieved today. We believe that building delightful end-user experiences requires innovating across the stack - from the UX all the way down to models that achieve the best user...

  • Automation Engineer

    1 month ago


    San Francisco, California, United States Invisible Technologies Full time

    Company DescriptionInvisible Technologies offers unusual services (a combination of outsourcing and automation) to fast-scaling, innovative companies. Each company understands and uses our services in a variety of ways. It's critical we can communicate to segments of users with visually appealing, and highly communicative assets. You're helping us explain...


  • San Francisco, United States BHO Tech Full time

    As an integral part of our technology team, you will work on our customer facing applications and internal tools. This means you’ll get to work with our groundbreaking natural language search technologies, helping us design, develop, and deploy our applications in a variety of contexts. You’ll wear many hats, touch many parts of our system, and have a...

  • Web Engineer

    4 weeks ago


    San Diego, United States PINT Full time

    Description PINT is seeking a web engineer with experience building and maintaining typical web systems including environments, networks, databases, deployment workflows, content management systems, etc. Engineers at PINT work across the application stack from api and endpoint service development to plugin and application work, as well as database management...


  • San Francisco, California, United States Block Full time

    Company DescriptionSince we opened our doors in 2009, the world of commerce has evolved immensely, and so has Square. After enabling anyone to take payments and never miss a sale, we saw sellers stymied by disparate, outmoded products and tools that wouldn't work together.To solve this problem, we expanded into software and built integrated solutions to help...


  • San Francisco, United States Square Full time

    Job Description Square recently consolidated all members of our web front-end engineering discipline into a centralized organization, specifically to elevate our velocity and quality. Our high-level objectives include deprecating older JavaScript frameworks and standardizing on React, which will include a substantial migration effort. This is a...

  • Lead Web Developer

    1 month ago


    San Francisco, California, United States Elajika Inc Full time

    Professional Degree BS in Computer Science or Engineering – preferable.Minimum 2+ years of strong real-time experience in developing React JS based applications with ReduxStrong exercise in working with Node JS Web Pack and other tools related to React Front End Development.Experience in designing & developing reusable front end frameworks node modules and...

  • Lead Web Developer

    1 week ago


    San Francisco, California, United States Elajika Inc Full time

    Designation Lead Web Developer (React)Location San Francisco CAJob Type ContractProfessional Degree BS in Computer Science or Engineering – preferable.Minimum 2+ years of strong real time experience in developing React JS based applications with ReduxStrong exercise in working with Node JS Web Pack and other tools related to React Front End...

  • Lead Web Developer

    1 week ago


    San Francisco, California, United States Elajika Inc Full time

    Designation Lead Web Developer (React)Location San Francisco CAJob Type ContractClient William SonomaProfessional Degree BS in Computer Science or Engineering – preferable.Minimum 2+ years of strong real-time experience in developing React JS based applications with ReduxStrong exercise in working with Node JS WebPack and other tools related to React Front...


  • San Francisco, United States Square Full time

    Company DescriptionSince we opened our doors in 2009, the world of commerce has evolved immensely, and so has Square. After enabling anyone to take payments and never miss a sale, we saw sellers stymied by disparate, outmoded products and tools that wouldn't work together.To solve this problem, we expanded into software and built integrated solutions to...


  • San Francisco, California, United States Block Full time

    Company DescriptionSince we opened our doors in 2009, the world of commerce has evolved immensely, and so has Square. After enabling anyone to take payments and never miss a sale, we saw sellers stymied by disparate, outmoded products and tools that wouldn't work together.To solve this problem, we expanded into software and built integrated solutions to help...

  • Web Developer

    2 days ago


    San Francisco, California, United States ITJobsList Full time

    Our client is actually looking for a senior-level software engineer that could design and create extremely scalable tolerant net APIs as well as distributed services. Essential Job Functions Participate in Agile preparation as well as execution Develop API characteristics bug fixes as well as automation tools. Strong master software growth encounters Strong...


  • San Francisco, United States Square Full time

    Job DescriptionJob DescriptionCompany DescriptionSince we opened our doors in 2009, the world of commerce has evolved immensely, and so has Square. After enabling anyone to take payments and never miss a sale, we saw sellers stymied by disparate, outmoded products and tools that wouldn’t work together. To solve this problem, we expanded into software and...

  • Lead Web Developer

    1 month ago


    San Francisco, United States Elajika Inc Full time

    Designation Lead Web Developer (React)Location San Francisco CAJob Type ContractProfessional Degree BS in Computer Science or Engineering – preferable.Minimum 2+ years of strong real time experience in developing React JS based applications with ReduxStrong exercise in working with Node JS Web Pack and other tools related to React Front End...

  • Lead Web Developer

    1 month ago


    San Francisco, United States Elajika Inc Full time

    Professional Degree BS in Computer Science or Engineering – preferable.Minimum 2+ years of strong real-time experience in developing React JS based applications with ReduxStrong exercise in working with Node JS Web Pack and other tools related to React Front End Development.Experience in designing & developing reusable front end frameworks node modules...

  • Lead Web Developer

    1 month ago


    San Francisco, United States Elajika Inc Full time

    Designation Lead Web Developer (React)Location San Francisco CAJob Type ContractClient William SonomaProfessional Degree BS in Computer Science or Engineering – preferable.Minimum 2+ years of strong real-time experience in developing React JS based applications with ReduxStrong exercise in working with Node JS WebPack and other tools related to...


  • San Francisco, United States Zitara Technologies, Inc. Full time

    Company In order to keep climate change to 1.5°C, we'll need 30% of global GDP (all of energy generation and transportation) to run on batteries by 2035. Zitara Technologies (YCombinator S20) builds predictive battery management software for transportation and energy customers with large deployments. Our customers operate >$100M deployments of batteries in...


  • San Francisco, United States Shimento, Inc. Full time

    Senior Infrastructure Automation Engineer9 MonthsLocation:- San Francisco, CA Qualifications: ***True Job Title-Senior Infrastructure Automation Engineer *** Location-this is a hybrid position, but we can consider remote candidates with exceptional technical expertiseThis role will be subject to level 2 screening. Demonstrated technical expertise across a...


  • San Francisco, United States JobRialto Full time

    Description: Demonstrated technical expertise across a variety of technology platforms and solutions Ability to work effectively independently or in a team environment Strong analytical, collaboration, leadership, critical thinking, multitasking, and time management skills Ability to influence successfully in a highly matrix or virtual organization...