Web Crawler Engineer

2 weeks ago

San Francisco, United States Acceler8 Talent Full time

Web Crawler Engineer


In the rapidly evolving landscape of artificial intelligence, the role of a Web Crawler Engineer is becoming increasingly critical. As technology progresses, the demand for high-quality data to fuel innovative models and applications grows significantly. This pivotal role supports the development of cutting-edge AI by ensuring a steady supply of well-curated data.


Our organization stands at the forefront of AI technology, backed by industry leaders and pioneering venture capital firms. With a mission to enhance human-computer interaction, we are committed to solving complex AI challenges through a synergy of technology and human creativity. Our team consists of dedicated professionals who bring their diverse expertise to tackle unique problems, driving forward the boundaries of what's possible with AI.


As a Web Crawler Engineer, you will play a fundamental role in sculpting the landscape of data that our AI models rely on. You'll be tasked with architecting and implementing robust web crawler systems that are scalable and efficient. This involves crafting high-performance data acquisition pipelines and ensuring the integrity and reliability of the data collected.


What we can offer you:

  • A collaborative environment with driven industry experts.
  • Opportunities to work on groundbreaking projects that shape the future of AI.
  • Competitive salary and comprehensive benefits.
  • Support for professional development and continuous learning.
  • Relocation assistance for qualified candidates.


Key responsibilities include:

  • Designing and developing large-scale distributed web crawler systems.
  • Implementing and maintaining web crawlers and scrapers for dynamic content handling.
  • Creating and managing data acquisition pipelines to process large data volumes efficiently.
  • Optimizing the performance and scalability of web crawlers.
  • Collaborating with teams across the organization to enhance data quality and utility.


This role is designed for innovators and problem solvers ready to make a significant impact in the AI field. If you are driven by challenging technical problems and have a passion for data and its capabilities in AI, we would love to hear from you.


Relevant technologies and skills for the role include proficiency in high-performance programming languages like Go, Rust, or C++, expertise in Docker/Kubernetes for orchestration, and experience with cloud services like GCP or AWS. A strong background in web crawling and a curiosity about the impact of data quality on large language models (LLMs) are also highly valued.

  • Web Crawler Engineer

    1 week ago

    San Francisco, United States Acceler8 Talent Full time

    Web Crawler EngineerIn the rapidly evolving landscape of artificial intelligence, the role of a Web Crawler Engineer is becoming increasingly critical. As technology progresses, the demand for high-quality data to fuel innovative models and applications grows significantly. This pivotal role supports the development of cutting-edge AI by ensuring a steady...

  • Web Crawler Engineer

    2 weeks ago

    San Francisco, United States Acceler8 Talent Full time

    Web Crawler EngineerIn the rapidly evolving landscape of artificial intelligence, the role of a Web Crawler Engineer is becoming increasingly critical. As technology progresses, the demand for high-quality data to fuel innovative models and applications grows significantly. This pivotal role supports the development of cutting-edge AI by ensuring a steady...

  • Web Crawler Engineer

    2 weeks ago

    San Francisco, United States Acceler8 Talent Full time

    Web Crawler EngineerIn the rapidly evolving landscape of artificial intelligence, the role of a Web Crawler Engineer is becoming increasingly critical. As technology progresses, the demand for high-quality data to fuel innovative models and applications grows significantly. This pivotal role supports the development of cutting-edge AI by ensuring a steady...

  • MTS: Data Acquisition

    4 weeks ago

    San Francisco, United States essential AI Full time

    Job DescriptionJob DescriptionEssential AI’s mission is to deepen the partnership between humans and computers, unlocking collaborative capabilities that far exceed what could be achieved today. We believe that building delightful end-user experiences requires innovating across the stack - from the UX all the way down to models that achieve the best user...

  • Automation Engineer

    4 weeks ago

    San Francisco, California, United States Invisible Technologies Full time

    Company DescriptionInvisible Technologies offers unusual services (a combination of outsourcing and automation) to fast-scaling, innovative companies. Each company understands and uses our services in a variety of ways. It's critical we can communicate to segments of users with visually appealing, and highly communicative assets. You're helping us explain...

  • Web Engineer

    1 week ago

    San Francisco, United States X (formerly Twitter) Full time

    Are you prepared to join the X team and help build the ultimate real-time information-sharing app, revolutionizing how people connect? At X, we're on a mission to become a trusted global digital public square, committed to minimal censorship within legal boundaries. Our goal is to empower every user to freely create and share ideas, fostering open public...

  • Web Engineer

    3 weeks ago

    San Diego, United States PINT Full time

    Description PINT is seeking a web engineer with experience building and maintaining typical web systems including environments, networks, databases, deployment workflows, content management systems, etc. Engineers at PINT work across the application stack from api and endpoint service development to plugin and application work, as well as database management...

  • Principal Engineer, Frontend/Fullstack, Web

    4 weeks ago

    San Francisco, California, United States Square Full time

    Job Description Square recently consolidated all members of our web front-end engineering discipline into a centralized organization, specifically to elevate our velocity and quality. Our high-level objectives include deprecating older JavaScript frameworks and standardizing on React, which will include a substantial migration effort. This is a...

  • Lead Web Developer

    4 weeks ago

    San Francisco, California, United States Elajika Inc Full time

    Professional Degree BS in Computer Science or Engineering – preferable.Minimum 2+ years of strong real-time experience in developing React JS based applications with ReduxStrong exercise in working with Node JS Web Pack and other tools related to React Front End Development.Experience in designing & developing reusable front end frameworks node modules and...

  • Lead Web Developer

    3 days ago

    San Francisco, California, United States Elajika Inc Full time

    Designation Lead Web Developer (React)Location San Francisco CAJob Type ContractProfessional Degree BS in Computer Science or Engineering – preferable.Minimum 2+ years of strong real time experience in developing React JS based applications with ReduxStrong exercise in working with Node JS Web Pack and other tools related to React Front End...

  • Lead Web Developer

    3 days ago

    San Francisco, California, United States Elajika Inc Full time

    Designation Lead Web Developer (React)Location San Francisco CAJob Type ContractClient William SonomaProfessional Degree BS in Computer Science or Engineering – preferable.Minimum 2+ years of strong real-time experience in developing React JS based applications with ReduxStrong exercise in working with Node JS WebPack and other tools related to React Front...

  • Sr Data Engineer

    4 days ago

    San Francisco, United States Varite Full time

    Pay rate range: $70-90/hr. True Job Title-Senior Infrastructure Automation Engineer Location-this is a hybrid position but we can consider remote candidates with exceptional technical expertise. This role will be subject to level 2 screening. In this role, you will get an opportunity to broadly apply your engineering skills across various technology...

  • Sr Data Engineer

    1 week ago

    San Francisco, United States Varite Full time

    Pay rate range: $80-90/hr. True Job Title-Senior Infrastructure Automation Engineer Location-this is a hybrid position but we can consider remote candidates with exceptional technical expertise. This role will be subject to level 2 screening. In this role, you will get an opportunity to broadly apply your engineering skills across various technology...

  • Sr. Software Engineer | Web

    4 weeks ago

    San Francisco, United States Zitara Technologies, Inc. Full time

    Company In order to keep climate change to 1.5°C, we’ll need 30% of global GDP (all of energy generation and transportation) to run on batteries by 2035. Zitara Technologies (YCombinator S20) builds predictive battery management software for transportation and energy customers with large deployments. Our customers operate >$100M deployments of batteries...

  • Lead Web Developer

    4 weeks ago

    San Francisco, United States Elajika Inc Full time

    Professional Degree BS in Computer Science or Engineering – preferable.Minimum 2+ years of strong real-time experience in developing React JS based applications with ReduxStrong exercise in working with Node JS Web Pack and other tools related to React Front End Development.Experience in designing & developing reusable front end frameworks node modules...

  • Lead Web Developer

    4 weeks ago

    San Francisco, United States Elajika Inc Full time

    Designation Lead Web Developer (React)Location San Francisco CAJob Type ContractProfessional Degree BS in Computer Science or Engineering – preferable.Minimum 2+ years of strong real time experience in developing React JS based applications with ReduxStrong exercise in working with Node JS Web Pack and other tools related to React Front End...

  • Lead Web Developer

    4 weeks ago

    San Francisco, United States Elajika Inc Full time

    Designation Lead Web Developer (React)Location San Francisco CAJob Type ContractClient William SonomaProfessional Degree BS in Computer Science or Engineering – preferable.Minimum 2+ years of strong real-time experience in developing React JS based applications with ReduxStrong exercise in working with Node JS WebPack and other tools related to...

  • Web UI Developer-Architect

    1 week ago

    San Francisco, United States Globant IT Services Corp. Full time

    *DUTIES: Responsible for leading multiple development projects of complex web and mobile applications that include both front-end and back-end development. Lead the collaboration with the clients and Globant Pods to thoroughly understand the market needs and product/project requirements. Control the representation of the product in the design and development...

  • Senior Software Engineer | Web and IoT

    1 week ago

    San Francisco, United States Zitara Technologies, Inc. Full time

    Company In order to keep climate change to 1.5°C, we'll need 30% of global GDP (all of energy generation and transportation) to run on batteries by 2035. Zitara Technologies (YCombinator S20) builds predictive battery management software for transportation and energy customers with large deployments. Our customers operate >$100M deployments of batteries in...

  • Senior Software Engineer | Web and IoT

    3 days ago

    San Francisco, United States Zitara Technologies, Inc. Full time

    Company In order to keep climate change to 1.5°C, we'll need 30% of global GDP (all of energy generation and transportation) to run on batteries by 2035. Zitara Technologies (YCombinator S20) builds predictive battery management software for transportation and energy customers with large deployments. Our customers operate >$100M deployments of batteries in...