Senior Data Engineering Lead

3 days ago


Palo Alto, California, United States Georgia IT Inc Full time

Georgia IT Inc is seeking a highly skilled Data Architect to lead the development of our data engineering capabilities. With a focus on cloud native technologies, this role will be responsible for designing and implementing scalable data pipelines, ensuring seamless data integration, and providing data-driven insights to inform business decisions.

As a key member of our team, you will work closely with cross-functional groups to understand business requirements, design and implement data solutions, and collaborate with engineers to ensure successful deployment. Your expertise in data orchestration, task management, database management, API/webhook implementation, and streaming data processing will be crucial in driving the success of our data initiatives.

Key responsibilities include:

  • Data Orchestration: Design, implement, and manage data workflows using Airflow to automate and orchestrate data processing tasks.
  • Task Management: Develop and maintain distributed task processing using Celery and ensure robust task queue management with Redis or RabbitMQ.
  • Database Management: Design and manage databases using Cosmos DB, MongoDB, and PostgreSQL.
  • API and Webhooks: Implement and manage FastAPI webhooks to handle data ingestion and integration tasks.
  • Streaming Data: Implement and manage Kafka Streams to handle real-time data processing and streaming requirements.

Required skills and qualifications include:

  • Proven experience with Airflow for data orchestration and workflow management.
  • Hands-on experience with Celery for task management and Redis or RabbitMQ for messaging.
  • Proficiency with Cosmos DB, MongoDB, and PostgreSQL for data storage and management.
  • Experience developing and managing webhooks using FastAPI and integrating with Azure Functions.
  • Familiarity with Iceberg for data lake management and optimization.

Estimated salary range is between $120,000 - $180,000 annually, depending on experience and qualifications. If you are a motivated and results-driven professional looking to make a significant impact in the field of data engineering, please submit your application.



  • Palo Alto, California, United States InDepth Engineering Solutions, LLC Full time

    Job DescriptionWe are seeking a highly skilled Senior Data Protection Engineer to join our team at InDepth Engineering Solutions, LLC. This role will play a critical part in enhancing our cybersecurity posture by focusing on tactical data protection operations, automation, and tool development.The ideal candidate will have a strong technical background, a...


  • Palo Alto, California, United States Machinify, Inc. Full time

    Machinify, Inc. is revolutionizing the healthcare industry with its AI-powered software products that transform claims and payment operations.About MachinifyWe are seeking a highly skilled Senior Data Engineer Lead to join our team and help us achieve our mission.Salary RangeThe estimated base salary for this position is $220k-$250k per year, which is...


  • Palo Alto, California, United States Insight Global Full time

    Company Overview">We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. Our team is dedicated to developing advanced AI applications that drive innovation and growth.">Job Summary">We are seeking a highly skilled Senior Data Annotation Engineer to join our small team...


  • Palo Alto, California, United States Woven Full time

    Unlock the Future of MobilityWe're seeking a Senior Data Engineer to elevate our data engineering practices, enhancing existing pipelines and driving the transition to a more scalable and centralized architecture. This role will play a critical part in advancing how data is stored, processed, and transformed to support diverse use cases across our...


  • Palo Alto, California, United States Machinify Inc. Full time

    Machinify Inc. is revolutionizing the healthcare claims and payment operations landscape with its cutting-edge AI-powered software products.The company has identified a pressing need to address the staggering $200B in annual claims mispayments, which translates to incredible waste, friction, and frustration for patients, providers, and payers alike.As a...


  • Palo Alto, California, United States HP Development Company, L.P. Full time

    HP Development Company, L.P. is seeking a highly skilled Senior Data Science Manager to lead its machine learning engineering team. The estimated salary for this role is $185,000 per year, based on industry standards and the location of HP Development Company, L.P.The Senior Data Science Manager will oversee a team of data scientists and engineers, guiding...

  • Data Scientist Lead

    3 days ago


    Palo Alto, California, United States SmartNews Full time

    About the RoleAs a Senior Data Scientist at SmartNews, you will play a critical role in driving the company's growth by providing actionable data insights that inform key decisions across Engineering, Product, Sales, and other departments.We are seeking an experienced Data Scientist to join our team, focusing on ad algorithm analysis, business performance...

  • Lead Data Engineer

    2 days ago


    Palo Alto, California, United States Tesla Full time

    Tesla is seeking a Lead Data Engineer to join our team and drive the development of data pipelines and analytics tooling to support the growth of our energy business. As a Lead Data Engineer, you will be responsible for designing and developing scalable data systems to extract insights from various data sources and collaborating with cross-functional teams...


  • Palo Alto, California, United States Tesla Full time

    Unlock Your Potential as a Senior AI Research Engineer at TeslaTesla is seeking an experienced Senior AI Research Engineer to lead our efforts in developing and optimizing large-scale data engines that power our autonomous driving systems. As a key member of our team, you will have access to unparalleled resources, including the world's largest self-driving...


  • Palo Alto, California, United States Woven by Toyota Full time

    At Woven by Toyota, we're shaping the future of mobility through innovative technologies. As a Senior Software Engineer on our Simulation Data and Analysis team, you'll play a pivotal role in developing the metrics and evaluation frameworks that assess the quality of our autonomy and validation stacks.About the RoleYou'll be responsible for designing and...


  • Palo Alto, California, United States AISERA Full time

    Aisera is a cutting-edge provider of AI Copilot solutions, leveraging AiseraGPT and Generative AI to drive business transformation and revenue growth through a self-service model.Our AI Copilot utilizes industry and domain-specific Large Language Models (LLMs) to deliver human-like experiences and auto-remediate requests through AI workflows.With 400+...


  • Palo Alto, California, United States Machinify, Inc. Full time

    Transforming Healthcare with AI: A Technical Leadership RoleMachinify, Inc. is revolutionizing the healthcare industry by leveraging AI to transform claims and payment operations. Our innovative platform has already demonstrated significant success in increasing the speed and accuracy of claims processing.We are seeking an exceptional Data Science Lead to...


  • Palo Alto, California, United States Qualified Health Full time

    Job Overview:Achieve excellence as a Senior Data Science Leader in our innovative healthcare team, focusing on AI-driven solutions that transform patient care and outcomes.About the Role:We are seeking a highly skilled and experienced Senior Data Science Leader to drive technical strategy, lead cross-functional teams, and deliver impactful machine learning...


  • Palo Alto, California, United States International Flavors and Fragrances Full time

    Job SummaryWe are seeking an experienced data engineer to accelerate our efforts in transforming the chemical industry from petroleum-based to bio-based. At International Flavors and Fragrances, we have a long history of developing cell factories that produce enzymes at scale for a wide range of applications.We are overhauling our workflows to streamline...


  • Palo Alto, California, United States Glean Full time

    We are building a world-class Data Organization at Glean, and we're seeking an experienced Data Science Lead to collaborate with our engineering, product management, and design teams. As a Data Science Lead, you will play a key role in defining and building data assets, creating quantitative frameworks and methodologies, and exploring different product...


  • Palo Alto, California, United States Snap Full time

    Ad Attribution Team LeaderSnap Inc is a pioneering technology company that harnesses the power of the camera to transform lives and foster global connections. We believe in empowering people to express themselves, live in the moment, learn about the world, and have fun together.Our products include Snapchat, Lens Studio, and Spectacles, which have reached...

  • Data Scientist Lead

    3 weeks ago


    Palo Alto, California, United States Stanford University Full time

    **Job Title:** Data Scientist Lead - Emergency CareCompany OverviewStanford University seeks a highly skilled and experienced Senior Research Scientist to join our Department of Emergency Medicine. This is a 3-year fixed-term position at 100% FTE.SalaryThe estimated salary for this role is $160,000 per annum, commensurate with experience.Job DescriptionAs a...


  • Palo Alto, California, United States Tesla Full time

    Job DescriptionCome and contribute to the development of Tesla's innovative data platforms, a cutting-edge platform that processes vast amounts of IoT data daily.About the RoleWe are seeking an experienced software engineer with a passion for distributed systems to join our team. As a Lead Data Engineer for Large-Scale Distributed Systems, you will be...


  • Palo Alto, California, United States eTek IT Services, Inc. Full time

    At eTek IT Services, Inc., we are seeking a highly skilled Senior Cloud Data Architect to join our team. This is a full-time opportunity for the right candidate.About the RoleThe ideal candidate will have experience in building data infrastructure for analytics teams, including writing code for processing large datasets in distributed cloud environments....


  • Palo Alto, California, United States Navan Full time

    About the OpportunityNavan is looking for a skilled Data Architect to lead the design and implementation of its data architecture strategy. The successful candidate will have a proven track record in driving innovation and excellence in data engineering.Responsibilities:Design and develop high-performance data architectures to meet business needs.Implement...