Senior Data Engineer

1 day ago


San Diego, California, United States Zapata Technology Full time
Job Description

Zapata Technology is seeking an experienced Data Engineer to lead the migration of our data infrastructure from Confluent with manual ETL processes to Apache NiFi.

Key Responsibilities
  • Plan, coordinate, and execute the migration from Confluent with manual ETL processes to Apache NiFi, ensuring minimal downtime and data integrity during the migration.
  • Design and implement automated ETL workflows using Apache NiFi to improve the efficiency and reliability of data processing.
  • Work with PostgreSQL, MongoDB, or Cassandra to ensure seamless data integration and management, optimizing database performance and reliability.
  • Deploy and manage all components within a Kubernetes environment to ensure scalability, high availability, and efficient resource utilization.
  • Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions while documenting processes, workflows, and configurations.
  • Implement monitoring solutions to track the performance of data pipelines and databases, continuously optimizing workflows for performance and cost-efficiency.
Requirements
  • Proven experience in data engineering, with a focus on ETL processes and data migration projects.
  • Proficiency in Apache NiFi for ETL automation.
  • Strong knowledge of PostgreSQL, MongoDB, or Cassandra.
  • Familiarity with Neo4j and its integration (preferred but not mandatory).
  • Experience with Kubernetes for container orchestration.
  • Understanding of data architecture, data modeling, and data integration best practices.
Preferred Qualifications
  • Experience in cloud platforms such as AWS, Google Cloud, or Azure.
  • Familiarity with additional data processing tools and frameworks.
  • Certification in Kubernetes or related technologies.
  • Ability to incorporate Neo4j into the data architecture if required.
Education and Clearance

Bachelor's degree in Information Technology or a related field, and two to three years of experience is preferred. Secret security clearance is required.


  • Senior Data Engineer

    3 weeks ago


    San Diego, California, United States BioPhase Solutions Full time

    Job Title: Senior Data EngineerAt BioPhase Solutions, we are seeking a highly skilled Senior Data Engineer to join our team. As a Senior Data Engineer, you will play a pivotal role in designing, implementing, and maintaining data pipelines that process and analyze large volumes of instrument data, clinical biomarker inputs, and patient information.Key...

  • Senior Data Engineer

    3 weeks ago


    San Diego, California, United States CoStar Group Full time

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at CoStar Group. As a Senior Data Engineer, you will be responsible for designing, building, testing, and deploying scalable, reusable, and maintainable applications that handle substantial amounts of data.Key Responsibilities:Design and develop cloud-based...


  • San Diego, California, United States Data Intelligence Full time

    Job Title: Senior RF Systems EngineerData Intelligence, LLC (DI) is a leading provider of full life cycle system development, systems engineering, cybersecurity, and analytical support to C4ISR and other complex systems. We are seeking a highly skilled Senior RF Systems Engineer to support our Department of Defense operations in San Diego, CA.Job SummaryWe...


  • San Diego, California, United States BRAIN CORP Full time

    Job Title: Senior Data EngineerBRAIN CORP is seeking a highly skilled Senior Data Engineer to lead the development of data-centric products that enhance our innovative BrainOS platform.This individual will be responsible for designing, constructing, and maintaining the systems and infrastructure to facilitate efficient collection, storage, and analysis of...


  • San Diego, California, United States Kyocera Full time

    Job Title: Senior Azure Data EngineerAt Kyocera, we are seeking a highly skilled Senior Azure Data Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, building, and maintaining large-scale data solutions on the Azure platform.Key Responsibilities:Design and implement data architectures for...


  • San Diego, California, United States Kyocera Full time

    Job Title: Senior Azure Data EngineerAt Kyocera, we are seeking a highly skilled Senior Azure Data Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, building, and maintaining large-scale data systems on the Azure platform.Key Responsibilities:Design and implement data architectures for enterprise...


  • San Diego, California, United States Kyocera Full time

    Job Title: Senior Azure Data EngineerWe are seeking a highly skilled Senior Azure Data Engineer to join our team at Kyocera International, Inc. As a key member of our data engineering team, you will be responsible for designing, building, and maintaining large-scale data systems on the Azure platform.Key Responsibilities:Design and implement data...

  • Senior Data Engineer

    4 weeks ago


    San Diego, California, United States Tek Hire Solutions Full time

    Job Title: Sr. QE Data EngineerWe are seeking a highly skilled Senior Quality Engineer (QE) with advanced expertise in data manipulation, AWS governance, API testing, and test automation. This role will focus on developing and executing comprehensive test plans, ensuring data integrity, and validating cloud-based applications.Key Responsibilities:Data...


  • San Diego, California, United States Ledgent Technology Full time

    Senior Data Visualization EngineerWe are seeking a highly skilled Senior Data Visualization Engineer to join our team at Ledgent Technology. As a key member of our organization, you will be responsible for designing and developing engaging, user-friendly front-end applications for data visualization using various libraries such as D3.js, Recharts, Victory,...


  • San Diego, California, United States BioPhase Full time

    Position OverviewThe Senior Data Engineer will play a pivotal role in designing, implementing, and maintaining data pipelines that process and analyze large volumes of instrument data, clinical biomarker inputs, and patient information. This role requires a strong technical foundation, experience in startup environments, and the ability to navigate the...

  • Senior Data Engineer

    4 weeks ago


    San Diego, California, United States CoStar Group Full time

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at CoStar Group. As a Senior Data Engineer, you will be responsible for designing, building, testing, and deploying scalable, reusable, and maintainable applications that handle substantial amounts of data.Responsibilities:Designing and building cloud-based...

  • Senior Data Engineer

    4 weeks ago


    San Diego, California, United States CoStar Group Full time

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at CoStar Group. As a Senior Data Engineer, you will be responsible for designing, building, testing, and deploying scalable, reusable, and maintainable applications that handle substantial amounts of data.Responsibilities:Designing and building cloud-based...


  • San Diego, California, United States Kyocera Full time

    Job DescriptionAt Kyocera, we are seeking a highly skilled Senior Azure Data Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, building, and maintaining large-scale data solutions on the Azure platform.The ideal candidate will have a strong background in data engineering, with experience in...


  • San Diego, California, United States Apple Full time

    Job Title: Senior Software EngineerAt Apple, we're committed to creating innovative products and services that make a difference in people's lives. As a Senior Software Engineer on our Data Cloud team, you'll play a key role in designing and developing scalable data systems that enable our teams to make data-driven decisions.About the RoleWe're looking for a...


  • San Diego, California, United States Apple Full time

    About the RoleWe are seeking a highly skilled Senior Software Engineer to join our Data Cloud team at Apple. As a key member of our team, you will be responsible for designing and implementing scalable data systems and cloud services that enable our development teams to make data-driven decisions.Key ResponsibilitiesDesign and implement data modeling and...


  • San Diego, California, United States Intuit Full time

    Job SummaryWe are seeking a highly skilled Senior Machine Learning Engineer to join our team at Intuit. As a key member of our data science team, you will be responsible for designing, developing, and deploying machine learning models at scale using the latest industry tools and technologies.Key Responsibilities:Design and develop machine learning models...


  • San Francisco, California, United States VAST Data Full time

    About the RoleVAST Data is seeking a highly skilled Senior Systems Engineer to contribute to our innovative team.This is an exceptional opportunity to be part of a pioneering company that is revolutionizing the data storage industry with its Universal Storage platform.As a key member of our team, you will play a crucial role in shaping the future of data...


  • San Diego, California, United States BioPhase Full time

    Senior Analytics Engineer, Data SolutionsAbout the Role:We are seeking a highly skilled Senior Analytics Engineer to join our Data Solutions team at BioPhase. As a key member of our analytics engineering team, you will be responsible for designing and implementing data platforms, patterns, and solutions that drive business insights and inform strategic...

  • Senior Data Engineer

    2 weeks ago


    San Francisco, California, United States Apex Systems, Inc. Full time

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at Apex Systems, Inc. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data systems and pipelines.Key Responsibilities:Lead the development of data pipelines for machine learning use casesDesign and...


  • San Diego, California, United States Apple Full time

    Job DescriptionWe are seeking a highly skilled Data Engineer to join our team at Apple. As a Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data solutions that support our software engineering group.Key Responsibilities:Develop and maintain Tableau dashboards and other analytical tools to provide visibility into...