Senior Data Pipeline Architect

3 days ago

San Francisco, California, United States Unreal Gigs Full time
Job Title: Senior Data Pipeline Developer

At Unreal Gigs, we're committed to building cutting-edge data pipeline solutions that enable efficient data ingestion, processing, and delivery. We're seeking a seasoned Senior Data Pipeline Developer to join our dynamic team and play a critical role in designing, building, and optimizing our data pipeline solutions.

Key Responsibilities:
  • Design and implement data pipeline solutions to ingest, process, and deliver data from various sources to target systems, ensuring data quality, integrity, and timeliness.
  • Develop and maintain data ingestion processes to collect data from diverse sources, including databases, APIs, files, and streaming sources, ensuring seamless data flow and interoperability.
  • Transform and cleanse data as it moves through the pipeline, applying business rules, data enrichment, and validation to meet business requirements and enable downstream analytics and reporting.
  • Implement workflow orchestration and automation solutions to schedule and manage data pipeline workflows, reducing manual intervention and improving operational efficiency.
  • Optimize data pipeline performance through parallel processing, partitioning, and other techniques, ensuring scalability and responsiveness for large-scale data processing needs.
  • Implement monitoring and alerting systems to track data pipeline performance and health, proactively identifying and resolving issues to minimize downtime and data loss.
  • Implement error handling and retry mechanisms to handle data processing failures and ensure data reliability and consistency.
  • Document data pipeline designs, processes, and best practices, providing clear and comprehensive documentation to facilitate understanding and collaboration among team members.
  • Collaborate with cross-functional teams, including data engineers, data scientists, and business analysts, to understand requirements and deliver data pipeline solutions that meet business needs.
  • Mentor junior developers, sharing expertise and best practices in data pipeline development, and facilitate knowledge sharing sessions within the team.
Requirements:
  • Bachelor's degree or higher in Computer Science, Engineering, Mathematics, or related field.
  • 5+ years of experience in data engineering, with a focus on designing, building, and optimizing data pipeline solutions.
  • Proficiency in data pipeline technologies such as Apache Kafka, Apache NiFi, Apache Airflow, or similar.
  • Strong programming skills in languages such as Python, Java, or Scala, with experience in data processing frameworks like Apache Spark or Apache Beam.
  • Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform, and services like AWS Glue, Azure Data Factory, or Google Dataflow.
  • Strong understanding of data integration concepts and techniques, with experience integrating data from diverse sources and systems.
  • Strong problem-solving skills and analytical thinking, with the ability to troubleshoot complex data pipeline issues and optimize system performance.
  • Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams and communicate technical concepts to non-technical stakeholders.
Benefits:
  • Competitive salary: The industry standard salary for Senior Data Pipeline Developers typically ranges from $170,000 to $230,000 per year, depending on experience and qualifications.
  • Comprehensive health, dental, and vision insurance plans.
  • Flexible work hours and remote work options.
  • Generous vacation and paid time off.
  • Professional development opportunities, including access to training programs, conferences, and workshops.
  • State-of-the-art technology environment with access to cutting-edge tools and resources.
  • Vibrant and inclusive company culture with opportunities for growth and advancement.
  • Exciting projects with real-world impact at the forefront of data-driven innovation.

  • Data Pipeline Architect

    15 hours ago

    San Francisco, California, United States Unreal Gigs Full time

    Job Title: Data Pipeline ArchitectAt Unreal Gigs, we're seeking a highly skilled Data Pipeline Architect to design and develop scalable and efficient data pipeline solutions. As a key member of our data engineering team, you'll work closely with cross-functional teams to ensure the reliability, scalability, and efficiency of our data pipeline infrastructure...

  • Senior Data Architect

    3 days ago

    San Francisco, California, United States Futran Tech Solutions Pvt. Ltd. Full time

    Job Title: Senior Data ArchitectWe are seeking a highly skilled Senior Data Architect to join our team at Futran Tech Solutions Pvt. Ltd. The ideal candidate will have a strong background in data engineering and a proven track record of designing and implementing large-scale data architectures.Key Responsibilities:* Design and implement data architectures...

  • Senior Data Pipeline Architect

    3 days ago

    San Francisco, California, United States Unreal Gigs Full time

    Company Overview:Welcome to Unreal Gigs, a pioneering company at the forefront of data-driven innovation. Our mission is to harness the power of data to drive transformative change and solve complex problems across industries.We're committed to building cutting-edge data pipeline solutions that enable efficient data ingestion, processing, and delivery. Our...

  • Lead Data Pipeline Architect

    18 hours ago

    San Francisco, California, United States Unreal Gigs Full time

    Job Title: Lead Data Pipeline DeveloperAt Unreal Gigs, we're seeking a highly skilled Lead Data Pipeline Developer to join our team. As a key member of our data engineering team, you'll be responsible for designing, building, and optimizing our data pipeline solutions.Key Responsibilities:Provide technical leadership and guidance to a team of data pipeline...

  • Senior Data Architect

    4 days ago

    San Francisco, California, United States Unreal Gigs Full time

    Senior Big Data EngineerAt Unreal Gigs, we're committed to harnessing the power of big data to drive transformative change and solve complex problems across industries. As a Senior Big Data Engineer, you'll play a pivotal role in designing, building, and maintaining our big data infrastructure and pipelines.Key Responsibilities:Big Data Infrastructure...

  • Senior Data Architect

    3 days ago

    San Francisco, California, United States Storm3 Full time

    Data Architect/Engineering Lead - AI/MLStorm3 is seeking an experienced Data Architect/Engineering Lead to join their fast-growing AI/ML team. This role will design and develop data model/lakehouse and work alongside ML architects/scientists to scale data curation and pipelines for ML on a modern tech stack.Responsibilities:Design, implement, and support...

  • Senior Big Data Architect

    3 days ago

    San Francisco, California, United States Unreal Gigs Full time

    Unlock the Power of Big DataWelcome to Unreal Gigs, where we're harnessing the potential of big data to drive innovation and solve complex problems. As a Senior Big Data Engineer, you'll play a pivotal role in designing, building, and maintaining our big data infrastructure and pipelines.Key Responsibilities:Big Data Infrastructure Design: Design, architect,...

  • Senior Data Architect

    1 month ago

    San Francisco, California, United States Storm3 Full time

    {"title": "Data Architect/Engineer Lead - ML", "company": "Storm3", "location": "Remote within US", "job_type": "Full-time", "industry": "HealthTech", "job_description": "Principal QM Research Scientist / EngineerAI Drug Discovery Unicorn (Series B)Competitive base + hefty equityThis start-up is a leader in deep learning & generative AI, merging cutting-edge...

  • Senior Data Architect

    5 days ago

    San Francisco, California, United States Storm3 Full time

    Data Architect/Engineering Lead - MLAI Drug Discovery PioneerRemote OpportunityCompetitive Compensation PackageStorm3 is a HealthTech recruitment firm partnering with a cutting-edge start-up in the field of deep learning and generative AI. This innovative company is merging advanced models with biology research to accelerate disease treatment and drug...

  • Senior Data Systems Architect

    4 weeks ago

    San Francisco, California, United States Unreal Gigs Full time

    Unreal GigsWelcome to the forefront of data-driven innovation. Our company is dedicated to harnessing the power of data to drive transformative change and solve complex problems across industries.Job OverviewAs a Senior Data Systems Engineer, you'll play a critical role in designing, building, and optimizing our data systems infrastructure. You'll work on...

  • Senior Data Systems Architect

    3 weeks ago

    San Francisco, California, United States Unreal Gigs Full time

    Job Title: Senior Data Systems EngineerCompany Overview:Welcome to Unreal Gigs, a leading company at the forefront of data-driven innovation. Our mission is to harness the power of data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge data systems that enable efficient data management,...

  • Senior Data Architect

    3 days ago

    San Francisco, California, United States Saxon Global Full time

    Job DescriptionThe Data Engineer will be responsible for designing and implementing data pipelines, optimizing data flow and collection, and ensuring data delivery architecture is consistent throughout ongoing projects.The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the...

  • Data Engineer

    3 days ago

    San Francisco, California, United States Figma Full time

    Join Figma's Data Engineering TeamFigma is a leading design and development platform that empowers teams to create innovative products. We're seeking an experienced Data Engineer to join our team and help us scale our data pipelines.As a Data Engineer at Figma, you'll be responsible for designing, building, and maintaining scalable data pipelines that...

  • Senior Data Architect

    2 weeks ago

    San Francisco, California, United States Unreal Gigs Full time

    Job Title: Senior Data EngineerCompany Overview:At Unreal Gigs, we're pushing the boundaries of data-driven innovation. Our mission is to harness the power of data to drive transformative change and solve complex problems across industries. We're committed to building scalable and reliable data infrastructure that enables advanced analytics, machine...

  • Senior Data Architect

    3 weeks ago

    San Francisco, California, United States Unreal Gigs Full time

    Job Title: Senior Data EngineerCompany Overview:At Unreal Gigs, we're at the forefront of data-driven innovation, harnessing the power of data to drive transformative change and solve complex problems across industries. Our commitment to building scalable and reliable data infrastructure enables advanced analytics, machine learning, and business...

  • Senior Data Architect

    4 days ago

    San Jose, California, United States DESIGNLIBRO INC Full time

    Job DescriptionWe are seeking a highly skilled Senior Data Engineer/Data Engineering Architect to join our team at DESIGNLIBRO INC.Key ResponsibilitiesDesign and implement robust data systems that collect, process, and store large volumes of data from IoT devices.Collaborate with product owners to translate data requirements into actionable data engineering...

  • Senior Data Systems Architect

    4 weeks ago

    San Francisco, California, United States Unreal Gigs Full time

    Company OverviewWelcome to Unreal Gigs, a leading innovator in data-driven solutions. Our mission is to harness the power of data to drive transformative change and solve complex problems across industries.We're committed to building cutting-edge data systems that enable efficient data management, processing, and analysis. Our team of experts is dedicated to...

  • Senior Data Architect

    4 days ago

    San Francisco, California, United States Veterans Sourcing Group Full time

    {"title": "Data Engineer", "description": "Job OverviewWe are seeking a highly skilled Data Engineer to join our team at Veterans Sourcing Group. As a Data Engineer, you will be responsible for designing, building, and maintaining large-scale data systems that support our business operations.Key ResponsibilitiesDesign and implement data pipelines to extract,...

  • Senior Data Systems Architect

    2 weeks ago

    San Francisco, California, United States Unreal Gigs Full time

    Senior Data Systems EngineerUnreal Gigs is seeking a highly skilled Senior Data Systems Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, building, and optimizing our data systems infrastructure.Key Responsibilities:Design and implement scalable and reliable data systems architecture to support...

  • Big Data Architect

    3 days ago

    San Francisco, California, United States Unreal Gigs Full time

    Job Title: Big Data ArchitectAt Unreal Gigs, we're seeking a highly skilled Big Data Architect to join our team. As a key member of our data engineering team, you'll be responsible for designing and building scalable and reliable big data infrastructure and platforms.Key Responsibilities:Design and architect big data infrastructure and platforms, including...