Data Pipeline Architect

13 hours ago


San Francisco, California, United States Unreal Gigs Full time
Job Title: Data Pipeline Architect

At Unreal Gigs, we're seeking a highly skilled Data Pipeline Architect to design and develop scalable and efficient data pipeline solutions. As a key member of our data engineering team, you'll work closely with cross-functional teams to ensure the reliability, scalability, and efficiency of our data pipeline infrastructure and workflows.

Key Responsibilities:

  • Design and architect scalable and reliable data pipeline solutions to automate data ingestion, processing, and transformation tasks.
  • Integrate data from diverse sources, including databases, files, APIs, and streaming platforms, into our data pipeline ecosystem, ensuring data consistency and coherence.
  • Develop and maintain data processing workflows to cleanse, enrich, and transform raw data into formats suitable for analysis and reporting.
  • Orchestrate and schedule data pipeline workflows using workflow management tools and frameworks (e.g., Apache Airflow, Luigi, Prefect).
  • Optimize data pipeline performance for speed, efficiency, and scalability, tuning data processing tasks and optimizing resource utilization.
  • Implement data quality checks, validation rules, and error handling mechanisms to ensure the accuracy, completeness, and consistency of data in the pipeline.
  • Monitor data pipeline performance and health, proactively identifying and addressing issues to minimize downtime and optimize resource utilization.
  • Document data pipeline architecture, processes, and best practices, and collaborate with cross-functional teams to ensure alignment and transparency.

Qualifications:

  • Bachelor's degree or higher in Computer Science, Engineering, or related field.
  • Strong background in data engineering, with hands-on experience in designing, building, and optimizing data pipeline solutions.
  • Proficiency in programming languages such as Python, Java, or Scala, and experience with data processing frameworks such as Apache Spark, Apache Beam, or Apache Flink.
  • Experience with workflow management tools and frameworks such as Apache Airflow, Luigi, or Prefect.
  • Familiarity with data integration techniques and technologies, including ETL (Extract, Transform, Load) processes and streaming data processing.
  • Strong problem-solving abilities and analytical thinking, with a keen attention to detail and a passion for tackling complex technical challenges.
  • Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams and communicate technical concepts to non-technical stakeholders.

Benefits:

  • Competitive salary: The industry standard salary for Data Pipeline Architects typically ranges from $150,000 to $220,000 per year, depending on experience and qualifications.
  • Comprehensive health, dental, and vision insurance plans.
  • Flexible work hours and remote work options.
  • Generous vacation and paid time off.
  • Professional development opportunities, including access to training programs, conferences, and workshops.
  • State-of-the-art technology environment with access to cutting-edge tools and resources.
  • Vibrant and inclusive company culture with team-building activities and social events.
  • Opportunities for career growth and advancement within the company.
  • Exciting projects with real-world impact across diverse industries.
  • Chance to work alongside top talent and industry experts in the field of data engineering.


  • San Francisco, California, United States Unreal Gigs Full time

    Job Title: Lead Data Pipeline DeveloperAt Unreal Gigs, we're seeking a highly skilled Lead Data Pipeline Developer to join our team. As a key member of our data engineering team, you'll be responsible for designing, building, and optimizing our data pipeline solutions.Key Responsibilities:Provide technical leadership and guidance to a team of data pipeline...


  • San Francisco, California, United States Unreal Gigs Full time

    Job Title: Senior Data Pipeline DeveloperAt Unreal Gigs, we're committed to building cutting-edge data pipeline solutions that enable efficient data ingestion, processing, and delivery. We're seeking a seasoned Senior Data Pipeline Developer to join our dynamic team and play a critical role in designing, building, and optimizing our data pipeline...


  • San Francisco, California, United States Unreal Gigs Full time

    Company Overview:Welcome to Unreal Gigs, a pioneering company at the forefront of data-driven innovation. Our mission is to harness the power of data to drive transformative change and solve complex problems across industries.We're committed to building cutting-edge data pipeline solutions that enable efficient data ingestion, processing, and delivery. Our...

  • Data Engineer

    3 days ago


    San Francisco, California, United States Figma Full time

    Join Figma's Data Engineering TeamFigma is a leading design and development platform that empowers teams to create innovative products. We're seeking an experienced Data Engineer to join our team and help us scale our data pipelines.As a Data Engineer at Figma, you'll be responsible for designing, building, and maintaining scalable data pipelines that...


  • San Francisco, California, United States Futran Tech Solutions Pvt. Ltd. Full time

    Job Title: Senior Data ArchitectWe are seeking a highly skilled Senior Data Architect to join our team at Futran Tech Solutions Pvt. Ltd. The ideal candidate will have a strong background in data engineering and a proven track record of designing and implementing large-scale data architectures.Key Responsibilities:* Design and implement data architectures...

  • Big Data Architect

    3 days ago


    San Francisco, California, United States Unreal Gigs Full time

    Job Title: Big Data ArchitectAt Unreal Gigs, we're seeking a highly skilled Big Data Architect to join our team. As a key member of our data engineering team, you'll be responsible for designing and building scalable and reliable big data infrastructure and platforms.Key Responsibilities:Design and architect big data infrastructure and platforms, including...


  • San Francisco, California, United States Unreal Gigs Full time

    Job Title: Data Infrastructure ArchitectWe're seeking a skilled Data Infrastructure Architect to design, develop, and optimize our data systems. As a key member of our team, you'll be responsible for constructing efficient, scalable data pipelines, ensuring data is accessible and usable for analysts, data scientists, and business stakeholders.Key...

  • Data Architect

    16 hours ago


    San Francisco, California, United States Diverse Lynx Full time

    Job Title: Data ArchitectJob Summary:We are seeking a highly skilled Data Architect to join our team at Diverse Lynx LLC. As a Data Architect, you will be responsible for designing, developing, and maintaining enterprise-level applications using the Snowflake platform.Key Responsibilities: Design and implement features for identity and access management...


  • San Francisco, California, United States Storm3 Full time

    Data Architect/Engineering Lead - AI/MLStorm3 is seeking an experienced Data Architect/Engineering Lead to join their fast-growing AI/ML team. This role will design and develop data model/lakehouse and work alongside ML architects/scientists to scale data curation and pipelines for ML on a modern tech stack.Responsibilities:Design, implement, and support...

  • Data Architect DBT

    3 weeks ago


    San Francisco, California, United States CriticalRiver Inc Full time

    Job Title: Data Architect DBTWe are seeking a highly skilled Data Architect DBT to join our team at CriticalRiver Inc. As a key player in shaping our data architecture strategies, you will drive innovative solutions and lead the charge in transforming data into actionable insights.Key Responsibilities:Implement and deploy net new pipelines in dbt for...


  • San Francisco, California, United States SmithRx Full time

    Transforming Pharmacy Benefit Management with Data Engineering ExpertiseAt SmithRx, we're revolutionizing the pharmacy benefit management industry with cutting-edge technology and innovative solutions. As a Principal Data Engineer, you'll play a pivotal role in driving this transformation by designing and developing robust data architectures that support...

  • Data Architect

    1 month ago


    San Francisco, California, United States Net2Source Inc. Full time

    Job SummaryWe are seeking a highly skilled Data Architect to join our team at Net2Source Inc. as a Cloud Data Engineer. As a key member of our data engineering team, you will be responsible for designing, building, and maintaining large-scale data systems on the cloud.Key ResponsibilitiesDesign and implement data architectures that meet the needs of our...

  • Senior Data Architect

    1 month ago


    San Francisco, California, United States Storm3 Full time

    {"title": "Data Architect/Engineer Lead - ML", "company": "Storm3", "location": "Remote within US", "job_type": "Full-time", "industry": "HealthTech", "job_description": "Principal QM Research Scientist / EngineerAI Drug Discovery Unicorn (Series B)Competitive base + hefty equityThis start-up is a leader in deep learning & generative AI, merging cutting-edge...


  • San Francisco, California, United States Storm3 Full time

    Data Architect/Engineering Lead - MLAI Drug Discovery PioneerRemote OpportunityCompetitive Compensation PackageStorm3 is a HealthTech recruitment firm partnering with a cutting-edge start-up in the field of deep learning and generative AI. This innovative company is merging advanced models with biology research to accelerate disease treatment and drug...


  • San Francisco, California, United States Unreal Gigs Full time

    Job Title: Data Systems ArchitectAt Unreal Gigs, we're seeking a highly skilled Data Systems Architect to join our team. As a key member of our data engineering team, you'll be responsible for designing and architecting scalable and resilient data systems infrastructure to meet business requirements and support data processing and analytics needs.Key...


  • San Francisco, California, United States Unreal Gigs Full time

    Senior Big Data EngineerAt Unreal Gigs, we're committed to harnessing the power of big data to drive transformative change and solve complex problems across industries. As a Senior Big Data Engineer, you'll play a pivotal role in designing, building, and maintaining our big data infrastructure and pipelines.Key Responsibilities:Big Data Infrastructure...


  • San Francisco, California, United States Saxon Global Full time

    Job DescriptionThe Data Engineer will be responsible for designing and implementing data pipelines, optimizing data flow and collection, and ensuring data delivery architecture is consistent throughout ongoing projects.The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the...

  • Data Architect

    3 weeks ago


    San Francisco, California, United States Parafin Full time

    Job Title: Analytics EngineerWe are seeking an experienced Analytics Engineer to join our team at Parafin. As an Analytics Engineer, you will be responsible for architecting business-critical datasets used across the company. You will drive alignment between teams by ensuring all of Parafin can be pointed to the same sources of truth.Key...

  • AI Pipeline Architect

    2 weeks ago


    San Francisco, California, United States Linea Full time

    Unlock the Power of AI with LineaThe Chief of Staff at Linea plays a critical role in driving the company's success by overseeing various business functions, including operations, recruiting, and marketing. This position requires a strong leader who can work closely with the executive team to identify opportunities for growth and implement strategies to...


  • San Francisco, California, United States Veterans Sourcing Group Full time

    {"title": "Data Engineer", "description": "Job OverviewWe are seeking a highly skilled Data Engineer to join our team at Veterans Sourcing Group. As a Data Engineer, you will be responsible for designing, building, and maintaining large-scale data systems that support our business operations.Key ResponsibilitiesDesign and implement data pipelines to extract,...