Real-Time Data Engineer

3 weeks ago

San Francisco, California, United States Unreal Gigs Full time
Job Overview

At Unreal Gigs, we're pushing the boundaries of real-time data engineering. As a key member of our team, you'll play a pivotal role in designing, building, and optimizing our real-time data infrastructure and systems.

As a Real-Time Data Engineer, you'll work closely with cross-functional teams of data scientists, software engineers, and business analysts to ensure the reliability, scalability, and efficiency of our real-time data solutions.

Key Responsibilities:

  1. Real-Time Data Pipeline Development: Design and develop real-time data pipelines to ingest, process, and analyze streaming data from various sources, ensuring low-latency and high-throughput.
  2. Stream Processing: Implement stream processing applications using technologies such as Apache Kafka, Apache Flink, Apache Spark Streaming, or AWS Kinesis to process and analyze real-time data streams.
  3. Data Integration: Integrate real-time data streams with existing data systems and applications, ensuring data consistency and coherence across systems.
  4. Data Transformation and Enrichment: Transform and enrich real-time data streams to derive meaningful insights and enable real-time decision-making, using techniques such as complex event processing (CEP) and real-time analytics.
  5. Data Quality and Governance: Implement data quality checks, validation rules, and error handling mechanisms to ensure the accuracy, completeness, and consistency of real-time data.
  6. Performance Optimization: Optimize real-time data processing pipelines for speed, scalability, and efficiency, tuning streaming applications and leveraging distributed computing technologies.
  7. Monitoring and Maintenance: Monitor real-time data infrastructure and systems for performance, reliability, and availability, proactively identifying and addressing issues to minimize downtime and ensure data integrity.
  8. Documentation and Collaboration: Document real-time data architecture, processes, and best practices, and collaborate with cross-functional teams to ensure alignment and transparency.

Requirements:

  • Bachelor's degree or higher in Computer Science, Engineering, or related field.
  • Strong background in real-time data engineering, with hands-on experience in designing, building, and optimizing real-time data solutions.
  • Proficiency in programming languages such as Python, Java, or Scala, and experience with stream processing frameworks such as Apache Kafka, Apache Flink, or Apache Spark Streaming.
  • Experience with cloud platforms such as AWS, Google Cloud Platform, or Microsoft Azure, and familiarity with cloud services for real-time data processing (e.g., AWS Kinesis, Google Cloud Dataflow, Azure Stream Analytics).
  • Knowledge of distributed computing technologies and concepts, including message brokers, distributed databases, and microservices architecture.
  • Strong problem-solving abilities and analytical thinking, with a keen attention to detail and a passion for tackling complex technical challenges.
  • Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams and communicate technical concepts to non-technical stakeholders.

Benefits:

  • Competitive salary: The industry standard salary for Real-Time Data Engineers typically ranges from $140,000 to $210,000 per year, depending on experience and qualifications.
  • Comprehensive health, dental, and vision insurance plans.
  • Flexible work hours and remote work options.
  • Generous vacation and paid time off.
  • Professional development opportunities, including access to training programs, conferences, and workshops.
  • State-of-the-art technology environment with access to cutting-edge tools and resources.
  • Vibrant and inclusive company culture with team-building activities and social events.
  • Opportunities for career growth and advancement within the company.
  • Exciting projects with real-world impact in the field of real-time data engineering.
  • Chance to work alongside top talent and industry experts in real-time data engineering.

  • Real-Time Data Engineer

    4 weeks ago

    San Francisco, California, United States Unreal Gigs Full time

    Unreal Gigs is seeking a highly skilled Data Engineer for Real-Time Solutions to join our team. As a key member of our data engineering team, you will play a pivotal role in designing, building, and optimizing our real-time data infrastructure and systems.**Key Responsibilities:*** Design and develop real-time data pipelines to ingest, process, and analyze...

  • Lead Real-Time Data Architect

    4 weeks ago

    San Francisco, California, United States Unreal Gigs Full time

    Job OverviewWelcome to Unreal Gigs, where we're pushing the boundaries of data-driven innovation. As the Lead Real-Time Data Architect, you'll be at the forefront of designing, building, and optimizing real-time data infrastructure. Your expertise in real-time data technologies will enable timely insights and actions, driving transformative change across...

  • Data Engineer

    3 weeks ago

    San Francisco, California, United States Fintool Full time

    About FintoolFintool is an AI Equity Research Copilot for institutional investors. We're building the AI Warren Buffett. Our team is on the fastest growing LLM vertical applications, with thousands of investors signed up for Fintool. We're backed by Y Combinator and entrepreneurs such as the co-founders of Datadog, Vercel, HuggingFace, or domain experts from...

  • Senior Data Engineer

    3 weeks ago

    San Jose, California, United States Tik Tok Full time

    Job Title: Senior Data Engineer - Real-Time Data WarehouseAbout the Role:We are seeking a highly skilled Senior Data Engineer to join our commercial real-time data warehouse team. As a Senior Data Engineer, you will be responsible for designing, developing, and optimizing our real-time data warehouse solution.Key Responsibilities:Design and develop a...

  • Senior Software Engineer, Real-Time Data Processing

    4 weeks ago

    San Francisco, California, United States Cisco Full time

    About the RoleCisco is seeking a highly skilled Lead Software Engineer to join our Events team. As a key member of this team, you will be responsible for building a highly scalable stream processing platform to detect anomalies and provide actionable insights to our customers.We are looking for a talented engineer with expertise in designing and building...

  • Data Engineer, Real Estate and Workplace

    3 weeks ago

    San Francisco, California, United States OpenAI Full time

    About the TeamAt OpenAI, our Real Estate and Workplace team is responsible for planning our Real Estate needs and elevating the employee experience through different delineations of the industry (Facilities, Operations, Experience) to support our growing company in its mission to ensure that AGI benefits all of humanity.We're looking for individuals who are...

  • Data Engineer

    4 weeks ago

    San Francisco, California, United States OpenAI Full time

    About the TeamOpenAI's Real Estate and Workplace team is responsible for planning and elevating the employee experience through data-driven insights.The team's mission is to ensure that artificial intelligence benefits all of humanity, and they strive to achieve this by creating a workplace that is both innovative and inclusive.About the RoleAs a Data...

  • Senior Data Engineer

    3 weeks ago

    San Jose, California, United States Tik Tok Full time

    Job SummaryAt TikTok, we're looking for a skilled Senior Data Engineer - Real-Time Data Warehouse to join our team. As a key member of our commercial data platform, you will be responsible for the development and optimization of our real-time data warehouse. This is a unique opportunity to work on a large-scale streaming data computing project, leveraging...

  • Senior Data Engineer

    4 weeks ago

    San Jose, California, United States Tik Tok Full time

    Job DescriptionTikTok is a leading destination for short-form mobile video, and our mission is to inspire creativity and bring joy. We're looking for a talented Data Streaming Engineer to join our team and help us achieve this goal.As a Data Streaming Engineer, you will be responsible for developing and optimizing TikTok's commercial real-time data...

  • Lead Software Engineer, Real-Time Data Platform

    3 weeks ago

    San Francisco, California, United States Cisco Full time

    About the RoleThe Events team at Cisco is responsible for building a highly scalable stream processing platform to detect anomalies and provide actionable insights to our customers.We work on problems such as identifying timely automated events, LLM/GenAI-based summarization and remediation, and building and deploying multiple stream processing platforms.We...

  • Data Engineer, Real Estate and Workplace Innovator

    4 weeks ago

    San Francisco, California, United States Openai Full time

    About the TeamOpenAI has a global presence with a custom-built headquarters in San Francisco and multiple offices worldwide where our teams collaborate.The Real Estate and Workplace team is responsible for planning our Real Estate needs and elevating the employee experience through different delineations of the industry (Facilities, Operations, Experience)...

  • Software Engineer, Real-time Data Ingestion

    3 weeks ago

    San Francisco, California, United States Orb Full time

    About OrbOrb is a modern billing infrastructure that powers revenue workflows for the next wave of AI and software companies. We believe that pricing and billing shouldn't be a blocker to building and shipping great products.We power billing for highly innovative companies like Vercel, Pinecone, Perplexity, Replit, and work with our customers on some of...

  • Senior Data Engineer

    4 weeks ago

    San Francisco, California, United States Gridware Full time

    Job Title: Senior Data EngineerGridware is a leading provider of advanced sensing systems for the electrical grid. Our team is responsible for designing, building, and optimizing data pipelines and infrastructure to support real-time grid monitoring systems.The Senior Data Engineer will be responsible for handling large-scale sensor data, ensuring data...

  • Senior Data Engineer

    4 weeks ago

    San Francisco, California, United States Gridware Full time

    Job OverviewGridware is a pioneering company that aims to enhance and protect the electrical grid, a critical infrastructure that underpins modern society. As a Senior Data Engineer, you will play a crucial role in designing, building, and optimizing data pipelines and infrastructure to support real-time grid monitoring systems.The ideal candidate will have...

  • Senior Data Engineer

    4 weeks ago

    San Francisco, California, United States Unreal Gigs Full time

    Unreal Gigs is seeking a highly skilled Senior Data Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, building, and maintaining our data infrastructure and pipelines.Key Responsibilities:* Design and implement scalable and efficient data pipelines for ingesting, processing, and transforming large...

  • Staff Software Engineer, Real Time Infrastructure

    4 weeks ago

    San Francisco, California, United States Discord Full time

    At Discord, we're building a platform that enables people to create spaces for belonging in their lives. We're looking for a skilled Staff Software Engineer to join our Real Time Infrastructure team, responsible for building, maintaining, and scaling the systems that power chat, push notifications, presence, and more for over 150 million active users.This...

  • Data Engineer

    3 weeks ago

    San Francisco, California, United States Bayone Full time

    Job Title: Data EngineerCompany Overview:At Bayone, we strive to create an environment where people are valued, and differences are celebrated. Our purpose is to empower our teams and inspire our customers to become the best versions of themselves.The Opportunity:Our technology team works fast and smart, developing the latest in mobile technologies, scalable...

  • Data Visualization Engineer

    4 weeks ago

    San Francisco, California, United States kbit LLC Full time

    Job DescriptionWe are seeking a skilled Data Visualization Engineer to join our team at kbit LLC. As a systematic trading firm, our success depends on the expertise of our engineers and researchers who collaborate to develop and implement profitable trading strategies.The ideal candidate is adaptable, takes initiative with projects, and consistently delivers...

  • Senior Data Engineer

    3 weeks ago

    San Francisco, California, United States EliseAI Full time

    About EliseAIEliseAI is a cutting-edge conversational AI technology company that focuses on developing innovative solutions for industries fundamental to our lives: housing and healthcare. Our mission is to leverage data as a strategic asset to drive decision-making processes. We're seeking a skilled Senior Data Engineer to join our Engineering team and play...

  • Data Visualization Engineer

    3 weeks ago

    San Francisco, California, United States kbit LLC Full time

    Job Title: Data Visualization EngineerWe are a successful crypto hedge fund with a strong track record of delivering returns for our investors. Our success depends on the expertise of our engineers and researchers who collaborate to develop and implement profitable trading strategies.We're seeking a skilled Data Visualization Engineer to join our team and...