Platform Data Engineer

4 weeks ago


San Jose, United States Bayone Full time

Location: San Francisco, Seattle, LA or PST
Enterprise Data Platform to enable timely, effective and safe sharing of data to multiple engineering, operations and business teams for building world class data products

Responsibilities
Build data ingestion and processing pipelines to enable data analytics and data science use-cases in areas of digital commerce, service operations, charging, reliability, finance, capex, warranty, customer service and others.
Build modular set of data services using Python, SQL, AWS Glue, lambdas, API Gateway, Kafka, data build tool (dbt), Apache Spark on EMR among others
Build automated unit and integration testing pipelines using frameworks like PySpark
Create and manage CICD pipelines with Gitlab CI and AWS Code Pipeline/CodeDeploy
Automate and schedule jobs using Managed Airflow
Build the ODS and reporting schemas and load the data into AWS Redshift or Snowflake
Design and build data quality management services with Apache Deequ and data observability tools like Splunk, DataDog , CloudWatch
Provide a variety of query services with REST, Athena/Presto, server sent events
Configure and setup the enterprise data lineage and meta data management and data catalog support using tools like Collibra/Alation
Assist the data scientist within the data engineering team as well as other software engineering teams with data cleansing, wrangling and feature engineering
Ensure green builds for deployment and work with program management and senior leads to burn down planned deliverables in a sprint cycle

Qualifications
At least 5+ years building data and analytics platforms using AWS Cloud, Python and SQL
Knowledge of AWS technologies specifically MSK, EMR, Athena, Glue, lambdas, API Gateway as well as Python, SQL is a must
Knowledge of modern data tools like dbt (data build tool) and Airflow orchestration is highly desired
Ability to assist SQL analysts and Tableau developers in business teams in creating the right set of materialized views in a SQL data warehouse like Redshift/Snowflake
Knowledge of automation and CICD best practices
Familiarity with machine learning and data science ecosystems especially AWS Sagemaker and Databricks is highly preferred
Hands-on experience in building and maintaining production data applications, current experience in both relational and distributed columnar date stores.
Deep experience using SQL, Python, and SparkHands-on experience with Big-data technologies (e.g. Redshift, Athena, Glue, EMR, Kinesis, Step Function, or equivalent in other web services)
Familiarity with timeseries database, data streaming applications, Kafka, Flink, and more is a plus

Familiarity with modern data science and product analytics tools and techniques such as R, Machine Learning, and advanced statistics is a plus.



  • San Jose, United States Mendel.ai Full time

    About Mendel: At Mendel, we are enabling different stakeholders in healthcare to make better decisions by learning from every patient's journey— from a physician deciding on the best treatment for a patient to a pharma company discovering the next blockbuster drug. In creating depth and breadth for a patient’s health record journey, Mendel enables the...


  • San Jose, United States Mendel.ai Full time

    About Mendel: At Mendel, we are enabling different stakeholders in healthcare to make better decisions by learning from every patient's journey— from a physician deciding on the best treatment for a patient to a pharma company discovering the next blockbuster drug. In creating depth and breadth for a patient’s health record journey, Mendel enables the...


  • San Jose, United States Mendel.ai Full time

    About Mendel: At Mendel, we are enabling different stakeholders in healthcare to make better decisions by learning from every patient's journey— from a physician deciding on the best treatment for a patient to a pharma company discovering the next blockbuster drug. In creating depth and breadth for a patient’s health record journey, Mendel enables the...


  • San Jose, United States HireIO Inc Full time

    Team Introduction: Our mission in experimentation and evaluation team is to build the next-gen A/B testing platform, that empowers the company to make data-driven decision for the products. The supported scenarios include recommendation, push, ads, search, mobile app, UI interaction and service upgrades etc. Our platform's capabilities cover the entire...


  • San Jose, United States Hireio, Inc. Full time

    Job DescriptionJob DescriptionTeam Introduction:Our mission in experimentation and evaluation team is to build the next-gen A/B testing platform, that empowers the company to make data-driven decision for the products. The supported scenarios include recommendation, push, ads, search, mobile app, UI interaction and service upgrades etc. Our platform's...


  • San Jose, United States RAMPS International Inc. Full time

    Job DescriptionJob DescriptionHadoop Platform EngineerSan Jose, CA4-12 MonthsPosition summaryThe Data and Analytics Platform team resides within Adobe's Information and Data Services team and we are looking for a Hadoop Platform Engineer who will be responsible for the implementation and ongoing administration of Hadoop infrastructure including...


  • San Francisco, CA, United States Pinecone Full time

    About Pinecone Pinecone is on a mission to build the search and database technology to power AI applications for the next decade and beyond. Our fully managed vector database makes it easy to add vector search to AI applications. Since creating the “vector database” category, demand has grown incredibly fast and it shows in our user base. We are a...


  • San Jose, United States Tik Tok Full time

    Responsibilities TikTok is the leading destination for short-form mobile video. Our mission is to inspire creativity and bring joy. TikTok has global offices including Los Angeles, New York, London, Paris, Berlin, Dubai, Singapore, Jakarta, Seoul and Tokyo. Why Join Us At TikTok, our people are humble, intelligent, compassionate and creative. We create to...

  • Software Engineer

    4 days ago


    San Francisco, CA, United States X Corp. Full time

    Are you prepared to join the X team and help build the ultimate real-time information-sharing app, revolutionizing how people connect? At X, we're on a mission to become a trusted global digital public square, committed to minimal censorship within legal boundaries. Our goal is to empower every user to freely create and share ideas, fostering open public...


  • San Francisco, California, United States Woven Full time

    About WovenConsumer Travel is Ripe for DisruptionToday's travelers are seeking personalized, immersive, and authentic experiences that reflect their unique tastes and interests. They want to be inspired, to feel the thrill of discovery, and to seamlessly benefit from technology throughout their journeys. Yet, the current travel landscape is a maze of...


  • San Francisco, California, United States Everyday Agents Full time

    About WovenConsumer Travel is Ripe for DisruptionToday's travelers are seeking personalized, immersive, and authentic experiences that reflect their unique tastes and interests. They want to be inspired, to feel the thrill of discovery, and to seamlessly benefit from technology throughout their journeys. Yet, the current travel landscape is a maze of...


  • San Antonio, United States Dunhill Professional Search Full time

    US Citizenship is required for this positon due to Government Clients.Offering sign-on bonus for relocation purposes if applicable Location: San Antonio, TX OR St. Louis, MO (Hybrid- Must relocate to the area)As part of our AI & Technology group, you will lead cloud technology innovation for our clients through robust delivery of world-class data platforms....


  • San Francisco, CA, United States HireIO Inc Full time

    Team IntroductionThe Data Platform team works on building data infrastructures and data products to support business engineering teams. As a Software Development Engineer in the data platform team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the opportunity to gain hands-on...


  • San Mateo, CA, United States Snowflake Full time

    Build the future of data. Join the Snowflake team. We're hiring talented Senior Engineers for our Container Platform team that are passionate about using software-based approaches to solve complex infrastructure challenges and automate those solutions. You'll be part of the cloud engineering organization where we have a strong focus on using...


  • San Francisco, United States OpenAI Full time

    About The Team You’ll manage the team that’s behind OpenAI’s data infrastructure that powers critical engineering, product, alignment teams that are core to the work we do at OpenAI. The systems we support include our data warehouse, batch compute infrastructure, data orchestration system, data lake, ingestion systems, critical integrations, and more....


  • San Francisco, CA, United States Equilibrium Energy Full time

    About Our Company Equilibrium Energy is a well-funded, Series A clean energy startup backed by some of the most prominent institutional investors in climate. We are building a digital native power company operating at the intersection of grid variability, market volatility, economic optimization, commercial structuring, and risk management, across the...


  • San Jose, CA, United States ByteDance Full time

    With a suite of more than a dozen products, including TikTok, Helo, and Resso, as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content. Our products are built to help imaginations thrive. The Applied Machine Learning (AML) team...

  • Software Engineer

    7 days ago


    San Jose, United States Cisco Full time

    You will be a member of a Cloud Infrastructure and Platform Automation (IPA) software engineering team that develops tools and integrations for a portfolio of cloud infrastructure services running Cisco’s critical business services. Our team is seeking a software engineer with extensive experience in enterprise-level software development, to join a dynamic...


  • San Mateo, CA, United States Snowflake Full time

    Build the future of data. We're at the forefront of the data revolution, committed to building the world's greatest data and applications platform. Snowflake started with a clear vision: develop a cloud data platform that is effective, affordable, and accessible to all data users. Snowflake developed an innovative new product with a built-for-the-cloud...


  • San Jose, United States Cisco Full time

    Who We Are The Cisco Security AI team delivers AI products and platform for all Cisco secure products and portfolios so businesses around the world defend against threats and safeguard the most vital aspects of your business with security resilience. We are passionate about making businesses secure and simplify security with zero compromise using AI...