Senior Data Engineer

1 month ago


San Francisco, California, United States Scroll Capital Full time
Lead Data Engineer - Python Expert

$45k – $90k
• 0.1% – 0.25%

About Our Company

We are a VC backed Fintech Startup in the Bay Area.

The Role

Staff/Lead Software Engineer

This is a core and very important role for the company. You'll have a lot of agency in architecting and implementing a corner store of modern financial infrastructure.

You'll be enhancing and maintainging the data pipeline that powers the backend of our company.

You'll be building a cutting edge fintech platform that helps SaaS companies get funded.

Qualifications
10+ years of experience with "Big data" data pipelines, architectures and data sets .
Knowledge of ETL and ELT processes etc.
10+ years of Python expeirence. You should be an exper in python & pandas.
You have experience in building pipelines. Luigi, Python, Pandas, Redshift, Airflow or any other pipeline building tools.
Experience with big data tools: Hadoop, Spark, Kafka, Spark & Kafka Streaming, (Preferred)
Advanced working SQL experience working with relational databases, query authoring (SQL) and working familiarity with a variety of databases
Expert in Docker, AWS Fargate, ECS, ELB - AWS Stack etc.
Knowledge of Data Science (pandas, sci-kit learn)
Experience with different databases: distributed DBs like Hadoop and Spark, relational DBs like PostgreSQL and mySQL, and NoSQL DBs such as MongoDB or Couchbase.
You have built, deployed, owned and maintained production systems with real world workloads which includes database schema design, api design, deploying code, product design and building websites, backend processing systems and more.
You have deployed production systems on AWS(preferred) or GCP
Write clean, readable and testable code and refactor code as capabilities evolve.
Can build and maintain end to end systems and put them into production on your own.

Tech Stack
Big Data - Python, Pandas, Luigi, Redshift, SQL, (Must Have Skills)
Deployment - AWS Fargate, Docker.
Experience interfacing via API and JSON with Ruby on Rails or
Web development - Ruby on Rails or
Backend - Postgres or Mysql database, SQL, Sidekiq
Data Science, ML - pytorch, (Highly preferred but not a requirement.)

Why you should Join
You should join our company if the following sounds exciting:

Well funded startup with 5+ years of runway.
You want to build your skills to work towards becoming a technical lead who can own the architecture and implementation of the entire system.
You want to take ownership for your work and have agency/voice in building systems.
You are a self-starter - you set ambitious goals and timelines for yourself and follow through.

Scroll Capital focuses on Financial Services and Finance Technology. Their company has offices in San Francisco. They have a small team that's between 1-10 employees.
You can view their website at

  • San Francisco, California, United States Trace3, Inc Full time

    Our major field office locations include Denver, Indianapolis, Grand Rapids, Lexington, Los Angeles, Louisville, Texas, San Francisco.Juice - The "Stuff" it takes to be a Needle MoverWe look forward to the goal, mentallymapping outevery checkpoint on the pathway to success, and visualizing what the final destination looks and feels like.We're looking to add...

  • Senior Data Engineer

    1 month ago


    San Francisco, California, United States Nightfall AI Full time

    Nightfall makes safeguarding sensitive data for every application simple and seamless. Organizations, from startups to global brands, trust Nightfall's software platform and APIs to discover, classify, and protect sensitive data.We're looking for a Senior Data Engineer to enhance Nightfall's core data models and infrastructure supporting our real-time and...


  • San Francisco, California, United States Archipelo Full time

    CompanyArchipelo is building a code security platform that gives organizations the ability to verify the authenticity and provenance of code within their software development lifecycle. They are solving a painful problem that affects every software developer on the planet: ensuring software security, authenticity, integrity, and compliance - by providing the...


  • San Francisco, California, United States Xero Full time

    About the Team The Data team is responsible for driving the adoption of data driven decision making across Xero - for both our internal users and our global audience of small business owners, bookkeepers and accountants. We create, manage and deliver a wide range of data services and products including core reporting, key data dimensions, extensible and...


  • San Francisco, California, United States Motion Recruitment Full time

    This company develops technology that protects the integrity, confidentiality, and privacy of healthcare data while enabling consumers to comprehend it. They create systems that exhibit an accurate picture of the underlying data, are auditable and automated as far as feasible, and, most importantly, are responsive to the needs of our end users.They're...


  • San Francisco, California, United States Syntegra Full time

    Syntegra has developed a state-of-the-art method to generate patient-level synthetic dataset for healthcare applications. By utilizing transformer models (like GPT-2 and GPT-3), our synthetic data engine enables us to create realistic equivalents of any healthcare dataset that accurately preserves the analytic characteristics of the original, while...


  • San Francisco, California, United States Quid Full time

    Senior DevOps EngineerAs a Senior DevOps engineer for the Quid product you will build software and processes to enable engineers to self-service the operation of NetBase Quid at scale and support 24x7 operations. Our developers are your customers. Your goal is to continuously assess and ease pain points of fellow engineers and of our Cloud infrastructure. We...

  • Data Engineer

    1 month ago


    San Francisco, California, United States iTvorks Inc Full time

    Required Skills Data engineer main skills Good knowledge of Data warehousing with expertise in data modeling and ETL. Advanced skills in SQL and HQL Should be good at these core tech AWS/EMR/Spark Hadoop Hive Vertica Qlikview Tableau Should have exp in CI/CD

  • Staff Data Engineer

    3 weeks ago


    San Francisco, California, United States Pachama Full time

    Who we are.Pachama is a mission-driven company looking to restore nature to help address climate change. Pachama brings the latest technology in remote sensing and AI to the world of forest carbon in order to enable forest conservation and restoration to scale. Pachama's core technology harnesses satellite imaging with artificial intelligence to measure...

  • Sr. Data Engineer

    4 weeks ago


    San Francisco, California, United States iTvorks Inc Full time

    Role Sr. Data Engineer (8+ years) Location SFO CA Duration Long TermExperience & Skills Extensive experience with Hadoop (or similar) Ecosystem (Map Reduce HDFS Hive Spark Pig HBase) Proficient in at least one of the SQL languages (MySQL PostgreSQL SqlServer Oracle) Good understanding of SQL Engine and able to conduct advanced performance tuning Strong...


  • San Francisco, California, United States Everyday Agents Full time

    About WovenConsumer Travel is Ripe for DisruptionToday's travelers are seeking personalized, immersive, and authentic experiences that reflect their unique tastes and interests. They want to be inspired, to feel the thrill of discovery, and to seamlessly benefit from technology throughout their journeys. Yet, the current travel landscape is a maze of...

  • Principal Engineer

    5 days ago


    San Francisco, California, United States Domino Data Lab Full time

    Who we areDomino Data Lab is a game-changer, providing a leading Enterprise AI platform trusted by more than 20% of the Fortune 100 companies. With Domino, businesses speed up data science work, foster collaboration, and uphold governance. Enterprises globally benefit from Domino's solutions, improving areas such as healthcare, agriculture, automotive...


  • San Francisco, California, United States Everyday Agents Full time

    About WovenConsumer Travel is Ripe for DisruptionToday's travelers are seeking personalized, immersive, and authentic experiences that reflect their unique tastes and interests. They want to be inspired, to feel the thrill of discovery, and to seamlessly benefit from technology throughout their journeys. Yet, the current travel landscape is a maze of...

  • Senior Engineer

    1 month ago


    San Francisco, California, United States The League Full time

    Company DescriptionThe League is a pre-series A mobile social dating app startup backed by IDG Ventures, xSeed Capital, Cowboy Ventures, Structure Capital, Sherpa Ventures, and many notable angels. The Founder is a Stanford MBA (ex-Google, ex-Salesforce) with a strong product sense (engineering degree from Carnegie Mellon) and a fierce determination to...


  • San Francisco, California, United States Block Full time

    Company DescriptionIt all started with an idea at Block in 2013. Initially built to take the pain out of peer-to-peer payments, Cash App has gone from a simple product with a single purpose to a dynamic ecosystem, developing unique financial products, including Afterpay/Clearpay, to provide a better way to send, spend, invest, borrow and save to our 47...


  • San Francisco, California, United States Databricks Full time

    While candidates in the listed locations are encouraged for this role, we are open to remote candidates in other locations.As a Data Infrastructure Engineer on the security data infrastructure team you will help build Lakehouse for Security organization. You will build reliable, large-scale, multi-geo data pipelines to support detecting threats(internal and...


  • San Francisco, California, United States Block Full time

    Company DescriptionIt all started with an idea at Block in 2013. Initially built to take the pain out of peer-to-peer payments, Cash App has gone from a simple product with a single purpose to a dynamic ecosystem, developing unique financial products, including Afterpay/Clearpay, to provide a better way to send, spend, invest, borrow and save to our 47...


  • San Francisco, California, United States Fathom Full time

    Fathom is on a mission to use AI to understand and structure the world's medical data, starting by making sense of the terabytes of clinician notes contained within the electronic health records of the world's largest health systems. Our deep learning engine automates the translation of patient records into the billing codes used for healthcare provider...


  • San Francisco, California, United States Fathom Full time

    Fathom is on a mission to use AI to understand and structure the world's medical data, starting by making sense of the terabytes of clinician notes contained within the electronic health records of the world's largest health systems. Our deep learning engine automates the translation of patient records into the billing codes used for healthcare provider...


  • San Francisco, California, United States QData Full time

    The goal of the ServiceNow Senior Engineer is to build help test and maintain a strategic enterprise service management platform that includes incident management request management hardware/software asset management and change management. The Senior Engineer will own the development tasks of the solution based on integration of ServiceNow with other...