Principal SWE, Data Lake Engineering

2 days ago


San Francisco, United States WEX Full time

This is a remote position; however, the candidate must reside within 30 miles of one of the following locations: Portland, ME; Boston, MA; Chicago, IL; and San Francisco Bay Area, CA.

About the Team/Role


The Data Lake Engineering team at WEX builds a unified, scalable data platform supporting AI and machine learning applications across the company. They collaborate with departments like Data & Analytics, Security, and Risk to enhance data accessibility, scalability, and innovation, aiming to drive WEX's tech transformation and support a growing customer base.

The Principal Software Engineer (SWE) will lead the technical architecture and development of WEX's Data Lake platform. Reporting to the VP of Data Lake and AI Engineering, they'll ensure quality, security, and scalability while integrating with company-wide platforms. The ideal candidate brings deep expertise in data lake environments and hands-on leadership skills.

How you'll make an impact

  • Lead and drive the development of technology and platform for the company's Data Lake needs, ensure the functional richness, reliability, performance, and flexibility of the Data Lake Platform.

  • Help design the architecture and lead the implementation of the Data Lake platform and services.

  • Challenge the status quo and hold a high bar to drive for better and better technical solutions to effectively serve all the needs in the broad risk area of Wex.

  • Guide your team to leverage data and/or AI technology or tools in your design and development for high productivity and better solution quality. Show strong thought leadership and influence your peers in this area.

  • Closely partner with the other senior engineers across the organizations at Wex to deeply understand the business needs and jointly come up with technical solutions to serve these needs effectively. Demonstrate strong thought leadership and partnership.

  • Partner closely with information security teams to ensure the security and privacy of data, implementing encryption and access controls as required.

  • Partner with solutions architecture and LOB product and engineering orgs on integrations of the Data Lake Platform with their products/systems, and ensure reliable and performed execution.

  • Engage with external partners and vendors to evaluate technologies and services that complement the company's Data Lake strategy.

  • Create, prioritize, communicate, manage, and execute roadmaps, project plans, and commitments.

  • Report on the status of development, quality, operations, and system performance to management and partners.

  • Drive the technical vision and strategy of Data Lake to serve the business needs for the near term and long term. A close partnership with the Data Engineering teams and the operational reporting teams of applications

  • Set a high bar, and provide technical guidance and effective training & mentorship to the engineering teams in your group, including engineering principles, skills, and best practices. Uphold strong DORA metrics across the org. Foster an environment of continuous learning and innovation. Build an inclusive, productive, innovative, and engaging Risk group, including attracting high talents from the industry.
    Guide the leaders in your group to do the same.

  • Set clear, measurable quality goals for your group in a data-driven way

  • Partner with the security org and SRE org to ensure continuous improvement and meeting of security requirements, SLOs, and error budget.

  • Take strong financial responsibility for your team.

  • Hold yourself and your team accountable for delivering quality results using defined OKRs.

  • Advise Senior Leadership team on highly complex situations that impact our organization and then lead teams through the initiative, driving excellent quality results and achieving defined OKRs across the organization.

  • Provide strategic thought leadership on business initiatives using your deep technical and business industry expertise.

Experience you'll bring

  • A technically deep and innovative Software Engineer leader able to act and deliver to the business needs.

  • Have 15-20 years of software design and development experience at a large scale.

  • Strong software development skills in his or her chosen programming language, but ready and willing to learn and program in Java and Python.

  • Broad understanding of Data Lakehouse formats like Apache Hudi, Apache Iceberg or Delta Lake is a big plus.

  • Demonstrable experience in Spark programming

  • Experience with Spark on DBT with AWS Glue or Apache Polaris is a plus

  • Broad understanding of cloud architecture tools and services, such as S3, EMR, Kubernetes, Lambda functions. Experience in AWS and Azure is highly desirable.

  • Deep technically; hands-on in architecture design, development, deployment, and production operation; familiar with agile software development and modern development tools and frameworks; holding strong engineering principles including automation and quality as well as best practices with a high bar. Extensive experience in full software development life cycle E2E including in production monitoring.

  • Rich experience and deep expertise in the areas of Big Data and large scale data platforms, especially in Data Lake.

  • Highly experienced and deep understanding of CI/CD automation.

  • Rich experience in combined engineering practice and agile development. Experienced in leading and guiding teams for effective development using this approach.

  • Excellent analytical skills; Capability in identifying and resolving complex problems.

  • An understanding of and experience in payment organizations is a plus.

  • A mentor to individual contributors of different seniority levels and expertise.

  • Ability to encourage and foster a culture of visibility and transparency across teams

  • Strong written and verbal communication skills

  • Strong philosophy of continuous improvement

  • Bachelor degree in Computer Science or a related field. A higher degree in a related field is a plus.

Leadership & Personal Characteristics

Beyond experience, the right leadership competencies and personal style are critical to success as the Sr Director of Data Lake engineering. The candidate will model WEX's commitment to innovation, integrity, execution, relationships, community, and the excellence that they commit to their customers. They will have a strong corporate stewardship philosophy and leadership style combined with the following:

  • Demonstrates a collaborative, engaging, mission-driven style with good listening skills combined with gravitas and self-confidence in order to effectively interact with senior leadership and other key constituencies with humility, empathy, transparency, cultural sensitivity, and agility.

  • Champions a culture of high standards, pushes for progress, embraces change and challenges the status quo based on integrity, strong partnership and financial responsibility. Possess uncompromising ethics and unwavering integrity.

  • Able to operate in a fast-paced entrepreneurial environment with a hands-on approach.

  • Leads with empathy. Recognizes others, listens intently, holds others accountable, and makes diversity and inclusion a priority. Promotes a culture of trust, transparency, fairness, innovation, collaboration, inclusion, and diversity of thought; empowers individuals.

  • Communicates effectively in small and large group formats. Able to articulate a compelling vision and purpose that excites the team and drives outstanding performance and service delivery excellence.

  • Talent magnet, talent developer and talent scout that proactively engages external talent and builds and nurtures internal talent; empowers individuals and manages diverse teams to drive engagement and performance.

  • Possesses a high level of intellectual curiosity and learning agility. A natural change agent with a combination of vision and execution skills.

  • Understands nuance and how to drive change through influence across a complex business. Superb stakeholder management and comfortable operating in a fast-changing environment.

The base pay range represents the anticipated low and high end of the pay range for this position. Actual pay rates will vary and will be based on various factors, such as your qualifications, skills, competencies, and proficiency for the role. Base pay is one component of WEX's total compensation package. Most sales positions are eligible for commission under the terms of an applicable plan. Non-sales roles are typically eligible for a quarterly or annual bonus based on their role and applicable plan. WEX's comprehensive and market competitive benefits are designed to support your personal and professional well-being. Benefits include health, dental and vision insurances, retirement savings plan, paid time off, health savings account, flexible spending accounts, life insurance, disability insurance, tuition reimbursement, and more. For more information, check out the "About Us" section. Pay Range: $195,000.00 - $260,000.00

  • San Francisco, CA, United States Atlassian Full time

    OverviewWorking at AtlassianAtlassians can choose where they work – whether in an office, from home, or a combination of the two. That way, Atlassians have more control over supporting their family, personal goals, and other priorities. We can hire people in any country where we have a legal entity. Interviews and onboarding are conducted virtually, a part...


  • San Francisco, United States Compunnel Inc. Full time

    Staff Data Lake EngineerFull timeRemotePay Range: $135,000.00 - $180,000.00 This is a remote position; however, the candidate must reside within 30 miles of one of the following locations: Portland, ME; Boston, MA; Chicago, IL; and San Francisco Bay Area, CA.About the Team/Role: The Data Lake Engineering team at our client builds a unified, scalable data...


  • San Francisco, United States WEX Full time

    This is a remote position; however, the candidate must reside within 30 miles of one of the following locations: Portland, ME; Boston, MA; Chicago, IL; and San Francisco Bay Area, CA. About the Team/Role: The Data Lake Engineering team at WEX builds a unified, scalable data platform supporting AI and machine learning applications across the company. They...


  • San Francisco, United States Keurig Dr Pepper Full time

    Why join us?Keurig Dr Pepper (NASDAQ: KDP) is a modern, leading coffee and beverage company with a bold vision built to deliver growth and opportunity. We operate with a differentiated business model and world-class brand portfolio, powered by a talented and engaged team that is anchored in our values. We seek diversity in our workforce and empower our team...


  • San Jose, United States Compunnel Inc. Full time

    Staff Data Lake EngineerFull timeRemotePay Range: $135,000.00 - $180,000.00 This is a remote position; however, the candidate must reside within 30 miles of one of the following locations: Portland, ME; Boston, MA; Chicago, IL; and San Francisco Bay Area, CA.About the Team/Role: The Data Lake Engineering team at our client builds a unified, scalable data...


  • San Jose, United States Compunnel Inc. Full time

    Staff Data Lake EngineerFull timeRemotePay Range: $135,000.00 - $180,000.00 This is a remote position; however, the candidate must reside within 30 miles of one of the following locations: Portland, ME; Boston, MA; Chicago, IL; and San Francisco Bay Area, CA.About the Team/Role: The Data Lake Engineering team at our client builds a unified, scalable data...


  • San Francisco, United States Snowflake Computing Full time

    Build the future of the AI Data Cloud. Join the Snowflake team. Snowflake is seeking an accomplished Principal Data Engineering Platform Sales Specialist to drive sales execution for our Data Engineering workload in the Enterprise segment. Reporting to the global Head of Data Engineering Platform Sales, this is a strategic role that works closely with the...


  • San Francisco, United States Yorke Engineering Full time

    Job DescriptionJob DescriptionJoin Yorke Engineering, LLC, an Environmental Consulting leader in California that implements Environmental Engineering and Compliance solutions for our clients throughout California. Our mission is to solve environmental compliance problems for industrial and governmental/infrastructure facilities. We are a growing and dynamic...


  • San Jose, United States OSI Engineering Full time

    Principal Test Engineer A leading provider of chip and silicon IP is looking for a talented Principal Test Engineer to join our Test Engineering team in San Jose, CA. This role offers the opportunity to work alongside some of the brightest inventors and engineers globally, developing products that enhance data speed and security. The Principal Test Engineer...


  • San Jose, United States OSI Engineering Full time

    Principal Test Engineer A leading provider of chip and silicon IP is looking for a talented Principal Test Engineer to join our Test Engineering team in San Jose, CA. This role offers the opportunity to work alongside some of the brightest inventors and engineers globally, developing products that enhance data speed and security. The Principal Test Engineer...


  • San Francisco, United States NobleAI Full time

    Job DescriptionJob DescriptionAt NobleAI, we believe that energy, material science and chemistry are key to building a sustainable world and that artificial intelligence is essential to unlock this potential. NobleAI leverages innovative Science-Based AI technology to revolutionize energy workflows, materials development, and chemical designs. We enable...


  • San Francisco, United States ZipRecruiter Full time

    Job DescriptionAt NobleAI, we believe that energy, material science and chemistry are key to building a sustainable world and that artificial intelligence is essential to unlock this potential. NobleAI leverages innovative Science-Based AI technology to revolutionize energy workflows, materials development, and chemical designs. We enable companies to...


  • San Francisco, United States Noble.AI Full time

    At NobleAI, we believe that energy, material science and chemistry are key to building a sustainable world and that artificial intelligence is essential to unlock this potential. NobleAI leverages innovative Science-Based AI technology to revolutionize energy workflows, materials development, and chemical designs. We enable companies to accelerate innovation...


  • San Francisco, United States Yorke Engineering, LLC Full time

    Join Yorke Engineering, LLC, an Environmental Consulting leader in California that implements Environmental Engineering and Compliance solutions for our clients throughout California. Our mission is to solve environmental compliance problems for industrial and governmental/infrastructure facilities. We are a growing and dynamic organization of highly...


  • San Francisco, United States hims & hers Full time

    Hims & Hers Health, Inc. (better known as Hims & Hers) is the leading health and wellness platform, on a mission to help the world feel great through the power of better health. We are revolutionizing telehealth for providers and their patients alike. Making personalized solutions accessible is of paramount importance to Hims & Hers and we are focused on...


  • San Francisco, United States Hire Score LLC Full time

    Excellent opportunity to join a well-funded, VC backed startup in the San Francisco Bay Area as the Founding Principal Data Engineer. This role offers a career-defining opportunity for ambitious data engineers to accelerate their growth and contribute to a quickly evolving startup. You’ll be joining an in-person team of engineers and physicists to continue...


  • San Francisco, United States Hire Score LLC Full time

    Excellent opportunity to join a well-funded, VC backed startup in the San Francisco Bay Area as the Founding Principal Data Engineer. This role offers a career-defining opportunity for ambitious data engineers to accelerate their growth and contribute to a quickly evolving startup. You’ll be joining an in-person team of engineers and physicists to continue...


  • San Francisco, California, United States Compunnel Inc. Full time

    About the RoleAs a Staff Data Lake Engineer at Compunnel Inc., you will play a key role in building a unified, scalable data platform that supports AI and machine learning applications across the company.You will collaborate with departments like Data & Analytics, Security, and Risk to enhance data accessibility, scalability, and innovation, aiming to drive...

  • Principal Engineer

    4 days ago


    San Francisco, California, United States Yorke Engineering Full time

    Our PhilosophyWe believe in fostering a collaborative environment where experts can thrive, sharing knowledge and expertise to achieve exceptional results. At Yorke Engineering, we value teamwork, innovation, and continuous learning, driving excellence in environmental consulting and compliance services.As a principal air quality engineer/scientist, you will...


  • San Francisco, California, United States Twilio Full time

    About UsWe're Twilio, a cloud communication platform that enables businesses to engage with their customers through a wide range of channels. Our goal is to provide a seamless and personalized experience for every interaction, whether it's a phone call, text message, or email. As a Principal, Machine Learning Engineer on our Customer and Developer Experience...