Principal or Staff AI/ML Engineer

Want to work on the next wave of AI? Join Wallaroo.AI and help us build the next generation of enterprise AI inference software

Wallaroo.AI is on the hunt for a Principal or Staff AI/ML Engineer to lead the charge in pushing the boundaries of state-of-the-art inference capabilities on the latest hardware. We're looking for an expert with a passion for LLM deployment, inference, and lifecycle, coupled with a deep understanding of LLM architectures and cutting-edge generative AI serving runtimes. Our mission at Wallaroo.AI is to enable inferencing on "any hardware," so if you thrive on mastering new technologies, diving deep, and extracting maximum performance, you'll feel right at home.

Wallaroo.AI is a Microsoft-backed startup at the forefront of AI, developing a powerful software platform that empowers AI teams to deploy AI inference at scale to any cloud, edge or on-prem environment with unparalleled speed and efficiency. We have marquee enterprise customers and technology partners, including some of the most globally recognized brands. We're poised for explosive growth in 2026 fueled by our go-to-market with leading AI hardware providers. This is your chance to be part of something truly transformative.

We've been fully distributed and remote-first since before it was cool This means we seek exceptional communicators who are self-motivated and adept at both independent and collaborative work. We also need individuals who thrive in dynamic, fast-paced environments. Successful Wallaroo.AI team members are comfortable experimenting with fresh ideas, taking initiative without constant oversight, adapting swiftly as priorities evolve, and bringing clarity to complex, emerging challenges. Prior startup experience or a similar high-velocity environment is a significant advantage.

Responsibilities:

At Wallaroo.AI, building a great product is a team endeavor. We expect active participation in discussions, proactive communication, and continuous feedback. You'll partner closely with customers and hardware teams to solve problems and build capabilities. As an experienced engineer, you'll negotiate tradeoffs with product managers, break down feature requests into incremental, high-quality deliverables, and mentor less experienced team members. You'll also propose process and workflow improvements, and confidently advocate for your perspective while thoughtfully listening to others.

Required Skills/Abilities (at least 4):

• Excellent verbal and written communication skills
• Expert Python, including extensive packaging experience using Pip, uv, etc.
• Strong experience with vLLM, SGLang, and other modern serving frameworks
• Experience developing high-scale production ML pipelines
• A strong foundation in deep learning architectures, especially the Transformer architecture, including experience with NLP tasks like text generation, summarization, and sentiment analysis using commonly available LLM APIs and SDKs (ie, OpenAI, Anthropic, Mistral, etc)

Desired Skills/Abilities:

• Experience developing in Rust
• Experience working with containers, Docker and/or Kubernetes clusters
• Experience working in AWS, GCP, and Azure
• Experience with AzureML, Google Vertex, Databricks, Sagemaker
• Experience with GPU-focused software environments like CUDA, ROCm, or OpenVino

Education and Experience:

• At least eight years related experience recommended

Interview Process

We know interviewing can be a daunting experience, so we would like to provide some visibility into our process.

• Resume Review: If the resume meets the requirements described in the job post then Wallaroo schedules an initial screen.
• Initial Screen: This is a 45 minute interview with questions that help us get a better understanding of your passion, your experience, and the way that you approach your work. There are a mix of technical and behavioral questions but no technical exercises.
• Technical Interview: If the initial screen was positive the next step is the technical interview. This step will be 90 minutes and will be conducted by two members of our engineering team. There may be a take-home component ahead of this interview.
• Behavioral Interview: A second one-hour interview will focus on team fit, and explore questions related to communication, decision-making, and development practices.
• Hiring Manager Interview: A third and final one-hour interview will be conducted by the engineering hiring manager, and will focus on perseverance, conflict resolution, and career growth.
• Offer

Candidates are encouraged to interview us at each stage as well, to ensure that they have as much insight into the role and company culture as possible and allow them to make an informed decision. Every interview will include at least 10 minutes for the candidate to ask questions of the interviewer.

Some of our benefits and perks for US employees include

• Medical, Dental, and Vision coverage plans starting at only $1
• One Medical membership
• Life insurance
• Healthcare, Limited Purpose, and Dependent FSAs
• Pet insurance options
• 401K
• Unlimited time off policy

WQ5ky2FRSJ