Data Engineer

Insight Global is looking for a Data Engineer to work 4 days on-site in Santa Clara, CA for a fast-growing technology company specializing in secure networking solutions-including SD-WAN, SASE, and SSE-that is helping global enterprises modernize their IT infrastructure for the cloud era. Led by industry veterans and backed by top-tier investors, the company is focused on innovation, scalability, and preparing for a future IPO.

The Data Engineer will be responsible for designing, building, and maintaining production-grade data pipelines that process and transform large-scale datasets. This role involves close collaboration with data scientists and software engineers to ensure the data infrastructure is scalable, reliable, and cost-efficient.
Pipeline Development & Deployment
• Architect and deploy batch and streaming pipelines using Airflow and containerized workflows for cybersecurity use cases.
• Containerize data-processing jobs with Docker, orchestrate with Kubernetes, and manage releases using Helm charts.
Distributed Computing
• Build high-throughput data transformations using Dask or Apache Spark.
• Maintain training data clusters across hybrid environments (on-prem and cloud).
• Optimize training jobs for performance, resiliency, and cost.
Monitoring & Reliability
• Implement observability tools (logging, metrics, alerting) to ensure pipeline health and SLA compliance.
• Troubleshoot and resolve data-processing failures in production environments.
Collaboration & Best Practices
• Define data contracts, schemas, and quality checks with cross-functional teams.
• Enforce engineering best practices including CI/CD, code reviews, automated testing, and documentation.
Data Modeling & Storage
• Design and maintain data models and schemas for continuous AI/ML training workflows.
• Load and manage data in cloud storage and data lakes for performance and accessibility.

Salary Range: $150k - $220k

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Required Skills & Experience
• 3-5 years of experience designing and operating production-scale data pipelines.
• Expertise in Docker, Kubernetes, and Helm for containerization and orchestration.
• Hands-on experience building DAG-based workflows in Apache Airflow.
• Strong proficiency in Python for data engineering tasks.
• Practical experience with Dask or Apache Spark for distributed data processing.
• Familiarity with cloud environments and services.
• Experience with compiled languages such as Go or Rust.
• Proficiency with Google Cloud Platform (e.g., Pub/Sub, BigQuery, Cloud Storage, GKE); equivalent experience in other cloud platforms is acceptable.
• Exposure to ML pipeline frameworks such as Ray.
• Familiarity with Terraform for infrastructure deployment.
• Understanding of data governance, encryption, and role-based access control.

Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.