Big Data Engineer
2 months ago
Job Description
Are you passionate about handling massive datasets and building the infrastructure that enables complex data analysis and machine learning at scale? Do you excel in creating robust, scalable data pipelines that fuel data-driven decision-making? If you’re ready to tackle the challenges of big data, our client has the perfect role for you. We’re seeking a Big Data Engineer (aka The Data Pipeline Innovator) to architect and maintain high-performance data systems that empower analytics and support advanced data processing needs.
As a Big Data Engineer at our client, you’ll collaborate with data scientists, analysts, and software engineers to design, implement, and optimize big data platforms. Your expertise in data engineering, distributed systems, and cloud infrastructure will be critical to ensuring that our data ecosystem is efficient, reliable, and scalable.
Key Responsibilities:- Design and Build Scalable Data Pipelines: Architect and implement data pipelines for ETL processes using tools like Apache Spark, Kafka, and Hadoop. You’ll create data workflows that handle high-volume, high-velocity data and ensure seamless integration across systems.
- Optimize Big Data Storage and Processing: Develop and manage data storage solutions (e.g., HDFS, S3, Cassandra) that are optimized for performance and cost-efficiency. You’ll configure distributed processing systems to support efficient data retrieval and transformation.
- Collaborate on Data Strategy and Integration: Work closely with data scientists, analysts, and other engineers to align big data architecture with analytics goals. You’ll ensure data availability and integrity across systems to support business objectives.
- Implement Data Quality and Governance Standards: Develop processes and tools to monitor data quality and enforce data governance policies. You’ll ensure data is accurate, reliable, and secure through regular checks and validation processes.
- Enhance Data Processing with Automation: Use tools like Apache Airflow or AWS Glue to automate data workflows and reduce manual processing. You’ll implement scripts and automation that streamline data handling and improve efficiency.
- Monitor and Troubleshoot Data Systems: Use monitoring tools to track system performance and address issues proactively. You’ll troubleshoot and resolve any bottlenecks or failures to maintain optimal data processing capabilities.
- Stay Updated on Big Data Trends and Technologies: Keep up with advancements in big data technologies and tools. You’ll integrate new techniques and platforms that align with business needs and promote innovation.
- Big Data Platform Proficiency: Extensive experience with big data technologies such as Apache Spark, Hadoop, Kafka, and Hive. You’re skilled at handling high-volume data and distributed processing.
- Data Pipeline and ETL Knowledge: Proven ability to design, build, and maintain ETL processes for massive datasets. You can handle both real-time and batch data processing requirements.
- Programming and Scripting: Proficiency in programming like Python, Java, or Scala for data processing and automation. Experience with SQL for data querying and manipulation is essential.
- Cloud Data Services Expertise: Familiarity with cloud platforms such as AWS, GCP, or Azure, including their big data and storage services (e.g., S3, BigQuery, Azure Data Lake).
- Data Quality and Governance: Strong understanding of data quality standards and governance practices, with experience in implementing data validation and monitoring frameworks.
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or a related field. Equivalent experience in data engineering or big data management may be considered.
- Certifications in big data or cloud technologies (e.g., Cloudera Certified Data Engineer, AWS Certified Big Data – Specialty, Google Professional Data Engineer) are a plus.
- 5+ years of experience in data engineering, with at least 3+ years focusing on big data technologies and high-scale data environments.
- Experience in distributed systems and large-scale data storage management.
- Familiarity with containerization (Docker, Kubernetes) for deploying data processing environments is advantageous.
- Health and Wellness: Comprehensive medical, dental, and vision insurance plans with low co-pays and premiums.
- Paid Time Off: Competitive vacation, sick leave, and 20 paid holidays per year.
- Work-Life Balance: Flexible work schedules and telecommuting options.
- Professional Development: Opportunities for training, certification reimbursement, and career advancement programs.
- Wellness Programs: Access to wellness programs, including gym memberships, health screenings, and mental health resources.
- Life and Insurance: Life insurance and short-term/long-term coverage.
- Employee Assistance Program (EAP): Confidential counseling and support services for personal and professional challenges.
- Tuition Reimbursement: Financial assistance for continuing education and professional development.
- Community Engagement: Opportunities to participate in community service and volunteer activities.
- Recognition Programs: Employee recognition programs to celebrate achievements and milestones.
-
Big Data Engineer
4 days ago
San Francisco, United States Unreal Gigs Full timeDescriptionAre you passionate about handling massive datasets and building the infrastructure that enables complex data analysis and machine learning at scale? Do you excel in creating robust, scalable data pipelines that fuel data-driven decision-making? If you’re ready to tackle the challenges of big data, our client has the perfect role for you. We’re...
-
Data Platform Engineer
3 weeks ago
San Francisco, California, United States ZipRecruiter Full timeJob Description Data Platforms and SystemsWe are seeking a skilled Software Development Engineer to join our Data Platform team, responsible for building and optimizing large-scale data infrastructures and products.You will design and implement reliable, scalable, and robust big data systems that support our core products and business, leveraging your...
-
Big Data Architect
4 weeks ago
San Francisco, California, United States ZipRecruiter Full timeJob Title:**Big Data Architect**OverviewWe are seeking an experienced Big Data Architect to join our team at ZipRecruiter. As a key member of our data engineering group, you will be responsible for designing and implementing scalable big data systems that support our analytics and business intelligence needs.The ideal candidate will have a strong background...
-
Big Data Engineer
4 weeks ago
San Antonio, Texas, United States Diverse Lynx Full timeWe are looking for a talented Big Data Engineer to join our team at Diverse Lynx LLC. In this role, you will design, develop, and deploy large-scale data processing systems that handle high volumes of data from various sources. You will work closely with our data science team to ensure that our data infrastructure meets the needs of our...
-
Big Data Platform Developer
1 month ago
San Francisco, California, United States Diverse Lynx Full timeJob Summary:Diverse Lynx LLC is seeking a highly skilled Big Data Platform Developer to join our team. As a key member of our big data engineering team, you will be responsible for designing, building, and maintaining scalable and efficient big data platforms using various big data engineering technologies such as Hadoop, Spark, HBase, Flume, and Storm. Your...
-
Senior Software Development Engineer
6 months ago
San Francisco, United States HireIO Inc Full timeTeam Introduction The Data Platform team works on building data infrastructures and data products to support business engineering teams. As a Software Development Engineer in the data platform team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the opportunity to gain hands-on experience...
-
Senior Software Development Engineer
2 weeks ago
San Francisco, United States ZipRecruiter Full timeJob DescriptionJob Description Team Introduction The Data Platform team works on building data infrastructures and data products to support business engineering teams. As a Software Development Engineer in the data platform team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the...
-
Senior Software Development Engineer
2 weeks ago
San Francisco, United States Hireio, Inc. Full timeJob DescriptionJob DescriptionTeam IntroductionThe Data Platform team works on building data infrastructures and data products to support business engineering teams.As a Software Development Engineer in the data platform team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the...
-
Senior Software Development Engineer
4 weeks ago
San Francisco, United States Hireio, Inc. Full timeTeam IntroductionThe Data Platform team works on building data infrastructures and data products to support business engineering teams.As a Software Development Engineer in the data platform team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the opportunity to gain hands-on experience...
-
Senior Software Development Engineer
2 months ago
San Francisco, United States ZipRecruiter Full timeJob DescriptionTeam IntroductionThe Data Platform team works on building data infrastructures and data products to support business engineering teams.As a Software Development Engineer in the data platform team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll gain hands-on experience on various...
-
Big Data Specialist
3 weeks ago
San Francisco, California, United States Jacobs Full timeJacobs is seeking a talented Senior Data Engineer to lead data engineering projects and help productize data products and algorithms into our scalable analytics platform. As a key member of our team, you will work closely with cross-functional teams to understand data needs and develop customized solutions for customers.The ideal candidate will have a strong...
-
Senior Big Data Developer
6 days ago
San Francisco, California, United States SS&C Full timeAbout UsSS&C is a leading provider of financial services and healthcare technology solutions with a global presence. Our team is dedicated to delivering high-quality products and services that meet the evolving needs of our clients.Job SummaryWe are seeking an experienced Senior Data Engineer to join our team and lead the design and implementation of our...
-
Big Data Specialist
4 weeks ago
San Francisco, California, United States Unity Technologies Full timeAbout UsWe are Unity Technologies, the world's leading platform for creating real-time 3D content. Our company overview highlights our commitment to innovation and customer satisfaction. We offer a dynamic work environment where you can grow your skills and advance your career.">This is an exciting opportunity to join our Unity Ads Data Engineering team as a...
-
Data Platform Software Engineer
4 weeks ago
San Francisco, California, United States HireIO Inc Full timeRole Summary">This position requires a highly skilled Software Development Engineer to join our Data Platform team at HireIO Inc.Key Responsibilities">">Build efficient and reliable data transformations for various use cases">Implement big data systems that meet scalability and reliability requirements">Develop and maintain high-quality software...
-
BIG Data Engineer
2 weeks ago
San Francisco, United States Saxon Global Full timemotivated Data Engineer to join our dynamic team. As a Data Engineer, you will play a critical role in developing and maintaining our data infrastructure, ensuring the smooth flow of data for our analytics and insights projects. The ideal candidate will have a strong background in SQL, data warehousing, ETL processes, and a preference for Snowflake...
-
Big Data Solutions Engineer
4 weeks ago
San Bruno, California, United States Tredence Full timeAbout the Company:Tredence is a global analytics services and solutions company that has been recognized as one of the fastest-growing private companies in the country for three consecutive years. Our unique selling proposition lies in our ability to build scalable big data solutions on-premises, GCP, or Azure clouds at a very cost-effective and easily...
-
Big Cloud | Machine Learning Engineer
2 weeks ago
san francisco, United States Big Cloud Full timeAre you passionate about building generative AI products with state-of-the art technologies?Do you thrive in an environment that combines research with hands-on product-building?Ready to work with a leading fintech platform for Silicon Valley?We’re working with a 40-person Series A start-up that’s transforming how startups manage their capital and...
-
Big Data Architect and Engineer
2 months ago
San Jose, California, United States RAMPS International Inc. Full timeAbout the Role:RAMPS International Inc. is seeking an experienced Big Data Architect and Engineer to join our team in San Jose, CA.Job Description:We are looking for a skilled professional with hands-on experience in designing, deploying, monitoring, tuning, and troubleshooting Hadoop infrastructure. The ideal candidate will have strong...
-
Senior Big Data Engineer
3 days ago
San Francisco, California, United States Plaid Inc Full timeWe're a dynamic team at Plaid Inc, focused on building innovative consumer products and developer APIs. We're seeking an experienced Data Engineer to join our team, who can help us drive business growth through data-driven decision-making.The successful candidate will have a proven track record of designing and implementing data pipelines that drive business...
-
Big Data Reliability Engineer
3 days ago
San Diego, California, United States Apple Full timeJob Overview: At Apple, we are committed to pushing the boundaries of innovation and excellence in the field of Big Data infrastructure. As a Site Reliability Engineer, you will play a critical role in designing, building, and operating scalable data systems that meet the needs of our customers.Key Responsibilities:Develop and maintain complex data...
