Big Data Engineer

2 days ago

San Francisco CA United States Unreal Gigs Full time

Are you passionate about handling massive datasets and building the infrastructure that enables complex data analysis and machine learning at scale? Do you excel in creating robust, scalable data pipelines that fuel data-driven decision-making? If you’re ready to tackle the challenges of big data, our client has the perfect role for you. We’re seeking a Big Data Engineer (aka The Data Pipeline Innovator) to architect and maintain high-performance data systems that empower analytics and support advanced data processing needs.

As a Big Data Engineer at our client , you’ll collaborate with data scientists, analysts, and software engineers to design, implement, and optimize big data platforms. Your expertise in data engineering, distributed systems, and cloud infrastructure will be critical to ensuring that our data ecosystem is efficient, reliable, and scalable.

Key Responsibilities:

  1. Design and Build Scalable Data Pipelines:
  • Architect and implement data pipelines for ETL processes using tools like Apache Spark, Kafka, and Hadoop. You’ll create data workflows that handle high-volume, high-velocity data and ensure seamless integration across systems.
Optimize Big Data Storage and Processing:
  • Develop and manage data storage solutions (e.g., HDFS, S3, Cassandra) that are optimized for performance and cost-efficiency. You’ll configure distributed processing systems to support efficient data retrieval and transformation.
Collaborate on Data Strategy and Integration:
  • Work closely with data scientists, analysts, and other engineers to align big data architecture with analytics goals. You’ll ensure data availability and integrity across systems to support business objectives.
Implement Data Quality and Governance Standards:
  • Develop processes and tools to monitor data quality and enforce data governance policies. You’ll ensure data is accurate, reliable, and secure through regular checks and validation processes.
Enhance Data Processing with Automation:
  • Use tools like Apache Airflow or AWS Glue to automate data workflows and reduce manual processing. You’ll implement scripts and automation that streamline data handling and improve efficiency.
Monitor and Troubleshoot Data Systems:
  • Use monitoring tools to track system performance and address issues proactively. You’ll troubleshoot and resolve any bottlenecks or failures to maintain optimal data processing capabilities.
Stay Updated on Big Data Trends and Technologies:
  • Keep up with advancements in big data technologies and tools. You’ll integrate new techniques and platforms that align with business needs and promote innovation.

Required Skills:

  • Big Data Platform Proficiency: Extensive experience with big data technologies such as Apache Spark, Hadoop, Kafka, and Hive. You’re skilled at handling high-volume data and distributed processing.
  • Data Pipeline and ETL Knowledge: Proven ability to design, build, and maintain ETL processes for massive datasets. You can handle both real-time and batch data processing requirements.
  • Programming and Scripting: Proficiency in programming languages like Python, Java, or Scala for data processing and automation. Experience with SQL for data querying and manipulation is essential.
  • Cloud Data Services Expertise: Familiarity with cloud platforms such as AWS, GCP, or Azure, including their big data and storage services (e.g., S3, BigQuery, Azure Data Lake).
  • Data Quality and Governance: Strong understanding of data quality standards and governance practices, with experience in implementing data validation and monitoring frameworks.

Educational Requirements:

  • Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or a related field. Equivalent experience in data engineering or big data management may be considered.
  • Certifications in big data or cloud technologies (e.g., Cloudera Certified Data Engineer, AWS Certified Big Data – Specialty, Google Professional Data Engineer) are a plus.

Experience Requirements:

  • 5+ years of experience in data engineering, with at least 3+ years focusing on big data technologies and high-scale data environments .
  • Experience in distributed systems and large-scale data storage management.
  • Familiarity with containerization (Docker, Kubernetes) for deploying data processing environments is advantageous.
  • Health and Wellness: Comprehensive medical, dental, and vision insurance plans with low co-pays and premiums.
  • Paid Time Off: Competitive vacation, sick leave, and 20 paid holidays per year.
  • Work-Life Balance: Flexible work schedules and telecommuting options.
  • Professional Development: Opportunities for training, certification reimbursement, and career advancement programs.
  • Wellness Programs: Access to wellness programs, including gym memberships, health screenings, and mental health resources.
  • Life and Disability Insurance: Life insurance and short-term/long-term disability coverage.
  • Employee Assistance Program (EAP): Confidential counseling and support services for personal and professional challenges.
  • Tuition Reimbursement: Financial assistance for continuing education and professional development.
  • Community Engagement: Opportunities to participate in community service and volunteer activities.
  • Recognition Programs: Employee recognition programs to celebrate achievements and milestones.
#J-18808-Ljbffr

  • Lead Big Data Engineer

    1 day ago

    San Francisco, CA, United States Unreal Gigs Full time

    DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to leveraging the power of big data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge big data solutions that enable advanced analytics, machine learning, and business intelligence. Join...

  • Big Data Engineer

    21 hours ago

    San Francisco, CA, United States ZipRecruiter Full time

    Job Description Are you passionate about handling massive datasets and building the infrastructure that enables complex data analysis and machine learning at scale? Do you excel in creating robust, scalable data pipelines that fuel data-driven decision-making? If you’re ready to tackle the challenges of big data, our client has the perfect role for you....

  • Lead Big Data Engineer

    3 weeks ago

    san francisco, United States Unreal Gigs Full time

    DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to leveraging the power of big data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge big data solutions that enable advanced analytics, machine learning, and business intelligence. Join...

  • Big Data Engineer

    3 days ago

    San Francisco, United States ZipRecruiter Full time

    Job Description Are you passionate about handling massive datasets and building the infrastructure that enables complex data analysis and machine learning at scale? Do you excel in creating robust, scalable data pipelines that fuel data-driven decision-making? If you’re ready to tackle the challenges of big data, our client has the perfect role for you....

  • Big Data Engineer

    2 weeks ago

    San Francisco, United States ZipRecruiter Full time

    Job DescriptionAre you passionate about handling massive datasets and building the infrastructure that enables complex data analysis and machine learning at scale? Do you excel in creating robust, scalable data pipelines that fuel data-driven decision-making? If you’re ready to tackle the challenges of big data, our client has the perfect role for you....

  • Big Data Engineer

    4 weeks ago

    San Francisco, United States Unreal Gigs Full time

    Are you passionate about handling massive datasets and building the infrastructure that enables complex data analysis and machine learning at scale? Do you excel in creating robust, scalable data pipelines that fuel data-driven decision-making? If you’re ready to tackle the challenges of big data, our client has the perfect role for you. We’re seeking a...

  • Senior Data Engineer

    1 month ago

    Sunnyvale, CA, United States Big Cloud Full time

    Would you like to shape a computer vision platform at an exciting start-up?Are you interested in a cross-functional team working across AI and 3D generative video?This start-up is revolutionizing video through AI-powered virtual environments. The team are architecting early-stage cutting-edge generative computer vision technology for iOS and web platforms...

  • Palantir Big Data Engineer

    4 weeks ago

    Rosemead, CA, United States APR Consulting Full time

    An electric utility client is looking for a Palantir Big Data Engineer who is responsible for developing and maintaining data ingestion and consumption pipelines for Palantir, deriving data insights using platform capabilities toward the realization of analytics and data science use cases.Location: Rosemead, CA 91770 (Hybrid)Position: Palantir Big Data...

  • Palantir Big Data Engineer

    4 weeks ago

    Rosemead, CA, United States APR Consulting Full time

    An electric utility client is looking for a Palantir Big Data Engineer who is responsible for developing and maintaining data ingestion and consumption pipelines for Palantir, deriving data insights using platform capabilities toward the realization of analytics and data science use cases.Location: Rosemead, CA 91770 (Hybrid)Position: Palantir Big Data...

  • Senior Data Engineer for Big Data Platforms

    3 days ago

    San Francisco, California, United States TEKsystems Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team in supporting our data platform.This is a 6-month contract-to-hire opportunity with excellent growth prospects.Key ResponsibilitiesSupport our data platform, responsible for collecting, storing, processing, and analyzing vast amounts of data across the organization.Work...

  • Senior Big Data Engineer

    4 weeks ago

    Irving, TX, United States Anblicks Full time

    Description: The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data...

  • Senior Software Development Engineer

    3 days ago

    San Francisco, United States HireIO Inc Full time

    Team Introduction The Data Platform team works on building data infrastructures and data products to support business engineering teams. As a Software Development Engineer in the data platform team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the opportunity to gain hands-on experience...

  • Senior Software Development Engineer

    4 months ago

    San Francisco, United States HireIO Inc Full time

    Team Introduction The Data Platform team works on building data infrastructures and data products to support business engineering teams. As a Software Development Engineer in the data platform team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the opportunity to gain hands-on experience...

  • Senior Software Development Engineer

    2 weeks ago

    San Francisco, United States CV Library Full time

    Team IntroductionThe Data Platform team works on building data infrastructures and data products to support business engineering teams.As a Software Development Engineer in the data platform team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll gain hands-on experience on all kinds of systems...

  • Senior Software Development Engineer

    2 weeks ago

    San Francisco, United States ZipRecruiter Full time

    Job DescriptionTeam IntroductionThe Data Platform team works on building data infrastructures and data products to support business engineering teams.As a Software Development Engineer in the data platform team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll gain hands-on experience on various...

  • Big Data Platform Staff Engineer

    2 days ago

    Santa Clara, CA, United States Integrated Resources Inc. Full time

    Our mission is to service with Integrity as both an employer of choice to our associates and a strategic partner to our clients. IRI’s vision is to become an industry-leading staffing services organization by maintaining ethical business practices, a passion for customer service, a commitment to quality, and our continued efforts to exceed expectations. ...

  • Big Data Developer

    4 months ago

    San Francisco, United States Sigmaways Inc Full time

    In this role, you can contribute to several high-quality data solutions and enhance your technical skills across many disciplines.Key ResponsibilitiesDesign, develop, and maintain end-to-end data solutions using open source, modern data lake, and enterprise data warehouse technologies (Hadoop, Spark, Cloud, etc.)Contribute to multiple data solutions...

  • Big Data Engineering Leader Opportunity

    1 week ago

    MS, United States Conviva Full time

    Job OverviewWe are seeking a seasoned Big Data Engineering leader to drive the development and deployment of our highly scalable, modular, and extensible Big Data platform. This individual will play a crucial role in driving innovation and supporting high-growth business by designing and developing solutions using mature and emerging big data storage...

  • S Slalom Data Engineer- Senior Consultant

    20 hours ago

    Atlanta, GA, United States Data Engineer Jobs Full time

    *Please note: This role is not eligible for 100% remote work. Employees must live within a commutable distance of the Atlanta Area and must be willing to be onsite at the client and/or Slalom Atlanta office up to 5 days a week.* Who You'll Work With As a modern technology company, our Slalom Technologists are disrupting the market and bringing to life the...

  • Senior Big Data Solutions Architect

    7 days ago

    San Francisco, California, United States Coatue Management L.L.C. Full time

    Job Title: Senior Big Data Solutions ArchitectAbout the Role:We are seeking an experienced Senior Big Data Solutions Architect to join our team at Coatue Management L.L.C. This is a unique opportunity to work with a leading investment firm and contribute to the design and implementation of big data solutions.Responsibilities:Design and implement scalable big...