Senior Big Data Infrastructure Administrator

1 week ago

Santa Clara, CA, United States InfoCepts Full time

Big Data Infrastructure Administrator

About Our Team:

The Big Data Operations team is dedicated to empowering organizations to leverage the potential of Big Data through a robust and dependable platform. We are in the process of developing an advanced Big Data infrastructure on AWS while ensuring the scalability and maintenance of our existing systems in the data centers. Our responsibilities include capacity planning, security, and disaster recovery for the innovative platform on AWS. We aim to enhance visibility into the operational metrics of our Big Data environment by collecting logs and metrics from various sources and establishing alarms to proactively identify issues.

Your Responsibilities:

  • Design, build, scale, and maintain the infrastructure in both data center and AWS environments for Big Data applications.
  • Oversee the complete availability of the Big Data platform in AWS and data centers.
  • Enhance the efficiency, reliability, and security of our Big Data infrastructure, ensuring a seamless experience for developers and analysts.
  • Implement automation for the establishment and maintenance of the new platform on AWS.
  • Create custom tools for the automation of daily operational tasks.
  • Set operational standards for our production environment.
  • Participate in a 24x7 on-call rotation with the team, addressing alerts and investigating platform issues.

Required Skills:

  • Extensive experience with the Hadoop ecosystem, including HDFS, Yarn, Hive, Spark, Oozie, Presto, and Ranger.
  • Proven expertise with Amazon EMR.
  • Solid experience with RDS and a good understanding of IaaS and PaaS.
  • Strong knowledge of Hadoop security, including SSL/TSL, Kerberos, and role-based authorization.
  • Performance tuning experience with Hadoop clusters, ecosystem components, and MR/Spark jobs.
  • Experience in infrastructure automation using Terraform, CI/CD pipelines (GIT, Jenkins, etc.), and configuration management tools like Ansible.
  • Proficiency in leveraging technologies such as Kubernetes/Docker (ELK) to support Data Engineers and Developers in creating innovative products.
  • Experience with monitoring solutions utilizing logs through CloudWatch, CloudTrail, and Lambda.
  • Capability to conduct post-mortem analyses of system failures, identifying root causes and generating detailed reports.
  • Proficiency in Bash and Python or Java.
  • Good understanding of JRE/JVM and garbage collection tuning.
  • Hands-on experience with RDBMS (Oracle, MySQL) and basic SQL.
  • Familiarity with Snowflake is a plus.
  • Experience with Qubole and Airflow is advantageous.

  • Infrastructure System Administrator

    3 days ago

    Santa Clara, CA, United States Sustainable Talent Full time

    Sustainable Talent is partnering with Nvidia a global leader who's been transforming computer graphics, PC gaming, and accelerated computing for over 25 years. We are looking for an Infrastructure System Administrator to support our client's Infrastructure team. This is a W-2 full-time contract based onsite in Santa Clara, CA. We offer competitive pay $70/hr...

  • Infrastructure System Administrator

    5 days ago

    Santa Clara, CA, United States Sustainable Talent Full time

    Sustainable Talent is partnering with Nvidia a global leader who's been transforming computer graphics, PC gaming, and accelerated computing for over 25 years. We are looking for an Infrastructure System Administrator to support our client's Infrastructure team. This is a W-2 full-time contract based onsite in Santa Clara, CA. We offer competitive pay $70/hr...

  • Big Data Engineer

    3 days ago

    Santa Monica, CA, United States KellyMitchell Group Full time

    Job Summary: Our client is seeking a Big Data Engineer to join their team! This position is located in Santa Monica, California. Duties: Design and build core components of a large-scale data platform for both real-time and batch processing, owning key features of big data applications that evolve with business needs Develop next-generation, cloud-based...

  • Big Data Engineer

    3 days ago

    Santa Monica, CA, United States KellyMitchell Group Full time

    Job Summary: Our client is seeking a Big Data Engineer to join their team! This position is located in Santa Monica, California. Duties: Design and build core components of a large-scale data platform for both real-time and batch processing, owning key features of big data applications that evolve with business needs Develop next-generation, cloud-based...

  • Principal Big Data Engineer

    3 days ago

    Santa Clara, CA, United States Right Skale, Inc. Full time

    Principal Big Data Engineer - Extendible Contract - Onsite (Santa Clara, CA) This role will be available in Santa Clara, CA, and local candidates are strongly preferred. About you: As a Principal Big Data Engineer, you will be an integral member of our data ingestion and processing platform team responsible for architecture, design, and development. Key...

  • Principal Big Data Engineer

    1 day ago

    Santa Clara, CA, United States Right Skale, Inc. Full time

    Principal Big Data Engineer - Extendible Contract - Onsite (Santa Clara, CA) This role will be available in Santa Clara, CA, and local candidates are strongly preferred. About you: As a Principal Big Data Engineer, you will be an integral member of our data ingestion and processing platform team responsible for architecture, design, and development. Key...

  • Principal Big Data Engineer

    5 days ago

    Santa Clara, CA, United States Right Skale, Inc. Full time

    Principal Big Data Engineer - Extendible Contract - Onsite (Santa Clara, CA) This role will be available in Santa Clara, CA, and local candidates are strongly preferred. About you: As a Principal Big Data Engineer, you will be an integral member of our data ingestion and processing platform team responsible for architecture, design, and development. Key...

  • Principal Big Data Engineer

    4 days ago

    Santa Clara, CA, United States Right Skale, Inc. Full time

    Principal Big Data Engineer - Extendible Contract - Onsite (Santa Clara, CA) This role will be available in Santa Clara, CA, and local candidates are strongly preferred. About you: As a Principal Big Data Engineer, you will be an integral member of our data ingestion and processing platform team responsible for architecture, design, and development. Key...

  • Senior Software Engineer, Big Data

    1 week ago

    Santa Monica, CA, United States ZipRecruiter Full time

    We offer a hybrid work environment. Most US-based positions can alsobeperformed remotely (any exceptions will be noted in the Minimum Qualifications below.) Our Mission: To actively connect people to their next great opportunity. Who We Are: ZipRecruiter is a leading online employment marketplace. Powered by AI-driven smart matching technology, the...

  • Senior Infrastructure Engineer

    3 days ago

    Santa Clara, CA, United States Sustainable Talent Full time

    Join the Sustainable Talent team, supporting NVIDIA as a Senior Infrastructure Engineer supporting the IPP (Infrastructure, Planning and Process) Cloud Infrastructure Team. This is a W-2 full-time 1 year contract based in Santa Clara, CA. We offer competitive pay $80- $100/hr based on factors like experience, education, location, etc. and provide full...