Hadoop Platform Engineer

1 week ago


Dallas, Texas, United States Photon Full time
Job Description


As a Hadoop Platform Engineer, you will be responsible for designing, implementing, and managing our company's Hadoop infrastructure and data ecosystem.

You will collaborate with cross-functional teams to understand data requirements, optimize data pipelines, and ensure the reliability and performance of our Hadoop clusters.

You will also be responsible for administering and monitoring the Hadoop environment, troubleshooting issues, and implementing security measures.

Responsibilities:
Design, implement, and maintain Hadoop clusters, including HDFS, YARN, and MapReduce components.

Collaborate with data engineers and data scientists to understand data requirements and optimize data pipelines for efficient processing and analysis.

Administer and monitor Hadoop clusters to ensure high availability, reliability, and performance.
Troubleshoot and resolve issues related to Hadoop infrastructure, data ingestion, data processing, and data storage.
Implement and manage security measures to protect data within the Hadoop ecosystem, including authentication, authorization, and encryption.
Collaborate with cross-functional teams to define and implement backup and disaster recovery strategies for Hadoop clusters.
Optimize Hadoop performance by fine-tuning configurations, capacity planning, and implementing performance monitoring and tuning techniques.
Work with DevOps teams to automate Hadoop infrastructure provisioning, deployment, and management processes.

Stay up to date with the latest developments in the Hadoop ecosystem and recommend and implement new technologies and tools that enhance the platform.

Document Hadoop infrastructure configurations, processes, and best practices.
Provide technical guidance and support to other team members and stakeholders.

Qualifications:


Bachelor's degree in Computer Science, Information Technology, or a related field (or equivalent work experience).Strong experience in designing, implementing, and administering Hadoop clusters in a production environment.

Proficiency in Hadoop ecosystem components such as HDFS, YARN, MapReduce, Hive, Spark, and HBase.
Experience with cluster management tools like Apache Ambari or Cloudera Manager.
Solid understanding of Linux/Unix systems and networking concepts.
Strong scripting skills (e.g., Bash, Python) for automation and troubleshooting.
Knowledge of database concepts and SQL.Experience with data ingestion tools like Apache Kafka or Apache NiFi.
Familiarity with data warehouse concepts and technologies.
Understanding of security principles and experience implementing security measures in Hadoop clusters.
Strong problem-solving and troubleshooting skills, with the ability to analyze and resolve complex issues.
Excellent communication and collaboration skills to work effectively with cross-functional teams.
Relevant certifications such as Cloudera Certified Administrator for Apache Hadoop (CCAH) or Hortonworks Certified Administrator (HCA) are a plus.

#J-18808-Ljbffr
  • Platform Engineer

    4 weeks ago


    Dallas, Texas, United States Atmosera Full time

    We are seeking a highly motivated and experienced Platform Engineer to join our Cloud Support Team. In this role, you will be an escalation point and lead to others technicians in addition to contributing to the excellence of the platform engineering team. You will be responsible for the overall health and performance of our platform, while also fostering a...

  • Platform Engineer

    3 weeks ago


    Dallas, Texas, United States Atmosera Full time

    We are seeking a highly motivated and experienced Platform Engineer to join our Cloud Support Team. In this role, you will be an escalation point and lead to others technicians in addition to contributing to the excellence of the platform engineering team. You will be responsible for the overall health and performance of our platform, while also fostering a...


  • Dallas, Texas, United States Resource Informatics Group Full time

    Position: Senior Big Data Engineer Location: Dallas, TX Duration: 12+ MonthsJob Description: Must be highly hands-on lead, writing code, leading team, reviewing other developers code, working closely with the architect to implement the proposed design 5+ years of experience writing enterprise grade Java or Scala or Python code (should be highly proficient)...


  • Dallas, Texas, United States American Airlines Full time

    Location: DFW Headquarters Building 7 (DFW-SV07) Cities: Dallas - TX Requisition ID: 73304 Job Description Intro Are you ready to explore a world of possibilities, both at work and duringyour time off? Join our American Airlines family, and you'll travel the world, grow your expertise and become the best version of you. As you embark on a new journey,...


  • Dallas, Texas, United States American Airlines Full time

    **Intro** Are you ready to explore a world of possibilities, both at work and during your time off? Join our American Airlines family, and youll travel the world, grow your expertise and become the best version of you. As you embark on a new journey, youll tackle challenges with flexibility and grace, learning new skills and advancing your career while...


  • Dallas, Texas, United States LTK USA Full time

    Title:Software Engineer, Backend Location:USA-Remote Job Purpose:Join the LTK Family As a part of the Brand Platform Engineering team, you will help us on our mission to be the largest digital marketing platform powered by influencers. Today, we're a global community of more than 700 team members, hundreds of thousands of the world's premium Creators, 7,000...


  • Dallas, Texas, United States Hireteq Solutions Inc. Full time

    We are looking for Director - Scheduling Platform Engineeringskills needed for this role - Scheduling Platform Engineering, IT Management, Leadership, Business Management, Banking, Financial Services, Wealth/Investment Management, Client Management, Data Analysis, SDLC, Agile, Mainframe Delivery, Distributed Hosting, Network, Storage, Messaging, Identity...


  • Dallas, Texas, United States TXU Energy Full time

    Job SummaryWe are seeking a highly skilled and experienced Principal Data Engineer to join our team. As a Principal Data Engineer, you will be responsible for architecting, designing, and implementing scalable and robust data solutions that enable efficient data processing, storage, and retrieval. You will provide technical leadership, drive innovation, and...


  • Dallas, Texas, United States TXU Energy Full time

    Job SummaryWe are seeking a highly skilled and experienced Principal Data Engineer to join our team. As a Principal Data Engineer, you will be responsible for architecting, designing, and implementing scalable and robust data solutions that enable efficient data processing, storage, and retrieval. You will provide technical leadership, drive innovation, and...


  • Dallas, Texas, United States Credera Experienced Hiring Job Board Full time

    ** REMOTE ROLE for Contractor anywhere in N. America **** 12 month contract and possible extension **** Rate = $80 - 90/hr USD Based On Experience Description: Data Engineers at TA Digital work closely with Subject Matter Experts (SMEs) to design the ontology (data model), develop data pipelines, and integrate Foundry with external systems containing the...

  • DevOps Engineer

    2 months ago


    Dallas, Texas, United States Binary tech Consulting corp Full time

    Job Title DevOps EngineerLocation Dallas TX / HybridDuration Longterm Contract /W2Job Description Skills Kubernetes Terraform CiCD JenkinsGood communication client management and analytical skills.Experience in Python Kubernetes OR TerraformMust be strong in following technologies.Experience in CICD setup with Jenkins GitHub and other relevant tools.Strong...

  • DevOps Engineer

    3 weeks ago


    Dallas, Texas, United States Binary tech Consulting corp Full time

    Job Title DevOps EngineerLocation Dallas TX / HybridDuration Longterm Contract /W2Job Description Skills Kubernetes Terraform CiCD JenkinsGood communication client management and analytical skills.Experience in Python Kubernetes OR TerraformMust be strong in following technologies.Experience in CICD setup with Jenkins GitHub and other relevant tools.Strong...


  • Dallas, Texas, United States Amazon Full time

    Are you a Data Analytics specialist? Do you have Data Warehousing, Hadoop/Data Lake experience? Do you like to solve the most complex and high scale (billions + records) data challenges in the world today? Do you like to work on-site in a variety of business environments, leading teams through high impact projects that use the newest data analytic...


  • Dallas, Texas, United States Enlink Full time

    Responsibilities EnLink Midstream is seeking a talented Power Platform Specialist that will be responsible for developing solutions using Microsoft Power Platform components, such as Power Apps and Power Automate. This individual supports EnLink's IT team in crafting and optimizing enterprise-level solutions using the Power Platform, catering to a diverse...


  • Dallas, Texas, United States Apptad Inc Full time

    Data Engineering Architect PositionLocation: Dallas, TX / Pittsburgh, PA (Hybrid)Job Type: Full-timeCompany Overview: We are currently seeking a Data Architect with experience in Neo4J to join our team. As a Data Engineering Architect, you will play a crucial role in developing data models, building data pipelines, and maintaining Neo4j servers. This is a...


  • Dallas, Texas, United States Foot Locker Full time

    Foot Locker is looking for a Technical Product Manager specializing in the Internal Developer Platform in Dallas. This role involves setting the strategic direction for services within an internal developer platform (IDP) by leveraging a deep understanding of modern application development. Collaborating with business product engineering teams, this position...

  • Data Engineer

    1 week ago


    Dallas, Texas, United States InterSources Full time

    Job Title: Data Engineer (GCP)Job Location: Dallas, TXJob Type: FulltimeJob Mode: OnsiteRequired Skills:4 + years of knowledge in Big Data environmentGCP Big Query, Hadoop architecture, HDFS commands, and designing and optimizing queries to build data pipelines.3 + years of experience in building data marts and data models to support Data Science and other...


  • Dallas, Texas, United States Tek Ninjas consulting services Full time

    Tekninjas solutions LLC is an E-Verified IT company. We are specialized in providing software development and consulting services. We have excellent training programs for individuals who are looking to start their career in the IT industry or individuals who would like to make a career change. In Short we position you for long-term success and on a solid...

  • Big Data Architect

    1 week ago


    Dallas, Texas, United States Resource Informatics Group Inc Full time

    Job DescriptionJob DescriptionRole: Senior Big Data ArchitectLocation: Dallas, TXDuration: 3 months Rates: All-InclusiveLocals prefer Another requirement should be a solid understanding of traditional DW/BI environments and how to migrate them to a Cloudera Hadoop based data lake environment. Big Data Architect SkillsExperience in working with Cloudera Big...

  • Big Data Architect

    1 week ago


    Dallas, Texas, United States Resource Informatics Group Full time

    Job Description Job Description Role: Senior Big Data ArchitectLocation: Dallas, TXDuration: 3 monthsRates: All-InclusiveLocals prefer Another requirement should be a solid understanding of traditional DW/BI environments and how to migrate them to a Cloudera Hadoop based data lake environment.Big Data Architect SkillsExperience in working with Cloudera Big...