Big Data Engineer with cloud experience

4 weeks ago


Austin, United States Diverse Lynx Full time
Title: Big Data Engineer with cloud experience
Location: Sunnyvale, CA/Austin, TX(Onsite)
Experience: 10+ Years

Technical/Functional Skills: Big Data, Tableau, Kafka, Iceberg, Spark

Roles & Responsibilities:
  • Having 10 years of professional experience fields of software Analysis, Design, Development, Deployment and Maintenance of software and Big Data applications.
  • Experience in Big data Implementation with strong experience on major components of Iceberg, Tableau, Kafka, Superset, Druid, Hive metastore, apache, ranger, security, AWS
  • Experience in creating iceberg tables and loading the data from different file formats.
  • Good Experience in Data importing and exporting to Hive and HDFS with Sqoop.
  • Experience in using Producer and Consumer APIs of Apache Kafka.
  • Skilled in integrating Kafka with Spark streaming for faster data processing.
  • Experience in using Spark Streaming programming model for Real-time data processing.
  • Experience dealing with the file formats like text files, Sequence files, JSON, Parquet, ORC.
  • Extensively used Apache Kafka to collect the logs and error messages across the cluster.
  • Excellent knowledge and understanding of Distributed Computing and Parallel processing frameworks.
  • Experienced with Analytics with Hive Megastore.
  • Experience with Superset, Druid.
  • Experience working with EC2 (Elastic Compute Cloud) cluster instances, setup data buckets on S3 (Simple Storage Service), setting up EMR (Elastic MapReduce).
  • Good experience working on Tableau and enabled the JDBC/ODBC data connectivity from those to Hive Metastore.
  • Good with version control systems like GIT.
  • Strong knowledge on UNIX/LINUX commands.
  • Adequate Knowledge on Python scripting Language.
  • Adequate knowledge of Scrum, Agile and Waterfall methodologies.
  • Highly motivated and committed to the highest levels of professionalism.
  • Exhibited strong written and oral communication skills. Rapidly learn and adapt quickly to emerging new technologies and paradigms.


Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.

  • Austin, United States Diverse Lynx Full time

    Title: Big Data Engineer with cloud experience Location: Sunnyvale, CA/Austin, TX(Onsite) Experience: 10+ Years Technical/Functional Skills: Big Data, Tableau, Kafka, Iceberg, Spark Roles & Responsibilities: Having 10 years of professional experience fields of software Analysis, Design, Development, Deployment and Maintenance of software and Big Data...

  • Cloud Data Engineer

    3 days ago


    Austin, Texas, United States Apple Full time

    SummaryPosted: May 24, 2024Role Number: The people here at Apple don't just create products - they create the kind of wonder that's revolutionized entire industries. It's the diversity of those people and their ideas that inspires the innovation that runs through everything we do, from amazing technology to industry-leading environmental efforts. Apple is...


  • Austin, United States Diverse Lynx Full time

    Role: Big Data Platform Administration Location: Austin, TX/ Dallas, TX (Onsite) Contract Skills - Platform: HPE/MapR Hadoop (Big Data), Programming: Python, UNIX Scripting CICD: Ansible, Platform Support, ITSM tools knowledge, ITIL 2 12 Years of IT experience in Big Data, Hadoop platform management & min 2 years experience in Big Data platform...

  • Systems Engineer

    3 days ago


    Austin, United States Hudson River Trading Full time

    Hudson River Trading (HRT) is looking for Systems Engineers to join our growing Research & Development team. This team builds and maintains an exceptionally large and growing distributed compute cluster, many-PB-scale storage systems, operating systems, automation software, and development tools. We are seeking a highly skilled engineer with expertise in...

  • Big Data Engineer

    2 weeks ago


    Austin, United States ShiftCode Analytics Full time

    Interview : Video Visa : All apart from H1b , cpt and TN This is hybrid from day-1. Description : Requirements: 5+ years' experience in data engineering work Previous experience as a Big Data Engineer In-depth knowledge of Hadoop, Spark, Airflow Technical expertise with data models, data mining, and segmentation techniques Hands-on experience with SQL...


  • Austin, United States Emerald Cloud Lab Full time

    Job DescriptionJob DescriptionSalary: The Emerald Cloud Laboratory (ECL) enables life scientists to move out of the lab, and to conduct research entirely from a computer. Stepping away from manual completion of experiments at the bench, scientists on the ECL leverage the remote, automated execution of all standard biology and chemistry experiments in...


  • Austin, Texas, United States Zello Full time

    IMPORTANT: Please be aware, scammers may try to impersonate Zello by reaching out regarding job opportunities. We will never ask you for bank account information, checks, or other sensitive information as part of our hiring process. All correspondence will come from the email domain. If you're unsure, please email with questions. About ZelloZello is a...

  • Azure Data Engineer

    2 weeks ago


    Austin, United States Epicor Inc Full time

    Epicor is seeking a highly skilled Azure Data Engineer to join our dynamic team. As Data Engineer, you will be responsible for overseeing all aspects of our Azure data processing pipelines, including development, implementation, and testing. In addition, you will work closely with our business functions and BI team to identify opportunities to optimize their...


  • Austin, United States snipeHIRE Full time

    Job DescriptionJob DescriptionCOMPANYOur client specializes in digital enablement and digital transformation solutions for Fortune companies. Using industry-leading data engineering, process mining and automation platforms, they help large enterprises become more process-efficient and improve their end-customer experience by incorporating the use of RPA, AI...


  • Austin, United States snipeHIRE Full time

    Job DescriptionJob DescriptionCOMPANYOur client specializes in digital enablement and digital transformation solutions for Fortune companies. Using industry-leading data engineering, process mining and automation platforms, they help large enterprises become more process-efficient and improve their end-customer experience by incorporating the use of RPA, AI...

  • Solutions Engineer

    5 days ago


    Austin, United States Wiz, Inc. Full time

    Come join the company that is reinventing cloud security and empowering businesses to thrive in the cloud. As the fastest-growing startup ever, Wiz is on a mission to help organizations secure cloud environments that will accelerate their businesses. Trusted by security teams all over the world, we have a proven track record of success and a culture that...

  • Solutions Engineer

    5 days ago


    Austin, United States Wiz, Inc. Full time

    Come join the company that is reinventing cloud security and empowering businesses to thrive in the cloud. As the fastest-growing startup ever, Wiz is on a mission to help organizations secure cloud environments that will accelerate their businesses. Trusted by security teams all over the world, we have a proven track record of success and a culture that...

  • Sr Data Engineer

    5 days ago


    Austin, Texas, United States Horizontal Talent Full time

    Join our dynamic team as we seek a Senior Data Engineer with a focus on finance to bolster our client's Data and Analytics division. This role offers an exhilarating chance to contribute to the evolution and modernization of analytics We're on the lookout for individuals with exceptional communication skills, adept at engaging with stakeholders at all levels...


  • Austin, Texas, United States teamworkonline Full time

    Who We Are:Endeavor is a global sports and entertainment company, home to many of the world's most dynamic and engaging storytellers, brands, live events, and experiences. The Endeavor network specializes in talent representation through entertainment agency WME; sports operations and advisory, event management, media production and distribution, and brand...

  • BI/DW SAP Architect

    4 days ago


    Austin, United States Cloud Analytics Technologies LLC Full time

    Local Candidates Preferred. Non-local candidates must be willing to pay for your own interview travel expenses and relocation costs **Candidates submitted over the max. bill rate will be automatically disqualified and counted as a submittal. For each position CBD allows only 2 (two) submittals from our sub-vendors/partners. So please send us only the exact...

  • Senior Data Engineer

    1 month ago


    Austin, United States 2K Full time

    Who We Are 2K is headquartered in Novato, California and is a wholly owned label of Take-Two Interactive Software, Inc. (NASDAQ: TTWO). Founded in 2005, 2K Games is a global video game company, publishing titles developed by some of the most influential game development studios in the world. Our studios responsible for developing 2K’s portfolio of...

  • System Analyst

    5 days ago


    Austin, United States Cloud Analytics Technologies LLC Full time

    Work Authorization: US Citizen, Green Card, H-1B, GC-EAD, OPT-EAD, L2-EAD, TN Visa US citizens (USC) and Permanent Residents (GC) are encouraged to apply. We are unable to offer any sort of Visa sponsorship for this position. Local Candidates Preferred. Non-local candidates must be willing to pay for your own interview travel expenses and relocation...

  • AWS API Data Engineer

    2 weeks ago


    Austin, United States JobRialto Full time

    Job Responsibilities: Develop, deploy, and maintain cloud-based APIs using AWS services. Design and implement RESTful APIs using Lambda functions, Python, and API Gateway. Collaborate with cross-functional teams to gather requirements and define API specifications. Ensure high availability, scalability, and security of APIs in the cloud environment....

  • Software Engineer

    3 days ago


    Austin, Texas, United States Apple Full time

    SummaryPosted: Apr 25, 2024Role Number: Do you love understanding every detail of how new technologies work? Join the team that serves as Apple's nerve center, our Information Systems and Technology group. There are countless ways you'll contribute here, whether you're coordinating technology needs for product launches, designing music solutions for retail...


  • Austin, United States H-E-B Full time

    Responsibilities: Since H-E-B Digital Technology's inception, we've been investing heavily in our customers' digital experience, reinventing how they find inspiration from food, how they make food decisions, and how they ultimately get food into their homes. This is an exciting time to join H-E-B Digital--we're using the best available technologies to...