Big Data Engineer with cloud experience

4 weeks ago

Austin, United States Diverse Lynx Full time
Title: Big Data Engineer with cloud experience
Location: Sunnyvale, CA/Austin, TX(Onsite)
Experience: 10+ Years

Technical/Functional Skills: Big Data, Tableau, Kafka, Iceberg, Spark

Roles & Responsibilities:
  • Having 10 years of professional experience fields of software Analysis, Design, Development, Deployment and Maintenance of software and Big Data applications.
  • Experience in Big data Implementation with strong experience on major components of Iceberg, Tableau, Kafka, Superset, Druid, Hive metastore, apache, ranger, security, AWS
  • Experience in creating iceberg tables and loading the data from different file formats.
  • Good Experience in Data importing and exporting to Hive and HDFS with Sqoop.
  • Experience in using Producer and Consumer APIs of Apache Kafka.
  • Skilled in integrating Kafka with Spark streaming for faster data processing.
  • Experience in using Spark Streaming programming model for Real-time data processing.
  • Experience dealing with the file formats like text files, Sequence files, JSON, Parquet, ORC.
  • Extensively used Apache Kafka to collect the logs and error messages across the cluster.
  • Excellent knowledge and understanding of Distributed Computing and Parallel processing frameworks.
  • Experienced with Analytics with Hive Megastore.
  • Experience with Superset, Druid.
  • Experience working with EC2 (Elastic Compute Cloud) cluster instances, setup data buckets on S3 (Simple Storage Service), setting up EMR (Elastic MapReduce).
  • Good experience working on Tableau and enabled the JDBC/ODBC data connectivity from those to Hive Metastore.
  • Good with version control systems like GIT.
  • Strong knowledge on UNIX/LINUX commands.
  • Adequate Knowledge on Python scripting Language.
  • Adequate knowledge of Scrum, Agile and Waterfall methodologies.
  • Highly motivated and committed to the highest levels of professionalism.
  • Exhibited strong written and oral communication skills. Rapidly learn and adapt quickly to emerging new technologies and paradigms.


Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.

  • Cloud Data Engineer

    1 week ago

    Austin, Texas, United States Apple Full time

    SummaryPosted: May 24, 2024Role Number: The people here at Apple don't just create products - they create the kind of wonder that's revolutionized entire industries. It's the diversity of those people and their ideas that inspires the innovation that runs through everything we do, from amazing technology to industry-leading environmental efforts. Apple is...

  • Senior Big Data Engineer

    2 days ago

    Austin, United States Grid Dynamics Full time

    Description Position at Grid Dynamics Our customer is an American multinational technology company headquartered in Cupertino, California. Our customer is one of the world's largest technology companies based in Silicon Valley with operations all over the world. On this project, we are working with bleeding-edge big data technologies to develop a...

  • Big Data Platform Administration

    3 weeks ago

    Austin, United States Diverse Lynx Full time

    Role: Big Data Platform Administration Location: Austin, TX/ Dallas, TX (Onsite) Contract Skills - Platform: HPE/MapR Hadoop (Big Data), Programming: Python, UNIX Scripting CICD: Ansible, Platform Support, ITSM tools knowledge, ITIL 2 12 Years of IT experience in Big Data, Hadoop platform management & min 2 years experience in Big Data platform...

  • Big Data Platform Administration

    2 days ago

    Austin, United States Diverse Lynx Full time

    Role: Big Data Platform Administration Location: Austin, TX/ Dallas, TX (Onsite) Contract Skills - Platform: HPE/MapR Hadoop (Big Data), Programming: Python, UNIX Scripting CICD: Ansible, Platform Support, ITSM tools knowledge, ITIL 2 12 Years of IT experience in Big Data, Hadoop platform management & min 2 years experience in Big Data platform...

  • Big Data Platform Administration

    3 days ago

    Austin, United States Diverse Lynx Full time

    Role: Big Data Platform Administration Location: Austin, TX/ Dallas, TX (Onsite) Contract Skills - Platform: HPE/MapR Hadoop (Big Data), Programming: Python, UNIX Scripting CICD: Ansible, Platform Support, ITSM tools knowledge, ITIL 2 12 Years of IT experience in Big Data, Hadoop platform management & min 2 years experience in Big Data platform...

  • Systems Engineer

    1 week ago

    Austin, United States Hudson River Trading Full time

    Hudson River Trading (HRT) is looking for Systems Engineers to join our growing Research & Development team. This team builds and maintains an exceptionally large and growing distributed compute cluster, many-PB-scale storage systems, operating systems, automation software, and development tools. We are seeking a highly skilled engineer with expertise in...

  • Big Data Engineer

    3 weeks ago

    Austin, United States ShiftCode Analytics Full time

    Interview : Video Visa : All apart from H1b , cpt and TN This is hybrid from day-1. Description : Requirements: 5+ years' experience in data engineering work Previous experience as a Big Data Engineer In-depth knowledge of Hadoop, Spark, Airflow Technical expertise with data models, data mining, and segmentation techniques Hands-on experience with SQL...

  • Big Data Engineer

    2 days ago

    Austin, United States ShiftCode Analytics Full time

    Interview : Video Visa : All apart from H1b , cpt and TN This is hybrid from day-1. Description : Requirements: 5+ years' experience in data engineering work Previous experience as a Big Data Engineer In-depth knowledge of Hadoop, Spark, Airflow Technical expertise with data models, data mining, and segmentation techniques Hands-on experience with SQL...

  • Staff Platform Engineer

    3 days ago

    Austin, United States Emerald Cloud Lab Full time

    The Emerald Cloud Laboratory (ECL) enables life scientists to move out of the lab, and to conduct research entirely from a computer. Stepping away from manual completion of experiments at the bench, scientists on the ECL leverage the remote, automated execution of all standard biology and chemistry experiments in Emerald's industrial lab facilities, working...

  • Staff Platform Engineer

    2 days ago

    Austin, United States Emerald Cloud Lab Full time

    The Emerald Cloud Laboratory (ECL) enables life scientists to move out of the lab, and to conduct research entirely from a computer. Stepping away from manual completion of experiments at the bench, scientists on the ECL leverage the remote, automated execution of all standard biology and chemistry experiments in Emerald's industrial lab facilities, working...

  • Senior Data Engineer

    2 days ago

    Austin, United States Saxon Global Full time

    SUMMARY: We are looking for passionate data engineers to join our growing Data and Analytics team. Join this exciting journey in modernizing our Advisor solutions to the next-generation cloud platform. You will be responsible for working in a cross-functional team to expand, optimize, and improve overall data quality and set up next generation data...

  • Staff Platform Engineer

    2 weeks ago

    Austin, United States Emerald Cloud Lab Full time

    Job DescriptionJob DescriptionSalary: The Emerald Cloud Laboratory (ECL) enables life scientists to move out of the lab, and to conduct research entirely from a computer. Stepping away from manual completion of experiments at the bench, scientists on the ECL leverage the remote, automated execution of all standard biology and chemistry experiments in...

  • Staff Platform Engineer

    2 days ago

    Austin, United States Emerald Cloud Lab Full time

    Job DescriptionJob DescriptionSalary: The Emerald Cloud Laboratory (ECL) enables life scientists to move out of the lab, and to conduct research entirely from a computer. Stepping away from manual completion of experiments at the bench, scientists on the ECL leverage the remote, automated execution of all standard biology and chemistry experiments in...

  • Senior Cloud Engineer

    2 weeks ago

    Austin, Texas, United States Zello Full time

    IMPORTANT: Please be aware, scammers may try to impersonate Zello by reaching out regarding job opportunities. We will never ask you for bank account information, checks, or other sensitive information as part of our hiring process. All correspondence will come from the email domain. If you're unsure, please email with questions. About ZelloZello is a...

  • Senior Cloud Engineer

    2 days ago

    Austin, Texas, United States Zello Full time

    IMPORTANT: Please be aware, scammers may try to impersonate Zello by reaching out regarding job opportunities. We will never ask you for bank account information, checks, or other sensitive information as part of our hiring process. All correspondence will come from the email domain. If you're unsure, please email with questions. About ZelloZello is a...

  • Azure Data Engineer

    3 weeks ago

    Austin, United States Epicor Inc Full time

    Epicor is seeking a highly skilled Azure Data Engineer to join our dynamic team. As Data Engineer, you will be responsible for overseeing all aspects of our Azure data processing pipelines, including development, implementation, and testing. In addition, you will work closely with our business functions and BI team to identify opportunities to optimize their...

  • Sr. Cloud Data Architect

    1 month ago

    Austin, United States snipeHIRE Full time

    Job DescriptionJob DescriptionCOMPANYOur client specializes in digital enablement and digital transformation solutions for Fortune companies. Using industry-leading data engineering, process mining and automation platforms, they help large enterprises become more process-efficient and improve their end-customer experience by incorporating the use of RPA, AI...

  • Sr. Cloud Data Architect

    1 month ago

    Austin, United States snipeHIRE Full time

    Job DescriptionJob DescriptionCOMPANYOur client specializes in digital enablement and digital transformation solutions for Fortune companies. Using industry-leading data engineering, process mining and automation platforms, they help large enterprises become more process-efficient and improve their end-customer experience by incorporating the use of RPA, AI...

  • Sr. Cloud Data Architect

    2 days ago

    Austin, United States snipeHIRE Full time

    Job DescriptionJob DescriptionCOMPANYOur client specializes in digital enablement and digital transformation solutions for Fortune companies. Using industry-leading data engineering, process mining and automation platforms, they help large enterprises become more process-efficient and improve their end-customer experience by incorporating the use of RPA, AI...

  • Data Engineer

    3 days ago

    Austin, United States RevOpsforce Full time

    Job DescriptionJob DescriptionAbout RevOpsforce:RevOpsforce is a revenue operations workforce management firm supporting clients to unlock revenue potential through solutions that better align your people, processes, data, and technology.We empower organizations with cutting-edge revenue operations management systems, seamlessly aligning sales, marketing,...