Data Engineer Pyspark_BigQuery

5 days ago

Phoenix, AZ, United States Diverse Lynx Full time

Job Title : Data Engineer Pyspark & BigQuery
Experience Required - 8+ Years
Must Have Technical/Functional Skills
• Design, develop, and maintain scalable ETL/ELT pipelines using PySpark, Airflow, and GCP-native tools.
• Build and optimize data warehouses and analytics solutions in BigQuery.
• Implement and manage workflow orchestration with Airflow/Cloud Composer.
• Write complex SQL queries for data transformations, analytics, and performance optimization.
• Ensure data reliability, security, and governance across pipelines.
• Conduct performance tuning and cost optimization of BigQuery and PySpark workloads.
• Collaborate with analysts and product teams to deliver reliable data solutions.
• Troubleshoot, debug, and resolve production issues in large-scale data pipelines.
• Contribute to best practices, reusable frameworks, and automation for data engineering.5+ years of experience within Data Engineering/ Data Warehousing using Big Data technologies will be ddon
• Expert on Distributed ecosystem
• Hands-on experience with programming using Python
• Expert on Hadoop and Spark Architecture and its working principle
• Hands-on experience on writing and understanding complex SQL(Hive/PySpark-dataframes),
• optimizing joins while processing huge amount of data
• Experience in UNIX shell scripting Ability to design and develop optimized Data pipelines for batch and real time data processing
• Should have experience in analysis, design, development, testing, and implementation of system applications
o Demonstrated ability to develop and document technical and functional specifications and analyze software and system processing flows.
Roles & Responsibilities
• Develop and maintain data pipelines using BigData processes
• Focus on ingesting, storing, processing, and analyzing large datasets
• Design, develop, and maintain application.
• Able to communicate complex data structures and associated components.
• Able to Design, code, test, maintain, and document application components.
• Lead reviews of colleagues' work.
• Defines test condi tions based on the functional and non-functional requirements provided.
• Has deep understanding of the core tools used in the planning, analyzing, crafting, building, testing, configuring, and maintaining of assigned application(s).
• Deep understanding of infrastructure, technologies, and components.
• Able to assess and interview team members to identify and develop talent.
• Making impactful changes by influencing leadership and making timely decisions.
• Monitors system performance and availability and improves software quality through root cause analysis.
• Proven track record to influence technological growth across teams.

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.

  • Data Engineer

    5 days ago

    Phoenix, AZ, United States RIT Solutions, Inc. Full time

    Data Engineer 3 days Phoenix, As a Data Engineer at our client, you will play a crucial role in designing, building, and maintaining the data architecture that powers our analytical and operational capabilities. You will work closely with developers, QA's, Scrum Master and other stakeholders to ensure data availability, reliability, and integrity. Your...

  • Data Engineering Consultant

    14 hours ago

    Phoenix, AZ, United States Plex Consulting, LLC Full time

    Job Title: Data Engineering Consultant Company: Plex Consulting Location: Phoenix, AZ About the company: Plex Consulting is a leading technology consulting firm offering data solutions to a diverse range of clients across industries. We combine our expertise in data engineering, analytics, and data science to provide innovative and efficient solutions that...

  • Data Engineering Consultant

    1 week ago

    Phoenix, AZ, United States Plex Consulting, LLC Full time

    Job Title: Data Engineering Consultant Company: Plex Consulting Location: Phoenix, AZ About the company: Plex Consulting is a leading technology consulting firm offering data solutions to a diverse range of clients across industries. We combine our expertise in data engineering, analytics, and data science to provide innovative and efficient solutions that...

  • Data Engineering Consultant

    6 days ago

    Phoenix, AZ, United States Plex Consulting, LLC Full time

    Job Title: Data Engineering Consultant Company: Plex Consulting Location: Phoenix, AZ About the company: Plex Consulting is a leading technology consulting firm offering data solutions to a diverse range of clients across industries. We combine our expertise in data engineering, analytics, and data science to provide innovative and efficient solutions that...

  • Big Data Engineer/Data Engineer

    5 days ago

    Phoenix, AZ, United States CrackaJack Digital Solutions LLC Full time

    In-Person round of interview mandatory. Tech Stack: Big Data, Spark, Python, SQL, GCP is must. Need a hardcore, heavy hitting Data Engineer who is extremely skilled and is able to function independently and manage their deliverables Capable of writing ETL pipelines using Python from scratch Expert in OOP principles and concepts Ability to independently write...

  • Big Data Engineer/Data Engineer

    2 weeks ago

    Phoenix, AZ, United States CrackaJack Digital Solutions LLC Full time

    In-Person round of interview mandatory. Tech Stack: Big Data, Spark, Python, SQL, GCP is must. Need a hardcore, heavy hitting Data Engineer who is extremely skilled and is able to function independently and manage their deliverables Capable of writing ETL pipelines using Python from scratch Expert in OOP principles and concepts Ability to independently write...

  • Big Data Engineer/Data Engineer

    7 days ago

    Phoenix, AZ, United States CrackaJack Digital Solutions LLC Full time

    In-Person round of interview mandatory. Tech Stack: Big Data, Spark, Python, SQL, GCP is must. Need a hardcore, heavy hitting Data Engineer who is extremely skilled and is able to function independently and manage their deliverables Capable of writing ETL pipelines using Python from scratch Expert in OOP principles and concepts Ability to independently write...

  • Data Engineer

    6 days ago

    Phoenix, AZ, United States Apolis Full time

    POSITION SUMMARY: We are looking for a savvy Data Engineer to join our growing team of analytics experts. The hire will be responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler...

  • Data Engineer

    4 days ago

    Phoenix, AZ, United States Apolis Full time

    POSITION SUMMARY: We are looking for a savvy Data Engineer to join our growing team of analytics experts. The hire will be responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler...

  • Data Engineer

    1 week ago

    Phoenix, AZ, United States Apolis Full time

    POSITION SUMMARY: We are looking for a savvy Data Engineer to join our growing team of analytics experts. The hire will be responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler...