Data Engineer Pyspark

1 week ago

Phoenix, Arizona, United States Tata Consultancy Services Full time $95,000 - $115,000 per year

Job Description
Job Title : Data Engineer Pyspark & BigQuery

Experience Required - 8+ Years

Must Have Technical/Functional Skills

  • Design, develop, and maintain scalable ETL/ELT pipelines using PySpark, Airflow, and GCP-native tools.
  • Build and optimize data warehouses and analytics solutions in BigQuery.
  • Implement and manage workflow orchestration with Airflow/Cloud Composer.
  • Write complex SQL queries for data transformations, analytics, and performance optimization.
  • Ensure data reliability, security, and governance across pipelines.
  • Conduct performance tuning and cost optimization of BigQuery and PySpark workloads.
  • Collaborate with analysts and product teams to deliver reliable data solutions.
  • Troubleshoot, debug, and resolve production issues in large-scale data pipelines.
  • Contribute to best practices, reusable frameworks, and automation for data engineering.5+ years of experience within Data Engineering/ Data Warehousing using Big Data technologies will be a addon
  • Expert on Distributed ecosystem
  • Hands-on experience with programming using Python
  • Expert on Hadoop and Spark Architecture and its working principle
  • Hands-on experience on writing and understanding complex SQL(Hive/PySpark-dataframes),
  • optimizing joins while processing huge amount of data
  • Experience in UNIX shell scripting Ability to design and develop optimized Data pipelines for batch and real time data processing
  • Should have experience in analysis, design, development, testing, and implementation of system applications
  • Demonstrated ability to develop and document technical and functional specifications and analyze software and system processing flows.

Roles & Responsibilities

  • Develop and maintain data pipelines using BigData processes
  • Focus on ingesting, storing, processing, and analyzing large datasets
  • Design, develop, and maintain application.
  • Able to communicate complex data structures and associated components.
  • Able to Design, code, test, maintain, and document application components.
  • Lead reviews of colleagues' work.
  • Defines test condi tions based on the functional and non-functional requirements provided.
  • Has deep understanding of the core tools used in the planning, analyzing, crafting, building, testing, configuring, and maintaining of assigned application(s).
  • Deep understanding of infrastructure, technologies, and components.
  • Able to assess and interview team members to identify and develop talent.
  • Making impactful changes by influencing leadership and making timely decisions.
  • Monitors system performance and availability and improves software quality through root cause analysis.
  • Proven track record to influence technological growth across teams.

Salary Range - $95,000 to $115,000 per year

TCS Employee Benefits Summary
Discretionary Annual Incentive.

Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.

Family Support: Maternal & Parental Leaves.

Insurance Options: Auto & Home Insurance, Identity Theft Protection.

Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.

Time Off: Vacation, Time Off, Sick Leave & Holidays.

Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

  • GCP engineer with Bigquery, Pyspark

    2 days ago

    Phoenix, Arizona, United States Tata Consultancy Services Full time

    Role: GCP engineer with Bigquery, PysparkLocation: Phoenix, Arizona, United StatesJob DescriptionJob Title : GCP engineer with Bigquery, PysparkExperience Required - 7+ YearsMust Have Technical/Functional SkillsGCP Engineer with Bigquery, Pyspark and Python experienceRoles & Responsibilities· 6+ years of professional experience with at least 4+ years of GCP...

  • Senior Data Engineer

    2 weeks ago

    Phoenix, Arizona, United States Hudson Manpower Full time $120,000 - $180,000 per year

    For over half a decade, Hudson Manpower has been a trusted partner in delivering specialized talent and technology solutions across IT, Energy, and Engineering industries worldwide. We work closely with startups, mid-sized firms, and Fortune 500 clients to support their digital transformation journeys. Our teams are empowered to bring fresh ideas, shape...

  • Senior Data Engineer

    1 week ago

    Phoenix, Arizona, United States Hudson Healthcare Solutions Full time $97,440 per year

    Job DescriptionMust Have Skills: (this should be the summary – cut through the one-page description into the must-haves)6- 9 years experienceGCPPythonSQLSparkRallyPySparkHiveFunctional TestingBig DataKey ResponsibilitiesDevelop and maintain robust data pipelines using Python, PySpark, and SQL.Design and optimize data workflows on Google Cloud Platform...

  • Senior Data Engineer

    10 hours ago

    Phoenix, Arizona, United States Saxon Global Full time

    The Senior Data Engineer & Test in Phoenix 85029 will play a pivotal role in delivering major data engineering initiatives within the Data & Advanced Analytics space. This position requires hands-on expertise in building, deploying, and maintaining robust data pipelines using Python, PySpark, and Airflow, as well as designing and implementing CI/CD processes...

  • GCP Data Engineer

    12 hours ago

    Phoenix, Arizona, United States Galent Full time

    JD:Job Title: GCP Data EngineerLocation: Phoenix AZ (Onsite)Skills: Python SQL Pyspark ETL GCPMust Have1.Extensive handson experience in objectoriented programming using Python PySpark APIs2Experience in building data pipelines for huge volume of data3Deep understanding of ETL concepts data warehousing principles SQL4Hands on experience in writing basic to...

  • Big Data Engineer

    5 days ago

    Phoenix, Arizona, United States Tata Consultancy Services Full time

    Job DescriptionJob Title : Big Data EngineerExperience Required - 5+ YearsMust Have Technical/Functional SkillsDesign, develop, and maintain scalable ETL/ELT pipelines using PySpark, Airflow, and GCP-native tools.Build and optimize data warehouses and analytics solutions in BigQuery.Implement and manage workflow orchestration with Airflow/Cloud...

  • Data Engineer

    5 days ago

    Phoenix, Arizona, United States enableIT Full time

    IT Data Engineer – Full Time | Ohio or Phoenix (Hybrid)Compensation:$170K–$180KIndustry:Bank / Financial ServicesA growing mid-sized financial institution is seeking afull-time IT Data Engineerto help modernize and scale its data environment in the cloud. This hands-on role focuses on building secure, compliant data pipelines inAzure, supporting key...

  • Data Engineer

    2 weeks ago

    Phoenix, Arizona, United States Synchrony Full time $80,000 - $120,000 per year

    Job Description:Role Summary/Purpose:The candidate for the Data Engineer role will join an Agile scrum team and perform functional & system development for Synchrony's Enterprise Data Lake.As a Big Data Engineer, the ability to integrate data across internal and external sources, provide analytical insights, and integrate with our critical systems are key...

  • Sr. Manager, Data Center Infrastructure Management

    13 hours ago

    Phoenix, Arizona, United States QTS Data Centers Full time

    TheSr. Manager, Data Center Infrastructure Management (DCIM)leads a team of engineers responsible for the management and administration of the Electric Power Monitoring Systems (EPMS) and Building Management Systems (BMS). This role requires a strategic thinker with keen planning skills and the ability to lead through influence. The ideal candidate will have...

  • AWS Data Engineer

    5 days ago

    Phoenix, Arizona, United States LTIMindtree Full time

    Job DetailsJob Title: AWS Data EngineerJob Location: Phoenix, AZ (Hybrid)Job Type: ContractClient: LTIMindtree Role OverviewWe are seeking a Specialist - Data Engineering with deep expertise in AWS cloud services and big data technologies to lead the development of scalable data pipelines and architectures.Skills Required:Apache SparkJavaPythonAWS EMRAWS...