Bigdata / Pyspark Developer

2 weeks ago


Plano, United States VirtusaPolaris - Virtusa Corporation Full time

Experience in building SparkStreaming process. Proficient in understanding distributed computing principles. Experience in managing Hadoop cluster with all services.Proficiency with Hadoop MapReduce HDFS Pig Hive and Impala. Experience with Nosql Databases and Messaging systems like Kafka. Designing building installing configuring and supporting Hadoop Perform analysis of vast data stores. Good understanding of cloud technology. Must have strong technical experience in Design Mapping specifications HLD LLD. Must have the ability to relate to both business and technical members of the team and possess excellent communication skills. Leverage internal tools and SDKs, utilize AWS services such as S3, Athena, and Glue, and integrate with our internal Archival Service Platform for efficient data purging. Lead the integration efforts with the internal Archival Service Platform for seamless data purging and lifecycle management. Collaborate with the data engineering team to continuously improve data integration pipelines, ensuring adaptability to evolving business needs. Performed database health checks and tuned the databases using Teradata Manager. Develop and maintain data platforms using Python Work with AWS and Big Data, design and implement data pipelines, and ensure data quality and integrity Collaborate with crossfunctional teams to understand data requirements and design solutions that meet business needs Implement and manage agents for monitoring, logging, and automation within AWS environments Handling migration from PySpark to AWS Experience in building SparkStreaming process. Proficient in understanding distributed computing principles. Experience in managing Hadoop cluster with all services.Proficiency with Hadoop MapReduce HDFS Pig Hive and Impala. Experience with Nosql Databases and Messaging systems like Kafka. Designing building installing configuring and supporting Hadoop Perform analysis of vast data stores. Good understanding of cloud technology. Must have strong technical experience in Design Mapping specifications HLD LLD. Must have the ability to relate to both business and technical members of the team and possess excellent communication skills. Leverage internal tools and SDKs, utilize AWS services such as S3, Athena, and Glue, and integrate with our internal Archival Service Platform for efficient data purging. Lead the integration efforts with the internal Archival Service Platform for seamless data purging and lifecycle management. Collaborate with the data engineering team to continuously improve data integration pipelines, ensuring adaptability to evolving business needs. Performed database health checks and tuned the databases using Teradata Manager. Develop and maintain data platforms using Python Work with AWS and Big Data, design and implement data pipelines, and ensure data quality and integrity Collaborate with crossfunctional teams to understand data requirements and design solutions that meet business needs Implement and manage agents for monitoring, logging, and automation within AWS environments Handling migration from PySpark to AWS

#J-18808-Ljbffr


  • Pyspark

    2 weeks ago


    Plano, United States E-Solutions INC Full time

    Job DescriptionJob DescriptionJob Title: Pyspark Location: Plano, TXFulltime/Contract(W2)Hands-on Pyspark SME with multiple project experience with Data planforms comprising of Hadoop, Teradata Data Warehouse, Ab Initio, Informatica, Java Spark (DPL), SSIS, AWS Lake Formation (S3), SnowflakeAbility to design, build and unit test applications on Spark...

  • Data Engineer

    4 weeks ago


    Plano, United States Promantus Inc Full time

    Data Engineer Location Plano Texas (Remote until COVID) Duration Long-term Job Description Develops Python/PySpark HQL queries. Develops new data models as necessary with Lead Backend Developer and Architect. Performs data visualization and analysis. Produces data samples for UI/UX Designer. Communicates insights to Architect Lead Backend Developer ...

  • Data Engineer

    4 weeks ago


    Plano, Texas, United States Promantus Inc Full time

    Data Engineer Location Plano Texas (Remote until COVID) Duration Long-term Job Description Develops Python/PySpark HQL queries. Develops new data models as necessary with Lead Backend Developer and Architect. Performs data visualization and analysis. Produces data samples for UI/UX Designer. Communicates insights to Architect Lead Backend Developer Lead...


  • Plano, United States Ascentt Full time

    Job Summary: We are seeking an experienced Mid-Senior Data Engineer (PySpark Engineer) to join our team of data professionals. In this role, you will collaborate closely with Data Scientists to prepare and transform large-scale datasets using PySpark, a popular open-source Python library for Apache Spark. You will play a crucial role in enabling effective...


  • Plano, United States Ascentt Full time

    Job Summary: We are seeking an experienced Mid-Senior Data Engineer (PySpark Engineer) to join our team of data professionals. In this role, you will collaborate closely with Data Scientists to prepare and transform large-scale datasets using PySpark, a popular open-source Python library for Apache Spark. You will play a crucial role in enabling effective...


  • Plano, United States Ascentt Full time

    Job Summary: We are seeking an experienced Mid-Senior Data Engineer (PySpark Engineer) to join our team of data professionals. In this role, you will collaborate closely with Data Scientists to prepare and transform large-scale datasets using PySpark, a popular open-source Python library for Apache Spark. You will play a crucial role in enabling effective...

  • Python Developer

    1 week ago


    Plano, United States Avacend Inc. Full time

    Responsibilities2-4 years of experience developing Data engineering, and ad-hoc transformation of unstructured raw dataUse of orchestration toolsDesign, build, and maintain workflows/pipelines to process a continuous stream of data with experience in end-to-end design and build process of Near-Real-Time and Batch Data Pipelines.Expected to work closely with...


  • Plano, United States Diverse Lynx Full time

    Role: AWS and Python and Java (Microservices) Developer Location - Plano TX, Northern VA (3 Days Office 2 Days remote) Experience: 10+ Year Duration: Long Term Mandatory Skills: AWS and Python and Java (Microservices) Job Description: • Strong working Python, AWS experience, web Application, UI experience (JSP, Angular), Java, Spring, Springboot, Databases...


  • Plano, United States Diverse Lynx Full time

    Role: AWS and Python and Java (Microservices) Developer Location - Plano TX, Northern VA (3 Days Office 2 Days remote) Experience: 10+ Year Duration: Long Term Mandatory Skills: AWS and Python and Java (Microservices) Job Description: • Strong working Python, AWS experience, web Application, UI experience (JSP, Angular), Java, Spring, Springboot, Databases...


  • Plano, United States Teleworld Solutions Full time

    OverviewTeleWorld Solutions is seeking a experienced Software engineer with focus on Data Engineering, ETL processes, preferably with exposure to both batch and streaming data. The candidate should have familiarity with use of Databases and DataLake infrastructure and associated tools for ingestion, transformation and efficient querying across distributed...


  • Plano, United States Avacend Full time

    Job DescriptionJob DescriptionOur client is looking for a Python Developer with Data Engineering @ Plano, TX. I am reaching you for a new job opportunity. Please let me know if you are interested, please share your updated resume and the following information.Responsibilities2-4 years of experience developing Data engineering, and ad-hoc transformation of...


  • Plano, United States AA SOFTWARE & NETWORKING PRIVATE LIMITED Full time

    Job Title: ETL/Data Warehousing DeveloperJob Location: Plano, TXJob Type: W2 OnlyJob responsibilities:Executes software solutions, design, development, and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problems.Creates secure and high-quality production code and maintains...


  • Plano, United States AA SOFTWARE & NETWORKING PRIVATE LIMITED Full time

    Job Title: ETL/Data Warehousing DeveloperJob Location: Plano, TXJob Type: W2 OnlyJob responsibilities:Executes software solutions, design, development, and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problems.Creates secure and high-quality production code and maintains...


  • Plano, United States AA SOFTWARE & NETWORKING PRIVATE LIMITED Full time

    Job Title: ETL/Data Warehousing DeveloperJob Location: Plano, TXJob Type: W2 OnlyJob responsibilities:Executes software solutions, design, development, and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problems.Creates secure and high-quality production code and maintains...

  • React.JS Developer

    4 weeks ago


    Plano, United States Pinnacle Group, Inc. Full time

    Role: React DeveloperLocation: Plano, TX- Hybrid 2/3 days in officeDuration: 6/12 Month Contract (Possible Extension or Hire)W2 ONLY NO C2CUSC/GC onlyMust have:Must Have: (1) React JS (2) AWS red shift and data bricks (3) PySparkReact and AWS CloudJob Description:Expert in React JS and design technique as well as experience working across large environments...

  • React.JS Developer

    4 weeks ago


    Plano, United States Pinnacle Group, Inc. Full time

    Role: React DeveloperLocation: Plano, TX- Hybrid 2/3 days in officeDuration: 6/12 Month Contract (Possible Extension or Hire)W2 ONLY NO C2CUSC/GC onlyMust have:Must Have: (1) React JS (2) AWS red shift and data bricks (3) PySparkReact and AWS CloudJob Description:Expert in React JS and design technique as well as experience working across large environments...

  • Business Analyst

    3 days ago


    Plano, United States Tata Consultancy Services Full time

    Job Title Data Engineering Tech LeadRelevant Experience (in Yrs) 10 -14 yearsMust Have Technical/Functional Skills1. Strong communication2. Has closely worked with Business users in understanding their requirements and processes3. Working knowledge of python, Pyspark , ETLs , Data loading Strategies - Ability to code and Troubleshoot4. Solid understanding of...


  • Plano, United States Cognizant Technology Solutions Full time

    Technical LeadQualification:Bachelors in science , engineering or equivalentResponsibility:Project Planning and Setup: Understand the project scope, identify activities/ tasks, task level estimates, schedule, dependencies, risks and provide inputs to Module Lead for review. Provide inputs to testing strategy, configuration, deployment, hardware/software...


  • Plano, United States JPMorgan Chase & Co. Full time

    Job responsibilities Executes software solutions, design, development, and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problemsCreates secure and high-quality production code and maintains algorithms that run synchronously with appropriate systemsProduces architecture...


  • Plano, United States JPMorgan Chase & Co. Full time

    Job responsibilities Executes software solutions, design, development, and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problemsCreates secure and high-quality production code and maintains algorithms that run synchronously with appropriate systemsProduces architecture...