PySpark Data Engineer

4 days ago

Irving, United States Diverse Lynx Full time

• 10 + Years Overall Experience in Data Management, Data Lake and Data Warehouse
• 6+ Years Hadoop, Hive, Sqoop, SQL, Teradata
• 6+ Years PySpark(Python and Spark), Unix
• Good to have Industry leading ETL experience
• Banking Domain experience
Key Responsibilities
• Ability to design, build and unit test applications on Spark framework on Python.
• Build PySpark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well.
• Develop and execute data pipeline testing processes and validate business rules and policies
• Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's.
• Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively.
• Ability to design & build real-time applications using Apache Kafka & Spark Streaming
• Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.
• Build data tokenization libraries and integrate with Hive & Spark for column-level obfuscation
• Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.
• Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories
• Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings
• Work collaboratively with onsite and offshore team.
• Develop & review technical documentation for artifacts delivered.
• Ability to solve complex data-driven scenarios and triage towards defects and production issues
• Ability to learn-unlearn-relearn concepts with an open and analytical mindset
• Participate in code release and production deployment.
• Challenge and inspire team members to achieve business results in a fast paced and quickly changing environment

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.

  • Data engineer

    3 weeks ago

    Irving, United States Tata Consultancy Services Full time

    Position: Data Engineer (PySpark)Duration: Full TimeLocation - Irving, TX (onsite)Pay -130-140k+benefits Job Description Big Data (PySpark) Tech Lead– • 10+ Years Overall Experience in Data Management, Data Lake and Data Warehouse• 8+ Years Hadoop, Hive, Sqoop, SQL, Teradata• 8+ Years PySpark(Python and Spark), Unix• Good to have Industry leading...

  • PySpark Developer

    6 days ago

    Irving, United States Procom Full time

    PySpark Developer with Kubernetes Experience Intro We are seeking a skilled PySpark Developer with expertise in Kubernetes to join our innovative team. This role involves designing, developing, and maintaining big data solutions using Apache Spark (PySpark), deploying scalable applications on Kubernetes, and ensuring efficient data pipeline operations. The...

  • PySpark Developer

    7 days ago

    Irving, United States Procom Full time

    PySpark Developer with Kubernetes Experience Intro We are seeking a skilled PySpark Developer with expertise in Kubernetes to join our innovative team. This role involves designing, developing, and maintaining big data solutions using Apache Spark (PySpark), deploying scalable applications on Kubernetes, and ensuring efficient data pipeline operations. The...

  • PySpark Developer with Kubernetes Experience

    7 days ago

    Irving, United States Mindlance Full time

    PySpark Developer with Kubernetes Experience Duration: 12 months plus (possible extension or conversion) Location: HYBRID W/ ONSITE REQUIREMENT - 3 days onsite Irving, TX 75039 Job Summary: We are looking for an experienced PySpark Developer with strong expertise in Kubernetes to join our dynamic team. The ideal candidate will be responsible for designing,...

  • PL/SQL Python/Pyspark Developer

    1 month ago

    Irving, United States NTT DATA Group Corporation Full time

    Company Overview: Req ID: 299699 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We have Python/Pyspark Developer available for 12 months and ONSITE/HYBRID. T/SQL or PL/SQL experience needed. W2 ONLY Job...

  • Big Data Engineer

    7 days ago

    Irving, Texas, United States Diverse Lynx Full time

    Job DescriptionWe are seeking a highly skilled Big Data Engineer to join our team at Diverse Lynx LLC.About UsDiverse Lynx LLC is an Equal Employment Opportunity employer committed to promoting diversity and inclusion in the workplace. We believe that a diverse workforce brings unique perspectives and ideas, leading to innovation and success.SalaryThe...

  • Senior Big Data Engineer

    4 weeks ago

    Irving, United States Anblicks Full time

    Description: The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data...

  • Senior Big Data Engineer

    2 months ago

    Irving, United States Anblicks Full time

    Description: The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data...

  • Data Engineer

    1 month ago

    Irving, United States Newt Global Full time

    Big Data EngineerIrving, TX - HybridW2 ContractDirect ClientQualifications:8+ years of experience in hadoop/big data technologies.3+ years of experience in spark.2+ years experience in Snowflake2+ year of experience working on Google or AWS cloud developing data solutions. Certifications preferred.Hands-on experience with Python/Pyspark/Scala and basic...

  • Data Scientist

    1 week ago

    Irving, United States SysMind Tech Full time

    Position: Data Scientist Location: Irving, TX (Hybrid) Duration: Long Term This position is for a hands-on experienced data scientist with experience in developing predictive models for business applications. The position involves working on data science projects and solutions that drive customer experience, customer service, network, and technical...

  • Senior Data Engineer

    3 weeks ago

    Irving, United States Newt Global Full time

    Greeting from Newt Global LLC!!!We are currently looking for Big Data Engineer to join our team for long term project with our Banking client. Interested with the below job description, please share your resume with expected rate/hr.Big Data EngineerIrving, TX (Hybrid – 2 Days a Week)Contract Type: W2Long TermExp: 6+ yrsBanking Exp RequiredBig Data...

  • Data Engineer Lead

    1 week ago

    Irving, Texas, United States Tata Consultancy Services Full time

    Company Overview:Tata Consultancy Services is a leading global IT services company.Salary: The estimated salary for this position is $130,000 - $140,000 per year, plus benefits.Job Description:This is a full-time opportunity as a Big Data (PySpark) Tech Lead in Irving, TX.Required Skills and Qualifications:10+ years of overall experience in data management,...

  • GCP Data Engineer @ Irving, TX

    1 week ago

    Irving, United States Diverse Lynx Full time

    Role: GCP Data Engineer Location: Irving, TX Type: Contract Technical Skills : SQL, Python, PySpark, GCP, BigQuery, Cloud Dataproc, Cloud Composer, Cloud Dataflow Technical Skills : Teradata/Oracle, Informatica/ AbInitio/ Datastage, Hadoop, Hive, Apache Spark, Cloud Pub/Sub, Cloud Spanner, Cloud SQL, Data Fusion Domain Skills : Technology :...

  • Lead Data Insights Developer

    7 days ago

    Irving, Texas, United States SysMind Tech Full time

    Job Title: Lead Data Insights DeveloperSalary: $140,000 - $160,000 per yearCompany Overview:SysMind Tech is a cutting-edge technology company that specializes in developing innovative solutions for businesses. We are currently seeking an experienced Data Scientist to join our team and lead our data insights development efforts.Job Description:We are looking...

  • IBM Streams Data Engineer

    4 weeks ago

    Irving, United States Abode Techzone LLC Full time

    Greetings from Abode Techzone, LLC! We are looking for a best match for one of our client's urgent requirements mentioned below. Let me know if you would be interested to move ahead, if yes please share your UPDATED RESUME with your Hourly Rates expectations along with below detail Work Authorization: Hourly Rates expectations : Current location: ...

  • Senior Big Data Engineer

    4 weeks ago

    Irving, TX, United States Anblicks Full time

    Description: The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data...

  • Big Data Analytics Specialist

    7 days ago

    Irving, Texas, United States Diverse Lynx Full time

    Job SummaryDiverse Lynx LLC seeks a highly skilled Big Data Analytics Specialist to leverage Apache Spark and PySpark expertise in our digital transformation initiatives. Key responsibilities include:Designing and implementing scalable data pipelines using Apache SparkDeveloping and maintaining complex data processing workflows with PySparkCollaborating with...

  • Business Intelligence Solutions Developer

    3 days ago

    Irving, Texas, United States Yoh Full time

    Job Title: Business Intelligence Solutions Developer - Data Engineering ExpertOverview:We are seeking a skilled Business Intelligence Solutions Developer to join our team. As a data engineering expert, you will be responsible for designing, developing, and maintaining business intelligence solutions using Microsoft Fabric and Power BI.Salary Range:$100,000 -...

  • Big Data Specialist

    4 weeks ago

    Irving, United States Tata Consultancy Services Full time

    Job DescriptionBig Data (PySpark) Tech Lead10 + Years Overall Experience in Data Management, Data Lake and Data Warehouse6+ Years Hadoop, Hive, Sqoop, SQL, Teradata6+ Years PySpark(Python and Spark), UnixGood to have Industry leading ETL experienceBanking Domain experience Key ResponsibilitiesAbility to design, build and unit test applications on Spark...

  • Big Data Specialist

    1 week ago

    Irving, United States Tata Consultancy Services Full time

    Job DescriptionBig Data (PySpark) Tech Lead10 + Years Overall Experience in Data Management, Data Lake and Data Warehouse6+ Years Hadoop, Hive, Sqoop, SQL, Teradata6+ Years PySpark(Python and Spark), UnixGood to have Industry leading ETL experienceBanking Domain experience Key ResponsibilitiesAbility to design, build and unit test applications on Spark...