Big Data Specialist

1 week ago


Irving, United States Tata Consultancy Services Full time

Job Description

Big Data (PySpark) Tech Lead

  • 10 + Years Overall Experience in Data Management, Data Lake and Data Warehouse
  • 6+ Years Hadoop, Hive, Sqoop, SQL, Teradata
  • 6+ Years PySpark(Python and Spark), Unix
  • Good to have Industry leading ETL experience
  • Banking Domain experience

Key Responsibilities

  • Ability to design, build and unit test applications on Spark framework on Python.
  • Build PySpark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well.
  • Develop and execute data pipeline testing processes and validate business rules and policies
  • Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's.
  • Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively.
  • Ability to design & build real-time applications using Apache Kafka & Spark Streaming
  • Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.
  • Build data tokenization libraries and integrate with Hive & Spark for column-level obfuscation
  • Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.
  • Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories
  • Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings
  • Work collaboratively with onsite and offshore team.
  • Develop & review technical documentation for artifacts delivered.
  • Ability to solve complex data-driven scenarios and triage towards defects and production issues
  • Ability to learn-unlearn-relearn concepts with an open and analytical mindset
  • Participate in code release and production deployment.
  • Challenge and inspire team members to achieve business results in a fast paced and quickly changing environment


  • Irving, Texas, United States Diverse Lynx Full time

    Job SummaryDiverse Lynx LLC seeks a highly skilled Big Data Analytics Specialist to leverage Apache Spark and PySpark expertise in our digital transformation initiatives. Key responsibilities include:Designing and implementing scalable data pipelines using Apache SparkDeveloping and maintaining complex data processing workflows with PySparkCollaborating with...

  • Big Data Specialist

    4 weeks ago


    Irving, United States Tata Consultancy Services Full time

    Job DescriptionBig Data (PySpark) Tech Lead10 + Years Overall Experience in Data Management, Data Lake and Data Warehouse6+ Years Hadoop, Hive, Sqoop, SQL, Teradata6+ Years PySpark(Python and Spark), UnixGood to have Industry leading ETL experienceBanking Domain experience Key ResponsibilitiesAbility to design, build and unit test applications on Spark...

  • Big Data Developer

    2 weeks ago


    Irving, United States Resource Informatics Group Full time

    Note: It is a W2 Opportunity, Client is accepting candidates from (New Jersey and New York) other location candidates will be straight away rejected and application candidate must have prior banking domain work experience. Role: Big Data Developer Location: Iselin, NJ In office: 3 days a week onsite Contract: 6-24 months to perm Interview process: screening...

  • Big Data Engineer

    6 days ago


    Irving, Texas, United States Diverse Lynx Full time

    Job DescriptionWe are seeking a highly skilled Big Data Engineer to join our team at Diverse Lynx LLC.About UsDiverse Lynx LLC is an Equal Employment Opportunity employer committed to promoting diversity and inclusion in the workplace. We believe that a diverse workforce brings unique perspectives and ideas, leading to innovation and success.SalaryThe...


  • Irving, United States Anblicks Full time

    Description: The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data...


  • Irving, United States Anblicks Full time

    Description: The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data...


  • Irving, United States America Technology Professionals LLC Full time

    Job Title: Lead / Sr. Big Data Engineer Location: Irving, TX (Hybrid – 2-3 days/week) Duration: 12+ Months USC/GC/H1B only C2C/1099 only This role can play the hand-on dev role along with tech lead skills Required Skills: Big data expert with 9+ years experience in Hadoop Big data ecosystem Spark - Batch Streaming (Python, Scala) Experience in cloud...


  • Irving, United States Infovision Full time

    Job Title: Lead / Sr. Big Data EngineerLocation: Irving, TX (Hybrid 2-3 days/week)Duration: 12+ MonthsRequired Skills: Big data expert with 9+ years experience in Hadoop Big data ecosystemSpark - Batch Streaming (Python, Scala)Experience in cloud environment specially Google Cloud PlatformExperience in developing both batch and real-time streaming data...


  • Irving, United States InfoVision Inc. Full time

    Job Title: Lead / Sr. Big Data EngineerLocation: Irving, TX (Hybrid 2-3 days/week)Duration: 12+ Months Required Skills: Big data expert with 9+ years experience in Hadoop Big data ecosystemSpark - Batch Streaming (Python, Scala)Experience in cloud environment specially Google Cloud PlatformExperience in developing both batch and real-time streaming data...


  • Irving, United States Infovision Full time

    Job title: Lead Big Data Engineer (GCP Cloud) Location: Irving, TX Duration: Long-term Skills Needed: 10+ years of experience in designing and building data pipelines in large-scale distributed systems. Lead the design, development, and maintenance of scalable batch and real-time data processing pipelines. Proficiency with Google Cloud Platform (GCP) and...


  • Irving, United States Infovision Full time

    Job title: Lead Big Data Engineer (GCP Cloud)Location: Irving, TXDuration: Long-termSkills Needed:10+ years of experience in designing and building data pipelines in large-scale distributed systems.Lead the design, development, and maintenance of scalable batch and real-time data processing pipelines.Proficiency with Google Cloud Platform (GCP) and tools...

  • Cloud Engineer

    1 week ago


    Irving, United States Resource Informatics Group Full time

    Job Title: Cloud Engineer - Senior / Big Data Location: McLean, VA / Dallas, TX (hybrid role) Duration: 6+ Months (possibility of extension or conversion) Experience: - 5 - 8 years Job Description 5+ Years of exp needed. Will be working in AWS environment. Product - Data Lake Platform in AWS This Developer will be working on Glue, DynamoDB, S3, Lambda...

  • Lead Big Data Engineer

    2 months ago


    Irving, United States InfoVision Inc. Full time

    Job title: Lead Big Data Engineer (GCP Cloud)Location: Irving, TXDuration: Long-term Skills Needed:10+ years of experience in designing and building data pipelines in large-scale distributed systems.Lead the design, development, and maintenance of scalable batch and real-time data processing pipelines.Proficiency with Google Cloud Platform (GCP) and tools...


  • Irving, TX, United States Anblicks Full time

    Description: The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data...


  • Irving, Texas, United States Forward Air Full time

    Job Title: Data EngineerJob Summary:Forward Air is seeking a skilled Data Engineer to work in the IT department. As a Data Engineer, you will be responsible for designing and implementing scalable cloud-based streaming data solutions that enable real-time analytics and mission-critical data visibility to our organization. You will work across departments...


  • Irving, Texas, United States NTT DATA Full time

    At NTT DATA, we are seeking a skilled Technical Project Manager to join our team in Irving, Texas. This is a hybrid position requiring 2-3 days on site and remote work.Job SummaryWe are currently looking for an experienced Technical Project Manager who can effectively manage projects from inception to closure. The ideal candidate will have a solid...


  • Irving, Texas, United States Verizon Full time

    Job DescriptionAt Verizon, we're looking for a highly skilled Senior Data Solutions Specialist to join our team. In this role, you'll be responsible for designing, building, and maintaining end-to-end data products that meet our business needs.Key Responsibilities:Develop high-quality code using cutting-edge technologies such as GCP, AWS, and Hadoop...


  • Irving, Texas, United States Donato Technologies Inc Full time

    Job DescriptionWe are seeking experienced professionals to manage and optimize Cassandra database clusters for our client in Irving, TX or Miami, FL. The ideal candidate will have a strong background in NoSQL database administration and performance tuning.Key Responsibilities:Install, configure, and manage Cassandra clusters to ensure high-performance data...

  • Senior Data Engineer

    2 weeks ago


    Irving, United States Newt Global Full time

    Greeting from Newt Global LLC!!!We are currently looking for Big Data Engineer to join our team for long term project with our Banking client. Interested with the below job description, please share your resume with expected rate/hr.Big Data EngineerIrving, TX (Hybrid – 2 Days a Week)Contract Type: W2Long TermExp: 6+ yrsBanking Exp RequiredBig Data...

  • Data Engineer

    1 month ago


    Irving, United States Newt Global Full time

    Big Data EngineerIrving, TX - HybridW2 ContractDirect ClientQualifications:8+ years of experience in hadoop/big data technologies.3+ years of experience in spark.2+ years experience in Snowflake2+ year of experience working on Google or AWS cloud developing data solutions. Certifications preferred.Hands-on experience with Python/Pyspark/Scala and basic...