Big Data Developer

2 months ago

Dallas, United States Anblicks Full time

Job title : Big Data Engineer (Python, Pyspark)

Location : Dallas, Texas

Job Type: Fulltime

Work type : Hybrid


Data Engineer:

The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.


Responsibilities:

  • Responsible for design and development of big data solutions. Partner with domain experts, product managers, analyst, and data scientists to develop Big Data pipelines in Hadoop
  • Responsible for moving all legacy workloads to cloud platform
  • Work with data scientist to build Client pipelines using heterogeneous sources and provide engineering services for data science applications
  • Ensure automation through CI/CD across platforms both in cloud and on-premises
  • Define needs around maintainability, testability, performance, security, quality and usability for data platform
  • Drive implementation, consistent patterns, reusable components, and coding standards for data engineering processes
  • Convert SAS based pipelines into languages like PySpark, Scala to execute on Hadoop and non-Hadoop ecosystems
  • Tune Big data applications on Hadoop and non-Hadoop platforms for optimal performance
  • Evaluate new IT developments and evolving business requirements and recommend appropriate systems alternatives and/or enhancements to current systems by analyzing business processes, systems and industry standards.
  • Applies in-depth understanding of how data analytics collectively integrate within the sub-function as well as coordinates and contributes to the objectives of the entire function.
  • Produces detailed analysis of issues where the best course of action is not evident from the information available, but actions must be recommended/taken.
  • Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Client, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency.

Qualifications:

  • 8+ years of total IT experience
  • 5+ years of experience with Hadoop (Cloudera)/big data technologies
  • Advanced knowledge of the Hadoop ecosystem and Big Data technologies Hands-on experience with the Hadoop eco-system (HDFS, MapReduce, Hive, Pig, Impala, Spark, Kafka, Kudu, Solr)
  • Experience on designing and developing Data Pipelines for Data Ingestion or Transformation using Java or Scala or Python.
  • Experience with Spark programming (pyspark or scala or java)
  • Expert level building pipelines using Apache Spark Familiarity with core provider services from AWS, Azure or GCP, preferably having supported deployments on one or more of these platforms
  • Hands-on experience with Python/Pyspark/Scala and basic libraries for machine learning is required;
  • Exposure to containerization and related technologies (e.g. Docker, Kubernetes)
  • Exposure to aspects of DevOps (source control, continuous integration, deployments, etc.)
  • Proficient in programming in Java or Python with prior Apache Beam/Spark experience a plus.
  • System level understanding - Data structures, algorithms, distributed storage & compute
  • Can-do attitude on solving complex business problems, good interpersonal and teamwork skills
  • Possess team management experience and have led a team of data engineers and analysts.
  • Experience in Snowflake is a plus.

Education:

  • Bachelor’s degree/University degree or equivalent experience

  • Big Data Developer

    4 weeks ago

    dallas, United States Anblicks Full time

    Job title : Big Data Engineer (Python, Pyspark)Location : Dallas, TexasJob Type: FulltimeWork type : HybridData Engineer: The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in...

  • Big Data Developer

    4 weeks ago

    dallas, United States Anblicks Full time

    Job title : Big Data Engineer (Python, Pyspark)Location : Dallas, TexasJob Type: FulltimeWork type : HybridData Engineer: The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in...

  • Big Data Developer

    3 months ago

    Dallas, United States Anblicks Full time

    Role: Big Data EngineerLocation : Dallas, TexasJob Type: Hybrid• Analyze and understand data sources & APIs• Design and Develop methods to connect & collect data from different data sources• Design and Develop methods to filter/cleanse the data• Design and Develop SQL , Hive queries, APIs to extract data from the store• Work closely with data...

  • Big Data Developer

    3 months ago

    dallas, United States Anblicks Full time

    Role: Big Data EngineerLocation : Dallas, TexasJob Type: Hybrid• Analyze and understand data sources & APIs• Design and Develop methods to connect & collect data from different data sources• Design and Develop methods to filter/cleanse the data• Design and Develop SQL , Hive queries, APIs to extract data from the store• Work closely with data...

  • Big Data Developer

    3 months ago

    dallas, United States Anblicks Full time

    Role: Big Data EngineerLocation : Dallas, TexasJob Type: Hybrid• Analyze and understand data sources & APIs• Design and Develop methods to connect & collect data from different data sources• Design and Develop methods to filter/cleanse the data• Design and Develop SQL , Hive queries, APIs to extract data from the store• Work closely with data...

  • Senior Python Developer for Big Data Processing

    4 weeks ago

    Dallas, Texas, United States Synechron Full time

    We are seeking a highly skilled Senior Python Developer to lead our big data processing efforts. The ideal candidate will have a strong background in data analysis and a proven track record of delivering high-quality results.Key Responsibilities:Develop and maintain data processing pipelines using PySpark for large-scale data processing and analytics.Work...

  • Big Data Engineer

    1 month ago

    Dallas, Texas, United States CloudBC Labs Full time

    Job SummaryWe are seeking a highly skilled Big Data Engineer to join our team at CloudBC Labs. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining large-scale data processing systems using Google Cloud technologies.Key Responsibilities- Design and implement data pipelines using Google Cloud...

  • Big Data Engineer

    4 weeks ago

    Dallas, Texas, United States Tekfortune Inc Full time

    Job Title: Hadoop DeveloperJob Description:We are seeking a highly skilled Hadoop Developer to join our team at Tekfortune Inc. As a Hadoop Developer, you will be responsible for designing, developing, and maintaining large-scale data processing systems using Hadoop and related technologies.Key Responsibilities:Design and develop Hadoop-based data processing...

  • Big Data and Analytics Engineer

    3 weeks ago

    Dallas, Texas, United States Anblicks Full time

    Job Title: Big Data and Analytics EngineerJob Summary:Analyze and understand data sources & APIsDesign and Develop methods to connect & collect data from different data sourcesDesign and Develop methods to filter/cleanse the dataDesign and Develop SQL , Hive queries, APIs to extract data from the storeWork closely with data Scientists to ensure the source...

  • Big Data Engineer

    3 weeks ago

    Dallas, United States Purple Drive Full time

    Job Title: Big Data Engineer with Python and Cloud Experience Candidate must be on our W2 Location: Santa Clara CA (Hybrid/Onsite) Experience Required: 410 years Job Type: FullTime Hadoop,AWS,Azure,Kafka

  • Big Data Infrastructure Specialist

    4 weeks ago

    Dallas, Texas, United States System One Full time

    Job SummaryWe are seeking a highly skilled Data Processing and Analytics Expert to join our team at System One. As a key member of our big data infrastructure team, you will be responsible for providing production support for tier 1 applications and enterprise platforms.Key ResponsibilitiesWork across multiple clusters supporting multi-tiered applications in...

  • Director of Development and Fundraising

    1 month ago

    Dallas, Texas, United States Big Thought Full time

    About the Role:The Director of Development and Fundraising will work closely with the Chief Development Officer to implement a strategy for sustainable fundraising for Big Thought. This individual will develop and manage a portfolio of philanthropic supporters and prospects who power the organization's mission and vision.About Big Thought:Big Thought equips...

  • Data Engineer

    1 month ago

    Dallas, Texas, United States NTT DATA Full time

    Job SummaryNTT DATA Services is seeking a highly skilled Data Engineer to join our team in Dallas, Texas. As a Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data systems to support our business operations.Key Responsibilities:Develop and maintain data pipelines to extract, transform, and load data from diverse...

  • Data Engineer

    5 months ago

    Dallas, United States NTT DATA Full time

    Job Description NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer (Onsite) to join our team in Dallas, Texas (US-TX), United States (US).Job Responsibilities...

  • Service Now Developer with Big Panda

    3 weeks ago

    dallas, United States Rockwoods Inc Full time

    Role: Service Now Developer with Big PandaLocation: Dallas or San antonio TexasSkills:Big Panda experienceServiceNow event managementServiceNow Dev experienceServiceNow playbook and auto remediation experienceExtras:AWS CertificationSNOW (Incident, Change, Problem, Kr)CI/CD Pipelines (preferred GitLab, Jenkins)Automation Tools – preferred Puppet,...

  • Corporate Development Analyst

    5 months ago

    Dallas, United States NTT DATA Full time

    Job Description Req ID: 283121 NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Corporate Development Analyst to join our team in Dallas, Texas (US-TX), United States...

  • Corporate Development Analyst

    1 month ago

    Dallas, Texas, United States NTT DATA Full time

    Job DescriptionReq ID: 283121 NTT DATA Services is seeking a highly analytical and financially astute Corporate Development Analyst to join its team in Dallas, Texas. The ideal candidate will possess a strong understanding of valuation methodologies and extensive experience in M&A, corporate development, or accounting.The successful candidate will develop...

  • Data Engineering Specialist

    1 month ago

    Dallas, Texas, United States Photon Full time

    Job Title:PySpark Data EngineerWe are seeking a skilled PySpark Data Engineer to drive the development of robust data processing and transformation solutions within our data platform.You will be responsible for designing, implementing, and maintaining PySpark-based applications to handle complex data processing tasks, ensure data quality, and integrate with...

  • Senior Data Engineer

    2 months ago

    Dallas, United States Anblicks Full time

    The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data solutions....

  • Senior Data Engineer

    1 month ago

    dallas, United States Anblicks Full time

    The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data solutions....