Big Data Developer

2 months ago

dallas, United States Anblicks Full time

Job title : Big Data Engineer (Python, Pyspark)

Location : Dallas, Texas

Job Type: Fulltime

Work type : Hybrid


Data Engineer:

The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.


Responsibilities:

  • Responsible for design and development of big data solutions. Partner with domain experts, product managers, analyst, and data scientists to develop Big Data pipelines in Hadoop
  • Responsible for moving all legacy workloads to cloud platform
  • Work with data scientist to build Client pipelines using heterogeneous sources and provide engineering services for data science applications
  • Ensure automation through CI/CD across platforms both in cloud and on-premises
  • Define needs around maintainability, testability, performance, security, quality and usability for data platform
  • Drive implementation, consistent patterns, reusable components, and coding standards for data engineering processes
  • Convert SAS based pipelines into languages like PySpark, Scala to execute on Hadoop and non-Hadoop ecosystems
  • Tune Big data applications on Hadoop and non-Hadoop platforms for optimal performance
  • Evaluate new IT developments and evolving business requirements and recommend appropriate systems alternatives and/or enhancements to current systems by analyzing business processes, systems and industry standards.
  • Applies in-depth understanding of how data analytics collectively integrate within the sub-function as well as coordinates and contributes to the objectives of the entire function.
  • Produces detailed analysis of issues where the best course of action is not evident from the information available, but actions must be recommended/taken.
  • Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Client, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency.

Qualifications:

  • 8+ years of total IT experience
  • 5+ years of experience with Hadoop (Cloudera)/big data technologies
  • Advanced knowledge of the Hadoop ecosystem and Big Data technologies Hands-on experience with the Hadoop eco-system (HDFS, MapReduce, Hive, Pig, Impala, Spark, Kafka, Kudu, Solr)
  • Experience on designing and developing Data Pipelines for Data Ingestion or Transformation using Java or Scala or Python.
  • Experience with Spark programming (pyspark or scala or java)
  • Expert level building pipelines using Apache Spark Familiarity with core provider services from AWS, Azure or GCP, preferably having supported deployments on one or more of these platforms
  • Hands-on experience with Python/Pyspark/Scala and basic libraries for machine learning is required;
  • Exposure to containerization and related technologies (e.g. Docker, Kubernetes)
  • Exposure to aspects of DevOps (source control, continuous integration, deployments, etc.)
  • Proficient in programming in Java or Python with prior Apache Beam/Spark experience a plus.
  • System level understanding - Data structures, algorithms, distributed storage & compute
  • Can-do attitude on solving complex business problems, good interpersonal and teamwork skills
  • Possess team management experience and have led a team of data engineers and analysts.
  • Experience in Snowflake is a plus.

Education:

  • Bachelor’s degree/University degree or equivalent experience

  • Big Data Developer

    2 months ago

    dallas, United States Anblicks Full time

    Job title : Big Data Engineer (Python, Pyspark)Location : Dallas, TexasJob Type: FulltimeWork type : HybridData Engineer: The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in...

  • Big Data Developer

    3 months ago

    Dallas, United States Anblicks Full time

    Job title : Big Data Engineer (Python, Pyspark)Location : Dallas, TexasJob Type: FulltimeWork type : HybridData Engineer: The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in...

  • Big Data Developer

    4 months ago

    dallas, United States Anblicks Full time

    Role: Big Data EngineerLocation : Dallas, TexasJob Type: Hybrid• Analyze and understand data sources & APIs• Design and Develop methods to connect & collect data from different data sources• Design and Develop methods to filter/cleanse the data• Design and Develop SQL , Hive queries, APIs to extract data from the store• Work closely with data...

  • Big Data Developer

    4 months ago

    dallas, United States Anblicks Full time

    Role: Big Data EngineerLocation : Dallas, TexasJob Type: Hybrid• Analyze and understand data sources & APIs• Design and Develop methods to connect & collect data from different data sources• Design and Develop methods to filter/cleanse the data• Design and Develop SQL , Hive queries, APIs to extract data from the store• Work closely with data...

  • Big Data Developer

    4 months ago

    Dallas, United States Anblicks Full time

    Role: Big Data EngineerLocation : Dallas, TexasJob Type: Hybrid• Analyze and understand data sources & APIs• Design and Develop methods to connect & collect data from different data sources• Design and Develop methods to filter/cleanse the data• Design and Develop SQL , Hive queries, APIs to extract data from the store• Work closely with data...

  • Senior Big Data Developer

    2 weeks ago

    Dallas, Texas, United States Photon Full time

    Job Title: Senior Big Data DeveloperCompany Overview:Photon is a leading organization in the field of data processing and analytics. We are seeking a skilled Senior Big Data Developer to join our team and drive the development of robust data processing solutions within our data platform.Job Description:We are looking for an experienced developer with strong...

  • Big Data Solutions Developer

    1 week ago

    Dallas, Texas, United States Resource Informatics Group Full time

    Resource Informatics Group is seeking a highly skilled Big Data Solutions Developer to join our team. As a key member of our technical staff, you will be responsible for designing, implementing, and maintaining large-scale data processing systems.Estimated Salary: $160,000 - $220,000 per yearThe ideal candidate will have experience working with big data...

  • Senior Big Data Analytics Developer

    4 weeks ago

    Dallas, Texas, United States Tekstrom Inc Full time

    About the RoleWe are seeking a highly skilled Senior Big Data Analytics Developer to join our team at Tekstrom Inc. This is an exciting opportunity to work with big data analytics engineers, develop automated testing systems, and contribute to building a flexible and scalable data testing automation solution.Key Responsibilities:Collaborate with...

  • Senior Big Data Developer

    7 days ago

    Dallas, Texas, United States VIZIO Full time

    About the JobWe are looking for a Senior Big Data Developer who will design, build, and launch new data extraction, transformation, and loading processes. You will also identify the data needed for business problems, implement logging, and work with data infrastructure to resolve issues. As a senior member of our team, you will leverage your expertise to...

  • Big Data Engineer

    3 weeks ago

    Dallas, United States JobRialto Full time

    Job Summary The Big Data Engineer will design, develop, and implement scalable data solutions using modern big data technologies and cloud platforms. This role requires expertise in data modeling, integration, and analysis, ensuring high-quality, actionable insights. The ideal candidate will have experience working with big data ecosystems and a strong...

  • Big Data Engineer

    2 weeks ago

    Dallas, Texas, United States Hunt Corporation Full time

    About the PositionHunt Corporation is seeking an experienced Big Data Engineer to join our team. This is a highly competitive role with a salary range of $150,000 - $200,000 per year.In this position, you'll be responsible for designing and developing scalable data pipelines, implementing strong data security measures, and partnering with data scientists and...

  • Big Data Specialist

    3 weeks ago

    Dallas, Texas, United States IT Vision Group Full time

    Job DescriptionTitle: Big Data SpecialistLocation: RemoteDuration: 12 Months+Client: TCS/PrudentialJob Description :Azure-based Data Engineer or Architect is needed to drive business growth with cutting-edge data solutions. As a Big Data Specialist, you will design, develop, and implement scalable data architectures that meet the client's needs.We are...

  • Big Data Solutions Specialist

    6 days ago

    Dallas, Texas, United States Anblicks Full time

    Job Title: Big Data Solutions SpecialistWe are looking for a highly skilled Hadoop Developer to join our team at Anblicks. The successful candidate will be responsible for designing and developing data pipelines using Hadoop and Spark.About the Job:In this role, you will have the opportunity to work with a talented team of engineers and scientists to design...

  • Big Data Deployment Expert

    7 days ago

    Dallas, Texas, United States KBR Full time

    About the Job">We are seeking an experienced Big Data Deployment Expert to join our team. As a key member of our Technical Solutions group, you will be responsible for deploying and integrating big data analytics systems for DoD testing Ranges and acquisition programs.Key ResponsibilitiesDeploy and integrate big data analytics systems in various...

  • Big Data Systems Architect

    1 week ago

    Dallas, Texas, United States Amazon Full time

    Are you passionate about building big data systems? Do you enjoy working with Machine Learning and AWS technologies? Look no further! We are seeking a Big Data Systems Architect to join our Finance Technology team at Amazon.In this role, you will be responsible for designing and developing scalable systems using Big Data and Machine Learning, building...

  • Big Data Integration Specialist

    2 weeks ago

    Dallas, Texas, United States KBR Full time

    Job Summary:KBR's Defense Systems Engineering (DSE) unit seeks a highly skilled Data Analytics Deployment Expert to support the Test Resource Management Center's (TRMC) Big Data (BD) and Knowledge Management (KM) Team. The successful candidate will be responsible for deploying and integrating Cloud Hybrid Edge-to-Enterprise Evaluation Test & Analysis Suite...

  • Big Data Engineering Expert

    4 weeks ago

    Dallas, Texas, United States eTek IT Services, Inc. Full time

    Job Title: Big Data EngineerAbout eTek IT Services, Inc.eTek IT Services, Inc. is a leading provider of innovative technology solutions.Salary:$120,000 - $150,000 per yearJob Description:Data Engineer PositionThe ideal candidate will have extensive experience in big data engineering, with a strong background in Google Cloud Platform (GCP) technologies such...

  • Big Data Deployment Expert

    7 days ago

    Dallas, Texas, United States KBR Full time

    About KBRKBR is a global provider of technical solutions to the Department of Defense and Intelligence Community. We are passionate about our people, sustainability, and Zero Harm culture. Our commitment to diversity, equity, and inclusion fosters an environment of collaboration and innovation across cultures and locations.We are seeking an experienced Big...

  • Big Data Solutions Leader

    1 week ago

    Dallas, Texas, United States evolv consulting Full time

    Job DescriptionWe are seeking a highly skilled Big Data Solutions Leader to join our team. The ideal candidate will have a strong background in big data technologies and a proven track record of delivering complex projects on time and within budget.The successful candidate will be responsible for designing and implementing large-scale data architectures,...

  • Senior Big Data Engineering Manager

    24 hours ago

    Dallas, Texas, United States Resource Informatics Group Inc Full time

    Job SummaryAs a Senior Big Data Architect at Resource Informatics Group Inc, you will be responsible for designing and implementing big data platforms that drive business insights and decision-making.Key Responsibilities:Design and implement scalable big data platforms using Cloudera Hadoop distributionsDevelop expertise in distributed, scalable big data...