Mid-Senior Data Engineer

4 weeks ago

Plano, United States Ascentt Full time

Job Summary: We are seeking an experienced Mid-Senior Data Engineer (PySpark Engineer) to join our team of data professionals. In this role, you will collaborate closely with Data Scientists to prepare and transform large-scale datasets using PySpark, a popular open-source Python library for Apache Spark. You will play a crucial role in enabling effective data analysis and modeling by ensuring the availability of high-quality, feature-rich datasets.


Responsibilities

1. Data Preparation and Transformation:

Leverage PySpark to efficiently process and transform large-scale datasets from various sources.

Develop robust and scalable PySpark code to handle data cleaning, munging, and feature engineering tasks.

Collaborate with Data Scientists to understand their data requirements and translate them into efficient PySpark workflows.

2. Feature Engineering:

Work closely with Data Scientists to identify and implement relevant feature engineering techniques.

Employ advanced feature engineering methods, such as one-hot encoding, scaling, binning, and feature creation/selection, to enhance the predictive power of machine learning models.

Stay up-to-date with the latest feature engineering techniques and best practices in the industry.

3. Distributed Computing:

Leverage Apache Spark's distributed computing capabilities to process and analyze large-scale datasets efficiently.

Optimize PySpark code for performance, scalability, and fault tolerance.

Implement and maintain data pipelines using PySpark to automate data preparation and transformation processes.

4. Code Quality and Documentation:

Write clean, maintainable, and well-documented PySpark code following best practices and coding standards.

Collaborate with team members through code reviews and knowledge sharing sessions.

Contribute to the development and maintenance of PySpark-related documentation and best practices within the organization.

5. Continuous Learning and Improvement:

Stay current with the latest developments in PySpark, Apache Spark, and related big data technologies.

Actively participate in professional development opportunities, such as attending conferences, workshops, or online training.

Identify areas for improvement in existing data preparation and transformation processes and propose solutions.

Qualifications:

Bachelor's or Master's degree in Computer Science, Data Science, or a related field.

Proven experience as a Senior PySpark Engineer or a similar role, with a minimum of 5 years of experience working with PySpark and Apache Spark.

Strong proficiency in Python programming and experience with PySpark APIs and libraries.

Solid understanding of distributed computing principles and experience working with large-scale datasets.

Familiarity with feature engineering techniques and their application in machine learning pipelines.

Experience in developing and maintaining data pipelines and workflows using PySpark.

Excellent problem-solving, analytical, and critical thinking skills.

Strong communication and collaboration skills to work effectively with Data Scientists and cross-functional teams.

Passion for staying up-to-date with the latest developments in the big data and data engineering domains.

  • Senior Data Engineer

    6 days ago

    Plano, United States Comprehensive Resources Inc Full time

    Only W2 Job Title: Senior Data Engineer (Location: Plano , Tx (hybrid)Client - Capital oneExperience: 10+ LOB - FS Data TeamJDResponsibilities:• building highly scalable resilient real time data platform which is critical for the organization in terms of deliver real time data insights• building this real time data platform which primarily encompasses...

  • Senior Data Engineer

    1 week ago

    Plano, United States Comprehensive Resources Inc Full time

    Only W2 Job Title: Senior Data Engineer (Location: Plano , Tx (hybrid)Client - Capital oneExperience: 10+ LOB - FS Data TeamJDResponsibilities:• building highly scalable resilient real time data platform which is critical for the organization in terms of deliver real time data insights• building this real time data platform which primarily encompasses...

  • Senior Data Engineer

    12 hours ago

    Plano, United States Comprehensive Resources Inc Full time

    Only W2 Job Title: Senior Data Engineer (Location: Plano , Tx (hybrid)Client - Capital oneExperience: 10+ LOB - FS Data TeamJDResponsibilities:• building highly scalable resilient real time data platform which is critical for the organization in terms of deliver real time data insights• building this real time data platform which primarily encompasses...

  • Senior Manager of Data Engineering

    3 weeks ago

    Plano, United States JPMorgan Chase Full time

    You have the opportunity to unleash your full potential at a world-renowned company and take the lead in shaping the future of technology.As a Senior Manager of Data Engineering at JPMorgan Chase within the Corporate Sector, Data Services, you serve in a leadership role by providing technical coaching and advisory for multiple technical teams, as well as...

  • Senior Lead Engineer

    1 week ago

    Plano, United States NTT DATA Full time

    Company Overview Req ID: 268435 NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Senior Lead Engineer to join our team in Plano, Texas (US-TX), United States (US). Job...

  • Senior Data Engineer

    1 week ago

    Plano, United States PepsiCo Full time

    OverviewPepsiCo operates in an environment undergoing immense and rapid change. Big-data and digital technologies are driving business transformation that is unlocking new capabilities and business innovations in areas like eCommerce, mobile experiences and IoT. The key to winning in these areas is being able to leverage enterprise data foundations built...

  • Senior Data Engineer

    20 hours ago

    Plano, United States PepsiCo Full time

    OverviewPepsiCo operates in an environment undergoing immense and rapid change. Big-data and digital technologies are driving business transformation that is unlocking new capabilities and business innovations in areas like eCommerce, mobile experiences and IoT. The key to winning in these areas is being able to leverage enterprise data foundations built...

  • Senior Databricks Engineer/Data Engineer

    3 weeks ago

    Plano, United States Elite Mente LLC Full time

    Job DescriptionJob DescriptionSenior Databricks Engineer/Data Engineer Plano, TX (Onsite) (open to all Visa W2 only)Long term ContractJob Description:They need to hire an expert someone who can direct the team and provide the guidance in how to modernize/re-write/re-build/etc.This person must be a Databricks expertMust be very strong in Python (java is a...

  • Senior GenAI Data Engineer

    1 week ago

    Plano, Texas, United States Amdocs Full time

    Job ID: 184155Required Travel :No TravelManagerial - NoLocation: :USA-TX, Plano (AM)Who are we?Amdocs helps those who build the future to make it amazing. With our market-leading portfolio of software products and services, we unlock our customers' innovative potential, empowering them to provide next-generation communication and media experiences for both...

  • Senior GenAI Data Engineer

    22 hours ago

    Plano, Texas, United States Amdocs Full time

    Job ID: 184155Required Travel :No TravelManagerial - NoLocation: :USA-TX, Plano (AM)Who are we?Amdocs helps those who build the future to make it amazing. With our market-leading portfolio of software products and services, we unlock our customers' innovative potential, empowering them to provide next-generation communication and media experiences for both...

  • Senior Civil Engineer

    3 weeks ago

    Plano, United States Olsson Full time

    Job Description As a Senior Civil Engineer on our Data Center Civil Team, you will be a part of the firms largest and most complex projects. You will serve as a project manager on some projects and lead design engineer on others. Prepare planning and design documents, process design calculations, and develop and maintain team and client standards. You may...

  • Senior Civil Engineer

    1 week ago

    Plano, United States Olsson Full time

    Company Description We are Olsson, a team-based, purpose-driven engineering and design firm. Our solutions improve communities and our people make it possible. Our most meaningful asset is our people, and we are dedicated to providing an environment where they can continue to learn, grow, and thrive. Our entrepreneurial spirit is what has allowed us...

  • Senior Civil Engineer

    21 hours ago

    Plano, United States Olsson Full time

    Company Description We are Olsson, a team-based, purpose-driven engineering and design firm. Our solutions improve communities and our people make it possible. Our most meaningful asset is our people, and we are dedicated to providing an environment where they can continue to learn, grow, and thrive. Our entrepreneurial spirit is what has allowed us...

  • Senior Data Engineer

    2 months ago

    Plano, United States Fannie Mae Full time

    Job DescriptionAs a valued colleague on our team, you will collaborate with team in designing, producing, testing, or implementing moderately complex software, technology, or processes, as well as create and maintain IT architecture, large scale data stores, and cloud-based systems.THE IMPACT YOU WILL MAKEThe Corporate Functions Technology - Software...

  • Cloud Security Engineer

    1 week ago

    Plano, United States Esolvit Full time

    Job Title: Mid-Level Cloud Security Engineer No of Openings: 5 Salary: $115000.00 Client Company: USAA - Plano Location: Plano, TX Required Skills • Cloud Security background • Programmed in the cloud • Stood up 3rd party applications on the cloud • Have done things like key management and search management • Certifications with the cloud providers...

  • Data Engineer II

    2 months ago

    Plano, United States Public Storage Full time

    Job Description As a key member of our leading-edge, full-stack team, the Data Engineer II role offers an unparalleled opportunity to advance your career in a stable, S&P 500 company renowned for its innovative spirit, collaborative team culture and commitment to technical excellence. In this elevated position, you are instrumental in advancing our...

  • Data Engineer II

    7 hours ago

    Plano, United States Public Storage Full time

    Job Description As a key member of our leading-edge, full-stack team, the Data Engineer II role offers an unparalleled opportunity to advance your career in a stable, S&P 500 company renowned for its innovative spirit, collaborative team culture and commitment to technical excellence. In this elevated position, you are instrumental in advancing our...

  • Data Engineer II

    1 week ago

    Plano, United States Public Storage Full time

    Job DescriptionJob DescriptionCompany DescriptionSince opening our first self-storage facility in 1972, Public Storage has grown to become the largest owner and operator of self-storage facilities in the world. With thousands of locations across the U.S. and Europe, and more than 170 million net rentable square feet of real estate, we're also one of the...

  • Data Engineer II

    3 hours ago

    Plano, United States Public Storage Full time

    Job DescriptionJob DescriptionCompany DescriptionSince opening our first self-storage facility in 1972, Public Storage has grown to become the largest owner and operator of self-storage facilities in the world. With thousands of locations across the U.S. and Europe, and more than 170 million net rentable square feet of real estate, we're also one of the...

  • Senior Lead Software Engineer

    19 hours ago

    Plano, United States JPMorgan Chase Bank, N.A. Full time

    Be an integral part of an agile team that's constantly pushing the envelope to enhance, build, and deliver top-notch technology products. As a Senior Lead Software Engineer at JPMorgan Chase within the Corporate & Investment Bank , you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products...