See more Collapse

big data engineer

2 months ago

Malvern, United States Randstad Full time

big data engineer.

  • malvern , pennsylvania

  • posted 1 day ago

job details

summary

  • $56.33 - $61.33 per hour

  • contract

  • bachelor degree

  • category computer and mathematical occupations

  • reference1057503

job details

job summary:

  • Integrate AWS Services such as Lambda, DynamoDB, EC2, RDS, S3, Athena, Data pipeline, API gateway, Glue, EMR etc. in reporting application to improve the accessibility/efficiency

  • Write scripts and troubleshoot for performance using relevant programming languages like SQL, Spark and Python

  • Develop and Implement ETL jobs using AWS Glue to extract, transform, and load data from various sources

  • Design and implement scalable data solutions using AWS, PySpark, Python and other related services

  • Work closely with Data and Analytics team to build Data Lake using various AWS services like EMR, S3, DynamoDB, Lambda etc., Apache Airflow, PySpark, and HIVE

  • Use AWS data pipeline for Data Extraction, Transformation and Loading from homogeneous or heterogeneous data sources

  • Work on migration of PySpark framework into AWS Glue for enhanced processing

  • Design and Develop web applications using Java EE, Python, Web Services and AWS Qualified candidates will be responsible for API design, implementation, testing and prod support.

  • They will work on a data modernization project which will utilize multiple AWS Services Tech needs: Java AWS Services - Glue, EMR API development SpringBoot services

  • They are aws tech experience. dynamo and/or azure db.

  • Knowledge of Resilancy patterns and chaos testing. Kafka knowledge

Nice to have:

  • Architecture experience in cloud or legacy systems. Building large scale systems. Java 11 understanding. spring boot experience and jersey framework knowledge , micro service architecture , graph ql implementation , Performance tuning of Java rest APIs.

  • Design, Model, Develop, Test and Implement using Big Data technologies like Python, Pyspark, Hadoop, UNIX and SQL

location: Malvern, Pennsylvania

job type: Contract

salary: $56.33 - 61.33 per hour

work hours: 8am to 5pm

education: Bachelors

responsibilities:

  • Integrate AWS Services such as Lambda, DynamoDB, EC2, RDS, S3, Athena, Data pipeline, API gateway, Glue, EMR etc. in reporting application to improve the accessibility/efficiency

  • Write scripts and troubleshoot for performance using relevant programming languages like SQL, Spark and Python

  • Develop and Implement ETL jobs using AWS Glue to extract, transform, and load data from various sources

  • Design and implement scalable data solutions using AWS, PySpark, Python and other related services

  • Work closely with Data and Analytics team to build Data Lake using various AWS services like EMR, S3, DynamoDB, Lambda etc., Apache Airflow, PySpark, and HIVE

qualifications:

  • Experience level: Experienced

  • Minimum 5 years of experience

  • Education: Bachelors

skills:

  • Data WarehouseEqual Opportunity Employer: Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status.At Randstad Digital, we welcome people of all abilities and want to ensure that our hiring and interview process meets the needs of all applicants. If you require a reasonable accommodation to make your application or interview experience a great one, please contact HRsupport@randstadusa.com.Pay offered to a successful candidate will be based on several factors including the candidate's education, work experience, work location, specific job duties, certifications, etc. In addition, Randstad Digital offers a comprehensive benefits package, including health, an incentive and recognition program, and 401K contribution (all benefits are based on eligibility).This posting is open for thirty (30) days.

We have other current jobs related to this field that you can find below

  • Big Data Engineer

    1 week ago

    Malvern, United States Vision It US Full time

    Job DescriptionJob DescriptionResponsibilitiesResponsible for working closely with the data analysts for data discovery, data engineers to provide technical guidance for data ingestion to bring data from various sources into the AEP. Responsible for data quality in the platform, monitor data ingestion ,solid understanding of the schemas, identity service,...

  • Big Data Consultant

    4 weeks ago

    Malvern, United States Veridic Solutions Full time

    Big Data EngineerLocation: Dallas, TX Required / Must HaveSpark/AWS Analytics/Lambda/Dynamo/Aurora Development experience with Python, APIM, Ansible, a little bit of C#, Hadoop (with parquet files) Experience with Microsoft Azure services (must have Azure Functions, Azure Storage Accounts, Azure Data Lake)

  • Big Data Consultant

    1 month ago

    Malvern, United States Veridic Solutions Full time

    Big Data EngineerLocation: Dallas, TX Required / Must HaveSpark/AWS Analytics/Lambda/Dynamo/Aurora Development experience with Python, APIM, Ansible, a little bit of C#, Hadoop (with parquet files) Experience with Microsoft Azure services (must have Azure Functions, Azure Storage Accounts, Azure Data Lake)

  • Data Engineer, Specialist

    2 weeks ago

    Malvern, United States The Vanguard Group Full time

    Data Engineer, Specialist (The Vanguard Group, Inc. / Malvern, PA) -- Analyze and process Extract, Transform and Load (ETL) Data Requests; support of NonProd and Production Environments, both domestic and internationally based; engage in projects related to design and enhancement of existing Infrastructure; provide technical consultation and support to...

  • Data Engineer

    4 weeks ago

    Malvern, United States Veridic Solutions Full time

    Title: Big Data AWS EngineerLocation: Malvern PAContractResponsibilitiesLearns and experiments with new technologies, tools and processes.Learns and understands client area business functions and requirements.Determines the appropriate technical tool to address the client's business needs.Designs, implements, and deploys cloud platforms to support and meet...

  • Data Engineer

    1 month ago

    Malvern, United States Veridic Solutions Full time

    Title: Big Data AWS EngineerLocation: Malvern PAContractResponsibilitiesLearns and experiments with new technologies, tools and processes.Learns and understands client area business functions and requirements.Determines the appropriate technical tool to address the client's business needs.Designs, implements, and deploys cloud platforms to support and meet...

  • Data Engineer

    1 month ago

    Malvern, United States Goli Tech Full time

    Job DescriptionJob DescriptionData Management AnalystProvides data domain aligned support for a given business area. Ensures the quality and accuracy of data. Serves as a thought partner and point of contact to resolve operational and service issues. Will perform data validation and documenting use casesData Management Analysts will be hybridized as a...

  • Data Engineer

    2 weeks ago

    Malvern, United States Cogent Infotech Corp Full time

    Desired Skills and Experience: *- 6+ years of experience in the Data Engineering Space - Experience writing ETL processes, designing database systems and developing tools for real-time and offline analytic processing. - Experience with AWS Technologies like: Glue ETL, S3 RDS etc. - Strong experience with PySpark, preferably for building Data Pipelines -...

  • Data Engineer

    3 days ago

    Malvern, United States Vision It US Full time

    Job DescriptionJob DescriptionResponsibilitiesWe would like an engineer that can build data and machine learning ingestion patterns and assist with basic data analytics and visualization. We are building a new data hub where we are loading and transforming new data into S3, building machine learning models in Sagemaker and helping our clients query, connect...

  • Data Engineer

    2 weeks ago

    Malvern, United States CyberTec Full time

    Client: Vanguard New Role: Data Engineer Pay Rate: $68.00/hr. on C2C) Location: Malvern, PA(Hybrid 3 days a week, must be onsite on day 1) Owns and manages the overall data strategy and architecture for Charitable (enterprise-wide) Understand the data and data needs across both SaaS applications and their on-prem legacy apps Go deep on evolving data &...

  • Data Engineer

    2 weeks ago

    Malvern, Arkansas, United States CubeSmart Full time

    OverviewThis is a hybrid role base in Malvern, PACubeSmart is currently seeking a Data Engineer to join the Information Technology team at our corporate office in Malvern, PA. The Data Engineer will be responsible for the implementation, development, and optimization of database, data objects, and database scripting with the core database platforms utilized...

  • Data Engineer

    3 weeks ago

    Malvern, United States CubeSmart Full time

    Overview This is a hybrid role base in Malvern, PA CubeSmart is currently seeking a Data Engineer to join the Information Technology team at our corporate office in Malvern, PA. The Data Engineer will be responsible for the implementation, development, and optimization of database, data objects, and database scripting with the core database platforms...

  • API Developer/Data Engineer

    1 month ago

    Malvern, United States Vision It US Full time

    Job DescriptionJob DescriptionDesign and Develop web applications using Java EE, Python, Web Services and AWS Qualified candidates will be responsible for API design, implementation, testing and prod support. They will work on a data modernization project which will utilize multiple AWS Services Tech needs: Java AWS Services - GlueEMR API development...

  • Data Engineer- Malvern, PA

    2 weeks ago

    Malvern, United States Georgia IT Inc Full time

    Job Title: Data Engineer Location: CA, PA Duration: Contract Rate: DOE(USC. GC, GCEAD and L2EADS are only preferred)Candidate Should be a local to Malvern or Charlotte, Relocation wont work on thisDuties and responsibilities: Provides advanced data solutions by using software to process, store, and serve data to others. Tests data quality and optimizes data...

  • FinTech Data Scientist

    2 weeks ago

    Malvern, United States Vanguard Full time

    FinTech Data Scientist Supports Vanguard's mission of exploring FinTech technologies that could dramatically improve investment performance or make capital markets work better and fosters a culture of innovation. Identifies potential experiments with financial technology related to alternative data sources, predictive analytics, and machine learning....

  • Data Scientist II

    2 weeks ago

    Malvern, United States The Vanguard Group Full time

    Data Scientist II (The Vanguard Group, Inc. / Malvern, PA) -- Develop queries and perform extensive programming to access, transform, and prepare data for statistical modeling; perform deep dive diagnostic, predictive, and prescriptive analytics to support data-driven business decision making; identify and diagnose data inconsistencies and errors, document...

  • Data Scientist

    3 weeks ago

    Malvern, United States Entech Full time

    We’re living and working in the midst of a digital revolution, and the demand for optimized technology practices has never been more critical for businesses. At Entech, it’s our mission to help companies navigate the changing environment and accelerate digital realization through the power of solutions, services, and product platforms. With comprehensive...

  • Data Scientist

    3 weeks ago

    Malvern, United States Entech Full time

    We’re living and working in the midst of a digital revolution, and the demand for optimized technology practices has never been more critical for businesses. At Entech, it’s our mission to help companies navigate the changing environment and accelerate digital realization through the power of solutions, services, and product platforms. With comprehensive...

  • AWS BIgdata Engineer

    3 months ago

    Malvern, United States Mergen IT Full time

    Design develop and deploy modern data warehouse solutions in AWS cloud platform.• Gather and document business requirements and translate it into technical design documents.• Create technical documentation e.g. data flow diagrams end user guide and technical operations guides.• Create technical documentation e.g. data flow diagrams end user guide...

  • AEP Data Engineer

    6 days ago

    Malvern, United States Outcome Logix ( A Tech 50 Finalist company 2022, by Pittsburgh Technology Council ) Full time

    An ideal candidate would be someone who has led the implementation of the Adobe Experience Platform (AEP) for seamless data ingestion, schema management, and identity stitching, ensuring data quality, governance compliance, and effective stakeholder collaboration across cross-functional teams.Position Overview:We are seeking a highly motivated Adobe...