Lead Data Engineer

3 weeks ago

Scottsdale, United States E-Solutions Full time

Role : Lead Data Engineer

Location : Scottsdale AZ (100% onsite)


Must have skill set: Spark, S3, Glue, AWS Redshift , python and stream set exp

6-8 years of IT experience focusing on enterprise data architecture and management.

• Experience in Conceptual/Logical/Physical Data Modelling & expertise in Relational and Dimensional Data Modelling

• Experience with Databricks & on Prem , Structured Streaming, Delta Lake concepts, and Delta Live Tables required

• Experience with Spark scala

• Data Lake concepts such as time travel and schema evolution and optimization

• Structured Streaming and Delta Live Tables with Databricks a bonus

• Experience leading and architecting enterprise-wide initiatives specifically system integration, data migration, transformation, data warehouse build, data mart build, and data lakes implementation / support

• Advanced level understanding of streaming data pipelines and how they differ from batch systems

• Formalize concepts of how to handle late data, defining windows, and data freshness

• Advanced understanding of ETL and ELT and ETL/ELT tools such as Data Migration Service etc

• Understanding of concepts and implementation strategies for different incremental data loads such as tumbling window, sliding window, high watermark, etc.

• Familiarity and/or expertise with Great Expectations or other data quality/data validation frameworks a bonus

• Familiarity with concepts such as late data, defining windows, and how window definitions impact data freshness

• Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design performance optimization)

• Indexing and partitioning strategy experience

• Debug, troubleshoot, design and implement solutions to complex technical issues

• Experience with large-scale, high-performance enterprise big data application deployment and solution

• Architecture experience in AWS environment a bonus

• Familiarity working with Lambda specifically with how to push and pull data, how to use AWS tools to view data for processing massive data at scale a bonus

• Experience with Gitlabs and CloudWatch and ability to write and maintain gitlabs for supporting CI/CD pipelines

• Experience working with AWS Lambdas for configuration and optimization and experience with S3

• Familiarity with Schema Registry, message formats such as Avro, ORC, etc.

• Ability to thrive in a team-based environment

• Experience briefing the benefits and constraints of technology solutions to technology partners, stakeholders, team members, and senior level of management

  • Lead Data Engineer

    3 weeks ago

    Scottsdale, United States E-Solutions Full time

    Role : Lead Data Engineer Location : Scottsdale AZ (100% onsite)Must have skill set: Spark, S3, Glue, AWS Redshift , python and stream set exp 6-8 years of IT experience focusing on enterprise data architecture and management.• Experience in Conceptual/Logical/Physical Data Modelling & expertise in Relational and Dimensional Data Modelling• Experience...

  • Lead Data Engineer

    3 weeks ago

    Scottsdale, United States E-Solutions Full time

    Role : Lead Data Engineer Location : Scottsdale AZ (100% onsite)Must have skill set: Spark, S3, Glue, AWS Redshift , python and stream set exp 6-8 years of IT experience focusing on enterprise data architecture and management.• Experience in Conceptual/Logical/Physical Data Modelling & expertise in Relational and Dimensional Data Modelling• Experience...

  • Hiring for Lead Data Engineer

    3 days ago

    Scottsdale, United States Persistent Systems LLC (New York) Full time

    About PersistentWe are a trusted Digital Engineering and Enterprise Modernization partner, combining deep technical expertise and industry experience to help our clients anticipate what's next. Our offerings and proven solutions create a unique competitive advantage for our clients by giving them the power to see beyond and rise above. We work with many...

  • Hiring for Lead Data Engineer

    7 days ago

    Scottsdale, United States Persistent Systems Full time

    About PersistentWe are a trusted Digital Engineering and Enterprise Modernization partner, combining deep technical expertise and industry experience to help our clients anticipate what’s next. Our offerings and proven solutions create a unique competitive advantage for our clients by giving them the power to see beyond and rise above. We work with many...

  • Data Engineer

    3 weeks ago

    Scottsdale, United States CBTS Full time

    CBTS is currently seeking a Data Engineer for a position located in Scottsdale, AZ. Responsibilities:Collaborate with fellow data engineering and cross-functional data teams to build and deliver custom enterprise data warehousing and database applications.Accurately estimate tasks, bugs, and spikes in your domain.Develop software that meets code quality...

  • Data Engineer

    1 month ago

    Scottsdale, United States CBTS Full time

    CBTS is currently seeking a Data Engineer for a position located in Scottsdale, AZ. Responsibilities:Collaborate with fellow data engineering and cross-functional data teams to build and deliver custom enterprise data warehousing and database applications.Accurately estimate tasks, bugs, and spikes in your domain.Develop software that meets code quality...

  • Senior Data Engineer

    4 weeks ago

    Scottsdale, United States RiverPoint Full time

    Job DescriptionJob DescriptionInfomatics is hiring a Senior Data Engineer on a direct hire/FTE basis to lead design, development and implementation efforts on our client’s Enterprise Analytics platform which includes an MPP Enterprise Data Warehouse, cloud-based Data Lake and other Big Data technologies. Essential Duties and Responsibilities:Guides the...

  • Senior Data Engineer

    3 weeks ago

    Scottsdale, United States RiverPoint Full time

    Job DescriptionJob DescriptionInfomatics is hiring a Senior Data Engineer on a direct hire/FTE basis to lead design, development and implementation efforts on our client’s Enterprise Analytics platform which includes an MPP Enterprise Data Warehouse, cloud-based Data Lake and other Big Data technologies. Essential Duties and Responsibilities:Guides the...

  • Data Engineer

    2 weeks ago

    Scottsdale, United States Stepstone Full time

    Company Description Appcast is the global leader in programmatic recruitment advertising technology and services. With advanced technology, unmatched market data and a team of the industry’s best recruitment marketers, Appcast’s technology and services manage job advertising annually for enterprise clients across industries. Headquartered in Lebanon,...

  • Principle Data Engineer

    1 month ago

    Scottsdale, United States CBTS Full time

    Position Information:Seeking skilled Principal Data Engineers who thrive in a highly collaborative, agile team environment. Engineers design elegant technical solutions for complex product requirements, deliver high quality code as part of a fast-moving agile team, and own the stability and quality of their domain. We would love to hear from talented...

  • Principal Data Engineer

    3 weeks ago

    Scottsdale, United States CBTS Full time

    Position Information:Seeking skilled Principal Data Engineers who thrive in a highly collaborative, agile team environment. Engineers design elegant technical solutions for complex product requirements, deliver high quality code as part of a fast-moving agile team, and own the stability and quality of their domain. We would love to hear from talented...

  • Gke data engineer

    3 weeks ago

    Scottsdale, United States Randstad Full time

    job summary: Our client is in need of a GKE Data Engineering Lead for a 2 year engagement to assist in solutioning the architecture as we Transform the rebate processing system from legacy platform. The potential candidate will be helping the engineers in rearchitecting, transforming, and designing data pipelines. This resource will come in and help do a...

  • Data Engineer

    3 days ago

    Scottsdale, United States Persistent Systems Full time

    About PersistentWe are a trusted Digital Engineering and Enterprise Modernization partner, combining deep technical expertise and industry experience to help our clients anticipate what’s next. Our offerings and proven solutions create a unique competitive advantage for our clients by giving them the power to see beyond and rise above. We work with many...

  • Data Engineer

    7 days ago

    Scottsdale, United States Persistent Systems Full time

    About PersistentWe are a trusted Digital Engineering and Enterprise Modernization partner, combining deep technical expertise and industry experience to help our clients anticipate what’s next. Our offerings and proven solutions create a unique competitive advantage for our clients by giving them the power to see beyond and rise above. We work with many...

  • Data Engineer III

    3 weeks ago

    Scottsdale, United States Troon Golf Full time

    Data Engineer is responsible for building a leading-edge Data & Analytics platform for enabling our new Troon Data Estate architecture & related analytics, business domain specific analytics & management (Finance, HR, Operations, & other), and general enterprise analytics needs. Designs, develops, maintains, and supports the cloud-based (Microsoft Azure) big...

  • Data Engineer III

    2 weeks ago

    Scottsdale, United States Troon Golf Full time

    Data Engineer is responsible for building a leading-edge Data & Analytics platform for enabling our new Troon Data Estate architecture & related analytics, business domain specific analytics & management (Finance, HR, Operations, & other), and general enterprise analytics needs. Designs, develops, maintains, and supports the cloud-based (Microsoft Azure) big...

  • Data Engineer III

    2 weeks ago

    Scottsdale, United States Troon Golf Full time

    Data Engineer is responsible for building a leading-edge Data & Analytics platform for enabling our new Troon Data Estate architecture & related analytics, business domain specific analytics & management (Finance, HR, Operations, & other), and general enterprise analytics needs. Designs, develops, maintains, and supports the cloud-based (Microsoft Azure) big...

  • Data Engineer

    1 week ago

    Scottsdale, United States Collabera Full time

    Description Home Search Jobs Job Description Data Engineer Contract: Scottsdale, Arizona, US Salary: $50.00 Per Hour Job Code: 350344 End Date: 2024-07-05 Days Left: 23 days, 3 hours left Apply We are looking for Sr Data Engineer with one of our direct client. Location:-Scottsdale, AZ Duration:-10 monthsAbout the Role: We are looking for a Sr. Data...

  • Hiring for Data Engineer

    2 months ago

    Scottsdale, United States Persistent Systems Full time

    About PersistentWe are a trusted Digital Engineering and Enterprise Modernization partner, combining deep technical expertise and industry experience to help our clients anticipate what’s next. Our offerings and proven solutions create a unique competitive advantage for our clients by giving them the power to see beyond and rise above. We work with many...

  • Data Engineer III ICP

    4 days ago

    Scottsdale, Arizona, United States HonorHealth Full time

    OverviewLooking to be part of something more meaningful? At HonorHealth, you'll be part of a team, creating a multi-dimensional care experience for our patients. You'll have opportunities to make a difference. From our Ambassador Movement to our robust training and development programs, you can select where and how you want to make an impact. HonorHealth...