Lead Data Engineer-AWS, Python
2 weeks ago
Role: Lead Data Engineer-AWS, Python
Location: Scottsdale AZ (100% onsite)
Duration:: FTE and CTH
Job Description:
Must have skill set: Spark, S3, Glue, AWS Redshift , python and stream set exp
6-8 years of IT experience focusing on enterprise data architecture and management.
Experience in Conceptual/Logical/Physical Data Modelling & expertise in Relational and Dimensional Data Modelling
Experience with Databricks & on Prem , Structured Streaming, Delta Lake concepts, and Delta Live Tables required
Experience with Spark scala
Data Lake concepts such as time travel and schema evolution and optimization
Structured Streaming and Delta Live Tables with Databricks a bonus
Experience leading and architecting enterprise-wide initiatives specifically system integration, data migration, transformation, data warehouse build, data mart build, and data lakes implementation / support
Advanced level understanding of streaming data pipelines and how they differ from batch systems
Formalize concepts of how to handle late data, defining windows, and data freshness
Advanced understanding of ETL and ELT and ETL/ELT tools such as Data Migration Service etc
Understanding of concepts and implementation strategies for different incremental data loads such as tumbling window, sliding window, high watermark, etc.
Familiarity and/or expertise with Great Expectations or other data quality/data validation frameworks a bonus
Familiarity with concepts such as late data, defining windows, and how window definitions impact data freshness
Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design performance optimization)
Indexing and partitioning strategy experience
Debug, troubleshoot, design and implement solutions to complex technical issues
Experience with large-scale, high-performance enterprise big data application deployment and solution
Architecture experience in AWS environment a bonus
Familiarity working with Lambda specifically with how to push and pull data, how to use AWS tools to view data for processing massive data at scale a bonus
Experience with Gitlabs and CloudWatch and ability to write and maintain gitlabs for supporting CI/CD pipelines
Experience working with AWS Lambdas for configuration and optimization and experience with S3
Familiarity with Schema Registry, message formats such as Avro, ORC, etc.
Ability to thrive in a team-based environment
Experience briefing the benefits and constraints of technology solutions to technology partners, stakeholders, team members, and senior level of management
-
Lead Data Engineer with AWS Redshift
2 weeks ago
Scottsdale, United States Vy Systems Full timeMust have skill set: Spark, S3, Glue, AWS Redshift , python and stream set exp 6-8 years of IT experience focusing on enterprise data architecture and management. Experience in Conceptual/Logical/Physical Data Modelling & expertise in Relational and Dimensional Data Modelling Experience with Databricks & on Prem , Structured Streaming, Delta Lake concepts,...
-
AWS Data Engineer
1 month ago
Scottsdale, Arizona, United States Cognizant Technology Solutions Full timeAbout AI & Analytics: Artificial intelligence (AI) and the data it collects and analyzes will soon sit at the core of all intelligent, human-centric businesses. By decoding customer needs, preferences, and behaviors, our clients can understand exactly what services, products, and experiences their consumers need. Within AI & Analytics, we work to design the...
-
Lead Data Engineer
2 weeks ago
Scottsdale, United States Enexus Global Full timeRole : Lead Data Engineer Location : Scottsdale AZ (100% onsite) Contract Type : W2/C2C/1099 Experience : 9+ Years Must have skill set: Spark, S3, Glue, AWS Redshift , python and stream set exp +8 years of IT experience focusing on enterprise data architecture and management. Experience in Conceptual/Logical/Physical Data Modelling & expertise in...
-
Senior Data Engineer
4 days ago
Scottsdale, United States RiverPoint Full timeJob DescriptionJob DescriptionInfomatics is hiring a Senior Data Engineer on a direct hire/FTE basis to lead design, development and implementation efforts on our client’s Enterprise Analytics platform which includes an MPP Enterprise Data Warehouse, cloud-based Data Lake and other Big Data technologies. Essential Duties and Responsibilities:Guides the...
-
Python Django Engineer
1 month ago
Scottsdale, United States Encora Digital Inc Full time**Python Django Engineer** Description **Job Summary:** The Developers primary function will be design, back-end development, and maintenance of a Django applications templates and DB data access. **Position details:** * Be responsible for the design, development, maintenance, and support of a Django application, which includes the development and...
-
Solution Architect
2 weeks ago
Scottsdale, United States MindWave Solutions Full timeLooking for local consultants ONLY (ARIZONA)OPT/CPT please excuse!Minimum 12+ years experience ONLY Job Description:The selected candidate will work with the Senior team to understand the needs and lead the development of compelling solutions. The person will bring these solutions forward through white papers, presentations, and proposal responses. Requires...
-
Staff Software Engineer
5 days ago
Scottsdale, United States Venmo Full timeAt PayPal (NASDAQ: PYPL), we believe that every person has the right to participate fully in the global economy. Our mission is to revolutionize commerce globally to make moving money, selling and shopping, personalized and secure. Job Description Summary: What you need to know about the role- As a Staff Software Engineer, you will act as a subject...
-
Scottsdale, United States Onsemi Full timeSenior Manager of Software Engineering, Data Integration Location: US (Scottsdale, AZ) About the Role: We are seeking a highly skilled and experienced software engineering leader to lead the design and development of our modern enterprise data integration platform. This role will focus on designing and implementing a new data...
-
Senior Data Engineer
1 month ago
Scottsdale, United States Cardone Ventures Full timePOSITION SUMMARYThe Senior Data Engineer will play a vital role in designing, developing, and maintaining the data pipelines and infrastructure that ingests, transforms, and stores massive datasets from various sources to support our evolving business needs. You'll be at the forefront of implementing a cutting-edge lake house architecture, ensuring efficient...
-
Senior Data Engineer
1 month ago
Scottsdale, United States Cardone Ventures Full timePOSITION SUMMARYThe Senior Data Engineer will play a vital role in designing, developing, and maintaining the data pipelines and infrastructure that ingests, transforms, and stores massive datasets from various sources to support our evolving business needs. You'll be at the forefront of implementing a cutting-edge lake house architecture, ensuring efficient...
-
Senior Data Engineer
1 month ago
Scottsdale, United States Cardone Ventures Full timePOSITION SUMMARYThe Senior Data Engineer will play a vital role in designing, developing, and maintaining the data pipelines and infrastructure that ingests, transforms, and stores massive datasets from various sources to support our evolving business needs. You'll be at the forefront of implementing a cutting-edge lake house architecture, ensuring efficient...
-
Hiring for Data Engineer
4 weeks ago
Scottsdale, United States Persistent Systems Full timeAbout PersistentWe are a trusted Digital Engineering and Enterprise Modernization partner, combining deep technical expertise and industry experience to help our clients anticipate what’s next. Our offerings and proven solutions create a unique competitive advantage for our clients by giving them the power to see beyond and rise above. We work with many...
-
Hiring for Data Engineer
4 weeks ago
Scottsdale, United States Persistent Systems Full timeAbout PersistentWe are a trusted Digital Engineering and Enterprise Modernization partner, combining deep technical expertise and industry experience to help our clients anticipate what’s next. Our offerings and proven solutions create a unique competitive advantage for our clients by giving them the power to see beyond and rise above. We work with many...
-
Senior Data Engineer
3 weeks ago
Scottsdale, United States Techwizens LLC Full timeMust have : Skillset: Java, Scala, S3, Glue, aws , Redshift. 6-8 years of IT experience focusing on enterprise data architecture and management.• Experience in Conceptual/Logical/Physical Data Modeling & expertise in Relational and Dimensional Data Modeling• Experience with Databricks & on Prem, Structured Streaming, Delta Lake concepts, and Delta Live...
-
Senior Data Engineer
3 weeks ago
Scottsdale, United States Techwizens LLC Full timeMust have : Skillset: Java, Scala, S3, Glue, aws , Redshift. 6-8 years of IT experience focusing on enterprise data architecture and management.• Experience in Conceptual/Logical/Physical Data Modeling & expertise in Relational and Dimensional Data Modeling• Experience with Databricks & on Prem, Structured Streaming, Delta Lake concepts, and Delta Live...
-
Senior Data Engineer
1 month ago
Scottsdale, United States Ignitec Inc Full timeJob DescriptionJob DescriptionWe are seeking a Senior Data Engineer - Direct Hire to become an integral part of our development team! You will design strategies for database systems as well as set standards for operations, programming and security. Design and construct large relational databases and expertise in Relational & Dimensional Data...
-
Lead Database Engineer
1 week ago
Scottsdale, United States PayPal Holdings, Inc. Full timeWhat you need to know about the role- At Venmo, we are creating a product that people love. We strive to create a delightful user experience while connecting the world and empowering people through payments. We are looking for intellectually curious Database Engineer, Engineer, Database, Lead, Operations, AWS, Technology
-
Lead Java AWS Developer
2 weeks ago
Scottsdale, Arizona, United States American Express Global Business Travel Full timeAmex GBT is a place where colleagues find inspiration in travel as a force for good and - through their work - can make an impact on our industry. We're here to help our colleagues achieve success and offer an inclusive and collaborative culture where your voice is valued.Ready to explore a career path? Start your journey.We are embarking on an exciting...
-
Quality Assurance Engineer
5 days ago
Scottsdale, United States VySystems Full timeMust have : Automation , java , AWS , ETL , Scala , Spark • 6-8 years of IT experience focusing on Testing and Automation experience using Java and Groovy.. Rest assured app concepts in automation. • Experience with Databricks & on Prem, Structured Streaming, Delta Lake concepts, and Delta Live Tables required• Experience with Spark Scala and Java...
-
Quality Assurance Engineer
4 days ago
Scottsdale, United States VySystems Full timeMust have : Automation , java , AWS , ETL , Scala , Spark • 6-8 years of IT experience focusing on Testing and Automation experience using Java and Groovy.. Rest assured app concepts in automation. • Experience with Databricks & on Prem, Structured Streaming, Delta Lake concepts, and Delta Live Tables required• Experience with Spark Scala and Java...