Direct Client – Sr Lead Data Engineer Apache Spark, Spark Core, Spark SQL– Data Bricks

3 weeks ago

Seattle, United States Chateck Inc Full time

Direct Client - Sr Lead Data Engineer Apache Spark, Spark Core, Spark SQL, Spark Streaming – Data Bricks
Location: Hybrid (Seattle, WA)
Length: 12+ Months

Job description
This position contributes to Client success by building enterprise data services for analytic solutions. This position is responsible for design, development, testing and support for data pipelines to enable continuous data processing for data exploration, data preparation and real-time business analytics. Models and acts in accordance with Client guiding principles.

Tops 3 Skills Needed

1

Data Engineering

5+ years

2

Data Bricks

5+ years

3

Sparks/ Dev ops practices

5+ years

Years of Experience:

5+ years

Technology requirements:

Proficiency in Apache Spark, including Spark Core, Spark SQL and Spark Streaming.
Proficiency in languages such as Python for data processing and scripting.

Basic Qualifications/ Experience:

Experience in designing and implementing ETL processes using Databricks notebooks for efficient data extraction, transformation, and loading.
In-depth knowledge of the Databricks Unified Analytics Platform and its features for collaborative big data analytics.
Understanding of data modeling concepts for designing database structures.
Proficiency in working with both relational databases and NoSQL databases.
Integration of data from diverse sources, including APIs, databases, streaming data, and external data feeds.
Implementation of processes to ensure data quality, including data validation, cleansing, and error handling.
Knowledge of cluster management, optimization, and scaling for efficient data processing.
Optimization of Spark jobs and Databricks clusters for better performance.
Proficiency in cloud platforms such as Azure for building scalable and flexible data architectures.
Use of tools like Apache Airflow, ADF, or Databricks to orchestrate and schedule data workflows.

Degree or certifications required:

Education (minimum education level, degree or certification necessary): Bachelor’s degree in computer science, management information systems, or related discipline

Skills (minimum skills required):

5-7+ years Architect and design large scale high performance distributed systems
5-7+ years SQL Platform
2+ years Exposure No-SQL Platform is a plus
5+ years Hadoop, YARN, MapReduce, Pig or Hive, Spark
2+ years Data platform implementation on Azure or AWS is a plus

Key Responsibilities:

Responsibilities and essential job functions include but are not limited to the following:
Demonstrate deep knowledge and ability to lead others in the data engineering team to build and support noninteractive (batch, distributed) & real-time, highly available data, data pipeline and technology capabilities
Translate strategic requirements into business requirements to ensure solutions meet business needs
Work with infrastructure provisioning & configuration tools to develop scripts to automate deployment of physical and virtual environments; to develop tools to monitor usage of virtual resources
Assist in the definition of architecture that ensure that solutions are built within a consistent framework
Lead resolution activities for complex data issues
Define & implement data retention policies and procedures
Define & implement data governance policies and procedures
Identify improvements in team coding standards and help in implementation of the improvements
Leverage subject matter expertise to coordinate issue resolution efforts across peer support groups, technical support teams, and vendors
Develop and maintain documentation relating to all assigned systems and projects
Perform systems and applications performance characterization and trade-off studies through analysis and simulation
Perform root cause analysis to identify permanent resolutions to software or business process issues
Lead by example by demonstrating the Client mission and value

Nice-to-Haves:

Knowledge of data security best practices and the implementation of measures to ensure data privacy and compliance.
Implementation of monitoring and logging solutions to track the health and performance of pipelines.
Familiarity with monitoring platforms like DataDog and New Relic
Azure

  • Lead Data Engineer

    7 days ago

    Seattle, United States JobRialto Full time

    Description: Key projects: Exploration of generative AI Technology requirements: Proficiency in Apache Spark, including Spark Core, Spark SQL and Spark Streaming. Proficiency in languages such as Python for data processing and scripting. Degree or certifications required: Bachelor's degree in computer science, management information systems, or related...

  • Senior Spark SQL Developer with Databricks exp

    2 weeks ago

    Seattle, United States Resource Informatics Group Inc Full time

    Job DescriptionJob Description Title: Senior Spark SQL Developer with Databricks Location: Seattle, WA # of Positions: 1Duration: 6 monthsRates: DOE Primary Skills: Spark, SQL, Databricks Job Description: Must Have Skills (Top 3 technical skills only) * Spark  SQL Databricks Detailed Job Description:5 to 8 years of post-college working experience...

  • Senior Spark SQL Developer with Databricks exp

    2 days ago

    Seattle, United States Resource Informatics Group Inc Full time

    Job DescriptionJob Description Title: Senior Spark SQL Developer with Databricks Location: Seattle, WA # of Positions: 1Duration: 6 monthsRates: DOE Primary Skills: Spark, SQL, Databricks Job Description: Must Have Skills (Top 3 technical skills only) * Spark  SQL Databricks Detailed Job Description:5 to 8 years of post-college working experience...

  • Sr. Data Engineer Data Activation

    5 days ago

    Seattle, United States INSPYR Solutions Full time

    Title: Sr. Data Engineer Data Activation & Sharing Location: Seattle, WA (Hybrid 2-3 days a week) Duration: 12+ month contract Compensation: $87.00-$95.40/hr Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S. We are looking for a Sr. Data Engineer to join the Data Activation team who can operate across the enterprise to ensure...

  • Data Engineer

    2 weeks ago

    Seattle, United States Resource Informatics Group Inc Full time

    Job DescriptionJob DescriptionResponsibilities:  1. Work in big data. Take existing solutions, optimize and build high-performance algorithms. Scale algorithms on terabytes of data. Improve time complexity and space complexity of data pre-processing. Recommend ways to improve data reliability, efficiency and quality.2. Process structured and unstructured...

  • Sr. Data Engineer Data Activation

    Found in: Jooble US O C2 - 4 days ago

    Seattle, WA, United States INSPYR Solutions Full time

    Title: Sr. Data Engineer Data Activation & Sharing Location: Seattle, WA (Hybrid 2-3 days a week) Duration: 12+ month contract Compensation: $87.00-$95.40/hr Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S. We are looking for a Sr. Data Engineer to join the Data Activation team who can operate across the enterprise to ensure...

  • Sr. Data Engineer Data Activation

    Found in: Appcast US C2 - 3 days ago

    Seattle, United States INSPYR Solutions Full time

    Title: Sr. Data Engineer Data Activation & SharingLocation: Seattle, WA (Hybrid 2-3 days a week)Duration: 12+ month contractCompensation: $87.00-$95.40/hrWork Requirements: US Citizen, GC Holders or Authorized to Work in the U.S.We are looking for a Sr. Data Engineer to join the Data Activation team whocan operate across the enterprise to ensure on-time,...

  • Sr. Data Engineer Data Activation

    Found in: Appcast Linkedin GBL C2 - 3 days ago

    Seattle, United States INSPYR Solutions Full time

    Title: Sr. Data Engineer Data Activation & SharingLocation: Seattle, WA (Hybrid 2-3 days a week)Duration: 12+ month contractCompensation: $87.00-$95.40/hrWork Requirements: US Citizen, GC Holders or Authorized to Work in the U.S.We are looking for a Sr. Data Engineer to join the Data Activation team whocan operate across the enterprise to ensure on-time,...

  • Senior Data Engineer

    7 days ago

    Seattle, United States Tavant Full time

    Own the end-to-end architecture of the data platform, ensuring its efficiency, cost-effectiveness, security, and governance. Collaborate closely with other engineers to design and build an optimal and cost-efficient platform solution. Work in partnership with other engineers and managers to design and develop foundational elements of the platform. Create...

  • Senior Data Engineer

    2 weeks ago

    Seattle, United States Tavant Full time

    Own the end-to-end architecture of the data platform, ensuring its efficiency, cost-effectiveness, security, and governance. Collaborate closely with other engineers to design and build an optimal and cost-efficient platform solution. Work in partnership with other engineers and managers to design and develop foundational elements of the platform. Create...

  • Sr. Data Engineer

    1 week ago

    Seattle, United States CyberTec Full time

    FTE - project location -: Seattle. WA or California Onsite 2 days in a week Note : Client is strictly needs a Senior Data Engineer Profile who can stand alone and work with all the Hadoop technology stack.Job Description Our Ads & Data Platforms team, a segment under my client is looking for a Lead Data Engineer. Data is essential for all our decision-making...

  • Data Engineer for Data Platform

    1 week ago

    Seattle, United States Jupiter Power Full time

    Description Team Introduction The Data Platform team works on building data infrastructures and data products to support business engineering teams. As a data engineer in the data platform team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the opportunity to gain hands-on experience on...

  • Data Engineer, Amazon Prime

    7 days ago

    Seattle, United States Amazon.com Inc Full time

    BASIC QUALIFICATIONS- 1 years of data engineering experience- Experience with data modeling, warehousing and building ETL pipelines- Experience with SQL- Experience with one or more query language (e.g., SQL, PL/ SQL, DDL, MDX, Hive. QL, Spark. SQ Data Engineer, Engineer, Customer Experience, Research Scientist, Science, AWS, Technology

  • Data Engineer for Data Platform

    2 weeks ago

    Seattle, United States Hireio, Inc. Full time

    Job DescriptionJob DescriptionTeam IntroductionThe Data Platform team works on building data infrastructures and data products to support business engineering teams.As a data engineer in the data platform team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the opportunity to gain...

  • Data Modellers

    Found in: Jooble US O C2 - 4 days ago

    Seattle, WA, United States INSPYR Solutions Full time

    Title: Sr. Data Engineer Data Activation & Sharing Location: Seattle, WA (Hybrid 2-3 days a week) Duration: 12+ month contract Compensation: $87.00-$95.40/hr Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S. We are looking for a Sr. Data Engineer to join the Data Activation team who can operate across the enterprise to ensure...

  • Data Engineer for Data Platform

    Found in: Appcast Linkedin GBL C2 - 4 days ago

    Seattle, United States HireIO, Inc. Full time

    Team IntroductionThe Data Platform team works on building data infrastructures and data products to support business engineering teams.As a data engineer in the data platform team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the opportunity to gain hands-on experience on all kinds of...

  • Azure Databricks Support Engineer

    Found in: Appcast US C2 - 2 weeks ago

    Seattle, United States Insight Global Full time

    Top 3 Hard Skills Required + Years of Experience 1. Minimum 3 years experience in Databricks 2. Minimum 3 years experience with Azure, 3. Minimum 3 years experience Spark knowledge and coding Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and/or Azure Databricks and/or HDInsights · Delivering...

  • Senior Data Engineer for E-Commerce

    2 weeks ago

    Seattle, United States Hireio, Inc. Full time

    Job DescriptionJob DescriptionThe Global E-Commerce team focuses on building data infrastructure and data product areas to support business engineering teams working directly.As a data engineer in the Global E-Commerce team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the...

  • Senior Data Engineer for E-Commerce

    Found in: Resume Library US A2 - 2 weeks ago

    Seattle, Washington, United States Hireio, Inc. Full time

    The Global E-Commerce team focuses on building data infrastructure and data product areas to support business engineering teams working directly. As a data engineer in the Global E-Commerce team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the opportunity to gain hands-on experience...

  • Senior Data Engineer for E-Commerce

    Found in: Appcast Linkedin GBL C2 - 4 days ago

    Seattle, United States HireIO, Inc. Full time

    The Global E-Commerce team focuses on building data infrastructure and data product areas to support business engineering teams working directly.As a data engineer in the Global E-Commerce team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the opportunity to gain hands-on experience on...