Sr Lead Data Engineer Apache Spark, Spark Core, Spark SQL, Spark Streaming – Databricks

Found in: Appcast US C2 - 1 week ago


Seattle, United States The AES Group Full time

Direct Client - Sr Lead Data Engineer Apache Spark, Spark Core, Spark SQL, Spark StreamingDatabricks

Location: Hybrid (Seattle, WA)

Length: 12+ Months


Job description

This position contributes to Client success by building enterprise data services for analytic solutions. This position is responsible for the design, development, testing, and support of data pipelines to enable continuous data processing for data exploration, data preparation, and real-time business analytics. Models and acts in accordance with Client guiding principles.


Tops 3 Skills Needed

  1. Data Engineering - 5+ years
  2. Data Bricks - 5+ years
  3. Sparks/ DevOps practices - 5+ years


Years of Experience:

  • 5+ years


Technology requirements:

  • Proficiency in Apache Spark, including Spark Core, Spark SQL and Spark Streaming.
  • Proficiency in languages such as Python for data processing and scripting.


Basic Qualifications/ Experience:

  • Experience in designing and implementing ETL processes using Databricks notebooks for efficient data extraction, transformation, and loading.
  • In-depth knowledge of the Databricks Unified Analytics Platform and its features for collaborative big data analytics.
  • Understanding of data modeling concepts for designing database structures.
  • Proficiency in working with both relational databases and NoSQL databases.
  • Integration of data from diverse sources, including APIs, databases, streaming data, and external data feeds.
  • Implementation of processes to ensure data quality, including data validation, cleansing, and error handling.
  • Knowledge of cluster management, optimization, and scaling for efficient data processing.
  • Optimization of Spark jobs and Databricks clusters for better performance.
  • Proficiency in cloud platforms such as Azure for building scalable and flexible data architectures.
  • Use of tools like Apache Airflow, ADF, or Databricks to orchestrate and schedule data workflows.


Degree or certifications required:

  • Education (minimum education level, degree, or certification necessary): Bachelor’s degree in computer science, management information systems, or related discipline


Skills (minimum skills required):

  • 5-7+ years of architecture and designing large-scale high-performance distributed systems
  • 5-7+ years SQL Platform
  • 2+ years Exposure No-SQL Platform is a plus
  • 5+ years of Hadoop, YARN, MapReduce, Pig or Hive, Spark
  • 2+ years of Data platform implementation on Azure or AWS is a plus


Key Responsibilities:

  • Responsibilities and essential job functions include but are not limited to the following:
  • Demonstrate deep knowledge and ability to lead others in the data engineering team to build and support non-interactive (batch, distributed) & real-time, highly available data, data pipeline, and technology capabilities
  • Translate strategic requirements into business requirements to ensure solutions meet business needs
  • Work with infrastructure provisioning & configuration tools to develop scripts to automate the deployment of physical and virtual environments; to develop tools to monitor the usage of virtual resources
  • Assist in the definition of architecture that ensures that solutions are built within a consistent framework
  • Lead resolution activities for complex data issues
  • Define & implement data retention policies and procedures
  • Define & implement data governance policies and procedures
  • Identify improvements in team coding standards and help in the implementation of the improvements
  • Leverage subject matter expertise to coordinate issue resolution efforts across peer support groups, technical support teams, and vendors
  • Develop and maintain documentation relating to all assigned systems and projects
  • Perform systems and applications performance characterization and trade-off studies through analysis and simulation
  • Perform root cause analysis to identify permanent resolutions to software or business process issues
  • Lead by example by demonstrating the Client's mission and value


Nice-to-Haves:

  • Knowledge of data security best practices and the implementation of measures to ensure data privacy and compliance.
  • Implementation of monitoring and logging solutions to track the health and performance of pipelines.
  • Familiarity with monitoring platforms like DataDog and New Relic
  • Azure



  • Seattle, United States Chateck Inc Full time

    Direct Client - Sr Lead Data Engineer Apache Spark, Spark Core, Spark SQL, Spark Streaming – Data Bricks Location: Hybrid (Seattle, WA) Length: 12+ Months Job description This position contributes to Client success by building enterprise data services for analytic solutions. This position is responsible for design, development, testing and support for...


  • Seattle, United States Resource Informatics Group Inc Full time

    Job DescriptionJob Description Title: Senior Spark SQL Developer with Databricks Location: Seattle, WA # of Positions: 1Duration: 6 monthsRates: DOE Primary Skills: Spark, SQL, Databricks Job Description: Must Have Skills (Top 3 technical skills only) * Spark  SQL Databricks Detailed Job Description:5 to 8 years of post-college working experience...


  • Seattle, United States Resource Informatics Group Inc Full time

    Job DescriptionJob Description Title: Senior Spark SQL Developer with Databricks Location: Seattle, WA # of Positions: 1Duration: 6 monthsRates: DOE Primary Skills: Spark, SQL, Databricks Job Description: Must Have Skills (Top 3 technical skills only) * Spark  SQL Databricks Detailed Job Description:5 to 8 years of post-college working experience...

  • Lead Data Engineer

    6 days ago


    Seattle, United States JobRialto Full time

    Description: Key projects: Exploration of generative AI Technology requirements: Proficiency in Apache Spark, including Spark Core, Spark SQL and Spark Streaming. Proficiency in languages such as Python for data processing and scripting. Degree or certifications required: Bachelor's degree in computer science, management information systems, or related...

  • Azure Databricks Support Engineer

    Found in: Appcast US C2 - 2 weeks ago


    Seattle, United States Insight Global Full time

    Top 3 Hard Skills Required + Years of Experience 1. Minimum 3 years experience in Databricks 2. Minimum 3 years experience with Azure, 3. Minimum 3 years experience Spark knowledge and coding Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and/or Azure Databricks and/or HDInsights · Delivering...

  • Sr. Product Manager, Databricks SQL

    Found in: Careerbuilder One Red US C2 - 6 days ago


    Seattle, WA, United States DataBricks Full time

    RDQ125R36At Databricks, we are passionate about enabling data teams to solve the world’s toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world’s best data and AI infrastructure platform so our customers can use deep data insights...


  • Seattle, United States INSPYR Solutions Full time

    Title: Sr. Data Engineer Data Activation & Sharing Location: Seattle, WA (Hybrid 2-3 days a week) Duration: 12+ month contract Compensation: $87.00-$95.40/hr Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S. We are looking for a Sr. Data Engineer to join the Data Activation team who can operate across the enterprise to ensure...

  • Sr. Data Engineer Data Activation

    Found in: Jooble US O C2 - 2 days ago


    Seattle, WA, United States INSPYR Solutions Full time

    Title: Sr. Data Engineer Data Activation & Sharing Location: Seattle, WA (Hybrid 2-3 days a week) Duration: 12+ month contract Compensation: $87.00-$95.40/hr Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S. We are looking for a Sr. Data Engineer to join the Data Activation team who can operate across the enterprise to ensure...

  • Sr. Product Manager

    Found in: Careerbuilder One Red US C2 - 6 days ago


    Seattle, WA, United States DataBricks Full time

    RDQ225R413This role requires coordinating all product activities from vision to implementation, including engaging customers to understand their needs, developing long-term product strategy, defining product roadmaps, working with engineering to build those products, and coordinating with various internal stakeholders (both pre- and post-launch) to ensure...

  • Sr. Data Engineer Data Activation

    Found in: Appcast US C2 - 1 day ago


    Seattle, United States INSPYR Solutions Full time

    Title: Sr. Data Engineer Data Activation & SharingLocation: Seattle, WA (Hybrid 2-3 days a week)Duration: 12+ month contractCompensation: $87.00-$95.40/hrWork Requirements: US Citizen, GC Holders or Authorized to Work in the U.S.We are looking for a Sr. Data Engineer to join the Data Activation team whocan operate across the enterprise to ensure on-time,...

  • Sr. Data Engineer Data Activation

    Found in: Appcast Linkedin GBL C2 - 1 day ago


    Seattle, United States INSPYR Solutions Full time

    Title: Sr. Data Engineer Data Activation & SharingLocation: Seattle, WA (Hybrid 2-3 days a week)Duration: 12+ month contractCompensation: $87.00-$95.40/hrWork Requirements: US Citizen, GC Holders or Authorized to Work in the U.S.We are looking for a Sr. Data Engineer to join the Data Activation team whocan operate across the enterprise to ensure on-time,...

  • Data Engineer

    1 week ago


    Seattle, United States Resource Informatics Group Inc Full time

    Job DescriptionJob DescriptionResponsibilities:  1. Work in big data. Take existing solutions, optimize and build high-performance algorithms. Scale algorithms on terabytes of data. Improve time complexity and space complexity of data pre-processing. Recommend ways to improve data reliability, efficiency and quality.2. Process structured and unstructured...


  • Seattle, United States DataBricks Full time

    Job DescriptionJob DescriptionSummaryWe are a product led organization transforming the way we work from how easy it is to use our IT services to the applications we develop that help us scale seamlessly in face of incredible growth.The Corporate Engineering team is responsible for building / providing services and infrastructure that enable bricksters to be...

  • Data Modellers

    Found in: Jooble US O C2 - 2 days ago


    Seattle, WA, United States INSPYR Solutions Full time

    Title: Sr. Data Engineer Data Activation & Sharing Location: Seattle, WA (Hybrid 2-3 days a week) Duration: 12+ month contract Compensation: $87.00-$95.40/hr Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S. We are looking for a Sr. Data Engineer to join the Data Activation team who can operate across the enterprise to ensure...

  • Staff Product Manager, Next Gen Analytics

    Found in: Careerbuilder One Red US C2 - 6 days ago


    Seattle, WA, United States DataBricks Full time

    RDQ125R35At Databricks, we have built the most capable platform for data teams to leverage AI and analytics to put their organization’s data to use.  Our ambition is to enable everyone, including those outside traditional data-specialist roles, to be able to do the same.You will be the product manager for leading the charge to make the Lakehouse simple to...

  • Sr. Data Engineer

    6 days ago


    Seattle, United States CyberTec Full time

    FTE - project location -: Seattle. WA or California Onsite 2 days in a week Note : Client is strictly needs a Senior Data Engineer Profile who can stand alone and work with all the Hadoop technology stack.Job Description Our Ads & Data Platforms team, a segment under my client is looking for a Lead Data Engineer. Data is essential for all our decision-making...


  • Seattle, United States Amazon.com Inc Full time

    BASIC QUALIFICATIONS- 1 years of data engineering experience- Experience with data modeling, warehousing and building ETL pipelines- Experience with SQL- Experience with one or more query language (e.g., SQL, PL/ SQL, DDL, MDX, Hive. QL, Spark. SQ Data Engineer, Engineer, Customer Experience, Research Scientist, Science, AWS, Technology

  • Machine Learning Engineer

    Found in: Jooble US O C2 - 1 week ago


    Seattle, WA, United States Expedia, Inc. Full time

    If you need assistance during the recruiting process due to a disability, please reach out to our Recruiting Accommodations Team through the Accommodation Request form . This form is used only by individuals with disabilities who require assistance or adjustments in applying and interviewing for a job. This form is not for inquiring about a position or the...


  • Seattle, United States Tavant Full time

    Own the end-to-end architecture of the data platform, ensuring its efficiency, cost-effectiveness, security, and governance. Collaborate closely with other engineers to design and build an optimal and cost-efficient platform solution. Work in partnership with other engineers and managers to design and develop foundational elements of the platform. Create...


  • Seattle, United States Tavant Full time

    Own the end-to-end architecture of the data platform, ensuring its efficiency, cost-effectiveness, security, and governance. Collaborate closely with other engineers to design and build an optimal and cost-efficient platform solution. Work in partnership with other engineers and managers to design and develop foundational elements of the platform. Create...