Sr Lead Data Engineer Apache Spark, Spark Core, Spark SQL, Spark Streaming – Databricks
Found in: Appcast US C2 - 1 week ago
Direct Client - Sr Lead Data Engineer Apache Spark, Spark Core, Spark SQL, Spark Streaming – Databricks
Location: Hybrid (Seattle, WA)
Length: 12+ Months
Job description
This position contributes to Client success by building enterprise data services for analytic solutions. This position is responsible for the design, development, testing, and support of data pipelines to enable continuous data processing for data exploration, data preparation, and real-time business analytics. Models and acts in accordance with Client guiding principles.
Tops 3 Skills Needed
- Data Engineering - 5+ years
- Data Bricks - 5+ years
- Sparks/ DevOps practices - 5+ years
Years of Experience:
- 5+ years
Technology requirements:
- Proficiency in Apache Spark, including Spark Core, Spark SQL and Spark Streaming.
- Proficiency in languages such as Python for data processing and scripting.
Basic Qualifications/ Experience:
- Experience in designing and implementing ETL processes using Databricks notebooks for efficient data extraction, transformation, and loading.
- In-depth knowledge of the Databricks Unified Analytics Platform and its features for collaborative big data analytics.
- Understanding of data modeling concepts for designing database structures.
- Proficiency in working with both relational databases and NoSQL databases.
- Integration of data from diverse sources, including APIs, databases, streaming data, and external data feeds.
- Implementation of processes to ensure data quality, including data validation, cleansing, and error handling.
- Knowledge of cluster management, optimization, and scaling for efficient data processing.
- Optimization of Spark jobs and Databricks clusters for better performance.
- Proficiency in cloud platforms such as Azure for building scalable and flexible data architectures.
- Use of tools like Apache Airflow, ADF, or Databricks to orchestrate and schedule data workflows.
Degree or certifications required:
- Education (minimum education level, degree, or certification necessary): Bachelor’s degree in computer science, management information systems, or related discipline
Skills (minimum skills required):
- 5-7+ years of architecture and designing large-scale high-performance distributed systems
- 5-7+ years SQL Platform
- 2+ years Exposure No-SQL Platform is a plus
- 5+ years of Hadoop, YARN, MapReduce, Pig or Hive, Spark
- 2+ years of Data platform implementation on Azure or AWS is a plus
Key Responsibilities:
- Responsibilities and essential job functions include but are not limited to the following:
- Demonstrate deep knowledge and ability to lead others in the data engineering team to build and support non-interactive (batch, distributed) & real-time, highly available data, data pipeline, and technology capabilities
- Translate strategic requirements into business requirements to ensure solutions meet business needs
- Work with infrastructure provisioning & configuration tools to develop scripts to automate the deployment of physical and virtual environments; to develop tools to monitor the usage of virtual resources
- Assist in the definition of architecture that ensures that solutions are built within a consistent framework
- Lead resolution activities for complex data issues
- Define & implement data retention policies and procedures
- Define & implement data governance policies and procedures
- Identify improvements in team coding standards and help in the implementation of the improvements
- Leverage subject matter expertise to coordinate issue resolution efforts across peer support groups, technical support teams, and vendors
- Develop and maintain documentation relating to all assigned systems and projects
- Perform systems and applications performance characterization and trade-off studies through analysis and simulation
- Perform root cause analysis to identify permanent resolutions to software or business process issues
- Lead by example by demonstrating the Client's mission and value
Nice-to-Haves:
- Knowledge of data security best practices and the implementation of measures to ensure data privacy and compliance.
- Implementation of monitoring and logging solutions to track the health and performance of pipelines.
- Familiarity with monitoring platforms like DataDog and New Relic
- Azure
-
Seattle, United States Chateck Inc Full timeDirect Client - Sr Lead Data Engineer Apache Spark, Spark Core, Spark SQL, Spark Streaming – Data Bricks Location: Hybrid (Seattle, WA) Length: 12+ Months Job description This position contributes to Client success by building enterprise data services for analytic solutions. This position is responsible for design, development, testing and support for...
-
Seattle, United States Resource Informatics Group Inc Full timeJob DescriptionJob Description Title: Senior Spark SQL Developer with Databricks Location: Seattle, WA # of Positions: 1Duration: 6 monthsRates: DOE Primary Skills: Spark, SQL, Databricks Job Description: Must Have Skills (Top 3 technical skills only) * Spark SQL Databricks Detailed Job Description:5 to 8 years of post-college working experience...
-
Senior Spark SQL Developer with Databricks exp
16 hours ago
Seattle, United States Resource Informatics Group Inc Full timeJob DescriptionJob Description Title: Senior Spark SQL Developer with Databricks Location: Seattle, WA # of Positions: 1Duration: 6 monthsRates: DOE Primary Skills: Spark, SQL, Databricks Job Description: Must Have Skills (Top 3 technical skills only) * Spark SQL Databricks Detailed Job Description:5 to 8 years of post-college working experience...
-
Lead Data Engineer
6 days ago
Seattle, United States JobRialto Full timeDescription: Key projects: Exploration of generative AI Technology requirements: Proficiency in Apache Spark, including Spark Core, Spark SQL and Spark Streaming. Proficiency in languages such as Python for data processing and scripting. Degree or certifications required: Bachelor's degree in computer science, management information systems, or related...
-
Azure Databricks Support Engineer
Found in: Appcast US C2 - 2 weeks ago
Seattle, United States Insight Global Full timeTop 3 Hard Skills Required + Years of Experience 1. Minimum 3 years experience in Databricks 2. Minimum 3 years experience with Azure, 3. Minimum 3 years experience Spark knowledge and coding Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and/or Azure Databricks and/or HDInsights · Delivering...
-
Sr. Product Manager, Databricks SQL
Found in: Careerbuilder One Red US C2 - 6 days ago
Seattle, WA, United States DataBricks Full timeRDQ125R36At Databricks, we are passionate about enabling data teams to solve the world’s toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world’s best data and AI infrastructure platform so our customers can use deep data insights...
-
Sr. Data Engineer Data Activation
3 days ago
Seattle, United States INSPYR Solutions Full timeTitle: Sr. Data Engineer Data Activation & Sharing Location: Seattle, WA (Hybrid 2-3 days a week) Duration: 12+ month contract Compensation: $87.00-$95.40/hr Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S. We are looking for a Sr. Data Engineer to join the Data Activation team who can operate across the enterprise to ensure...
-
Sr. Data Engineer Data Activation
Found in: Jooble US O C2 - 2 days ago
Seattle, WA, United States INSPYR Solutions Full timeTitle: Sr. Data Engineer Data Activation & Sharing Location: Seattle, WA (Hybrid 2-3 days a week) Duration: 12+ month contract Compensation: $87.00-$95.40/hr Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S. We are looking for a Sr. Data Engineer to join the Data Activation team who can operate across the enterprise to ensure...
-
Sr. Product Manager
Found in: Careerbuilder One Red US C2 - 6 days ago
Seattle, WA, United States DataBricks Full timeRDQ225R413This role requires coordinating all product activities from vision to implementation, including engaging customers to understand their needs, developing long-term product strategy, defining product roadmaps, working with engineering to build those products, and coordinating with various internal stakeholders (both pre- and post-launch) to ensure...
-
Sr. Data Engineer Data Activation
Found in: Appcast US C2 - 1 day ago
Seattle, United States INSPYR Solutions Full timeTitle: Sr. Data Engineer Data Activation & SharingLocation: Seattle, WA (Hybrid 2-3 days a week)Duration: 12+ month contractCompensation: $87.00-$95.40/hrWork Requirements: US Citizen, GC Holders or Authorized to Work in the U.S.We are looking for a Sr. Data Engineer to join the Data Activation team whocan operate across the enterprise to ensure on-time,...
-
Sr. Data Engineer Data Activation
Found in: Appcast Linkedin GBL C2 - 1 day ago
Seattle, United States INSPYR Solutions Full timeTitle: Sr. Data Engineer Data Activation & SharingLocation: Seattle, WA (Hybrid 2-3 days a week)Duration: 12+ month contractCompensation: $87.00-$95.40/hrWork Requirements: US Citizen, GC Holders or Authorized to Work in the U.S.We are looking for a Sr. Data Engineer to join the Data Activation team whocan operate across the enterprise to ensure on-time,...
-
Data Engineer
1 week ago
Seattle, United States Resource Informatics Group Inc Full timeJob DescriptionJob DescriptionResponsibilities: 1. Work in big data. Take existing solutions, optimize and build high-performance algorithms. Scale algorithms on terabytes of data. Improve time complexity and space complexity of data pre-processing. Recommend ways to improve data reliability, efficiency and quality.2. Process structured and unstructured...
-
Jira and Atlassian Solutions Engineer | 157
1 week ago
Seattle, United States DataBricks Full timeJob DescriptionJob DescriptionSummaryWe are a product led organization transforming the way we work from how easy it is to use our IT services to the applications we develop that help us scale seamlessly in face of incredible growth.The Corporate Engineering team is responsible for building / providing services and infrastructure that enable bricksters to be...
-
Data Modellers
Found in: Jooble US O C2 - 2 days ago
Seattle, WA, United States INSPYR Solutions Full timeTitle: Sr. Data Engineer Data Activation & Sharing Location: Seattle, WA (Hybrid 2-3 days a week) Duration: 12+ month contract Compensation: $87.00-$95.40/hr Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S. We are looking for a Sr. Data Engineer to join the Data Activation team who can operate across the enterprise to ensure...
-
Staff Product Manager, Next Gen Analytics
Found in: Careerbuilder One Red US C2 - 6 days ago
Seattle, WA, United States DataBricks Full timeRDQ125R35At Databricks, we have built the most capable platform for data teams to leverage AI and analytics to put their organization’s data to use. Our ambition is to enable everyone, including those outside traditional data-specialist roles, to be able to do the same.You will be the product manager for leading the charge to make the Lakehouse simple to...
-
Sr. Data Engineer
6 days ago
Seattle, United States CyberTec Full timeFTE - project location -: Seattle. WA or California Onsite 2 days in a week Note : Client is strictly needs a Senior Data Engineer Profile who can stand alone and work with all the Hadoop technology stack.Job Description Our Ads & Data Platforms team, a segment under my client is looking for a Lead Data Engineer. Data is essential for all our decision-making...
-
Data Engineer, Amazon Prime
6 days ago
Seattle, United States Amazon.com Inc Full timeBASIC QUALIFICATIONS- 1 years of data engineering experience- Experience with data modeling, warehousing and building ETL pipelines- Experience with SQL- Experience with one or more query language (e.g., SQL, PL/ SQL, DDL, MDX, Hive. QL, Spark. SQ Data Engineer, Engineer, Customer Experience, Research Scientist, Science, AWS, Technology
-
Machine Learning Engineer
Found in: Jooble US O C2 - 1 week ago
Seattle, WA, United States Expedia, Inc. Full timeIf you need assistance during the recruiting process due to a disability, please reach out to our Recruiting Accommodations Team through the Accommodation Request form . This form is used only by individuals with disabilities who require assistance or adjustments in applying and interviewing for a job. This form is not for inquiring about a position or the...
-
Senior Data Engineer
5 days ago
Seattle, United States Tavant Full timeOwn the end-to-end architecture of the data platform, ensuring its efficiency, cost-effectiveness, security, and governance. Collaborate closely with other engineers to design and build an optimal and cost-efficient platform solution. Work in partnership with other engineers and managers to design and develop foundational elements of the platform. Create...
-
Senior Data Engineer
1 week ago
Seattle, United States Tavant Full timeOwn the end-to-end architecture of the data platform, ensuring its efficiency, cost-effectiveness, security, and governance. Collaborate closely with other engineers to design and build an optimal and cost-efficient platform solution. Work in partnership with other engineers and managers to design and develop foundational elements of the platform. Create...