Azure Data Engineer

3 weeks ago


Sunnyvale, United States JobRialto Full time
Description:

Data Bricks(Must Have)

Strong hands on in Pyspark and Apache Spark

Strong hands on in Medallion architecture

Experience in Native Spark Migration to Databricks.

Experience in Building Data Governance Solutions like Unity Catalog, Azure Purview etc.

Highly experienced in Usability Optimization (Auto Compaction, Ordering, Vacuuming), Cost Optimization and Performance Optimization.

Build Very Strong Orchestration Layer in Databricks/ADF.... Workflows.

Build CICD for Databricks in Azure Devops.

Process near Real time Data thru Auto Loader, DLT Pipelines.

Implement Security Layer in Delta Lake.

Implement Massive Parallel Processing Layers in Spark SQL and PySpark.

Implement Cost effective Infrastructure in Databricks.

Experience In extracting logic and from on prem layers, SAP, ADLS into Pyspark/ADLS using ADF/Databricks.

Azure Synapse Analytics/Azure data Factory (ADF)(Must Have)

Hands on Experience in Azure Synapse Analytics, Azure Data Factory and Data Bricks, Azure Storage, Azure Key Vault, SQL Pools CI/CD Pipeline Designing and other Azure services like functions, logic apps

Linked services, Various Runtimes, Datasets, Pipelines, Activities

Strong Hands-on Experience in Various Activities like Control flow logic and conditions (For Each, if, switch, until), Lookup, Stored procedure, scripts, validations, Copy Data, Data flow, Azure functions, Notebooks, SQL Pool Stored procedures and etc

Strong hands on exp in deployment of code throughout landscape (Dev -> QA -> Prod), Git Hub, CI/CD pipelines and etc

SQL Server stored procedures(Must have)

strong hands on creating the SQL stored procedures

Functions, Stored Procedures, how to call one SP into another, How to process record-by-record

Dynamic SQL

Python(Must have)

Must have strong background about the Python libraries like PySpark, Pandas, NumPy, pymysql, Oracle, Pyspark libraries

Must have strong hands on to get data through APIs

Must be able to install libraries and help users to troubleshoot issues

Must have knowledge to get the data through stored procedures via Python

Should be able to debug the Python code

Sparks(Must have)

Hands on experience in Spark Pools, PySpark

Should be able to merge data/delta loads through Notebooks

Must have strong background about the Python libraries and PySpark

Education: Bachelors Degree

  • Sunnyvale, California, United States Intuitive Surgical Full time

    Job DescriptionWe need a skilled Senior Azure cloud platform engineer to join our team. You'll design, implement, and manage infrastructure on Azure. Work with development teams to deploy and manage Azure AI/ML/Cognitive services environments effectively.Roles and Responsibilities:Infrastructure as Code (IaC)Design, implement, and manage infrastructure using...

  • Data Engineer

    6 days ago


    Sunnyvale, United States Achmea Full time

    De afdeling Je bent werkzaam binnen het chapter Data Engineering & Science, onderdeel van de afdeling Customer Center of Data (ongeveer 40 collega’s). Het chapter bestaat uit 12 collega’s (data engineers en data scientists). De afdeling Customer Center of Data werkt primair voor het merk Zilveren Kruis, maar bedient ook de andere zorgmerken van Achmea:...

  • Azure DevOps Engineer

    2 weeks ago


    Sunnyvale, United States Redolent Infotech Pvt. Ltd. Full time

    One of our direct client is urgently looking for an Azure DevOps Engineer @ Sunnyvale CA TITLE: Azure DevOps Engineer LOCATION:Sunnyvale, CA Duration: 6 to 12+ Months Rate: DOE Description: Docker, Kubernetes and environment creation using scripts or java / ruby are the key requirements. 1. Resource should have minimum 2+ years on cloud technologies. 2....

  • Backend Engineer

    2 weeks ago


    Sunnyvale, United States Vision It US Full time

    Job DescriptionJob DescriptionDescriptionJob Duties:Develops innovation strategies, processes, and best practices.Drives the execution of multiple business plans and projects.Supports and ensures business objectives being met.Leads the discovery phase of medium to large projects to come up with high level design.Leads the work of other small group of 3 to 4...


  • Sunnyvale, California, United States Illumio Full time

    On-site work model with 5 days in office/week in Sunnyvale, CA About the Team:Illumio's new Machine Learning (ML) team embodies a culture of thought leadership, independence, and responsibility, holding a significant and highly visible role within our Engineering division. This dynamic team is tasked with developing state-of-the-art ML solutions for complex...

  • Data Scientist

    4 days ago


    Sunnyvale, United States Achmea Full time

    De afdeling De afdeling Product- en Portfoliomanagement Schade Bedrijven bestaat uit productmanagers, data analisten, pricing analisten, data scientists, business intelligence en scrum masters, in totaal zo’n 60 collega’s. Deze professionals werken voor de merken Centraal Beheer, Interpolis en Avéro Achmea. Ze houden zich specifiek bezig met het...

  • AI Tooling Engineer

    1 month ago


    Sunnyvale, United States Figure Full time

    Figure is an AI Robotics company developing a general purpose humanoid. Our Humanoid is designed for corporate tasks targeting labor shortages and jobs that are undesirable or unsafe. We are based in Sunnyvale, CA and require 5 days/week in-office collaboration. Figure's vision is to deploy autonomous humanoids at a global scale. Our AI team is looking for...


  • Sunnyvale, United States Caspex Corporation Full time

    Prefer a candidate with more ML modeling experience. Qualifications•Solid background in machine learning, deep learning, statistical analysis, and software development•Proficient in data mining, machine learning, and deep learning packages in Python/Tensorflow/Keras•Experience working with large data sets and cloud computing (Hive, Spark, Google Cloud...

  • BIG Data Engineer

    1 month ago


    Sunnyvale, United States QData Full time

    Required Skills & Experience Experience range 8-14 years Java Development Experience Data Engineer. Python. Hadoop Stack Data Pipeline Using ETL or another tool spark – spark streaming RDD SQL NoSQL Cassandra Handling High Volume Data AWS Experience Experience On Big Data Migration Experience is PLUS

  • BIG Data Engineer

    1 month ago


    Sunnyvale, California, United States QData Full time

    Required Skills & Experience Experience range 8-14 years Java Development Experience Data Engineer. Python. Hadoop Stack Data Pipeline Using ETL or another tool spark – spark streaming RDD SQL NoSQL Cassandra Handling High Volume Data AWS Experience Experience On Big Data Migration Experience is PLUS


  • Sunnyvale, United States Chemix, Inc. Full time

    Job DescriptionJob DescriptionChemix is seeking a highly-motivated data scientist to develop and expand our AI platform for battery materials discovery. Our AI platform is the core of Chemix. Though data is first and foremost in any application of AI, it is typically very scarce in materials development. We've designed our R&D operation to generate...


  • Sunnyvale, United States Koko Home Full time

    Koko Home is seeking a Data Pipeline Engineer to develop and maintain data pipelines from our IoT devices all the way to our model building infrastructure. As this is a healthcare focused company, we’re seeking candidates with experience working within clinical and regulatory environments. MedTech/SaMD/HCIT experience is a plus.This role entails:Insuring...


  • Sunnyvale, United States Koko Home Full time

    Koko Home is seeking a Data Pipeline Engineer to develop and maintain data pipelines from our IoT devices all the way to our model building infrastructure. As this is a healthcare focused company, we’re seeking candidates with experience working within clinical and regulatory environments. MedTech/SaMD/HCIT experience is a plus.This role entails:Insuring...


  • Sunnyvale, United States Koko Home Full time

    Koko Home is seeking a Data Pipeline Engineer to develop and maintain data pipelines from our IoT devices all the way to our model building infrastructure. As this is a healthcare focused company, we’re seeking candidates with experience working within clinical and regulatory environments. MedTech/SaMD/HCIT experience is a plus.This role entails:Insuring...

  • AI Tooling Engineer

    2 weeks ago


    Sunnyvale, United States Figure Full time

    Figure is an AI Robotics company developing a general purpose humanoid. Our Humanoid is designed for corporate tasks targeting labor shortages and jobs that are undesirable or unsafe. We are based in Sunnyvale, CA and require 5 days/week in-office collaboration. Figure's vision is to deploy autonomous humanoids at a global scale. Our AI team is looking for...


  • Sunnyvale, United States Chemix, Inc. Full time

    Chemix is seeking a highly-motivated software engineer to develop and expand our AI platform for autonomous battery materials discovery and optimization. Better batteries are a critical piece of humanity's transition to sustainable energy. As a software engineer at Chemix, you will contribute to our mission by building and maintaining the data pipelines that...

  • Data Engineer

    2 weeks ago


    Sunnyvale, United States Lorven technologies Full time

    Job DescriptionJob DescriptionHi,Our client is looking Data Engineer for Long term project in Austin, TX/Sunnyvale, CA (Hybrid) below is the detailed requirements.Job Title : Data EngineerLocation : Austin, TX/Sunnyvale, CADuration : Long term Job description:Bachelor's degree in Computer science or equivalent, with minimum 10+ Years of relevant...


  • Sunnyvale, United States Illumio Full time

    Senior Backend Software Engineer (Python (Golang a plus))On-site work model of 5 days in office/week in Sunnyvale, CA In this role, you will focus on the Azure Firewall Management Program over the next year. You will work in conjunction with Illumio's Core product and our newest product, CloudSecure, to deliver on customer-facing products and initiatives. As...

  • Data Engineer

    15 hours ago


    Sunnyvale, California, United States Randstad USA Full time

    job summary: Additional Job Details: Need two data engineers with big data experience.3+ years of experience to work on specific code in our Ads space to implement new privacy controls. Description: Designs, develops, and implements Hadoop eco-system based applications to support business requirements.Follows approved life cycle methodologies, creates design...


  • Sunnyvale, United States Wilco Source Full time

    1. Bachelor's or Master's degree in Computer Science, Engineering, or related field.2. 10+ years of experience in data engineering, with a focus on GCP technologies.3. Strong proficiency in SQL, Scala, and Python for data manipulation and analysis.4. Hands-on experience with GCP services, including BigQuery, Cloud Storage, andDataflow.5. Solid understanding...