Big Data Engineer

1 day ago

Sunnyvale, CA, United States Texas State Library and Archives Commision Full time
Big Data Engineer:

Must Have:
• Proficiency in managing and manipulating huge datasets in the order of terabytes (TB) is essential
• Expertise in big data technologies like Hadoop, Apache Spark (Scala preferred), Apache Hive, or similar frameworks on the cloud (GCP preferred, AWS, Azure etc.) to build batch data pipelines with strong focus on optimization, SLA adherence and fault tolerance.
• Expertise in building idempotent workflows using orchestrators like Automic, Airflow, Luigi etc.
• Expertise in writing SQL to analyze, optimize, profile data preferably in BigQuery or SPARK SQL
• Strong data modeling skills are necessary for designing a schema that can accommodate the evolution of data sources and facilitate seamless data joins across various datasets
• Ability to work directly with stakeholders to understand data requirements and translate that to pipeline development / data solution work
• Strong analytical and problem-solving skills are crucial for identifying and resolving issues that may arise during the data integration and schema evolution process
• Ability to move at rapid pace with quality and start delivering with minimal ramp up time will be crucial to succeed in this initiative
• Effective communication and collaboration skills are necessary for working in a team environment and coordinating efforts between different stakeholders involved in the project

Nice to have:
• Experience building complex near real time (NRT) streaming data pipelines using Apache Kafka, Spark streaming, Kafka Connect with a strong focus on stability, scalability and SLA adherence.
• Good understanding of REST APIs - working knowledge on Apache Druid, Redis, Elastic search, GraphQL or similar technologies. Understanding of API contracts, building telemetry, stress testing etc.
• Exposure in developing reports/dashboards using Looker/Tableau
• Experience in eCommerce domain.

Tech stack: Google cloud, HDFS, SPARK, Scala, Python (optional), Automic/Airflow, BigQuery, Kafka, API, Druid

  • Big Data Engineer

    1 day ago

    Sunnyvale, CA, United States Purple Drive Full time

    Key Responsibilities Design, develop, and maintain big data pipelines leveraging Hadoop, Spark, and Kafka. Develop and optimize J2EE backend services for large-scale data processing and integration. Program in Python, Scala, and Java to build robust data and backend solutions. Architect and implement cloud-native solutions on GCP and Azure platforms. Design...

  • Big Data Engineer

    6 days ago

    Sunnyvale, CA, United States Purple Drive Full time

    Key Responsibilities Design, develop, and maintain big data pipelines leveraging Hadoop, Spark, and Kafka. Develop and optimize J2EE backend services for large-scale data processing and integration. Program in Python, Scala, and Java to build robust data and backend solutions. Architect and implement cloud-native solutions on GCP and Azure platforms. Design...

  • Senior Full Stack Big Data Engineer

    2 weeks ago

    Sunnyvale, CA, United States ClifyX Full time

    Senior Full Stack Big Data Engineer Profile: Senior full stack big data engineer with focus on Spark, Scala, Kafka & Cassandra. Extensive experience of Java based full stack big data implementation on HDP platform (Hortonworks). Hands-on experience in Spark, Scala is must and experience in Kafka, Cassandra. Work experience in handling huge volume of data....

  • Senior Full Stack Big Data Engineer

    6 days ago

    Sunnyvale, CA, United States ClifyX Full time

    Senior Full Stack Big Data Engineer Profile: Senior full stack big data engineer with focus on Spark, Scala, Kafka & Cassandra. Extensive experience of Java based full stack big data implementation on HDP platform (Hortonworks). Hands-on experience in Spark, Scala is must and experience in Kafka, Cassandra. Work experience in handling huge volume of data....

  • Senior Full Stack Big Data Engineer

    1 week ago

    Sunnyvale, CA, United States ClifyX Full time

    Senior Full Stack Big Data Engineer Profile: Senior full stack big data engineer with focus on Spark, Scala, Kafka & Cassandra. Extensive experience of Java based full stack big data implementation on HDP platform (Hortonworks). Hands-on experience in Spark, Scala is must and experience in Kafka, Cassandra. Work experience in handling huge volume of data....

  • Senior Full Stack Big Data Engineer

    6 days ago

    Sunnyvale, CA, United States ClifyX Full time

    Senior Full Stack Big Data Engineer Profile: Senior full stack big data engineer with focus on Spark, Scala, Kafka & Cassandra. Extensive experience of Java based full stack big data implementation on HDP platform (Hortonworks). Hands-on experience in Spark, Scala is must and experience in Kafka, Cassandra. Work experience in handling huge volume of data....

  • Contract Big Data Engineer Real-Time Analytics

    8 hours ago

    Sunnyvale, CA, United States Procom Consultants Group Full time

    A leading IT staffing company is seeking a Big Data Software Engineer for a 7+ month contract in Sunnyvale, CA. This role involves enhancing a Retail Engagement Analytics product by collaboratively implementing data solutions and ensuring high data quality and reliability. Ideal candidates will possess strong programming skills and experience with No-SQL...

  • Contract Big Data Engineer Real-Time Analytics

    6 hours ago

    Sunnyvale, CA, United States Procom Consultants Group Full time

    A leading IT staffing company is seeking a Big Data Software Engineer for a 7+ month contract in Sunnyvale, CA. This role involves enhancing a Retail Engagement Analytics product by collaboratively implementing data solutions and ensuring high data quality and reliability. Ideal candidates will possess strong programming skills and experience with No-SQL...

  • IT - Technology Architect | Big Data - Data Processing | Spark

    4 days ago

    Sunnyvale, CA, United States SysMind Tech Full time

    Client: Apple Do you have a pre-identified candidate? New sourcing? Existing Subcon?- New sourcing Do you have a preferred interview scpersondule?: Yes Would you require tperson candidates to meet you for an in-person interview?: No Is Skype/WebEx interview, OK?: Yes Work Location with ZIP: Sunnyvale-CA-94085 Is remote work an option?: No If remote work,...

  • IT - Technology Architect | Big Data - Data Processing | Spark

    4 days ago

    Sunnyvale, CA, United States SysMind Tech Full time

    Client: Apple Do you have a pre-identified candidate? New sourcing? Existing Subcon?- New sourcing Do you have a preferred interview scpersondule?: Yes Would you require tperson candidates to meet you for an in-person interview?: No Is Skype/WebEx interview, OK?: Yes Work Location with ZIP: Sunnyvale-CA-94085 Is remote work an option?: No If remote work,...