Senior Data Engineer

Senior Data Engineer

Truckee, CA 96160

We are seeking a highly skilled Senior Data Engineer to join our data team and drive the development and optimization of our data infrastructure, indexing pipelines, and Solr collections. In this critical role, you will be responsible for designing and implementing efficient indexing data pipelines, optimizing Solr schemas and queries, and collaborating with the API team to modify Solr collections based on changing requirements. You will work closely with cross-functional teams to understand data needs and build efficient, scalable data architectures. Additionally, you will collaborate with the DevOps team to understand geospatial and mapping data requirements and gradually take ownership of related functionalities.

ABOUT YOU:

• Solr Indexing and Schema Expertise: Solid understanding of Apache Solr indexing processes, schema design, and query optimization. Experience in designing and implementing efficient indexing pipelines and data modeling techniques.
• Proficient in Python: Strong programming skills in Python, with expertise in developing data pipelines, automation scripts, and Solr collection modifications.
• Relational Database Expertise: Strong expertise in working with relational databases such as Postgres, MySQL, and Aurora, optimizing them for efficient data storage and retrieval.
• Streaming Architecture and Message Queues: Experience working with streaming architectures and message queues for real-time data processing and integration. Familiarity with Apache Kafka is a plus.
• Data Architecture Proficiency and Improvement: Proficiency in developing and maintaining robust, efficient data architectures, ensuring scalability, performance, and data integrity. Proactive in conducting research and exploring future state architectures for improving indexing performance and scalability. Ability to investigate innovative approaches, develop proof-of-concepts, and make data-driven recommendations for architectural enhancements.
• Cross-Team Collaboration: Strong collaboration skills to work effectively with the API team in modifying Solr collections based on evolving requirements. Ability to understand and translate technical needs across teams.
• AWS Experience: Hands-on experience working with AWS services and tools, such as EC2, S3, RDS, and AWS Glue. Experience with Redshift is a plus but not a hard requirement.
• Analytical and Problem-Solving Skills: Excellent analytical and problem-solving skills, enabling the ability to analyze complex data problems, identify patterns, and derive meaningful insights to drive decision-making and optimizations.

THINGS YOU MAY DO:

• Design and Implement Indexing Data Pipelines: Design and implement efficient indexing data pipelines from relational databases to Solr, ensuring data accuracy, consistency, and timely updates.
• Optimize Solr Schemas and Queries: Analyze data requirements, design optimal Solr schemas, and develop efficient queries to support search functionality and data retrieval.
• Collaborate with API Team on Solr Collection Modifications: Work closely with the API team to modify Solr collections based on changing requirements. Assist in planning and executing Solr collection updates.
• Build Efficient Data Architectures: Design and build efficient, scalable data architectures to support the organization's data needs. Ensure high performance and reliability of the data infrastructure.
• Collaborate with DevOps on Geospatial and Mapping Functionalities: Collaborate with the DevOps team to understand geospatial and mapping data requirements, provide insights, and gradually take ownership of related functionalities.
• Implement Streaming Architecture and Message Queues: Implement streaming architecture and message queues, such as Apache Kafka, for real-time data processing and integration. Ensure seamless data flow and scalability.
• Research and Improve Indexing Architecture: Conduct research and explore future state architectures for improving indexing performance and scalability. Investigate innovative approaches, develop proof-of-concepts, and make data-driven recommendations for architectural improvements.

req24-00539