Data Engineer

About the Role: The Hybrid Site Reliability Engineer/Data Engineer at Lumin Digital will play a critical role in ensuring the reliability, scalability, and performance of the Lumin Analytics data platform, while also designing, implementing, and managing robust data pipelines. Lumin Analytics provides financial institutions with critical insights about digital banking metrics and user engagement, enabling them to understand, predict, and influence customer behavior.

This role combines operational excellence with data engineering proficiency, requiring deep expertise in automation, infrastructure management, continuous deployment, data streaming, and data enrichment.

Responsibilities:

Design, develop, and manage scalable and reliable data pipelines and infrastructure for Lumin Analytics.

Develop and manage robust CI/CD pipelines to facilitate rapid, consistent deployment.

Monitor system health, troubleshoot infrastructure and data pipeline issues, and proactively meet Service Level Objectives (SLOs).

Engage in capacity planning, demand forecasting, and performance optimization to ensure scalability and reliability.

Implement monitoring and alerting systems emphasizing automation, self-healing mechanisms, and incident response effectiveness.

Collaborate closely with Development, Security, and Operations teams to ensure compliance with regulatory and security standards.

Conduct and participate in code and architecture reviews.

Participate in an on-call rotation to support system reliability and availability.

Advocate effectively for project resources with senior leadership.

Technology Stack:

AWS infrastructure with Terraform (Infrastructure-as-Code)

Docker and Kubernetes (Amazon EKS)

Data streaming with Amazon MSK (Kafka), Debezium CDC, ksqlDB

Data enrichment and querying with Amazon Redshift

Java (Kafka Streams, custom transformations)

Python (AWS Lambda, Glue, pandas, boto3, awswrangler)

Scala (AWS EMR, for ETL)

CI/CD pipelines

Apache Spark (Scala and Python)

Requirements:

Software engineering experience (Python, Scala, and Java preferred).

Strong understanding of data structures, data movement, and troubleshooting data issues.

Experience with streaming platforms and data transformation/enrichment (Kafka, Debezium, ksqlDB).

Solid experience with AWS (preferred), Azure, Google Cloud, Databricks, and/or Snowflake.

Proficiency in Linux/Unix environments, git version control, Docker, Kubernetes, Terraform, and CI/CD practices.

Advanced problem-solving skills, with an ability to anticipate operational and data-related issues.

Knowledge of networking protocols (HTTP, DNS, TCP/IP).

Strong written and verbal communication skills.

Commitment to continuous learning and self-improvement.

Desired Skills:

Familiarity with big data frameworks (Spark, Hadoop, Hive) and related programming languages (Python, Scala, Java).

Experience with relational and non-relational databases (MySQL, PostgreSQL, DynamoDB, Redis).

Knowledge of data quality assurance tools (Great Expectations, deequ, etc.).

Understanding of various data formats (YAML, JSON, Avro, Parquet, protobuf, Iceberg, Delta, Hudi).

Experience with Kafka broker and topic management (performance, sizing, ACL security).

Knowledge of distributed systems concepts (concurrency, race conditions, deadlocks).

Education:

Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience.

Additional Information:

Minimal travel required (~12 days/year, ~2 team gatherings annually).

Participation in occasional incident response during off-hours.

LIFE AT LUMIN DIGITAL

Lumin Digital is a trailblazer in digital banking solutions, driven by a unique approach to technology, service, and people. We empower credit unions and banks by creating cutting-edge digital experiences that continuously serve, engage, and grow their membership base. Lumin is 100% cloud-native, purpose-built to unlock the full advantages of the cloud for financial institutions and their users.

At Lumin, we thrive on curiosity and innovation. Our culture fosters trust - in our expertise and decisions, respect - for diverse perspectives and talents, and boldness - in pursuing innovative paths. These values guide us, shaping a workplace where collaboration thrives, ideas flourish, and new possibilities are discovered. Focused on continuous improvement and innovation, we encourage our team to explore, experiment, and put new ideas into action, challenging the usual way of doing things.

All qualified applicants, including those with arrest or conviction records, will be considered for employment. Any conditional offer will include a notice regarding the review of the candidate's criminal history as part of the hiring process.

For more information, visit