Lead PostgreSQL Database Administrator

4 days ago

Los Angeles, CA, United States Green Dot Corporation Full-time $109,000 - $164,000 Contract

Location Requirement

This position is open only to candidates who currently reside in one of the following approved locations: West Los Angeles, CA



Key responsibilities

  1. High availability and disaster recovery (critical) Active-passive (primary/standby): Design, deploy, and operate zone-redundant high availability on Azure Database for PostgreSQL flexible server: synchronous standby in a second availability zone, automatic failover in 60–120 seconds with zero data loss, and the 99.99% uptime SLA that configuration carries. Decide between zone-redundant and same-zone (zonal) HA per workload based on latency tolerance and zone-failure risk, and document the trade‑off. Run and document planned, unplanned, and forced failover tests at least quarterly; measure downtime from the application side, not from the service side. Size max_replication_slots and max_wal_senders correctly (four reserved for HA plus one per read replica and per logical slot) so failovers never fail for lack of slots. Build equivalent active-passive clusters on self‑managed Azure VMs using streaming replication with Patroni and etcd (Linux) or a supported cluster manager on Windows, fronted by HAProxy, PgBouncer, or an Azure internal load balancer. Design cross‑region disaster recovery: geo‑redundant backups, cross‑region read replicas, and promotion runbooks with tested RPO/RTO targets. Preserve logical replication across failover: pg_failover_slots on PostgreSQL 16 and earlier, native sync_replication_slots = on with hot_standby_feedback = on on PostgreSQL 17 and later. Active-active (bi‑directional / multi‑master): Design and operate active‑active topologies across Azure regions using the pglogical extension or native logical replication, following Microsoft's published bi‑directional replication guidance for Azure Database for PostgreSQL. Define and enforce conflict‑avoidance and conflict‑resolution strategy: partitioned write ownership, non‑overlapping sequence ranges or UUIDv7 keys, last‑writer‑wins rules where appropriate, and monitoring for conflict counts. Manage DDL propagation across nodes (pglogical.replicate_ddl_command() or controlled DDL rollout for native replication) and sequence synchronization (PostgreSQL 19 adds native sequence replication via ALL SEQUENCES and REFRESH SEQUENCES). Monitor replication lag, slot retention, and retain_dead_tuples settings on every node; alert before WAL retention threatens primary disk. Own the decision of when active-active is and is not appropriate, and push back on it where a well‑run active‑passive design meets the requirement at lower risk.
  2. Azure platform administration (critical) Provision and manage Azure Database for PostgreSQL flexible server via the portal, Azure CLI, Az.
    PostgreSQLFlexibleServer PowerShell module, ARM/Bicep, Terraform, and REST API. Configure networking: VNet integration, private endpoints, private DNS zones, firewall rules, and migration from VNet‑injected to private‑endpoint servers (now supported for HA‑enabled servers). Implement Microsoft Entra ID authentication, Entra group login, managed identities, customer‑managed keys (including cross‑tenant CMK), and SCRAM‑SHA-256 as the password default. Manage compute and storage tiers, including Premium SSD v2 (GA with HA, read replicas, geo‑DR, and major version upgrade support) and online storage scaling. Operate built‑in PgBouncer (currently 1.25.x) for connection pooling and tune pool modes per application. Run in‑place major version upgrades on flexible server (PostgreSQL 18 upgrade GA June 2026), including pre‑upgrade extension and large‑object checks. Manage read replicas (in‑region, cross‑region, cascading), promote them when required, and keep replica lag within SLA. Configure backup retention, long‑term retention via Azure Backup, point‑in‑time restore, and recovery of deleted servers; test restores on a schedule. Evaluate and deploy elastic clusters (sharded flexible server) for horizontally scaled workloads where appropriate. Set custom maintenance windows and coordinate Azure‑initiated minor version patching with application owners.
  3. Self‑managed PostgreSQL on Linux and Windows (critical) Install, patch, and tune PostgreSQL 15–19 on RHEL 8/9, CentOS Stream, and Windows Server 2019/2022 running as Azure VMs. On Linux: manage systemd units, SELinux contexts, firewalld, LVM/managed‑disk layout, kernel and huge‑page settings, and PGDG repository packaging. On Windows: manage the PostgreSQL service, NTFS permissions, Windows Firewall, scheduled tasks for maintenance, and Windows‑specific installer and upgrade behavior. Operate backup tooling: pgBackRest or Barman on Linux, pg_basebackup and WAL archiving to Azure Blob Storage on both platforms, with incremental backups where supported.
  4. Performance engineering Diagnose and tune queries using EXPLAIN (ANALYZE, BUFFERS, WAL, IO), pg_stat_statements, auto_explain, and Azure Query Performance Insight. Tune autovac