Lead Data Pipeline Developer
2 weeks ago
Company Overview: Welcome to the forefront of data-driven innovation Our company is dedicated to leveraging the power of data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge data pipeline solutions that enable efficient data ingestion, processing, and delivery. Join us and lead our efforts in shaping the future of data pipeline development.
Position Overview: As the Lead Data Pipeline Developer, you will play a pivotal role in leading our data pipeline development initiatives and driving the design, implementation, and optimization of our data pipeline solutions. You will lead a team of skilled developers, collaborating closely with cross-functional teams to deliver high-quality data pipeline solutions that meet the evolving needs of our data-driven organization. If you're a seasoned developer with expertise in data pipeline technologies and a proven track record of leadership in delivering successful data projects, we invite you to join us in this exciting opportunity.
Requirements
Key Responsibilities:
- Technical Leadership: Provide guidance, mentorship, and technical leadership to a team of data pipeline developers, fostering a culture of excellence and innovation.
- Data Pipeline Design: Lead the design and architecture of data pipeline solutions, including data ingestion, transformation, and delivery, to meet business requirements and ensure efficient data processing and analytics.
- Data Ingestion: Lead efforts to develop and maintain data ingestion processes to collect data from diverse sources, including databases, APIs, files, and streaming sources, ensuring seamless data flow and interoperability.
- Data Transformation: Lead data transformation efforts to cleanse, enrich, and transform data as it moves through the pipeline, applying business rules, data validation, and enrichment to meet business needs.
- Orchestration and Automation: Lead the implementation of workflow orchestration and automation solutions to schedule and manage data pipeline workflows, reducing manual intervention and improving operational efficiency.
- Performance Optimization: Lead efforts to optimize data pipeline performance through parallel processing, partitioning, and other techniques, ensuring scalability and responsiveness for large-scale data processing needs.
- Monitoring and Alerting: Implement monitoring and alerting systems to track data pipeline performance and health, proactively identifying and resolving issues to minimize downtime and data loss.
- Error Handling and Retry Mechanisms: Implement error handling and retry mechanisms to handle data processing failures and ensure data reliability and consistency.
- Documentation and Best Practices: Define and promote best practices for data pipeline development, ensuring clear and comprehensive documentation to facilitate understanding and collaboration among team members.
- Collaboration: Collaborate with cross-functional teams, including data engineers, data scientists, and business analysts, to understand requirements and deliver data pipeline solutions that meet business needs.
- Mentorship and Development: Mentor and coach junior developers, providing guidance, support, and opportunities for skill development and career growth.
Qualifications:
- Bachelor's degree or higher in Computer Science, Engineering, Mathematics, or related field.
- 8+ years of experience in data engineering, with significant experience in designing, building, and optimizing data pipeline solutions.
- Proven leadership experience, with a track record of successfully leading data pipeline development teams and delivering complex projects.
- Proficiency in data pipeline technologies such as Apache Kafka, Apache NiFi, Apache Airflow, or similar.
- Strong programming skills in languages such as Python, Java, or Scala, with experience in data processing frameworks like Apache Spark or Apache Beam.
- Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform, and services like AWS Glue, Azure Data Factory, or Google Dataflow.
- Strong understanding of data integration concepts and techniques, with experience integrating data from diverse sources and systems.
- Strong problem-solving skills and analytical thinking, with the ability to design and troubleshoot complex data pipeline issues.
- Excellent communication and collaboration skills, with the ability to effectively interact with technical and non-technical stakeholders.
Benefits
- Competitive salary: The industry standard salary for Lead Data Pipeline Developers typically ranges from $190,000 to $270,000 per year, depending on experience and qualifications.
- Comprehensive benefits package, including health insurance, retirement plans, and wellness programs.
- Flexible work arrangements, including remote work options and flexible hours.
- Generous vacation and paid time off.
- Professional development opportunities, including access to training programs, conferences, and workshops.
- State-of-the-art technology environment with access to cutting-edge tools and resources.
- Vibrant and inclusive company culture with opportunities for growth and advancement.
- Exciting projects with real-world impact at the forefront of data-driven innovation.
Join Us: Ready to lead the charge in data pipeline development? Apply now to join our team and be part of the data revolution
-
Data Pipeline Developer
2 weeks ago
San Francisco, United States Unreal Gigs Full timeJob DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! At our company, we're committed to harnessing the power of data to drive insights and empower decision-making. Our mission is to develop scalable and efficient data pipeline solutions that enable organizations to streamline data processing and analysis....
-
Senior Data Pipeline Developer
2 weeks ago
San Francisco, United States Unreal Gigs Full timeJob DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to leveraging the power of data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge data pipeline solutions that enable efficient data ingestion, processing, and...
-
Senior Data Pipeline Developer
3 days ago
San Francisco, CA, United States Unreal Gigs Full timeJob Description Job Description Company Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to leveraging the power of data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge data pipeline solutions that enable efficient data ingestion, processing,...
-
Software Developer, Data Pipelines
6 days ago
San Francisco, California, United States Autodesk Full timeJob Requisition ID #24WD77850Job Title: Software Engineer, Data Pipelines - Generative AIPosition OverviewAutodesk is leading the transformation of the AEC industry, integrating AI technology into our products. We're enhancing our applications with cloud-native capabilities, including data at scale, edge computing, AI-based solutions, and advanced 3D...
-
Lead Data Engineer
2 weeks ago
San Francisco, United States Unreal Gigs Full timeJob DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is dedicated to harnessing the power of data to drive transformative change and solve complex problems across industries. We're committed to building scalable and reliable data infrastructure that enables advanced analytics, machine learning,...
-
Senior Data Engineer
7 days ago
San Francisco, United States People Data Labs Full timeJob DescriptionJob DescriptionAbout UsAt People Data Labs, we're committed to democratizing access to high-quality B2B data and leading the emerging DaaS economy. We empower developers, engineers, and data scientists to create innovative, compliant data products at scale with our clean, easy-to-use datasets of resume, company, location, and education...
-
Sales Development Representative
4 days ago
San Francisco, United States Pipeline Gurus Full time**Role Overview**: As a Sales Development Representative at Pipeline Gurus, you will be a pivotal part of our sales team, responsible for generating new business opportunities and driving our growth strategy. Your role will focus on identifying potential clients, engaging them through targeted outreach, and nurturing leads to create sales-ready...
-
San Francisco, United States Stripe Full timeWho we are About Stripe Stripe is a financial infrastructure platform for businesses. Millions of companies - from the world's largest enterprises to the most ambitious startups - use Stripe to accept payments, grow their revenue, and accelerate new business opportunities. Our mission is to increase the GDP of the internet, and we have a staggering amount...
-
Sales Development Representative
1 day ago
San Francisco, United States Pipeline Gurus Full timeRole Overview:As a Sales Development Representative at Pipeline Gurus, you will be a pivotal part of our sales team, responsible for generating new business opportunities and driving our growth strategy. Your role will focus on identifying potential clients, engaging them through targeted outreach, and nurturing leads to create sales-ready opportunities. You...
-
Sales Development Representative
3 days ago
San Francisco, United States Pipeline Gurus Full timeJob DescriptionJob DescriptionRole Overview:As a Sales Development Representative at Pipeline Gurus, you will be a pivotal part of our sales team, responsible for generating new business opportunities and driving our growth strategy. Your role will focus on identifying potential clients, engaging them through targeted outreach, and nurturing leads to create...
-
Data Analyst
5 days ago
San Francisco, United States People Data Labs Full timeJob DescriptionJob DescriptionAbout UsAt People Data Labs, we're committed to democratizing access to high-quality B2B data and leading the emerging DaaS economy. We empower developers, engineers, and data scientists to create innovative, compliant data products at scale with our clean, easy-to-use datasets of resume, company, location, and education...
-
Dir. of Philanthropy, Pipeline Development
2 months ago
San Diego, CA, United States St. Vincent De Paul Village Full timeDirector of Philanthropy, Pipeline Development Senior Director of Philanthropy The director of philanthropy, pipeline development (DOP, PD) plays a key strategic role in elevating the office of philanthropy into a disciplined, data driven operation, advising on every phase of the donor cycle, broadly and on the individual donor level, to include...
-
Lead Cloud Data Engineer
2 weeks ago
San Francisco, United States Unreal Gigs Full timeJob DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is a trailblazer in leveraging the power of data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge cloud data solutions that harness the scalability, flexibility, and power of...
-
Lead Cloud Data Engineer
7 days ago
San Francisco, United States Unreal Gigs Full timeJob DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is a trailblazer in leveraging the power of data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge cloud data solutions that harness the scalability, flexibility, and power of...
-
Lead Cloud Data Engineer
2 weeks ago
San Francisco, United States Unreal Gigs Full timeCompany Overview: Welcome to the forefront of data-driven innovation! Our company is a trailblazer in leveraging the power of data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge cloud data solutions that harness the scalability, flexibility, and power of cloud platforms. Join us and lead...
-
Technical Lead, Data Engineering
7 days ago
San Francisco, California, United States Verantos Full timeOverviewVerantos ) is a leading provider of high-accuracy Real-World Evidence (RWE) generation services. Our RWE platform seamlessly integrates diverse real-world data sources and harnesses the precision required for regulatory and reimbursement purposes. Leveraging data science, artificial intelligence, and advanced data sources, such as Electronic Health...
-
Lead Real-Time Data Engineer
2 weeks ago
San Francisco, United States Unreal Gigs Full timeJob DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is at the cutting edge of leveraging real-time data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge real-time data solutions that enable timely insights and actions. Join us...
-
Lead Real-Time Data Engineer
10 hours ago
San Francisco, United States Unreal Gigs Full timeJob DescriptionJob DescriptionCompany Overview: Welcome to the forefront of data-driven innovation! Our company is at the cutting edge of leveraging real-time data to drive transformative change and solve complex problems across industries. We're committed to building cutting-edge real-time data solutions that enable timely insights and actions. Join us...
-
Lead Data Engineer
6 days ago
San Francisco, United States Candid Health Full timeThe Role At Candid Health, we’re searching for our founding Lead Data Engineer to design, build, and support our next generation of data infrastructure. As a key strategic investment for the company and product, you will be working on highly impactful and visible outcomes. This is an opportunity to get in at the ground floor of designing and building...
-
Lead Data Engineer
6 days ago
San Francisco, California, United States Candid Health Full timeThe RoleAt Candid Health, we're searching for our founding Lead Data Engineer to design, build, and support our next generation of data infrastructure. As a key strategic investment for the company and product, you will be working on highly impactful and visible outcomes. This is an opportunity to get in at the ground floor of designing and building...
