Data Pipeline Engineer
2 weeks ago
CarOffer, a CarGurus Company, is seeking a motivated and detail-oriented
Data Pipeline Engineer
to join our data engineering team. In this role, you will be responsible for designing, implementing, and maintaining data pipelines to efficiently ingest and stream data into our Snowflake data warehouse from diverse sources, including web analytics, third-party APIs, and FTP servers. As a key member of our team, you will play a crucial role in ensuring the reliability, scalability, and performance of our data infrastructure, enabling timely and accurate data insights for business decision-making. This position offers an exciting opportunity to work with cutting-edge technologies and industry-standard practices in data engineering while contributing to the success of our data-driven initiatives.
What You'll Do:
Collaborate with stakeholders to understand data requirements and source systems, identifying data sources and defining data ingestion strategies.
Design and implement robust data pipelines to ingest, process, and stream data into our Snowflake data warehouse from various sources, ensuring data quality and reliability.
Develop connectors and integrations with web analytics platforms, third-party APIs, and FTP servers to automate data extraction and ingestion processes.
Implement industry-standard practices for data pipelining, including error handling, data validation, and monitoring, to ensure data integrity and availability.
Optimize data pipeline performance and resource utilization, identifying bottlenecks and opportunities for improvement in data processing and streaming.
Collaborate with data engineers, software engineers, and infrastructure teams to integrate data pipelines into our data ecosystem and ensure seamless data flow.
Document data pipeline architecture, configurations, and dependencies, ensuring clear and comprehensive documentation for future reference and troubleshooting.
Monitor data pipeline health and performance metrics, proactively identifying and addressing issues to minimize downtime and ensure data availability.
Stay current with emerging technologies and best practices in data engineering and streaming technologies, continuously evaluating and adopting new tools and techniques.
Participate in team meetings, code reviews, and knowledge-sharing sessions, contributing to a collaborative and supportive team culture.
Who You Are:
Bachelors degree in Computer Science, Engineering, or a related field; Masters degree preferred.
1-3 years of experience in data engineering, software development, or a related field, with a focus on building and maintaining data pipelines.
Hands-on experience with data pipelining tools and technologies, such as Apache Kafka, Apache NiFi, or AWS Glue, preferred.
Proficiency in programming languages such as Python, Java, or Scala, with experience in scripting and automation.
Familiarity with cloud platforms and services, such as AWS, Azure, or Google Cloud Platform, and experience with cloud-based data services (e.g., S3, Redshift, BigQuery).
Strong understanding of data formats and protocols, including JSON, XML, REST APIs, and FTP/SFTP.
Excellent problem-solving and analytical skills, with the ability to troubleshoot and resolve complex data pipeline issues.
Strong communication and collaboration skills, with the ability to work effectively in a team environment and interact with stakeholders at all levels.
Self-motivated and proactive, with a passion for learning and staying abreast of new technologies and industry trends.
Ability to prioritize tasks, manage time efficiently, and meet project deadlines in a fast-paced, dynamic environment.
#J-18808-Ljbffr
-
Pipeline Technician
2 weeks ago
Addison, United States MRINetwork Jobs Full timeJob DescriptionJob DescriptionWe are representing a full service, engineering firm for the natural gas pipeline industry. Our client is currently seeking Pipeline Technicians in Texas, preferably the Austin, TX area.Job Requirements:3+ years of field experience operating natural gas pipeline facilities and equipment.Experience performing preventative...
-
Data Engineer
2 weeks ago
Addison, United States CareerBuilder Full timeHTC A brief profile Established in 1990, HTC Inc., a CMM Level 5 company with headquarters in Troy, Michigan, is a leading global Information Technology solution and BPO provider. HTC assists clients across multiple industry verticals, offering turnkey project lifecycle in, e-business, data warehousing, embedded systems, ECM, SCM, CRM, and ERP solutions. HTC...
-
Engineer, Data
2 weeks ago
Addison, Texas, United States Concentra Career Choice Full timeOverview: The Data Engineer is primarily focused on analytical processes with ability to implement database solutions and best practices in the realm of machine learning projects. Essential software engineering skills with strong foundational knowledge on data movement and orchestration both on-premises and cloud environment. The Data Engineer supports and...
-
Manager of Data Analytics
3 days ago
Addison, United States IDR (Internal Data Resources) Full timeImmediate and permanent opportunity for a Manager of Enterprise Analytics with industry leading company headquartered in the North Dallas, TX area. The Manager of Enterprise Analytics will lead a team of data analysts and business intelligence specialists responsible for the delivery of analytics services, as well as converting data into actionable...
-
Manager of Data Analytics
6 hours ago
Addison, United States IDR (Internal Data Resources) Full timeImmediate and permanent opportunity for a Manager of Enterprise Analytics with industry leading company headquartered in the North Dallas, TX area. The Manager of Enterprise Analytics will lead a team of data analysts and business intelligence specialists responsible for the delivery of analytics services, as well as converting data into actionable insights....
-
Engineer, Data
2 weeks ago
Addison, Texas, United States Concentra Full timeOverview: The Data Engineer is primarily focused on analytical processes with ability to implement database solutions and best practices in the realm of machine learning projects. Essential software engineering skills with strong foundational knowledge on data movement and orchestration both on-premises and cloud environment. The Data Engineer supports and...
-
Data Architect
2 weeks ago
Addison, United States Reece USA Full timeThe Data Architect we are hiring will provide technical leadership and guidance throughout the lifecycle of data and data systems. The Data Architect works closely with various internal departments and teams to drive effectiveness through a total data solution mindset and process-orientated team. Participate in strategy planning and solution negotiations....
-
Data Architect
2 weeks ago
Addison, United States Reece USA Full timeThe Data Architect we are hiring will provide technical leadership and guidance throughout the lifecycle of data and data systems. The Data Architect works closely with various internal departments and teams to drive effectiveness through a total data solution mindset and process-orientated team. Participate in strategy planning and solution negotiations....
-
Data Architect
3 weeks ago
Addison, United States Reece USA Full timeThe Data Architect we are hiring will provide technical leadership and guidance throughout the lifecycle of data and data systems. The Data Architect works closely with various internal departments and teams to drive effectiveness through a total data solution mindset and process-orientated team. Participate in strategy planning and solution negotiations....
-
Electrical Engineer, Commercial Avionics
2 weeks ago
Addison, United States International Staff Consulting Full timeElectrical Engineer, Full Product Life Cycle, Commercial Avionics One of our clients needs a passionate Hardware Engineer who can think outside the box to find new and better ways to design quality electrical products. Someone with a strong drive who's drawn to creative problem solving and to the challenges of designing prototypes and improving existing...
-
AWS DevOps Engineer
2 weeks ago
Addison, United States CarOffer Full timeCarOffer, a CarGurus company, is seeking a skilled and experienced AWS DevOps Engineer to join its engineering team. Our engineering team is at the forefront of developing world-class solutions in the automotive space, specifically tailored for used car dealerships. We are committed to revolutionizing the industry by leveraging cutting-edge technologies and...
-
SQL Data Engineer
3 weeks ago
Addison, United States Motion Recruitment Full timeOutstanding long-term contract opportunity! A well-known Financial Services Company is looking for a BI Tools Developer in Addison, TX (Hybrid). Work with the brightest minds at one of the largest financial institutions in the world. This is long-term contract opportunity that includes a competitive benefit package! Our client has been around for over 150...
-
DevOps Engineer
2 weeks ago
Addison, United States Codonix Technologies Inc Full timeJob DescriptionJob DescriptionTitle: DevOpsEngineerDuration: 12 +months Location:Addison, TX/Atlanta, GA(Onsite) W2 OnlyJob Description .ConfigureCI-CD pipelinesConfigure and manageSonar dashboardConfigure and manageCode quality toolsMerge/Retrofit. Responsible for supporting feature teams and other consumers with environmental support function:Environment...
-
DevOps Engineer
4 days ago
Addison, United States Codonix Technologies Inc Full timeJob DescriptionJob DescriptionTitle: DevOpsEngineerDuration: 12 +months Location:Addison, TX/Atlanta, GA(Onsite) W2 OnlyJob Description .ConfigureCI-CD pipelinesConfigure and manageSonar dashboardConfigure and manageCode quality toolsMerge/Retrofit. Responsible for supporting feature teams and other consumers with environmental support function:Environment...
-
DevOps Engineer II
4 days ago
Addison, United States Concentra Full timeOverview: Are you looking for a career that transcends the ordinary? At Concentra, we offer opportunities beyond patient care. As a valued member of our team, you'll be part of our efforts to provide exceptional service to our employer clients and exceptional care to their employees. Our values define our path forward always working to ensure welcoming,...
-
DevOps Engineer II
1 week ago
Addison, United States Concentra Career Choice Full timeOverview: Are you looking for a career that transcends the ordinary? At Concentra, we offer opportunities beyond patient care. As a valued member of our team, you'll be part of our efforts to provide exceptional service to our employer clients and exceptional care to their employees. Our values define our path forward always working to ensure welcoming,...
-
Senior Software/Data warehouse Developer
9 hours ago
Addison, United States Tyton Holdings Full timeThe Senior Software Developer/Database Administrator shall possess knowledge and experience in applications software development principles and methods sufficient to participate in the design, development, testing and implementation of new or modified applications software; including operating systems installation and configuration procedures;...
-
Senior Software/Data warehouse Developer
5 days ago
Addison, United States Tyton Holdings Full timeJob DescriptionJob DescriptionThe Senior Software Developer/Database Administrator shall possess knowledge and experience in applications software development principles and methods sufficient to participate in the design, development, testing and implementation of new or modified applications software; including operating systems installation and...
-
Senior Software/Data warehouse Developer
2 days ago
Addison, United States Tyton Holdings Full timeJob DescriptionJob DescriptionThe Senior Software Developer/Database Administrator shall possess knowledge and experience in applications software development principles and methods sufficient to participate in the design, development, testing and implementation of new or modified applications software; including operating systems installation and...
-
DevOps AWS/Azure Engineer
3 weeks ago
Addison, United States Code Lab Technology Full time• Define architect and implement CI/CD pipelines for enterprise-grade applications using AWS/Azure DevOps Services• Experience in provisioning and configuration of AWS/Azure PaaS services including the managed database services• First-hand experience in AWS/Azure infrastructure automation with AWS/Azure Resource Manager (ARM) and Terraform•...
