![The Carlyle Group](https://media.trabajo.org/img/noimg.jpg)
Data Engineer
2 months ago
Basic information
Job Name:
Data Engineer
Location:
Washington, DC
Line of Business:
Global Technology & Solutions
Job Function:
Investor Services
Date:
Thursday, May 16, 2024
Position Summary
As a Data Engineer at Carlyle, you will join an innovative team that leverages data as the driving force of our cutting-edge solutions. You will design, build and maintain the data infrastructure and pipelines that power our data-driven products and insights. You will work with tools like Snowflake, Spark, Kafka, Airflow and cloud data platforms to create robust data architectures and enable scalable and efficient data collection, storage, processing and analysis.
You will also ensure data quality, security and governance by implementing data validation processes, access controls and monitoring systems. You will collaborate with data consumers like analysts, data scientists and engineers to understand their requirements and deliver trusted data products.
You should have substantial experience in distributed systems, data modeling, pipeline orchestration and programming languages like Python/Scala. You should also have strong problem-solving abilities and excellent communication skills. If you are passionate about building scalable data architectures and turning raw data into analytical insights, join our team and help drive our data-centric products and strategy.
Responsibilities
- Design, implement, and support cloud data platforms such as Snowflake and Databricks, and leverage their features and capabilities to optimize data performance and scalability.
- Architect and administer data lakes and cloud data warehouses that provide secure, reliable, and flexible data storage and access for analytics and machine learning.
- Design, build, and maintain scalable and robust data pipelines using various cloud services and tools such as AWS, Azure, SnapLogic, Apache Airflow, and Prefect.
- Develop and optimize data processing workflows with Python, Scala, and Spark, and manage and support data warehouse and data lake solutions in Snowflake, Amazon Aurora, and other platforms.
- Utilize Git, GitHub, and Azure DevOps for version control and collaboration, and apply Terraform and Infrastructure as Code (IaC) principles to automate and manage infrastructure.
- Champion the implementation of CI/CD pipelines to streamline development and deployment processes.
- Ensure data integrity and compliance with best practices in SQL and NoSQL database systems, and troubleshoot issues with data quality, security, and privacy.
- Continuously explore new technologies to enhance data reliability, efficiency, and quality.
- Collaborate with data consumers like analysts, data scientists, engineers, and product managers to understand their requirements and deliver trusted data products.
- Create and maintain data documentation, metadata, and data dictionaries to ensure data accessibility and usability.
- Perform data testing and validation to ensure data accuracy and consistency.
- Provide data engineering support and guidance to junior data engineers and other data team members.
- Stay updated with the latest trends and developments in data engineering and related fields.
- Apply best practices and standards for data governance, security, and quality across cloud data platforms, and ensure compliance with data policies and regulations.
- Evaluate and select appropriate data tools, frameworks, and technologies to meet the data engineering needs of the organization.
- Design and implement data APIs and services to enable data consumption and integration across different systems and applications.
- Monitor and improve data pipeline performance, efficiency, and reliability, and troubleshoot any data issues or failures.
- Conduct data analysis and provide insights and recommendations to support data-driven decision making.
- Implement and integrate machine learning models using AWS Sagemaker, MLFlow, and Jupyter Notebooks into production systems.
- Mentor and coach other data team members on data engineering best practices, standards, and methodologies.
Qualifications
Education & Certificates
- Bachelor's degree in Computer Science, Engineering, or related field
- Relevant certifications in AWS, Azure, and other modern data technologies are highly desirable.
Professional Experience
- 5+ years of relevant experience in data engineering, data analysis, and data pipeline development.
- Proficient in AWS data services, such as S3, Glue, Redshift, EMR, Athena, and Kinesis, and able to design, build, and optimize scalable and reliable data pipelines using AWS tools and best practices.
Competencies & Attributes
- Proficient in Snowflake cloud data platform, and able to leverage its features and capabilities for data ingestion, storage, processing, and analysis.
- Proficient in Databricks unified data analytics platform, and able to use its collaborative notebooks, integrated APIs, and optimized clusters for data engineering and machine learning.
- Proficient in Azure data services, such as Blob Storage, Data Factory, Synapse Analytics, Databricks, and HDInsight, and able to design, build, and optimize scalable and reliable data pipelines using Azure tools and best practices.
- Expert skills in SQL, Python, Scala, and Spark for data extraction, transformation, and loading (ETL).
- Experience with AWS Sagemaker, MLFlow, Jupyter Notebooks, and other machine learning frameworks and applications.
- Experience with Git, GitHub, Azure DevOps, Terraform, and CI/CD practices for data pipeline automation and deployment.
- Knowledge of data warehouse, data lake, and data mart concepts and architectures.
- Experience with pipeline orchestration tools like Apache Airflow, Prefect, or Luigi.
- Ability to design, optimize, monitor, and troubleshoot data pipelines for performance, reliability, and quality.
- Proficient in SnapLogic, able to leverage rich set of connectors to build scalable and robust data pipelines for various use cases.
- Demonstrated experience in using SnapLogic's features such as pipelines, tasks, snaps, patterns, and ultra tasks to design, develop, test, and deploy data solutions.
- Experience with data governance, data security, data validation, and error handling best practices.
- Alation data catalog experience a plus
- Expertise in Alation a plus, able to use its features such as data search, data lineage, data quality, and data stewardship to enable data governance and discovery across various data sources and platforms.
- Understanding of data modeling, data mining, and data analysis techniques and methods.
- Familiarity with other industry-standard data tools like Kafka, Hive, Redis, MongoDB, etc.
- Excellent communication and collaboration skills, and ability to mentor and coach other data team members.
Benefits/Compensation
The compensation range for this role is specific to Washington, D.C. and takes into account a wide range of factors including but not limited to the skill sets required/preferred; prior experience and training; licenses and/or certifications.
The anticipated base salary range for this role is $170,000 to $190,000.
In addition to the base salary, the hired professional will enjoy a comprehensive benefits package spanning retirement benefits, health insurance, life insurance and disability, paid time off, paid holidays, family planning benefits and various wellness programs. Additionally, the hired professional may also be eligible to participate in an annual discretionary incentive program, the award of which will be dependent on various factors, including, without limitation, individual and organizational performance.
Due to the high volume of candidates, please be advised that only candidates selected to interview will be contacted by Carlyle.
Company Information
The Carlyle Group (NASDAQ: CG) is a global investment firm with $425 billion of assets under management and more than half of the AUM managed by women, across 595 investment vehicles as of March 31, 2024. Founded in 1987 in Washington, DC, Carlyle has grown into one of the world's largest and most successful investment firms, with more than 2,200 professionals operating in 28 offices in North America, Europe, the Middle East, Asia and Australia. Carlyle places an emphasis on development, retention and inclusion as supported by our internal processes and seven Employee Resource Groups (ERGs). Carlyle's purpose is to invest wisely and create value on behalf of its investors, which range from public and private pension funds to wealthy individuals and families to sovereign wealth funds, unions and corporations. Carlyle invests across three segments - Global Private Equity, Global Credit and Investment Solutions - and has expertise in various industries, including: aerospace, defense & government services, consumer & retail, energy, financial services, healthcare, industrial, real estate, technology & business services, telecommunications & media and transportation.
At Carlyle, we know that diverse teams perform better, so we seek to create a community where we continually exchange insights, embrace different perspectives and leverage diversity as a competitive advantage. That is why we are committed to growing and cultivating teams that include people with a variety of perspectives, people who provide unique lenses through which to view potential deals, support and run our business.
-
Data Engineer
4 months ago
Washington, Washington, D.C., United States Non-Departmental Agency Full timeSummary Data Engineers work with data consumers to create and populate optimal data architectures, structures, and systems to meet CIA's business needs. Duties As a Data Engineer for CIA, you will focus on the design, implementation, and operation of data management systems to meet the CIA's business needs. This includes designing how the data will be...
-
Data Engineer
3 months ago
Washington, Washington, D.C., United States Atechstar Full timeKey Responsibilities Design implement and support applications that provide structured and timely access to actionable business information addressing stakeholder needs. Interface directly with stakeholders gathering requirements and owning automated end-to-end reporting solutions. Partner with analysts data engineers business intelligence engineers and...
-
Data Engineer
4 weeks ago
Washington, Washington, D.C., United States BigBear Full timeData Engineer Washington, DCOverview: is seeking a Data Engineer to join our team and help us build and maintain scalable and reliable data pipelines for our clients. You will be responsible for developing and designing data pipelines to support an end-to-end solution, integrating data pipelines with AWS cloud services to extract meaningful insights, and...
-
Data Engineer
3 weeks ago
Washington, Washington, D.C., United States World Bank Full timeData EngineerDescriptionWorking at the World Bank provides a unique opportunity for you to help our clients solve their greatest development challenges. The World Bank is one of the largest sources of funding and knowledge for developing countries; a unique global partnership of five institutions dedicated to ending extreme poverty, increasing shared...
-
Big Data Engineer
4 months ago
Washington, Washington, D.C., United States Dash techology Full timeWe're looking for a Big Data Engineer who can find creative solutions to tough problems. As a Big Data Engineer you'll create and manage our data infrastructure and tools including collecting storing processing and analyzing our data and data systems. You know how to work quickly and accurately using the best solutions to analyze mass data sets and you know...
-
Data Engineer
4 months ago
Washington, Washington, D.C., United States Atechstar Full timeResponsibilities Design build and own all the components of a high-volume data warehouse end to end. Build efficient data models using industry best practices and metadata for ad-hoc and pre-built reporting Provide wing-to-wing data engineering support for project lifecycle execution (design execution and risk assessment) Interface with business customers...
-
Senior Data Engineer
4 weeks ago
Washington, Washington, D.C., United States Amentum Full timeAmentum is looking for a Senior Data Engineer specializing in Research, Development, Test, and Evaluation (RDT&E) to assist with a contract for the DIA Analytic Innovations Office Advanced Analytics & Product Assessment. The role is based in Washington, D.C. Initially, all employees will work from the D.C. office but may, upon request and client approval,...
-
Data Engineering Lead
1 month ago
Washington, Washington, D.C., United States Analytica Full timeAnalytica is seeking a remote Data Engineering Lead to lead cloud data engineering implementations that support business intelligence, machine learning, data science and/or graph analytics on federal client projects. As the Data Engineering Lead, you'll be responsible for building and scaling advanced cloud data pipelines that support effective data...
-
Data Engineer
1 month ago
Washington, Washington, D.C., United States AARP Full timeOverviewAARP is the nation's largest nonprofit, nonpartisan organization dedicated to empowering people 50 and older to choose how they live as they age. With a nationwide presence, AARP strengthens communities and advocates for what matters most to the more than 100 million Americans 50-plus and their families: health security, financial stability and...
-
Mid Data Engineer
6 days ago
Washington, Washington, D.C., United States Amentum Full timeAmentum is looking for a Mid Data Engineer (RDT&E) to support a DIA Analytic Innovations Office Advanced Analytics & Product Evaluation project based in Washington, D.C. This role involves providing enhanced scientific/engineering research, capability analysis, and data management systems design for defense analytical requirements. The ideal candidate should...
-
Staff Data Engineer
2 months ago
Washington, Washington, D.C., United States Danaher Full timeWondering what's within Beckman Coulter Diagnostics? Take a closer look.At first glance, you'll see that for more than 80 years we've been dedicated to advancing and optimizing the laboratory to move science and healthcare forward. Join a team where you can be heard, be supported, and always be yourself. We're building a culture that celebrates backgrounds,...
-
AWS Senior Data Engineer
4 months ago
Washington, Washington, D.C., United States Atechstar Full timeJob description Skills RequiredProficient in languages Python.Experience in AWS StackGlue Athena Quick sight RDS Redshift Kafka pySpark.Experience setting up data pipelines archiving data data lakes.Bachelors or Master's degree in computer science Maths statistics or related field.Expertise with designing complex Data Models and Data Engineering...
-
Data Scientist
4 weeks ago
Washington, Washington, D.C., United States The World Bank Full timeProven AI and machine learning expertise, with experience in NLP, deep learning, large language models, and predictive analyticsA technical team within that Office brings together experts in statistics, data science, data engineering, economics and econometrics, and geography to support the mission of the Chief StatisticianFor more information, visit you a...
-
Data Analyst
4 weeks ago
Washington, Washington, D.C., United States LMI Consulting, LLC Full timeOverviewLMI is seeking a Data Analyst to support our Intelligence Community client.LMI: Innovation at the Pace of NeedTMAt LMI, we're reimagining the path from insight to outcome at The New Speed of PossibleTM. Combining a legacy of over 60 years of federal expertise with our innovation ecosystem, we minimize time to value and accelerate mission success. We...
-
Azure Data Engineer
4 months ago
Washington, Washington, D.C., United States Atechstar Full timeJob description Desired Candidate ProfileExperience with setup of Azure data factory Synapse Azure Data lake Azure Data bricks Azure Cosmos DB Azure Streaming Analytics Event-hub Strong expertise in SQL Experience across end to end data management concepts including Data Quality Meta data Data Security Coding skills (Python additional programming skills such...
-
Big Data Engineer
3 months ago
Washington, Washington, D.C., United States Atechstar Full timeJob descriptionRoles & Responsibilities You would be responsible for evaluating developing maintaining and testing big data solutions for advanced analytics projects The role would involve big data pre-processing & reporting workflows including collecting parsing managing analyzing and visualizing large sets of data to turn information into business insights...
-
Data Analytics Internship
4 weeks ago
Washington, Washington, D.C., United States teamworkonline Full timeData Analytics InternshipPosition Summary:The Data Analyst Intern at D.C. United will assist in enhancing processes and soccer operations efficiency through data-driven insights. This position involves collecting, analyzing, and interpreting large datasets related to player performance and league dynamics. The intern will work closely with the Strategy &...
-
Azure Data Solutions Architect
3 months ago
Washington, Washington, D.C., United States Analytica Full timeAnalytica is seeking a talented Azure Data Solutions Architect to support one or more dynamic, long-term federal government enterprise data programs. The ideal candidate will lead the architecture and implementation of cloud and on-prem data solutions. Candidates having experience working with the Microsoft Fabric platform will be prioritized. This will be a...
-
Senior Data Analyst
3 months ago
Washington, Washington, D.C., United States Atechstar Full timeWhat we are looking for M.S. in a quantitative discipline (Business Analytics Engineering Operations Research Mathematics Computer Science Statistics etc.) with a minimum 2 years of relevant work experience Profound understanding of data structure and database management and real-world experience with large-scale databases Proficiency in SQL Python R and/or...
-
Data Operations Engineer_TEST_CHANGE
2 months ago
Washington, Washington, D.C., United States Partners Internal Quality Control Full timeCompany DescriptionThis is a test2This is a test job, do not apply. Our mission is to build connections between our clients and their potential customer base by creating a standard of excellence and providing top notch service while, fostering our teams' growth through a rewarding and progressive environment. The growth of our team members is our highest...