Data Engineer

Data Engineer

Application Deadline:

30 April 2024 Department:

ODC

- Technology Employment Type:

Contract / Temp Location:

ODC - India Description

We are a data driven asset management firm with a very wide variety of datasets with varying frequency of updates that essentially drive our research models which ultimately determine the daily trading activity. We also trade a wide range of asset classes like Equities, Futures, Options, FX, Swaps, etc. We are currently undergoing a data platform modernization program and building out our next gen data platform and associated data governance capabilities. This has led to internal team re-structuring and creation of two main data product streams: Macro and Equity, each supported by a dedicated team that does all the new development, BAU and ongoing support. This role is for a talented Data Engineer to join the Macro Data Product team that covers all the non-Equity asset classes. Primary responsibilities of the team are: New Data Product Delivery, Data Quality Monitoring, Ongoing BAU and Data Operations. The team will leverage a common Data Platform and follow uniform standards to deliver Macro data to the research and/or other business teams and to production systems.

Most of our existing Macro data tech stack is Java and MS SQL Server based. We use Snowflake as our strategic data warehouse in AWS. We want to migrate to cloud based data ingestion pipelines built using Python and Snowflake. We are currently running a PoC to finalize our pipeline orchestrator and the team may get involved in existing or new PoCs to trial new tools and contribute to the capabilities of the strategic data platform. We use JIRA for change management and follow an Agile approach to delivery. Key Responsibilities Build a deep understanding of our current Macro data products and their usage across various user groups and system components.

Replace existing Macro data loaders with Python based data pipelines on Snowflake and AWS. The pipelines cover both raw data ingestion as well as generation of derived Data Products as per the business requirements.

Build the necessary microservices to support the pipelines.

Collaborate with the research teams to understand new data requirements and translate that to a set of technical requirements and implement the appropriate solution working in close conjunction with the Manager.

Maintaining Data Quality is a key responsibility of the team. Work with stakeholders to understand the data quality requirements and implement the appropriate data checks and notifications.

Participate in the team rota to provide BAU & Data Ops support. We expect everyone in the team to be on the rota. This includes providing L2/L3 support to our L1 support team (PSG) and monitoring the pipelines end of end and fix any issues escalated to the team.

Put your Data Analyst hat on to investigate and fix any data quality issues, identify the impact and urgency by utilizing information on the wikis or through Lineage information from the Data Catalog. Maintain high degree of communication with the internal stakeholders or external vendors during incident management.

Run PoCs to trial new tools and contribute to the strategic data platform particularly around Data Catalog, Lineage & Observability but this could extend to other areas.

Maintain good level of documentation on the internal wiki.

Maintain high quality of delivery by following good development practices including thorough unit testing.

Participate in daily standups and regular Sprint planning meetings and provide effort estimates and task updates.

Any other duties commensurate with the post holder’s position and seniority.

All employees should understand that it is their personal responsibility to comply with all organisational, statutory and regulatory policies and procedures.

Skills, Knowledge and Expertise

Essential 7+ years’ strong Python programming experience.

Strong foundation of OO software engineering concepts.

Strong SQL skills including writing complex queries, views, and query optimization.

Data wrangling experience with libraries like Pandas ideally using financial market data.

Solid understanding of Snowflake concepts and troubleshooting query performance.

Willing to carry out operational data tasks alongside developing new solutions.

Well organised with the ability to prioritise and manage multiple streams of work independently.

Ability to handle high pressure, complex environments.

Build positive working relationships with external and internal teams.

Strong problem solving and analytical skills to solve complex issues.

Excellent verbal and written communication skills.

Excellent interpersonal skills and be able to quickly understand and retain knowledge of business processes and the technology systems.

Beneficial Experience onboarding Market or Reference data using Bloomberg Data License.

Experience building microservices in Python.

Familiarity with Docker and CI/CD in the cloud would be an advantage.

Experience in handling financial data sets and familiarity with financial instruments (equities, futures, FX).

Experience working with Orchestration tools like Prefect or Dagster.

Educational & Professional Qualifications

·

Educated to degree (or equivalent) level or higher, from a leading university.

·

A degree with mathematical and computing content (e.g. Computer Science, Maths, Physics, Economics) #J-18808-Ljbffr