Market Data Engineer
Full-time
Company Description
We are a global algo trading company fully dedicated to technology. Our brilliant team comprises 100+ individuals scattered across the globe. Seventy percent of our team members are technical specialists, encompassing development, infrastructure, testing, and analytics, while the rest comprise business units, including operations, legal, finance, etc.
Job Description
Historical Data Capture and Storage: Design, develop, and maintain systems for the acquisition, storage, and retrieval of historical market data from multiple financial exchanges, brokers, and market data vendors.
Data Integrity and Accuracy: Ensure the integrity and accuracy of historical market data, including implementing data validation, cleansing, and normalization processes.
Data Architecture Development: Build and optimize data storage solutions, ensuring they are scalable, high-performance, and capable of managing large volumes of time-series data.
Versioning and Reconciliation: Develop systems for data versioning and reconciliation to ensure that changes in exchange formats or corrections to past data are properly handled.
Data Source Integration: Implement robust integrations with various market data providers, exchanges, and proprietary data sources to continuously collect and store historical data.
Data Access Tools: Build internal tools to provide easy access to historical data for research and analysis, ensuring performance, ease of use, and data integrity
Collaborate with Trading and Research Teams: Work closely with quantitative researchers and traders to understand their data requirements and optimize the systems for data retrieval and analysis for backtesting and strategy development.
Performance and Scalability: Develop scalable solutions to handle growing volumes of historical market data, including ensuring efficient queries and data retrieval for research and backtesting needs.
Optimize Storage Costs: Work on optimizing data storage solutions, balancing cost-efficiency with performance, and ensuring that large datasets are managed effectively.
Compliance and Auditing: Ensure historical market data systems comply with regulatory requirements and assist in data retention, integrity, and reporting audits.
Qualifications
Required Skills and Experience:
Commercial experience of
financial instruments and markets (equities, futures, options, forex, etc.), particularly understanding how historical data is used for algorithmic trading. Familiarity with
market data formats (e.g., MDP, ITCH, FIX, SWIFT, proprietary exchange APIs) and market data providers. Strong programming skills in Python and Go, or Rust. Familiarity with
ETL (Extract, Transform, Load) processes (or other data pipeline architecture) and tools to clean, normalize, and validate large datasets. Experience in working with
distributed data systems and tools such as
Hadoop, Kafka, Spark, or similar technologies.
Linux/Unix expertise, particularly in managing and optimizing systems for data storage and processing.
Commercial experience in building and maintaining large-scale time series or historical market data in the financial services industry. Strong
SQL proficiency: aggregations, joins, subqueries, window functions (first, last, candle, histogram), indexes, query planning, and optimization.
Bachelor’s degree in Computer Science, Engineering, or related field.
Preferred Qualifications:
Experience in a
proprietary trading firm or
buy-side environment working with historical market data and its vendors. Experience with
data governance and
compliance related to financial data storage and retrieval. Proficiency in
containerization, orchestration - Docker, Airflow, SLURM tools. Experience with
cloud-based storage solutions such as AWS S3, Google Cloud Storage, or Azure, and the ability to optimize for performance and cost. Familiarity with
machine learning and data science workflows to support quantitative research teams.
Additional Information
What we offer:
Working in a modern international technology company without bureaucracy, legacy systems, or technical debt. Excellent opportunities for professional growth and self-realization. We work remotely from anywhere in the world, with a flexible schedule. We offer compensation for health insurance, sports activities, and non-professional training.
Market Data Engineer
Dubai - United Arab Emirates * Full-time
