Data Engineer

My client is a not-for-profit organisation focused on developing the blockchain/DLT ecosystem. Due to ongoing growth they are now looking for an experienced Data Engineer to join their team. This role is pivotal in managing, curating, optimising, and securing datasets specifically related to cryptocurrency discussions across various platforms. The ideal candidate will be adept in web scraping, data quality assurance using AI, data integration, ensuring data security and compliance, and maintaining detailed documentation. Blockchain experience is not needed but a solid background with Python, AWS and web scraping is a must. Role and Responsibilities * Data Collection: Identify relevant chat sources, groups, and forums on platforms discussing particular topics. Maintain and develop web scraping tools or APIs for periodic data extraction. * Data Quality Assurance: Develop and implement AI-based procedures for quality control of data and data sources to eliminate inaccuracies and anomalies. Create tools for monitoring data sources for changes and updates, adapting data collection and cleaning processes accordingly. * Data Integration: Collaborate with data scientists and analysts to integrate collected data into various projects and analysis tools. Ensure smooth data flow and integration with other data sources within the organisation. * Data Security and Compliance: Uphold the security and privacy of collected data in compliance with relevant regulations and company policies. * Documentation: Maintain clear and comprehensive documentation of data sources, collection methods, and workflows. Produce reports and documentation for both internal and external stakeholders as required. * Monitoring and Reporting: Develop and maintain systems to monitor the performance and health of data collection processes. Key skills needed; * Bachelor’s degree in Computer Science, Data Science, or a related field. * Demonstrable experience in a data engineering role, with a focus on web scraping and data collection. * Proficient in programming languages such as Python, SQL. * Strong AWS experience * Knowledge of Data Structures and Databases * Knowledge of data privacy laws and compliance requirements. * Experience with Data Analytics/algorithms would be nice to have but training will be provided for the right candidate * Excellent communication and collaboration abilities. Desirable * Experience with Typescript * Experience with big data technologies and cloud services. * Proficiency in AI and machine learning techniques for data quality assurance. They are happy for this role to be predominately remote based with once or twice a month in the office