Corpus Platform Development

Language corpora gain increasing relevance in the Human Language Technology areas. World leading universities, research groups and NLP and speech technology companies apply corpora with vetted linguistic data for an array of purposes from studying the dynamics of language development to training NLP models.

Oxford University Press is known as the creator of reliable human language technology tools and services. For years, OUP has been developing and using New Monitoring Corpus building software that harvested RSS feeds and collected data. Up to this date, NMC has stored more than 26 million documents with around 9 million tokens.

However, latest technologies allow to improve these numbers and excel the capabilities of corpora. With this in mind, OUP decided to create a system able to collect more data and stand no performance limits.

The main goal was to build a platform scalable in terms of productivity and functionality. To address this challenge, OUP has chosen cloud technology – not a standard approach to making a corpus. As a result, Super Corpus Platform has proved to become a powerful system which exceeded all expectations.

The system harvests as much as 80 thousand documents per day in a cold mode. For the record, 80 thousand documents include all news articles in English collected by Event Registry throughout the Internet daily. Super Corpus Platform can process this one-day worth news feed in 3 hours. Moreover, it can handle roughly 1 million documents a day in a hot mode, with the ultimate performance of 8.6 million documents in 24 hours.

Today, Super Corpus Platform compiles 2 million documents which corresponds to 800-900 thousand documents of balanced corpus content monthly. Just for comparison: the NMC system has collected 26 million documents in 5 years. It makes the new Super Corpus Platform almost 5 times more efficient than the old system. Not to mention the quality of the balanced corpus content created by the new system.

Moreover, the modular architecture of Super Corpus Platform makes it a highly scalable tool, both in performance and functionality. Thus, new components such as sentiment analysis or genre recognition can be added to a given data creation process. So as other data sources (Twitter, hybrid search using Bing API) and additional languages. Technically, even the core components – processing pipeline and data storage – can be substituted from Azure components to equivalent modules of other platforms.

All in all, Super Corpus Platform represents the solution for corpus building with huge potential and applicability.