The biggest game-changer for my research recently has been using Surge AI for human data collection. With Surge, the workflow for collecting human data now looks closer to “launching a job on a cluster” which is wild to me.

Nobody understands how to generate high-quality human data like Surge AI. Our engineering team was spending 50% of their time managing contractors, after unsuccessfully iterating with other data providers for 18 months, which was a poor use of their time. The Surge AI team came in, redesigned our data collection methods, and doubled the amount of high-quality data for training our models in two weeks. That data gave us the biggest gain in metrics I’ve seen since I joined.

Anytime it's pro level NLP data labeling for hard problems, it inevitably leads to the team at Surge AI.

The Surge AI team are experts at collecting data for training and evaluating large language models. They worked closely with us every step of the way, from helping us understand what types of data we should collect to designing our tasks and guidelines. Their experience and expertise accelerated our timelines for training new human feedback models from 1 year to 1 month.

If you want to do an NLP data collection/labeling process and don't want/need to be managing annotators directly, Surge is remarkably easy to work with and their team does very good work.

When I joined our company, one of my first realizations was that low-quality training data was holding our machine learning models back. It was taking our team 6 months to gather datasets to train new models, and our data scientists were saying half the data was unusable due to quality issues. We talked to Surge AI, and in 2 weeks, they replaced a year’s worth of training data. Retraining our models gave us a 63% boost in model AUC – which was larger than our entire team of ML engineers had produced in 2 years.

Thanks to Surge AI for the incredible work they are doing on search human eval.

The data Surge AI produces is worth its weight in gold. I used to work at Google and Facebook, and we’d have accelerated our ML development by years if we’d had something like Surge AI. Their speed and quality has enabled new machine learning products, and they get it all right on the first try, without needing to iterate for months.