NSF CAREER Award for Simplifying Crowdsource Requests

NJIT's Senjuti Basu Roy, assistant professor of computer science, received a prestigious National Science Foundation award announced this month for her research addressing inefficiencies of deploying tasks in crowdsourced labor services. 

The award is from the NSF's Faculty Early Career Development Program, widely known as CAREER in academic spheres. It is an acknowledgement for early career activities of scholars who integrate research and education in the context of their organizations, the NSF states.

Basu Roy, an expert on optimizing machine learning techniques, won more than $100,000 for her first year of work which could exceed $500,000 as the project continues through 2025. "It's more of an acknowledgement of the research capability of the principal investigator," she noted. "My project compliments, and my hope is that it will augment, all existing crowdworking research."

Crowdsourcing is the concept of solving complicated jobs by dividing the work among large groups of dispersed workers, who are often lay people rather than professionals. The most popular crowdsourcing platform is Amazon's Mechanical Turk. Unfortunately, the process of planning and deploying tasks through such systems can be almost as complicated as the job itself, Basu Roy explained.

"Task deployment on such platforms requires identifying appropriate deployment strategies to satisfy deployment parameters, provided by requesters as thresholds on quality, latency and cost, and also requires analysis of the workforce that is available to undertake the deployed tasks," Basu Roy stated in an NSF abstract.

"To date, task deployment remains a painstakingly manual process, as there is little to no help for requesters in deciding how to organize the workforce, in what style and in what structure to satisfy deployment parameters. Consequently, requesters and workers are mostly confined to one platform, as there is no easy portability of deployment processes across platforms."

To date, task deployment remains a painstakingly manual process, as there is little to no help for requesters in deciding how to organize the workforce, in what style and in what structure to satisfy deployment parameters

Basu Roy is developing middleware to make life easier for the task deployers. She said it's called SLOAN, short for Scalable, decLarative, Optimization-driven, Adaptive and uNified. Her aim is to make it a user-friendly and customizable framework for specifying deployment goals and constraints.

SLOAN will have three technical components. The first part models and recommends deployment strategies for batches of requests. The second part analyzes the workers' preferences and availability, and then sends its results to the recommendation component. Lastly, a results aggregation component estimates the quality of the completed tasks undertaken by the workers. This last component feeds to the other two.

Basu Roy gave an example. "Consider a sample data transcription task, typically seen in Mechanical Turk or Figure Eight. Such tasks contains thousands of audio utterances for common medical symptoms like knee pain or headaches, totaling more than 8 hours. Workers first write text phrases to describe symptoms given. For example, for headaches, a worker might write 'I need help with my migraines'. The requester wants to deploy such a task on the platform. For deployment, the requester naturally has certain constraints in mind: they want the transcribed sentences to be at least 80% as good as the work of a domain expert, in a span of at most 5 days, and by spending at most $350. These are considered as deployment parameters (or constraints)," she wrote.

In that situation, she said, "SLOAN will return a deployment strategy such as, "Hire 150 workers with skills in healthcare and English and divide the 8 hours of audio in chunks of 10 minutes. Pay each of the workers $2 and deploy these small chunks for one day. For each chunk, assign a group of 3 workers and let them collaboratively translate the audio. At the end of this process, assign the 10 most-skilled workers to combining the small pieces and creating the full transcription. Pay these workers $5 each and deploy this latter part for 2 days. These 10 workers should work sequentially, one after the other, and continue to improve the quality of transcription."

Basu Roy hopes to deploy the framework on the non-profit website Crowd4u.org in the next 2-3 years.