Pinterest had already done this kind of moderation work on terms like “anti-vaccination ,” using a combination of humans and machine learning to clean up the search results. Now, after training its algorithms to recognize content that promotes self-harm, the company says reports of those pins are down 88 percent. When someone does flag a pin that violates its community guidelines, which ban content that encourages suicide or self-injury, it’s three times faster at removing it.
Removing bad content was only part of the problem. The fact remained that people were coming to Pinterest to look for stuff related to self-harm, feeling overwhelmed and in pain. “Lots and lots of people are using it for that,” says Evan Sharp, Pinterest’s cofounder and chief of design. “So, how do we hold up our responsibility to do our best to help them?”
The company launched an initiative over the summer called “compassionate search ,” which aimed to help Pinterest users combat stress and anxiety by channeling negative emotions into exercises like guided breathing or journaling. Now the company has built another series of exercises specifically geared toward the emotional turbulence of self-injury. Developed in collaboration with a group of mental health organizations, the exercises take techniques from dialectical behavior therapy—a type of psychotherapy used to treat mood disorders, self-harm, and suicidal ideation—and reimagine them for the smartphone.
When a pinner enters a related search term, the site will surface a prompt for these exercises. (It also displays the number for the National Suicide Prevention Lifeline, one of Pinterest’s partners.) Pinterest does not store data about who uses these exercises, and participation will never affect advertisements or pin recommendations on the platform.“We do in fact have evidence-based treatments, based in dialectical behavioral therapy, that work well,” says Nina Vasan, a clinical assistant professor at Stanford's School of Medicine and the founder and director at Brainstorm, Stanford’s lab for mental health innovation, which is one of the groups that worked with Pinterest. “The problem is that people don’t know about them or don’t have access to the treatments, so as physicians we feel like it’s urgent for us to think about creative ways of educating people about and increasing access to the treatments that we know work.”