"We’re introducing safety notices in Messenger that will pop up in a chat and provide tips to help people spot suspicious activity and take action to block or ignore someone when something doesn’t seem right," reads a blog post from Facebook's director of product management for Messenger privacy and safety Jay Sullivan. "As we move to end-to-end encryption, we are investing in privacy-preserving tools like this to keep people safe without accessing message content."
Facebook hasn't revealed many details about how its machine-learning abuse detection tricks will work. But a Facebook spokesperson tells WIRED the detection mechanisms are based on metadata alone: who is talking to whom, when they send messages, with what frequency, and other attributes of the relevant accounts—essentially everything other than the content of communications, which Facebook's servers can't access when those messages are encrypted. "We can get pretty good signals that we can develop through machine learning models, which will obviously improve over time," a Facebook spokesperson told WIRED in a phone call. They declined to share more details in part because the company says it doesn't want to inadvertently help bad actors circumvent its safeguards.
Facebook Had an Incredibly Busy Weekend
The company's blog post offers the example of an adult sending messages or friend requests to a large number of minors as one case where its behavioral detection mechanisms can spot a likely abuser. In other cases, Facebook says, it will weigh a lack of connections between two people's social graphs—a sign that they don't know each other—or consider previous instances where users reported or blocked a someone as a clue that they're up to something shady.
One screenshot from Facebook, for instance, shows an alert that asks if a message recipient knows a potential scammer. If they say no, the alert suggests blocking the sender, and offers tips about never sending money to a stranger. In another example, the app detects that someone is using a name and profile photo to impersonate the recipient's friend. An alert then shows the impersonator's and real friend's profiles side-by-side, suggesting that the user block the fraudster.