Inside the harrowing ecosystem of bots, scammers, catfish, self-promoters, and rogue AI on modern dating apps

fakes fakes fakes!!!

"What's something that feels like it should be illegal to know?"

Today is Valentine’s Day, so I’m thinking a lot about romance, and the things that make romance…difficult. One thing in particular has lived rent-free in my mind ever since I first learned about it in 2018: "animation" — the dating industry term for fake users, fake activity, anything to make a dating app feel like there are signs of life on it.

I've come across many horrors and subjected myself to innumerable traumas over my 13 years working in the dating industry and 25 years of dating. I've been assaulted, kidnapped, held at knifepoint, catfished, manipulated, and mugged. I’ve had all my profile photos stolen for use in someone else’s profile. One time someone stole all my profile text, including the part where I talk about my open relationship with my then-partner! But despite all of this, “animation” is still among the creepiest concepts I’ve ever come across.

It's not just a matter of catfish, scammers, and some Tinder accounts having modelesque photos, blank bios, and/or a link to their Instagram or OnlyFans profile. Dating apps can actually choose which degree of “animation” they prefer to integrate into their system! They can also, by virtue of choosing not to invest in adequate moderation, invite particularly harmful forms of unintentional animation onto their platforms, from catfishing and romance scams to stalking and abuse.

A few years ago I was part of an ABC News 4 Corners investigation into dating apps’ faulty/nonexistent safety & reporting protocols. Earlier this month, I spoke at length about the darkest depths of AI-powered animation and fakery on the Birds and the Bee podcast.

For now, though, let me take you on a little journey through types of "animation" I’ve come across, in the hopes you can protect yourselves now, and well into the future…

Animation Level 1: Static Fake Profiles

Apps can pay ~$50-500 for instant access to hundreds of thousands or even millions of fake profiles — just the photos and fake bios — basically as a .zip file that they can load into their backend servers. This is typically used for testing out features and populating user rolls, so they can see what an experience might look and feel like; but it’s also used to pad user numbers when presenting to investors, just like how you can pay a few bucks for thousands of fake Instagram followers.

Animation Level 2: Real Person, Fake Profile

This is where traditional catfishing comes into play, in which a real person uses photos that aren’t theirs, or fabricates their profile in some way, in order to mislead their intended targets. The 2012 show Catfish and 2022 documentary The Tinder Swindler brought these types of behavior to the national spotlight.

But another variant of this deception also exists. Some companies offer free modeling shoots to women, on the condition that they provide their photo ID and consent to having their images used across 1 or more dating/networking apps. This is a common occurrence in the “international premium dating” industry (aka mail order brides). Users will message these fake profiles, and the dating company can pay random people anywhere in the world to manage a dozen or more of these fake profiles at time. If at some point the end user wants to meet up with their prospective bride, the company will contact the original model and ask them if they want to meet; if they do not want to, the profile's ghostwriter will say something like, “My love, I desperately want to marry you but my father is being stubborn and won’t let me! I’m so sad! Can we keep talking in case he changes his mind?”

Keep in mind, these services may charge you on a per-message basis, so the goal is always to keep you talking, endlessly, online, sometimes totaling $50k or more annually in accumulated fees. Interestingly, if the model actually says yes to meeting, then the app actually can arrange for the end user to meet up with them.

Animation Level 3: Real Person, Real Profile, Misleading Intentions

i.e., influencers, escorts, OnlyFans/camsite performers, and trolls

Platforms typically like having attractive people on their platforms who are keen to message people with some regularity. This boosts their metrics for total members, total daily logins, and total message counts. However, there are entire cohorts of users who systematically violate the spirit of the app by not intending to date, but really just using the platform as a marketing channel for their own content. Influencers usually push you to their IG or Tiktok; escorts usually push you to Kik, Snap, Whatsapp, or text; camsite and OnlyFans performers typically push to either IG, Snap, or direct to their preferred broadcasting platform, depending on the app's profile text affordances and the level of content moderation the dating app employs.

Trolls, on the other hand, could be authentic users, but they're usually so burned out and jaded by online dating that they just throw random shit at their profile and see what sticks.

Trollloloolol

escort profile, most likely

Animation Level 4: Real users (aka a living human on the other end) who happen to be scammers

Most platforms deal with this problem constantly, and many to pay upwards of $1million/month to stem the tide. It results from either lax profile creation standards, or lax content moderation standards, or both. Consequently, any bad actors can simply come in with fake photos/profiles of their own and begin extorting and manipulating users.

Animation Level 5: Fake users who are also scammers — aka “bots”

Platforms fight tooth and nail to cull bots whenever possible, but there are entire industries built upon creating more advanced and nuanced bot networks. If you can deploy 10,000 AI bots to a dating platform, sending malicious links and pre-scripted text flows to millions of users in just a few hours, it’s possible to make millions of dollars. Most of these accounts get banned very quickly, but it only takes a few getting through to wreak havoc. In the US alone, online scams result in the loss of $4.5 BILLION annually, and in 2022, romance scams alone robbed Americans of $740 million!

Animation Level 6: AI Everywhere

With the age of artificial intelligence upon us, where freely-available AI services can instantly generate fake photos, profiles, audio, and even video, the dating ecosystem (and our wider world) is on the precipice of the largest-scale fuckery we’ve ever witnessed. I’m honestly not ready to put Animation Level 6 into words just yet, but I’d be willing to bet that your Instagram ads will soon be giving you hints at what’s to come…

In the coming months, try to keep your wits about you. Stay vigilant. Seek outside opinions, sanity checks, and disconfirming evidence. Be mindful of what information you give out. Be equally mindful of what information you take in. Remember to go outside. Breathe fresh air. Talk to living, breathing people. Remind yourself that you’re still human…for now.

---

If you're curious to learn more about the inner workings of content moderation, you can check out the white paper I wrote on the topic, or delve into the ongoing efforts of Integrity Institute to build up better globally coordinated systems of user/data/privacy/election protection.

Here are two snippets from my white paper which capture the ridiculous complexity (and cost) of moderation across modern internet platforms:

I also just spoke at length about the darkest depths of AI-powered animation and fakery on the Birds and the Bee podcast! We recorded for over 2 hours, so be sure to subscribe, as there are still 1-2 more segments to be released!

If you ever want to discuss these issues further, or learn more about how I coach daters through these minefields, please reach out! You can schedule a free chat with me anytime via the calendar link on my website, Dateworking.com.

my 2 primary vibes in life are 1) lofi massagey cuddle puddle Steve, and 2) overcaffeinated, bedraggled, underslept nerdy ranting Steve

PS — I’ve compiled my last 25 years of dating experience and wisdom into an 8-week group seminar series! My goal is to set all my students up with a deep dive into their dating and relationship self-understanding, coupled with extremely practical step-by-step guidance through selecting dating apps, making profiles, choosing photos, sending messages, flirting, designing dates, and laying the groundwork for sustainable relationships. You can sign up for the next cohort’s waitlist here!