Bookmark: "Inadvertent Algorithmic Cruelty"

Sebastian Greger

Bookmarked:

A spot-on example of scenarios not taken into account in design inadvertently turning something intended to be “joyful” into something “awful”: Eric Meyer really did not want to be reminded of his past year, but still faced Facebook pushing his terrible year's content on him in a “Your Year in Review” feature:

This inadvertent algorithmic cruelty is the result of code that works in the overwhelming majority of cases, reminding people of the awesomeness of their years, showing them selfies at a party or whale spouts from sailing boats or the marina outside their vacation house.

But for those of us who lived through the death of loved ones, or spent extended time in the hospital, or were hit by divorce or losing a job or any one of a hundred crises, we might not want another look at this past year.

The algorithm is even the problem here; it’s the difficulty to grasp that something designers may believe to be great for their target personas might not be for people in other circumstances. What Eric says:

If I could fix one thing about our industry, just one thing, it would be that: to increase awareness of and consideration for the failure modes, the edge cases, the worst-case scenarios.

As he points out in a follow-up post, this can happen to anybody who creates something – this is not about blame for an inconsiderate decision; it’s a reminder that maybe the non-intended consequences of design might be worth more contemplation than its actually intended goals?!

I'm Sebastian, Sociologist and Interaction Designer. This journal is mostly about bringing toge­ther social science and design for inclusive, privacy-focused, and sustainable "human-first" digital strategies. I also tend to a "digital garden" with carefully curated resources.

My monthly email newsletter has all of the above, and there are of course also an RSS feed and Twitter.