Some other more concrete examples I want to briefly highlight are: (1) facial recognition in surveillance systems, (2) military drone control UI, (3) military shooter games, and (4) deceptive UX patterns.
- This example ties back to AI being as biased as their creators. We’ve seen and heard stories of facial recognition software racially profiling people of darker skin, Black or not (but more likely Black than not). Much has already been cited and said about this, but a new disturbing case study of AI usage of deceased or missing children can be seen here.
- Many innocent civilians abroad, from Yemen to Somalia to Afghanistan, have died at the hands of drone bombings. While one can argue that in times of conflict minimizing infantry casualties and fatalities is a noble goal, where that argument becomes plagued is when the solution for that problem is applied to wanton murder and genocide (ass seen with the bomb drops on Hiroshima and Nagasaki). And we designers play a hand in it by architecting the interfaces this is all done on.
- While this trend is dying down, there have been too many shooters, especially set in modern times, that depict American troops versus insert-race-here (usually Arabs, Kurds, and sometimes Russians), where you can guess which side is “good” and “evil.” Entertainment value shouldn’t be discounted; people absorb what they consume, and we design those narratives and those characters. As such, we can make or break harmful images, stereotypes, and racial profiling.
- Deceptive UX patterns are an immortal cancer in the digital universe. But in a strange case of possible justice, Amazon is being sued over its purposeful misleading of their customer base, tricking them into signing up for Prime. This is a step in the right direction, and this is a tricky subject to broach since designers likely are forced into designing these disgusting patterns. But at what point does it become too much?
The more and more we look away from the repercussions, and even the horrors, of our own work coming to life, the further we perpetuate the lie that we help humanity while helping it to its ruin. As Oppenheimer says during the countdown sequence to the Trinity test, “these things are hard on your heart.”
In the movie, the military aggressively insists on compartmentalization in the name of national security (since Russia and Communism were perceived as subtle threats at the same time as Nazis being the explicit enemy). Oppenheimer then brainstorms a way to maintain physical separations of the different groups and establishments tied to the Manhattan Project while also combining resources and knowledge at one central location (that being Los Alamos). He justifies this pushback of sorts by mentioning that suboptimal security may cost America the race, but the Nazis finishing first will cost them the race (though there was also a scientific curiosity in creating the bomb). In challenging militaristic dogmas, the team at Los Alamos created the first successful atom bomb.
Different designers possess different skill sets, strengths, career ambitions, personalities, experiences, and weaknesses. Siloing ourselves by project or lane or vertical, especially within our own organizations, would only slow or diminish our growth (professionally and personally). If we each wish to progress a project faster, sure, we can design it out individually. But to go farther with a project, working together is the best way (based on one of my favorite proverbs). You see more when you get out of tunnel vision and enjoy a bridge view instead.
I do however wish to emphasize the importance of having perspectives and opinions that push back against your own, in a productive way, just as those at Los Alamos had expressed their concerns with the Manhattan Project’s necessity after Hitler’s death. Having an entire room of yes-people not only limits design potential, but can further increase the likelihood and degree of harm designed and deployed to the world.
“These things are hard on your heart…” until you’ve reflected on the implications of your work going beyond your control and into the world. In the movie, we can see and feel that shift in tone from pride and accomplishment to fear, regret, and eventual dread once the US army decides to take the bombs from Los Alamos to drop on Japan. There’s an irony in which at this point, Oppenheimer speaks briefly with Edward Teller, and he says to Teller that we don’t have a right to dictate how this new weapon will be used since it’s out of the scientists’ hands now. Soon after, Teller asks Oppenheimer to support his desire to build a hydrogen bomb, which Oppenheimer quickly dismisses; he had begun to consider what dangers have and could arise from what they have done at Los Alamos.
Oppenheimer also asks Groves to relay some of his concerns to President Truman around how the bomb is perceived (subtly echoing what Neils Bohr had said to Oppenheimer earlier on in the film: “We have to make the politicians understand this isn’t a new weapon. It’s a new world. I’ll be out there doing what I can but you, you are an American Prometheus. The man who gave them the power to destroy themselves. And they’ll respect that”). Truman exclaims on radio about the bombs that have been dropped on Hiroshima and Nagasaki, and the power and destruction they created. From this point on we begin to see the crippling remorse that plagues Oppenheimer for the rest of his life, and his failed attempts to encourage peace talks and knowledge-sharing among nations and institutions (especially Russia).
One of the greatest scenes in the movie is the congregation following Truman’s announcement of the bombs falling and Japan surrendering. Oppenheimer comes out to the Los Alamos crowd in a small auditorium, and while he spouts congratulatory remarks to rile the crowd up, we get an alternate glimpse of him having some kind of out-of-body experience, the type that a person would have when something just feels too unreal to comprehend. It was a chilling interpretation of these famous and equally chilling words spoken by Oppenheimer in real life:
As designers, we need to cement this understanding in our collective psyche, in our curricula, our practices and processes that what we do and don’t design and advocate for on behalf of people will impact them. Design can have multiple impacts, a single impact, it can be direct, or indirect, and vary in severity or magnitude. As Niels Bohr said to Oppenheimer in the film, building the bomb will bring about a new world, not merely a weapon. Design proposals and our willingness to create those designs have that same effect, though some solutions will have a slower or more subtle effect than others.
We can and should take pride in our work, we can and should celebrate our wins and launches, but at what cost? Retaliation is also real in tech (as with Timnit Gebru’s case); speaking out against juggernauts driving profit at the cost of respect for people’s experiences, needs, and lives is never easy. How may we navigate that as practitioners? As history has shown, as the present shows, and as the future will show if we stay this course, the cost of unchecked progress will always be too great, burdensome, even bloody.