autotldr@lemmings.world
on 07 May 2024 12:45
nextcollapse
This is the best summary I could come up with:
The latest culprit in this area is Meta’s AI chatbot, which, for some reason, really wants to add turbans to any image of an Indian man.
We tried prompts with different professions and settings, including an architect, a politician, a badminton player, an archer, a writer, a painter, a doctor, a teacher, a balloon seller, and a sculptor.
For instance, it constantly generated an image of an old-school Indian house with vibrant colors, wooden columns, and styled roofs.
In the gallery bellow, we have included images with content creator on a beach, a hill, mountain, a zoo, a restaurant, and a shoe store.
In response to questions TechCrunch sent to Meta about training data an biases, the company said it is working on making its generative AI tech better, but didn’t provide much detail about the process.
If you have found AI models generating unusual or biased output, you can reach out to me at im@ivanmehta.com by email and through this link on Signal.
The original article contains 956 words, the summary contains 164 words. Saved 83%. I’m a bot and I’m open source!
kokesh@lemmy.world
on 07 May 2024 12:48
nextcollapse
Makes sense
0x0@programming.dev
on 07 May 2024 13:08
nextcollapse
There are a lot of men in India who wear a turban, but the ratio is not nearly as high as Meta AI’s tool would suggest. In India’s capital, Delhi, you would see one in 15 men wearing a turban at most.
Probably because most Sikhs are from the Punjab region?
FMT99@lemmy.world
on 07 May 2024 13:24
nextcollapse
Why would you ask a bot to generate a stereotypical image and then be surprised it generates a stereotypical image. If you give it a simplistic prompt it will come up with a simplistic response.
0x0@programming.dev
on 07 May 2024 15:11
nextcollapse
A traditional outfit
How traditional? How statistically relevant is it? Most Indians i know do not wear turbans at all.
If these stats are trustworthy (and i think they are), the only Indians that wear turbans are Sikhs (1.7%) and Muslims (14.2%). I’d say 15.9% is not statistically significant.
HopeOfTheGunblade@kbin.social
on 07 May 2024 15:24
nextcollapse
You don't think nearly 1/6th is statistically significant? What's the lower bound on significance as you see things?
To be clear, it's obviously dumb for their generative system to be overrepresenting turbans like this, although it's likely to be a bias in the inputs rather than something the system came up with itself, I just think that 5% is generally enough to be considered significant and calling three times that not significant confuses me.
0x0@programming.dev
on 07 May 2024 15:36
nextcollapse
You don’t think nearly 1/6th is statistically significant?
FarceOfWill@infosec.pub
on 07 May 2024 16:14
collapse
5/6 not wearing them seems more statistically significant
tabular@lemmy.world
on 07 May 2024 17:58
nextcollapse
The fact less people of that group actually wear it than do is significant when you want an average sample. When categorizing a collection of images then, naturally, the traditional garments of a group is associated more with that group than any other group: 1/6 is bigger than any other race.
On a footnote: why should the concept of a traditional dress be offensive?
Ain’t to me, couldn’t care less. I was just trying to point out that most Indians do not seem to wear turbans (and based my reasoning on the religions dress alone).
otp@sh.itjust.works
on 07 May 2024 17:14
nextcollapse
I think the idea is that it’s what makes a person “an Indian” and not something else.
Only a minority of Indians wear turbans, but more Indians than other people wear turbans. So if someone’s wearing a turban, then that person is probably Indian.
I’m not saying that’s true necessarily (though it may be), but that’s how the AI interprets it…or how pictures get tagged.
It’s like with Canadians and maple leaves. Most Canadians aren’t wearing maple leaf stuff, but I wouldn’t be surprised if an AI added maple leaves to an image of “a Canadian”.
ramble81@lemm.ee
on 07 May 2024 18:24
nextcollapse
Except if they trained it on something that has a large proportion of turban wearers. It is only as good as the data fed to it, so if there was a bias, it’ll show the bias. Yet another reason this really isn’t “AI”
catsarebadpeople@sh.itjust.works
on 07 May 2024 18:49
nextcollapse
I think you’re looking at it wrong. The prompt is to make an image of someone who is recognizable as Indian. The turban is indicative clothing of that heritage and therefore will cause the subject to be more recognizable as Indian to someone else. The current rate at which Indian people wear turbans isn’t necessarily the correct statistic to look at.
What do you picture when you think, a guy from Texas? Are they wearing a hat? What kind? What percentage of Texans actually wear that specific hat that you might be thinking of?
sugar_in_your_tea@sh.itjust.works
on 08 May 2024 02:18
collapse
A surprising number of Texans wear cowboy and trucker hats (both stereotypical). A surprising number of Indians don’t wear turbans since it’s by far a minority.
catsarebadpeople@sh.itjust.works
on 08 May 2024 03:56
collapse
Woosh
rimjob_rainer@discuss.tchncs.de
on 07 May 2024 21:19
nextcollapse
Imagine a German man from Bavaria… You just thought of a man wearing Lederhosen and holding a beer, didn’t you? Would you be surprised if I told you that they usually don’t look like that outside of a festival?
mriormro@lemmy.world
on 08 May 2024 01:32
collapse
I don’t picture real life people as though they were caricatures.
sugar_in_your_tea@sh.itjust.works
on 08 May 2024 02:17
nextcollapse
But AI does, because we feed it caricatures.
catsarebadpeople@sh.itjust.works
on 08 May 2024 04:01
nextcollapse
Are you literally the second coming of Jesus? Hey everybody! I found a guy who doesn’t see race! I can’t believe it but he doesn’t think anyone is changed in any way by the place that they grew up in or their culture! Everyone is a blank slate to this guy! It’s amazing!
mriormro@lemmy.world
on 11 May 2024 12:01
collapse
No, I just don’t lob groups of people together. It’s not that hard to do, everyone’s a different person.
admin@lemmy.my-box.dev
on 08 May 2024 04:11
collapse
He was imaginary though.
Duamerthrax@lemmy.world
on 08 May 2024 04:33
collapse
What’s the data that the model is being fed? What percentage of imaging featuring Indian men are tagged as such? What percentage of imaging featuring men wearing Turbans are tagged as Indian Men? Are there any images featuring Pakistan men wearing Turbans? Even if only a minority of Indian feature Turbans, if that’s the only distinction between Indian and Pakistan men in the model data, the model will favor Turbans for Indian Men. That’s just a hypothetical explanation.
FiskFisk33@startrek.website
on 08 May 2024 04:13
nextcollapse
By that logic americans should always be depicted in cowboy hats.
Duamerthrax@lemmy.world
on 08 May 2024 04:28
nextcollapse
VirtualOdour@sh.itjust.works
on 08 May 2024 08:43
nextcollapse
Put in western or Texas and that’s what you get, the west is a huge area even just of America but the word is linked to a lot of movie tropes and stuff so that’s what you get.
This is also only when the language is English, ask in urdu or Bengali and you get totally different results, in fact just use urdu instead of Indian and get less turbans or put in Punjabi and you’ll get more turbuns.
Or just put turban in the negatives if you want
skulblaka@startrek.website
on 08 May 2024 13:57
collapse
Ask an AI for pictures of Texans and see how many cowboy hats it gives back to you.
Eyck_of_denesle@lemmy.zip
on 08 May 2024 05:05
collapse
It’s a traditional outfit of sikhs, not indians. Pick up a book
vin@lemmynsfw.com
on 08 May 2024 06:16
nextcollapse
Pre-independence, most Indian males had some sort of headgear. E.g. look at any old photos of Bombay
Eyck_of_denesle@lemmy.zip
on 08 May 2024 09:05
collapse
I’m not sure how AI could be possibility racist. (Image is of a supposed Native American but my point still stands)
TrickDacy@lemmy.world
on 07 May 2024 16:23
nextcollapse
What point is that?
possiblylinux127@lemmy.zip
on 07 May 2024 17:15
collapse
Stereotypes are everywhere
demonsword@lemmy.world
on 07 May 2024 18:28
collapse
the AI itself can’t be racist but it will propagate biases contained in its training data
possiblylinux127@lemmy.zip
on 07 May 2024 21:26
collapse
It definitely can be racist it just can’t be held responsible as it just regurgitates.
sugar_in_your_tea@sh.itjust.works
on 08 May 2024 02:32
nextcollapse
No, racism requires intent, AI aren’t (yet) capable of intent.
demonsword@lemmy.world
on 08 May 2024 11:33
collapse
It definitely can be racist
that requires reasoning, no AI that currently exist can do that
admin@lemmy.my-box.dev
on 07 May 2024 17:33
nextcollapse
Meanwhile on DALL-E…<img alt="DALL-E drawing a girl in 18th century clothing, standing in front of a stereo typical (by by no means average) house." src="https://lemmy.my-box.dev/pictrs/image/5f749d0f-70d0-458c-bdf8-fa274ebceb65.jpeg">
admin@lemmy.my-box.dev
on 07 May 2024 17:35
nextcollapse
I’m just surprised there’s no windmill in either of them. Canals, bikes, tulips… Check check check.
admin@lemmy.my-box.dev
on 08 May 2024 17:50
collapse
<img alt="windmills do not work that way!" src="https://lemmy.my-box.dev/pictrs/image/4e3bde4f-928f-45fc-94a5-7705b69d96ba.webm">
emidio@lemmy.blahaj.zone
on 12 May 2024 16:52
collapse
That looks just like the town Delft
gerryflap@feddit.nl
on 07 May 2024 17:40
nextcollapse
Kinda makes sense though. I’d expect images where it’s actually labelled as “an Indian person” to actually over represent people wearing this kind of clothing. An image of an Indian person doing something mundane in more generic clothing is probably more often than not going to be labelled as “a person doing X” rather than “An Indian person doing X”. Not sure why these authors are so surprised by this
captainlezbian@lemmy.world
on 08 May 2024 02:20
nextcollapse
Yeah I bet if you searched desi it wouldn’t have this problem
NocturnalEngineer@lemmy.world
on 08 May 2024 05:00
collapse
Like most of my work’s processes… Shit goes in, shit comes out…
Kolanaki@yiffit.net
on 07 May 2024 18:12
nextcollapse
Would they be equally surprised to see a majority of subjects in baggy jeans with chain wallets if they prompted it to generate an image of a teen in the early 2000’s? 🤨
Linkerbaan@lemmy.world
on 07 May 2024 19:16
nextcollapse
That’s Sikh
jivandabeast@lemmy.browntown.dev
on 07 May 2024 23:14
nextcollapse
Indians can be Sikh, not all indians are Hindu
Eyck_of_denesle@lemmy.zip
on 08 May 2024 05:04
nextcollapse
Yes but the gentlemen in the images are also sikhs
UnderpantsWeevil@lemmy.world
on 08 May 2024 14:30
collapse
glances at the current policies of the Indian government
Well… for now, anyway.
sugar_in_your_tea@sh.itjust.works
on 08 May 2024 02:16
nextcollapse
Not necessarily. Hindus and Muslims in India also wear them.
rimjob_rainer@discuss.tchncs.de
on 07 May 2024 21:15
nextcollapse
Wouldn’t surprise me if most of them are depicted shitting on the street
dwalin@lemmy.world
on 07 May 2024 23:23
nextcollapse
Overfitting
It happens
Sam_Bass@lemmy.world
on 08 May 2024 04:50
nextcollapse
Get down with the Sikhness
Sorgan71@lemmy.world
on 08 May 2024 08:46
collapse
not me calling in sikh to work
VirtualOdour@sh.itjust.works
on 08 May 2024 09:06
nextcollapse
Articles like this kill me because the nudge it’s kinda sorta racist to draw images like the ones they show which look exactly like the cover of half the bollywood movies ever made.
Yes, if you want to get a certain type of person in your image you need to choose descriptive words, imagine gong to an artist snd saying ‘I need s picture and almost nothing matters beside the fact the look indian’ unless they’re bad at their job they’ll give you a bollywood movie cover with a guy from rajistan in a turbin - just like their official tourist website does
Ask for an business man in delhi or an urdu shop keeper with an Elvis quiff if that’s what you want.
UnderpantsWeevil@lemmy.world
on 08 May 2024 14:29
collapse
the ones they show which look exactly like the cover of half the bollywood movies ever made.
Almost certainly how they’re building up the data. But that’s more a consequence of tagging. Same reason you’ll get Marvel’s Iron Man when you ask an AI generator for “Draw me an iron man”. Not as though there’s a shortage of metallic-looking people in commercial media, but by keyword (and thanks to aggressive trademark enforcement) those terms are going to pull back a superabundance of a single common image.
imagine gong to an artist snd saying ‘I need s picture and almost nothing matters beside the fact the look indian’
I mean, the first thing that pops into my head is Mahatma Gandhi, and he wasn’t typically in a turbine. But he’s going to be tagged as “Gandhi” not “Indian”. You’re also very unlikely to get a young Gandhi, as there are far more pictures of him later in life.
Ask for an business man in delhi or an urdu shop keeper with an Elvis quiff if that’s what you want.
I remember when Google got into a whole bunch of trouble by deliberately engineering their prompts to be race blind. And, consequently, you could ask for “Picture of the Founding Fathers” or “Picture of Vikings” and get a variety of skin tones back.
So I don’t think this is foolproof either. Its more just how the engine generating the image is tuned. You could very easily get a bunch of English bankers when querying for “Business man in delhi”, depending on where and how the backlog of images are sources. And urdu shopkeeper will inevitably give you a bunch of convenience stores and open-air stalls in the background of every shot.
I’m guessing this relates to training data. Most training data that contains skin cancer is probably coming from medical sources and would have a ruler measuring the size of the melanoma, etc. So if you ask it to generate an image it’s almost always going to contain a ruler. Depending on the training data I could see generating the opposite as well, ask for a ruler and it includes skin cancer.
threaded - newest
This is the best summary I could come up with:
The latest culprit in this area is Meta’s AI chatbot, which, for some reason, really wants to add turbans to any image of an Indian man.
We tried prompts with different professions and settings, including an architect, a politician, a badminton player, an archer, a writer, a painter, a doctor, a teacher, a balloon seller, and a sculptor.
For instance, it constantly generated an image of an old-school Indian house with vibrant colors, wooden columns, and styled roofs.
In the gallery bellow, we have included images with content creator on a beach, a hill, mountain, a zoo, a restaurant, and a shoe store.
In response to questions TechCrunch sent to Meta about training data an biases, the company said it is working on making its generative AI tech better, but didn’t provide much detail about the process.
If you have found AI models generating unusual or biased output, you can reach out to me at im@ivanmehta.com by email and through this link on Signal.
The original article contains 956 words, the summary contains 164 words. Saved 83%. I’m a bot and I’m open source!
Makes sense
Probably because most Sikhs are from the Punjab region?
Why would you ask a bot to generate a stereotypical image and then be surprised it generates a stereotypical image. If you give it a simplistic prompt it will come up with a simplistic response.
So the LLM answers what’s relevant according to stereotypes instead of what’s relevant… in reality?
It just means there’s a bias in the data that is probably being amplified during training.
It answers what’s relevant according to its training.
Please remember what the A in AI stands for.
.
How traditional? How statistically relevant is it? Most Indians i know do not wear turbans at all.
If these stats are trustworthy (and i think they are), the only Indians that wear turbans are Sikhs (1.7%) and Muslims (14.2%). I’d say 15.9% is not statistically significant.
You don't think nearly 1/6th is statistically significant? What's the lower bound on significance as you see things?
To be clear, it's obviously dumb for their generative system to be overrepresenting turbans like this, although it's likely to be a bias in the inputs rather than something the system came up with itself, I just think that 5% is generally enough to be considered significant and calling three times that not significant confuses me.
For statistics’ sake? Yes.
For the LLM bias? No.
.
5/6 not wearing them seems more statistically significant
The fact less people of that group actually wear it than do is significant when you want an average sample. When categorizing a collection of images then, naturally, the traditional garments of a group is associated more with that group than any other group: 1/6 is bigger than any other race.
so if there was a country where 1 in 6 people had blue skin you would consider that insignificant because 5 out of 6 didn’t?
For a caricature of the population? Yes, that’s not what the algorithm should be optimising for.
.
.
Point taken.
Ain’t to me, couldn’t care less. I was just trying to point out that most Indians do not seem to wear turbans (and based my reasoning on the religions dress alone).
.
I think the idea is that it’s what makes a person “an Indian” and not something else.
Only a minority of Indians wear turbans, but more Indians than other people wear turbans. So if someone’s wearing a turban, then that person is probably Indian.
I’m not saying that’s true necessarily (though it may be), but that’s how the AI interprets it…or how pictures get tagged.
It’s like with Canadians and maple leaves. Most Canadians aren’t wearing maple leaf stuff, but I wouldn’t be surprised if an AI added maple leaves to an image of “a Canadian”.
.
Except if they trained it on something that has a large proportion of turban wearers. It is only as good as the data fed to it, so if there was a bias, it’ll show the bias. Yet another reason this really isn’t “AI”
I think you’re looking at it wrong. The prompt is to make an image of someone who is recognizable as Indian. The turban is indicative clothing of that heritage and therefore will cause the subject to be more recognizable as Indian to someone else. The current rate at which Indian people wear turbans isn’t necessarily the correct statistic to look at.
What do you picture when you think, a guy from Texas? Are they wearing a hat? What kind? What percentage of Texans actually wear that specific hat that you might be thinking of?
A surprising number of Texans wear cowboy and trucker hats (both stereotypical). A surprising number of Indians don’t wear turbans since it’s by far a minority.
Woosh
Imagine a German man from Bavaria… You just thought of a man wearing Lederhosen and holding a beer, didn’t you? Would you be surprised if I told you that they usually don’t look like that outside of a festival?
I don’t picture real life people as though they were caricatures.
But AI does, because we feed it caricatures.
Are you literally the second coming of Jesus? Hey everybody! I found a guy who doesn’t see race! I can’t believe it but he doesn’t think anyone is changed in any way by the place that they grew up in or their culture! Everyone is a blank slate to this guy! It’s amazing!
No, I just don’t lob groups of people together. It’s not that hard to do, everyone’s a different person.
He was imaginary though.
What’s the data that the model is being fed? What percentage of imaging featuring Indian men are tagged as such? What percentage of imaging featuring men wearing Turbans are tagged as Indian Men? Are there any images featuring Pakistan men wearing Turbans? Even if only a minority of Indian feature Turbans, if that’s the only distinction between Indian and Pakistan men in the model data, the model will favor Turbans for Indian Men. That’s just a hypothetical explanation.
By that logic americans should always be depicted in cowboy hats.
I see you’ve watched anime featuring Americans.
.
Put in western or Texas and that’s what you get, the west is a huge area even just of America but the word is linked to a lot of movie tropes and stuff so that’s what you get.
This is also only when the language is English, ask in urdu or Bengali and you get totally different results, in fact just use urdu instead of Indian and get less turbans or put in Punjabi and you’ll get more turbuns.
Or just put turban in the negatives if you want
Ask an AI for pictures of Texans and see how many cowboy hats it gives back to you.
It’s a traditional outfit of sikhs, not indians. Pick up a book
Pre-independence, most Indian males had some sort of headgear. E.g. look at any old photos of Bombay
Are we still in “pre independence”?
Oh… didn’t know that traditional meant what you wore yesterday
.
You’re wrong.
Whenever I try, I get Ravi Bhatia screaming "How can she slap?!"
This isn’t what I call news.
But is it what you call technology?
I call everything technology
<img alt="1000002349" src="https://lemmy.zip/pictrs/image/0558ac22-d120-4880-ba6e-915ba05eab05.webp">
I’m not sure how AI could be possibility racist. (Image is of a supposed Native American but my point still stands)
What point is that?
Stereotypes are everywhere
the AI itself can’t be racist but it will propagate biases contained in its training data
It definitely can be racist it just can’t be held responsible as it just regurgitates.
No, racism requires intent, AI aren’t (yet) capable of intent.
that requires reasoning, no AI that currently exist can do that
Meanwhile on DALL-E…<img alt="DALL-E drawing a girl in 18th century clothing, standing in front of a stereo typical (by by no means average) house." src="https://lemmy.my-box.dev/pictrs/image/5f749d0f-70d0-458c-bdf8-fa274ebceb65.jpeg">
I’m just surprised there’s no windmill in either of them. Canals, bikes, tulips… Check check check.
Careful, the next generated image is gonna contain a windmill with clogs for blades
Well, they do run on air…
<img alt="windmills do not work that way!" src="https://lemmy.my-box.dev/pictrs/image/4e3bde4f-928f-45fc-94a5-7705b69d96ba.webm">
That looks just like the town Delft
Kinda makes sense though. I’d expect images where it’s actually labelled as “an Indian person” to actually over represent people wearing this kind of clothing. An image of an Indian person doing something mundane in more generic clothing is probably more often than not going to be labelled as “a person doing X” rather than “An Indian person doing X”. Not sure why these authors are so surprised by this
Yeah I bet if you searched desi it wouldn’t have this problem
Like most of my work’s processes… Shit goes in, shit comes out…
Would they be equally surprised to see a majority of subjects in baggy jeans with chain wallets if they prompted it to generate an image of a teen in the early 2000’s? 🤨
That’s Sikh
Indians can be Sikh, not all indians are Hindu
Yes but the gentlemen in the images are also sikhs
glances at the current policies of the Indian government
Well… for now, anyway.
Sikh pun
Not necessarily. Hindus and Muslims in India also wear them.
Wouldn’t surprise me if most of them are depicted shitting on the street
Overfitting
It happens
Get down with the Sikhness
not me calling in sikh to work
Articles like this kill me because the nudge it’s kinda sorta racist to draw images like the ones they show which look exactly like the cover of half the bollywood movies ever made.
Yes, if you want to get a certain type of person in your image you need to choose descriptive words, imagine gong to an artist snd saying ‘I need s picture and almost nothing matters beside the fact the look indian’ unless they’re bad at their job they’ll give you a bollywood movie cover with a guy from rajistan in a turbin - just like their official tourist website does
Ask for an business man in delhi or an urdu shop keeper with an Elvis quiff if that’s what you want.
Almost certainly how they’re building up the data. But that’s more a consequence of tagging. Same reason you’ll get Marvel’s Iron Man when you ask an AI generator for “Draw me an iron man”. Not as though there’s a shortage of metallic-looking people in commercial media, but by keyword (and thanks to aggressive trademark enforcement) those terms are going to pull back a superabundance of a single common image.
I mean, the first thing that pops into my head is Mahatma Gandhi, and he wasn’t typically in a turbine. But he’s going to be tagged as “Gandhi” not “Indian”. You’re also very unlikely to get a young Gandhi, as there are far more pictures of him later in life.
I remember when Google got into a whole bunch of trouble by deliberately engineering their prompts to be race blind. And, consequently, you could ask for “Picture of the Founding Fathers” or “Picture of Vikings” and get a variety of skin tones back.
So I don’t think this is foolproof either. Its more just how the engine generating the image is tuned. You could very easily get a bunch of English bankers when querying for “Business man in delhi”, depending on where and how the backlog of images are sources. And urdu shopkeeper will inevitably give you a bunch of convenience stores and open-air stalls in the background of every shot.
.
its the “skin cancer is where there’s a ruler” phenomena.
I don’t get it.
I’m guessing this relates to training data. Most training data that contains skin cancer is probably coming from medical sources and would have a ruler measuring the size of the melanoma, etc. So if you ask it to generate an image it’s almost always going to contain a ruler. Depending on the training data I could see generating the opposite as well, ask for a ruler and it includes skin cancer.
Ooooohhhh nice explanation!
Turbans are cool and distinct.