Ayhem@lemmy.world to Artificial Intelligence@lemmy.worldEnglish · 2 years agoWebsites blocking OpenAIlemmy.worldimagemessage-square14fedilinkarrow-up171arrow-down14file-text
arrow-up167arrow-down1imageWebsites blocking OpenAIlemmy.worldAyhem@lemmy.world to Artificial Intelligence@lemmy.worldEnglish · 2 years agomessage-square14fedilinkfile-text
minus-squaresenoro@lemmy.mllinkfedilinkEnglisharrow-up6·2 years agoCompanies do honour robots.txt. Maybe not the small project made by some guy somewhere. But large companies do.
Companies do honour robots.txt. Maybe not the small project made by some guy somewhere. But large companies do.