VK instructed the robot from OpenAI not to collect data from Zen – Kommersant
[ad_1]
Content platform “Zen” (owned by VK; MOEX: VKCO) indicated that the GPTBot robot from OpenAI (developer of ChatGPT and GPT-4) is prohibited from bypassing the pages of the dzen.ru website to collect data. Kommersant discovered a corresponding directive in the robots.txt file, intended for programs for automatically collecting information from websites. The VK press service explained to Kommersant that they blocked GPTBot to reduce the load on Zen servers.
“The Zena recommendation system is one of the largest in the country. Highly loaded services operate continuously and process more than 150 thousand requests per second. The decision not to include GPTBot from OpenAI in the file was made for the proper use of a technical resource, so as not to create additional load. “Zen regularly creates millions of new publications: both in text and in video format – we direct resources to provide a quality experience for our users and authors,” VK said.
The robots.txt file is advisory in nature: technically, nothing prohibits robots from ignoring the directives written in it. In 2019, the Internet Archive project, which creates archival copies of websites, announced that it would not follow the settings from the file, but by 2022 it reversed the decision.
GPTBot is used to collect information that is subsequently used to create OpenAI neural network products. The company does not provide access to them in Russia and has blocked access to the site for users from the Russian Federation. There are similar “crawler” robots in Yandex and Google, with the help of which search engines maintain up-to-date data about sites. Both companies are also developing their neural network products. At the same time, they allow site administrators to refuse service to robots created for neural networks (YandexAdditional and Google-Extended, respectively), but at the same time not close the site from information collectors for search results. In December 2023, the Main Radio Frequency Center, subordinate to Roskomnadzor, recommended that hosting providers block GPTBot requests.
Read more in the Kommersant article. “And he leaves her for a bot”.
[ad_2]
Source link
