ccunning@lemmy.world to News@lemmy.world · 26 天前Pay-per-output? AI firms blindsided by beefed up robots.txt instructions.arstechnica.comexternal-linkmessage-square8linkfedilinkarrow-up126cross-posted to: [email protected]
arrow-up126external-linkPay-per-output? AI firms blindsided by beefed up robots.txt instructions.arstechnica.comccunning@lemmy.world to News@lemmy.world · 26 天前message-square8linkfedilinkcross-posted to: [email protected]
minus-squareTheMightyCat@ani.sociallinkfedilinkarrow-up2·25 天前If there is a way to combine this with something like anubis this could be very interesting, then you are also not dependent on the honesty of reading the robots.txt Have a RSL license? here you go scrape the content. Trying to freeload? anubis time.
If there is a way to combine this with something like anubis this could be very interesting, then you are also not dependent on the honesty of reading the robots.txt
Have a RSL license? here you go scrape the content.
Trying to freeload? anubis time.