Anubis (software)
Software designed to prevent scraping
From Wikipedia, the free encyclopedia
Anubis is an open source software program that adds a proof of work challenge to websites before users can access them in order to deter web scraping. It has been adopted mainly by Git forges and free and open-source software projects.[4][5]
.png)
| Anubis | |
|---|---|
The page shown by Anubis during its challenge. | |
| Original author | Xe Iaso |
| Developer | Techaro[1] |
| Initial release | January 19, 2025[1][2] |
| Stable release | |
| Written in | Go, JavaScript[1] |
| License | MIT License |
| Website | anubis |
| Repository | github |
Anubis was created by Xe Iaso in response to Amazon's web crawler overloading their Git server, as the crawler did not respect the robots.txt exclusion protocol and would work around restrictions.[4][6] Iaso lists Hashcash as having inspired the project.[7] The application supports inspecting request elements such as headers like the User-Agent header to determine if the request should require proof of work.
The name Anubis is taken from the Ancient Egyptian god of funerals and judgement, who weighs the hearts of the dead to determine if they are allowed passage into the afterlife, whereas the Anubis software "weighs the soul of incoming HTTP requests".[8]
Design
Anubis temporarily blocks access to websites until the client completes a cryptographic challenge.[9] This challenge is intended to be a design hurdle and incur a compute cost for web crawlers while minimizing impact on typical visitors (see § Criticism). Challenge solutions remain valid for a tunable period of time using EdDSA-signed tokens, which may be stored as HTTP cookies.[1][10]
The challenge itself is similar to the proof of work algorithms implemented by Hashcash and Bitcoin. Anubis presents a challenge to the client in the form of a random number, to which the client must respond with another number (the nonce) such that when the two numbers are concatenated, the SHA-256 hash of both numbers contains a pre-specified number of leading zeros.[9] SHA-256 is chosen specifically because of its (assumed) pre-image resistance (ie. finding such a nonce is computationally expensive) while simultaneously being cheap to verify. Moreover, tuning the number of required leading zeros controls the difficulty of the challenge: a typical browser can find a solution with four leading zeros (the default) in seconds, while one with six can take several minutes.[1]
By default, Anubis challenges any client claiming to be a browser, by checking if the User-Agent header contains "Mozilla", unless the client is requesting "low-harm" content (eg. robots.txt, the well-known URI).[1][10] Additionally, Anubis exempts Common Crawl "so [other] scrapers have less incentive to scrape".[1]
Although Anubis could be altered to mine cryptocurrency to serve as proof of work, Iaso has rejected this idea: "I don't want to touch cryptocurrency with a 20 foot pole. I realize I'm leaving money on the table by doing this, but I don't want to alienate the kinds of communities I want to protect."[7]
Mascot
The software's loading screen is branded with a commissioned artwork of Anubis as a jackal-eared anime girl by the European artist CELPHASE.[1][8] The mascot is depicted with a hoodie, skirt and magnifying glass. Before the artwork was ordered, Anubis used an AI-generated placeholder image.[1]
The Anubis mascot is shown to all end users and cannot be altered in the software configuration.[1] The image's feel may clash with websites that have more formal atmospheres, surprising or confusing users of those sites.[8] Altering the branding is an enterprise feature and Iaso has requested that operators not attempt to change it themselves unless they have made financial contributions to the project.[1]
Duke University, which has deployed Anubis for its digital archives, was "hesitant" to use it due to the mascot but has reached an agreement to use the software with custom branding.[1] Jamie Zawinski describes the mascot as "cutesey kawaii bullshit".[7]
Adoption
It has been used by a number of projects, including:[11]
Criticism
Tavis Ormandy wrote that he believes that the cost of scraping websites protected by Anubis is negligible: "I don’t think we reach a single cent per month in compute costs until several million sites have deployed Anubis."[16]
