Anubis (software)
Software designed to prevent scraping
From Wikipedia, the free encyclopedia
Anubis is an open source software program that adds a proof of work challenge to websites before users can access them in order to deter web scraping. It has been adopted mainly by Git forges and free and open-source software projects.[4][5]
.png)
| Anubis | |
|---|---|
The page shown by Anubis during its challenge. | |
| Original author | Xe Iaso |
| Developer | Techaro[1] |
| Initial release | January 19, 2025[1][2] |
| Stable release | |
| Written in | Go, JavaScript[1] |
| License | MIT License |
| Website | anubis |
| Repository | github |
Anubis was created by Xe Iaso in response to Amazon's web crawler overloading their Git server, as the crawler did not respect the robots.txt exclusion protocol and would work around restrictions.[4][6] Iaso lists Hashcash as having inspired the project.[7] The application supports inspecting request elements such as headers like the User-Agent header to determine if the request should require proof of work.
The name Anubis is taken from the Ancient Egyptian god of funerals and judgement, who weighs the hearts of the dead to determine if they are allowed passage into the afterlife, whereas the Anubis software "weighs the soul of incoming HTTP requests".[8]
Design
Anubis temporarily blocks access to websites until the client completes a cryptographic challenge.[9] This challenge is intended to be a design hurdle and incur a compute cost for web crawlers while minimizing impact on typical visitors (see § Technical criticism). Challenge solutions remain valid for a tunable period of time using EdDSA-signed tokens, which may be stored as HTTP cookies.[1][10]
The challenge itself is similar to the proof of work algorithms implemented by Hashcash and Bitcoin. Anubis presents a challenge to the client in the form of a random number, to which the client must respond with another number (the nonce) such that when the two numbers are concatenated, the SHA-256 hash of both numbers contains a pre-specified number of leading zeros.[9] SHA-256 is chosen specifically because of its (assumed) pre-image resistance (i.e. finding such a nonce is computationally expensive) while simultaneously being cheap to verify. Moreover, tuning the number of required leading zeros controls the difficulty of the challenge: a typical browser can find a solution with five leading zeros (the default) in seconds, while one with six can take several minutes.[1]
By default, Anubis challenges any client claiming to be a browser, by checking if the User-Agent header contains "Mozilla", unless the client is requesting "low-harm" content, e.g. robots.txt or the well-known URI.[1][10] Additionally, Anubis exempts Common Crawl "so [other] scrapers have less incentive to scrape".[1]
Although Anubis could be altered to mine cryptocurrency to serve as proof of work, Iaso has rejected this idea: "I don't want to touch cryptocurrency with a 20 foot pole. I realize I'm leaving money on the table by doing this, but I don't want to alienate the kinds of communities I want to protect."[7]
Mascot
The software's loading screen is branded with a commissioned artwork of Anubis as a jackal-eared anime girl by the European artist CELPHASE.[1][8] The mascot is depicted with a hoodie, skirt and magnifying glass. Before the artwork was ordered, Anubis used an AI-generated placeholder image.[1]
The Anubis mascot is shown to all end users and cannot be altered in the software configuration.[1] The image's feel may clash with websites that have more formal atmospheres, surprising or confusing users of those sites.[8] Altering the branding is an enterprise feature and Iaso has requested that operators not attempt to change it themselves unless they have made financial contributions to the project.[1]
Duke University, which has deployed Anubis for its digital archives, was "hesitant" to use it due to the mascot but has reached an agreement to use the software with custom branding.[1]
Adoption
It has been used by a number of projects, including:[11]
Technical criticism
Tavis Ormandy estimated that the cost of scraping websites protected by Anubis is negligible: "In fact, I don’t think we reach a single cent per month in compute costs until several million sites have deployed Anubis. I’m just not convinced this math works… this is literally nothing for a soulless AI vendor with a monthly cloud services budget in the 8 figures."[17] Ormandy demonstrates that it is technically easy to develop a native code solver, which can solve Anubis's proof-of-work challenges more quickly and efficiently than the JavaScript code that ordinary human users must run on their devices.[17]
By contrast, Anubis imposes substantial costs on human users attempting to access the sites it protects, particularly those with older devices,[17] on which Anubis challenges may be extremely slow and battery-depleting. LWN decided not to deploy Anubis itself in part because Anubis will completely "block users who turn off JavaScript" in their browsers.[1]
LWN noted in July 2025 that "there is no guarantee that Anubis will maintain its edge against scrapers in the long run,"[1] and indeed one month later Codeberg reported that many AI scraper bots had "learned how to solve the Anubis challenges"; at the same time Codeberg praised Anubis for having offered a reliable and practical anti-bot solution for many months.[18]
