Shadow library

Shadow libraries, or pirate libraries, are online repositories of freely available digital media that are normally paywalled, access-controlled, or otherwise not readily accessible.^[1][2] Shadow libraries usually contain textual works like academic papers and ebooks, and may include other digital media like software, music, or films.

Anna's Archive, Library Genesis, Sci-Hub, and Z-Library are some of the most popular shadow libraries for books and academic literature.^[1][3]

History

Growth of Library Genesis, 2009–2022

Early predecessors to shadow libraries were informal collections of unauthorized digital copies of books, scholarly literature, and other textual media, often shared with small groups via mailing lists, forums, or social media websites.^[1]: 1 Online communities of scientists also collaborated to share paywalled literature among themselves.^[4]

Russian samizdat and photo negatives of unofficial literature

Many shadow libraries originate in Russia, which has a rich history of samizdat stemming from the Soviet era. There was strict state censorship and control of print materials, which gave rise to the dissident activity of copying and disseminating censored or underground works. Even after the dissolution of the Soviet Union and the end of the official censorship program, these sharing practices continued as a result of widespread economic hardship.^{[1]: 31–33} Texts were widely digitized and shared on Russian FidoNet systems as computer and internet access became more widespread in Russia. One early collection of digitized texts was Maksim Moshkow's 1994 Lib.ru.^{[1]: 34–35} The Russian Kolkhoz collection, named for the kolkhoz collective farms, was created by a community that worked in the early 2000s to download or digitize scientific texts, which they stored on FTP servers and DVDs. This collection eventually grew to around 50,000 documents.^[1]: 37

Some of these early collections later became shadow libraries as they attracted volunteer librarians who catalogued the archives' contents. Early academic shadow libraries in the 2000s included Textz.org, monoskop, and Gigapedia (later Library.nu). Gigapedia focused more on academic texts than other shadow libraries, which mainly contained literature.^{[1]: 26–27} Around 2006 or 2007, it incorporated the files amassed by the Kolkhoz collectors,^[1]: 37 and had become the largest shadow library by 2010.^{[1]: 26–27} Gigapedia, by then renamed to Library.nu, was shut down in 2012 through a lawsuit from a coalition of seventeen publishing companies including HarperCollins, Oxford University Press, and MacMillan.^{[1]: 26–27 [5]}

Library Genesis (also known as LibGen) was founded in approximately 2007 or 2008 by a group of Russian scientists, who began by organizing a collection of Russian science and technology texts made available on a torrent site, aggregated from sources including the Kolkhoz collection and lib.ru.^{[1]: 27–28, 38} In 2011, LibGen absorbed the Library.nu collection, keeping it accessible even as Library.nu was forced to shut down. At the time, LibGen was unique in its focus on its open library infrastructure, prioritizing the free sharing of its collection, catalog, and source code to encourage many others to increase shadow libraries' collective resiliency by mirroring and forking the project.^{[1]: 27–28}

Motivation

See also: Open access and Serials crisis

Shadow libraries are part of the open access and open knowledge movements.^{[1]: 6 [6]} They seek to more freely disseminate academic scholarship and other media, often citing a moral imperative to make knowledge freely available.^[2]

LibGen's operators have described the site's mission as enabling access to information for poor people and opposing the gating of knowledge by elite academic institutions, with one administrator writing "the target groups for LibGen are poors: Africa, India, Pakistan, Iran, Iraq, China, Russia and post-USSR etc., and on a separate note, people who do not belong to academia. If you are not at a university, you can't access anything or at least your access will be so much troubled that you won't be able to progress at all."^[1]: 28 Alexandra Elbakyan, the creator of Sci-Hub, has justified the site by arguing that the lack of open access to scholarship violates the human right to science and culture, captured in Article 27 of the United Nations Universal Declaration of Human Rights, which states: "Everyone has the right freely to participate in the cultural life of the community, to enjoy the arts and to share in scientific advancement and its benefits."^[7] Elbakyan has also argued that "Any law against knowledge is fundamentally unjust".^[8] American activist Aaron Swartz captured the motivations of many shadow libraries in his 2008 Guerilla Open Access Manifesto,^{[1]: 28–29} writing:

The world's entire scientific and cultural heritage, published over centuries in books and journals, is increasingly being digitized and locked up by a handful of private corporations. ... Those with access to these resources—students, librarians, scientists—you have been given a privilege. You get to feed at this banquet of knowledge while the rest of the world is locked out. But you need not—indeed, morally, you cannot—keep this privilege for yourselves.

— Aaron Swartz, Guerilla Open Access Manifesto^[9]

Shadow libraries have also cited the increasing cost of academic literature and books, also termed the "serials crisis".^[10]

Technologies

Some shadow libraries (or their content databases) make use of BitTorrent (mainly for database dumps), dark web, and InterPlanetary File System (IPFS) technologies to increase their resilience or distribute loads.^{[11][12][3][2][13]} Shadow libraries including LibGen and Anna's Archive develop and make their software accessible as open source software, enabling code development by any volunteer and encouraging mirrors or forks.^{[1]: 27–28 [14]} Anna's Archive claims that "if we get taken down we'll just pop right up elsewhere, since all our code and data is fully open source".^[14]

Legal status

Shadow libraries often host or link to copyrighted material without the consent of copyright holders, making them illegal or dubiously legal in many countries.^[1] Such libraries are also described as pirate libraries.^[8][1]: 4 Many shadow libraries maintain bibliographic catalogs separate from the hosting of files themselves. This is both an organizational convenience and a protection against legal challenges, since the law is often ambiguous on the distinction between hosting and indexing copyrighted content. However, several shadow library catalogs have been the target of injunctions and takedown threats.^{[1]: 25–26}

The aggressive legal strategies pursued by Western music and film industries against online filesharing websites during the 2000s were not widely mirrored by academic or literary publishers against shadow libraries. However, as shadow libraries have grown larger and more visible, they have attracted more legal challenges. Library.nu (previously Gigapedia) was shut down in 2012 by a lawsuit from a coalition of seventeen publishing companies including HarperCollins, Oxford University Press, and MacMillan.^{[1]: 26–27 [5]} In 2015, the academic publisher Elsevier sued LibGen and Sci-Hub in American courts, accusing them of "operat[ing] an international network of piracy and copyright infringement".^[15] Elsevier won a default judgment against the two groups, and was awarded $15 million in damages, but has not collected the money as LibGen's operators are unknown and Sci-Hub's are outside the reach of the US legal system.^[16] Although the judge in the Elsevier case granted an injunction against several domains used by the shadow libraries, briefly taking them offline, the libraries quickly moved to new domains and onion sites.^[17][15] A lawsuit by the American Chemical Society in 2017 against Sci-Hub also resulted in a judgment order for $4.8 million in damages.^[16] In November 2022, the FBI seized domains associated with Z-Library and charged two of its operators with criminal copyright infringement, wire fraud, and money laundering.^[18] Courts have ordered Internet service providers in countries including Denmark, France, Germany, Russia and the United Kingdom to block access to pirate libraries,^[19][20] although these blocks are of limited effectiveness.^[21]

The legality of directing individuals to shadow libraries is undetermined. While there are legal theories that linking to copyright infringing material hosted by shadow libraries could constitute vicarious or contributory copyright infringement, there have been no cases brought with these theories. In 2019, Elsevier threatened legal action against Citationsy, the developer of a bibliography management tool, for publishing a blog post directing readers to Sci-Hub and Citationsy removed the link.^[22]

Although most academics are not penalized for distributing their own published works for free, academic publishers have threatened scientists for sharing or republishing their work.^[23]

Some publishers have accused shadow libraries including Sci-Hub of illegally obtaining login credentials to academic databases, though Sci-Hub says the credentials are voluntarily donated.^[24]

A class action lawsuit filed in June 2023 against ChatGPT developer OpenAI, led by authors Paul Tremblay and Mona Awad, alleged that the company used shadow libraries to source training data for their large language model.^[25][26][27] Meta has also been alleged to have used data from from shadow libraries to train its AI model.^[28][29] DeepSeek's Vision-Language (VL) model was trained with data from the shadow library Anna's Archive.^[30]

Reception

By academics

Some academics have tacitly or explicitly endorsed shadow library efforts,^[1] with many viewing them as morally acceptable acts of civil disobedience against the abusive business models of academic publishers.^[31] Furthermore, shadow libraries may increase the impact of academics whose work is made available. According to one study from Cornell University, articles that are available on Sci-Hub receive 1.72 times as many citations as articles from journals of similar quality that are not available on Sci-Hub.^[32]

By non-academic authors

Non-academic writers have been more vocally opposed to shadow libraries.^[8]

In February 2022, after joining a lawsuit with Amazon Publishing and Penguin Random House against a Ukrainian website selling pirated e-books, American bestselling fiction authors John Grisham and Scott Turow published an op-ed in The Hill calling on US lawmakers to pass a law prohibiting search engines from linking to piracy websites.^[8][33]

In October 2022, the US-based Authors Guild submitted a complaint to the United States Trade Representative about LibGen and Z-Library, describing digital book piracy as "one of the biggest threats facing authors’ livelihoods today".^[34] The Authors Guild and the UK-based Publishers Association both worked with the FBI in efforts against Z-Library, which culminated with November 2022 the arrest of two of its operators.^[18] However, some authors and writers' organizations have opposed such efforts. British novelist Alison Rumfitt wrote in Dazed that she was not celebrating the site's takedown, and that "the hunger to read is something to be encouraged, something which, in my opinion, is a societal good; even as publishing grows ever more overtly capitalist and monopolised, reading still thrives, and piracy allows it to take place despite borders and Digital Rights Management. Not everyone has access to a library, and not every library in the world is well-stocked."^[35] Dave Hansen, executive director of the Authors Alliance nonprofit, expressed that students and researchers would be negatively impacted by attempts to shut down shadow libraries, and expressed that such projects were "a kind of symptom of how broken the system is, particularly when you’re looking at access to scientific articles".^[2]

See also

• Academic journal publishing reform
• #ICanHazPDF
• Intellectual property infringement
• Open Library