Bookshelves
Bookshelves Wiki
Installation and usage
Last update: 2021-07-26

About

This Wiki is about Bookshelves project, you will find two parts covered here: the back-end part made in Laravel which is clearly the most important part in Bookshelves and the front-end part in NuxtJS which retrieves data from the API in order to display it in a nice user interface.

If you are interested in Bookshelves, you can keep only the back-end part and create your own front-end with the technology you want. All the logic of Bookshelves is in the backend and it is even possible to not use an external frontend and use Bookshelves with the internal backend interface.

Concept

The goal of Bookshelves is to create a database from a list of eBooks that are analyzed by the back-end in Laravel (PHP). All the metadata of each eBook are extracted and processed to create each book with its relationships: authors, publisher, year of release, language, identifiers (ISBN, ISBN13), tags, summary, cover. The series and volume are also retrieved if they are present according to the system created by Calibre, this information not being listed by the EPUB format. The tags are separated into tags and genres to create main and secondary tags, the genres are taken from Wikipedia.

From these relations, all the books of the same series can be listed but also the books and series close by tag. It is interesting to note that a book can have several authors even if only one "main" author is considered for URL generation. Of course, with such a system if the eBooks have a single error on important data, this can have unexpected consequences.

Example "Language": let's take the language of an English eBook that is mistakenly indicated as French and that is part of a "D'Artagan Romances" series, then this eBook will be part of its own specific series "D'Artagan Romances" indicated as French while the others will be part of the "D'Artagan Romances" series indicated as English. This allows to have two series with the same name in several languages but requires a precise work in setting up the metadata with Calibre for example.

  • The Three Musketeers from Alexandre Dumas in french: /dumas-alexandre/d-artagnan-romances-fr
  • Twenty Years After from Alexandre Dumas in english: /alexandre-dumas/d-artagnan-romances-fr

Example "Author": an eBook "The Three Musketeers" has a lastname-firstname author name such as "Alexandre Dumas" while another eBook of the same series "Twenty Years After" has a lastname-firstname author name such as "Dumas Alexandre". During the generation, two different authors will be created and thus two different series, so all the eBooks must have the name of each author indicated in the same way, the proposal of Bookshelves is to prefer firstname-lastname. It's important because with this order, Bookshelves can seperate firstname and lastname to order authors. You can reverse firstname-lastname into config/bookshelves.php

  • The Three Musketeers from Alexandre Dumas: /dumas-alexandre/three-musketeers-en from serie /dumas-alexandre/d-artagnan-romances-en
  • Twenty Years After from Dumas Alexandre: /alexandre-dumas/twenty-years-after from serie /alexandre-dumas/d-artagnan-romances-en

The back-end can work without the front-end by using only Catalog as an interface

  • Generation of eBooks with additional data generation with GoogleBooks and Wikipedia API
    • eBooks : number of pages
    • Authors: description, URL link, photo
    • Series: description, link
    • Each information can be overridden with JSON files presenting the data or JPG files for the pictures
  • Generation of an API with documentation
  • Wiki with a documentation for installation and use
  • An OPDS (Open Publication Distribution System) feed
  • An interface called Catalog presenting the data in a very simplified way in order to allow access from the browser of an eReader and thus to download eBooks with an integrated search
  • An eBook reader in the browser, WebReader, in order to consult directly an eBook

The front-end offers a more modern interface by providing

  • More data for each entry
  • An advanced search
  • A pagination on the data in collection
  • The download of eBooks but also of all the books of a specific author or series as a ZIP file.
  • A commenting and bookmarking system for logged-in users.
  • Retrieval of additional data: publishers, tags, genres and associated eBooks & series
  • Guides to inform users on how to use an eReader or eBooks
  • Dark mode
  • Contact form

What Bookshelves is not

Bookshelves is not like Calibre with dynamic database from EPUB into a specific directory, with Bookshelves you have to parse all EPUB in a directory and, if you add another, you have to parse for new EPUB (fastly than first parse) to generate new eBooks for database. You can use this application like Calibre but it's not same. If you want a Calibre app for web, check these projects:

Links

🚀 bookshelves.ink: demo of Bookshelves

📚 bookshelves.ink/wiki: wiki for Bookshelves usage
📚 bookshelves.ink/docs: API documentation
📚 bookshelves.ink/opds: OPDS feed for applications which can read this feed
📚 bookshelves.ink/catalog: Catalog, a basic interface for eReader browser to download eBook from eReader
📚 bookshelves.ink/webreader: Webreader, to read any Bookshelves eBook into your browser

📀 gitlab.com/ewilan-riviere/bookshelves-back : back-end of Bookshelves
🎨 gitlab.com/ewilan-riviere/bookshelves-front : front-end of Bookshelves

I. Setup

a. Dependencies

Extensions for PHP, here for php8.0

sudo apt-get install -y php8.0-xml php8.0-gd

For spatie image optimize tools

sudo apt-get install -y jpegoptim optipng pngquant gifsicle webp
npm install -g svgo

You need this to use Bookshelves

  • PHP v8.0
  • Composer v2.0
  • MySQL v8.0
  • NodeJS v14.16
  • Yarn v1.2

b. Setup

Download dependencies

composer install

Setup command (easy way)

Execute setup and follow guide

php artisan setup

Manual

Create .env

cp .env.example .env

Generate key

php artisan key:generate

Set database informations, you can set more data if you want, check II. .env

DB_DATABASE=<database_name>
DB_USERNAME=<database_user>
DB_PASSWORD=<database_password>

Download NodeJS dependencies

yarn

Execute Laravel mix

yarn dev

Generation API documentation

php artisan scribe:generate

Migrate database

php artisan migrate

II. .env

a. For local

APP_URL=http://localhost:8000

SANCTUM_STATEFUL_DOMAINS=localhost:3000
SESSION_DOMAIN=localhost

TELESCOPE_ENABLED=true

BOOKSHELVES_ADMIN_EMAIL=admin@mail.com
BOOKSHELVES_ADMIN_PASSWORD=password

Setup for Mailtrap

MAIL_HOST=smtp.mailtrap.io
MAIL_PORT=587
MAIL_USERNAME=mailtrap_username
MAIL_PASSWORD=mailtrap_password
MAIL_FROM_ADDRESS=noreply@bookshelves.ink
MAIL_FROM_NAME="${APP_NAME}"
MAIL_TO_ADDRESS=contact@bookshelves.ink
MAIL_TO_NAME="${APP_NAME}"

b. For production

APP_URL=https://www.mydomain.com

SANCTUM_STATEFUL_DOMAINS=www.mydomain.com
SESSION_DOMAIN=.mydomain.com

TELESCOPE_ENABLED=false

Setup for Mailgun

For credentials

  • Create an account
  • After setup domain
  • Sending -> Domain settings -> SMTP credentials
MAIL_HOST=smtp.eu.mailgun.org
MAIL_PORT=587
MAIL_USERNAME=mailgun_user_login
MAIL_PASSWORD=mailgun_user_password
MAIL_FROM_ADDRESS=noreply@bookshelves.ink
MAIL_FROM_NAME="${APP_NAME}"
MAIL_TO_ADDRESS=contact@bookshelves.ink
MAIL_TO_NAME="${APP_NAME}"

III. Generate eBooks data

a. Add your own eBooks

Add EPUB files in public/storage/raw/books and execute Epub Parser

php artisan bookshelves:generate -h to check options

# for fresh installation (erase current database) with force option for production
php artisan bookshelves:generate -fF

b. Test with demo eBook

If you want to test Bookshelves, you can use bookshelves:sample to generate data from libre eBooks

php artisan bookshelves:sample -h to check options

php artisan bookshelves:sample

e. Mails

TODO

g. MetadataExtractor

TODO

i. Wikipedia

TODO

  • WikipediaProvider

IV. Packages

a. Auth

laravel/sanctum: Laravel Sanctum provides a featherweight authentication system for SPAs (single page applications), mobile applications, and simple, token based APIs. Sanctum allows each user of your application to generate multiple API tokens for their account. These tokens may be granted abilities / scopes which specify which actions the tokens are allowed to perform.

Routes can be protected like this

Route::middleware(['auth:sanctum'])->group(function () {
  // routes
}

Login 419 error: "CSRF token mismatch"

php artisan cache:clear
php artisan route:clear
php artisan config:clear
php artisan view:clear
php artisan optimize:clear

b. API documentation

knuckleswtf/scribe: Scribe helps you generate API documentation for humans from your Laravel/Lumen/Dingo codebase. See a live example at demo.scribe.knuckles.wtf.

If you use php artisan serve, knuckleswtf/scribe is available on /docs, like localhost:8000/docs. You can set parameters into each Controller, check scribe.knuckles.wtf/laravel to know more, like this

<?php

/**
 * @group Author
 *
 * Endpoint to get Authors data.
 */
class AuthorController extends Controller
{
  /**
   * GET Author collection
   *
   * <small class="badge badge-blue">WITH PAGINATION</small>
   *
   * You can get all Authors with alphabetic order on lastname with pagination.
   *
   * @queryParam per-page int Entities per page, '32' by default. No-example
   * @queryParam page int The page number, '1' by default. No-example
   * @responseFile public/storage/responses/authors.index.get.json
   */
  public function index(Request $request)
  {
    // ...
  }
}

And generate documentation

php artisan scribe:generate

c. Code linter

IDE helper

barryvdh/laravel-ide-helper: to generate magic methods for each model to help IDE completion

composer helper

larastan

nunomaduro/larastan: Adds static analysis to Laravel improving developer productivity and code quality.

php artisan larastan

PHP CS Fixer

friendsofphp/php-cs-fixer: A tool to automatically fix PHP Coding Standards issues

composer helper

d. Database and models

  • fakerphp/faker: Faker is a PHP library that generates fake data for you

Enum

spatie/laravel-enum: Laravel support for spatie/enum

Media library

spatie/laravel-medialibrary: Associate files with Eloquent models. If you update registerMediaConversions() in any Model, you can regenerate conversions

php artisan media-library:regenerate

Tags

spatie/laravel-tags: Add tags and taggable behaviour to your Laravel app

e. Tools

Clockwork

itsgoingd/clockwork: Clockwork is a development tool for PHP available right in your browser. Clockwork gives you an insight into your application runtime - including request data, performance metrics, log entries, database queries, cache queries, redis commands, dispatched events, queued jobs, rendered views and more - for HTTP requests, commands, queue jobs and tests.

To use Clockwork, you have to install browser extension: Chrome or Firefox. When it's done, just open DevTools and choose Clockwork.

CORS

fruitcake/laravel-cors: Adds CORS (Cross-Origin Resource Sharing) headers support in your Laravel application

Images

Markdown

XML

spatie/array-to-xml: A simple class to convert an array to xml

Routing

spatie/laravel-route-attributes: Use PHP 8 attributes to register routes in a Laravel app

Telescope

Only available on routes/web.php

  • laravel/telescope: Telescope makes a wonderful companion to your local Laravel development environment. Telescope provides insight into the requests coming into your application, exceptions, log entries, database queries, queued jobs, mail, notifications, cache operations, scheduled tasks, variable dumps, and more.
  • fruitcake/laravel-telescope-toolbar: A toolbar for Laravel Telescope, based on the Symfony Web Profiler.

To enable Telescope, just change variable in .env

TELESCOPE_ENABLED=true

Note: only useful in local

You can use laravel/telescope on Bookshelves at http://localhost:8000/telescope if you serve project with php artisan serve (adapt URL if you have VHost).

f. Tests

php artisan migrate --database=testing

You can run Pest and PHP Unit tests

php artisan pest:run

V. Deployment

a. VHost: NGINX

server {
  server_name bookshelves.ink;

  error_page 500 502 503 504 /index.html;
  location = /index.html {
    root /usr/share/nginx/html;
    internal;
  }

  location / {
    include proxy_params;
    proxy_pass http://localhost:3004;
  }

  location ~ ^/(admin|api|css|media|uploads|storage|docs|packages|cache|sanctum|login|logout) {
    include proxy_params;
    proxy_pass http://127.0.0.1:8000;
    proxy_buffer_size 128k;
    proxy_buffers 4 256k;
    proxy_busy_buffers_size 256k;
  }
}
server {
  listen 8000;
  listen [::]:8000;

  server_name bookshelves.ink;

  error_log /var/log/nginx/bookshelves.log warn;
  access_log /var/log/nginx/bookshelves.log;

  root /home/ewilan/www/bookshelves-back/public;
  index index.php;

  add_header X-Frame-Options "SAMEORIGIN";
  add_header X-XSS-Protection "1; mode=block";
  add_header X-Content-Type-Options "nosniff";

  charset utf-8;

  location / {
    try_files $uri $uri/ /index.php?$query_string;
  }

  error_page 404 /index.php;

  location ~ ^/cache/resolve {
    try_files $uri $uri/ /index.php?$query_string;
  }

  location ~ ^/docs/ {
    try_files $uri $uri/ /index.php?$query_string;
  }

  location ~* \.(js|css|png|jpg|jpeg|gif|ico|eot|svg|ttf|woff|woff2)$ {
    expires max;
    log_not_found off;
  }

  location ~ \.php$ {
    include snippets/fastcgi-php.conf;
    fastcgi_pass unix:/run/php/php8.0-fpm.sock;
    fastcgi_buffer_size 128k;
    fastcgi_buffers 4 256k;
    fastcgi_busy_buffers_size 256k;
  }

  location ~ /\.(?!well-known).* {
    deny all;
  }
}