Usenet Indexer Installation - Debian / Ubuntu

System Vorbereitungen

1. Aktualisieren des Systems:

   # apt-get update
   # apt-get upgrade
   
   # apt-get dist-upgrade

2. Installation der benötigten Grundpakete:

   # apt-get install vim screen wget git apache2 apache2-utils curl mariadb-server mariadb-client php7.0-fpm php7.0-mysql php7.0-common php7.0-gd php7.0-json php7.0-cli php7.0-curl php7.0-mcrypt php-imagick php-pear php7.0-dev php7.0-mbstring php7.0-xml php7.0 gcc make curl

Konfiguration des Webservers

Im ersten Schritt, erstellen wir unter: /etc/apache2/sites-available/ einen neuen Virtual-Host, für unseren späteren Usenet Indexer:

# vim /etc/apache2/sites-available/nzedb.conf

<VirtualHost *:80>
    ServerName 127.0.0.1
    ServerAdmin webmaster@localhost
    ServerSignature Off

    DocumentRoot "/var/www/nZEDb/www"

    LogLevel warn
    ErrorLog /var/log/apache2/error.log

    <Directory "/var/www/nZEDb/www">
        Options FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
    
    Alias /covers /var/www/nZEDb/resources/covers

    ProxyPassMatch ^/(.*\.php(/.*)?)$ unix:/run/php/php7.0-fpm.sock|fcgi://localhost/var/www/nZEDb/www
</VirtualHost>

# sed -i "s/ServerTokens OS/ServerTokens Prod/" /etc/apache2/conf-available/security.conf
# sed -i "s/ServerSignature On/ServerSignature Off/" /etc/apache2/conf-available/security.conf

# vim /etc/apache2/apache2.conf     #->> change at "<Directory /var/www/>... AllowOverride None" to -> AllowOverride All

Nun wird als erstes der Default-Virtual-Host deaktiviert, unser neuer aktiviert und das Rewrite-Modul enabled:

# mkdir -p /var/www/nZEDb/www

# a2dissite 000-default
# a2ensite nzedb.conf

# a2enmod rewrite

# systemctl enable apache2

Konfiguration von PHP 7.0

We already installed all needed PHP extensions earlier. For best performance, we also need to configure some PHP settings. :

# php --ini |grep Loaded

Loaded Configuration File:         /etc/php/7.0/cli/php.ini

In our case, we have to make changes first to the /etc/php/7.0/cli/php.ini file. We will increase the default upload limit to 2000 MB. You can set the values according to your needs. Run the following commands:

# sed -i "s/max_execution_time = 30/max_execution_time = 300/" /etc/php/7.0/cli/php.ini
# sed -i "s/;date.timezone =/date.timezone = Europe\/Zurich/" /etc/php/7.0/cli/php.ini

Now we have to do the same for our php7.0-fpm Configuration:

# sed -i "s/max_execution_time = 30/max_execution_time = 300/" /etc/php/7.0/fpm/php.ini
# sed -i "s/post_max_size = 8M/post_max_size = 64M/" /etc/php/7.0/fpm/php.ini
# sed -i "s/memory_limit = 128M/memory_limit = 2048M/" /etc/php/7.0/fpm/php.ini
# sed -i "s/;date.timezone =/date.timezone = Europe\/Zurich/" /etc/php/7.0/fpm/php.ini

# echo "<?php phpinfo(); ?>" > /var/www/nZEDb/www/phpinfo.php
# systemctl start php7.0-fpm
# systemctl enable php7.0-fpm

# a2enmod proxy_fcgi
# systemctl restart apache2

Check PHP-Configuration at: IP-From-SERVER/phpinfo.php

Konfiguration Mariadb

Erstellen einer neuen MariaDB spezifischen Konfiguration:

# vim /etc/mysql/mariadb.conf.d/50-server.cnf

# configurations by nZEDb
innodb_file_per_table = 1
max_allowed_packet = 128M
group_concat_max_len = 8192

Nach dem Speichern, wird im nächsten Schritt die MariaDB enabled und gestartet!

# systemctl start mysql.service
# systemctl enable mysql.service

# /usr/bin/mysql_secure_installation

Enter current password for root (enter for none): Enter
Set root password? [Y/n]: Y
New password: <your-password>
Re-enter new password: <your-password>
Remove anonymous users? [Y/n]: Y
Disallow root login remotely? [Y/n]: Y
Remove test database and access to it? [Y/n]: Y
Reload privilege tables now? [Y/n]: Y

Erstellen der Datenbank & Konfiguration von AppArmor

Erstellen der von nZEDb benötigten Datenbank und Users:

mysql -u root -p

MariaDB [(none)]> CREATE DATABASE nzedb;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON nzedb.* TO 'nzedbuser'@'localhost' IDENTIFIED BY 'YOUR-PASSWORD';
MariaDB [(none)]> GRANT FILE ON *.* TO 'nzedbuser'@'localhost';
MariaDB [(none)]> GRANT RELOAD ON *.* TO nzedbuser@'localhost';   #  (used when optimising DB)
MariaDB [(none)]> FLUSH PRIVILEGES;
MariaDB [(none)]> quit

Anschliessend, wenn die DB erstellt worden ist, muss nun AppArmor konfiguriert werden.

# apt install apparmor-utils
# aa-complain /usr/sbin/mysqld

You might encounter the following error. It tells you that MariaDB isn’t restricted by AppArmor so just go to the next step.

ERROR: /etc/apparmor.d/usr.sbin.mysqld contains no profile

Installation der Media Tools

1. Installation der packetierten Media-Tools:

   # apt-get install time unrar p7zip-full mediainfo lame ffmpeg libav-tools

2. Installation von yEnc, um das header und message prozessing durch das Indexing zu beschleunigen:

    
   # mkdir /tmp/yenc
   # cd /tmp/yenc
   # wget https://ayera.dl.sourceforge.net/project/yydecode/yydecode/0.2.10/yydecode-0.2.10.tar.gz
   # tar xzf yydecode-0.2.10.tar.gz
   # cd yydecode-0.2.10
   
   # ./configure
   # make
   # make install
   # cd /tmp
   # rm -rf yenc

3. Installation der php-yenc extension, für noch schnelleres Encoding:

   # wget https://github.com/niel/php-yenc/releases/download/v1.3.0/php7.0-yenc_1.3.0_amd64.deb
   
   # dpkg -i php7.0-yenc_1.3.0_amd64.deb

Download und Installation von nZEDb

# rm -Rf /var/www/nZEDb/
# cd /var/www/
# git clone https://github.com/nZEDb/nZEDb.git

# chown -R www-data:www-data /var/www/nZEDb
# chmod -R g+w /var/www/nZEDb/

# cp /var/www/nZEDb/_install/install.sh /var/www/
# chmod a+x /var/www/install.sh

# ./install.sh

During the installation, Composer will install all needed PHP dependencies and you will need to enter your sudo password to set permissions. Once the script finished, we need to set up the web server.

Tmux Version 2 installieren

# apt-get install libevent-dev build-essential git autotools-dev automake pkg-config ncurses-dev
# apt-get remove tmux
# cd /tmp/
# git clone https://github.com/tmux/tmux.git --branch 2.0 --single-branch
# cd tmux
# ll
# ./autogen.sh
# ./configure
# make -j4
# make install
# make clean

Wichtige Arbeiten, von vor dem konfigurieren / einrichten von nZEDb

• CA Zertifikat installieren:

  # wget -P /etc/ssl/certs/ http://curl.haxx.se/ca/cacert.pem
  # chmod 744 /etc/ssl/certs/cacert.pem

• Environment Ergänzungen:

  # mysql_tzinfo_to_sql /usr/share/zoneinfo | sudo mysql -u root -p mysql
  
  # mkdir -p /var/www/nZEDb/resources/tmp/unrar
  # cp /var/www/nZEDb/configuration/settings.example.php /var/www/nZEDb/configuration/settings.php
  # chown www-data:www-data /var/www/nZEDb/configuration/settings.php
  
  # chmod 777 /var/www/nZEDb/resources

nZEDb konfigurieren / Einrichten

server-ip/install

or

domain.com/install

Step1 → System Check should be OK! Step2 → Setup Database! Step3 → Enter CA-Cert Path: → /etc/ssl/certs/cacert.pem Step4 → Configure primary News-Server! ..

Einrichten von PreDB

PreDB is a database that contains information about releases. Utilizing PreDB will assist nZEDb in renaming releases and picking up valid releases. It will also allow nZEDb to mark bad releases. Using a preDB will use up disk/database space however, so if you plan on utilizing it, be aware.

# cd /var/www/nZEDb/cli
# php data/predb_import_daily_batch.php 0 local true

Einrichten von IRCScraper

Prior to setting up the IRCScraper, you may want to import previous preDB dumps. It’s not necessary, but will help matching/renaming releases. Follow the guide to do this.

1. We need to add a PPA so we can install a better version of ZNC.

   # add-apt-repository ppa:teward/znc && sudo apt-get update

2. Install ZNC and the extras:

   # apt-get install znc znc-dbg znc-dev znc-perl znc-python znc-tcl

3. Run the configuration wizard:

   # znc --makeconf

   Answer the wizard with the following answers, change to suit your needs. To avoid confusion, it is advisable to set both your username and nick as the same.

   Listen on Port: **6664, you can change this if you want**
   Listen using SSL: **no, but up to you (this is SSL for the scraper and ZNC, not to IRC**
   Listen on IPv4 and IPv6: **no, but up to you**
   Username: **Make up a unique one, will be used to connect to ZNC**
   Password: **Make up a unique one, will be used to connect to ZNC**
   Nick: **Make up a unique one, will be used to connect to IRC**
   Alt nick: **Make up a unique one, will be used to connect to IRC**
   Ident: **Leave as default**
   Real Name: **Leave as default**
   Bind host: **Leave as default**
   
   Setup Network: **yes**
   Name: **synIRC**
   Server host: 192.168.1.14
   Server SSL: **no**
   Server port: **6667**
   Server password: **leave empty**
   Initial channels: **#nZEDbPRE**
   
   Launch ZNC: **yes**
   

4. Change some settings in ZNC

   Point your web browser to the IP and port of your ZNC machine, i.e. http://192.168.1.14:6664, and login.

   • Under Global Settings, change Max Buffer Size to 1000 and click Save.

   • Under Your Settings → Networks, click Edit next to syncIRC.

   • (Nur falls SSL = yes) Paste the following into the textbox for Trusted SSL fingerprints:

     0b:35:ba:24:e7:1c:f6:9e:1f:82:1d:9a:4e:0d:9f:70:8e:91:74:26:57:13:9e:f9:c7:8e:9c:5c:a6:8e:30:62
     23:2d:7d:fd:79:09:d1:20:ad:6a:88:f1:fc:49:b5:34:cc:00:2a:7f:95:10:07:e7:b7:d7:90:af:7d:eb:7f:07
     54:86:50:b5:7e:08:cb:b4:95:d8:54:9e:fb:8d:f3:6b:97:8a:b7:25:95:d6:3e:38:4c:fb:42:b0:4e:2a:d8:de
     67:7b:f5:25:0d:c0:2d:06:b8:57:2a:ef:9f:5c:2f:c9:48:e9:17:f0:43:22:2e:67:0a:56:ca:f8:ee:98:79:71
     72:26:00:f7:f0:7f:1d:13:a6:20:88:73:ba:42:6c:8b:5e:ef:fd:04:b3:98:90:f7:23:63:bf:08:46:6d:2e:41
     8e:17:a3:cd:ea:5c:55:ff:06:14:91:23:3b:d8:26:d5:b0:d5:8f:69:88:5a:b7:60:dd:73:01:54:d0:b2:18:65
     b3:75:8a:5a:a7:ed:5e:ef:22:45:d4:07:bd:06:32:1e:b8:92:07:49:72:cc:7e:7a:63:fb:3f:e1:92:c3:9a:5a
     d9:10:d9:8a:96:0d:f2:89:d1:a0:87:d0:26:42:b8:51:f9:1d:72:fc:fb:ee:d4:32:14:e8:0c:0d:f3:7a:fa:63
     ef:da:d9:13:be:ad:97:aa:64:65:42:ed:77:24:89:65:97:44:81:da:3d:38:97:56:86:27:67:90:99:57:48:7c
     0b:35:ba:24:e7:1c:f6:9e:1f:82:1d:9a:4e:0d:9f:70:8e:91:74:26:57:13:9e:f9:c7:8e:9c:5c:a6:8e:30:62
     23:2d:7d:fd:79:09:d1:20:ad:6a:88:f1:fc:49:b5:34:cc:00:2a:7f:95:10:07:e7:b7:d7:90:af:7d:eb:7f:07
     67:7b:f5:25:0d:c0:2d:06:b8:57:2a:ef:9f:5c:2f:c9:48:e9:17:f0:43:22:2e:67:0a:56:ca:f8:ee:98:79:71
     72:26:00:f7:f0:7f:1d:13:a6:20:88:73:ba:42:6c:8b:5e:ef:fd:04:b3:98:90:f7:23:63:bf:08:46:6d:2e:41
     8e:17:a3:cd:ea:5c:55:ff:06:14:91:23:3b:d8:26:d5:b0:d5:8f:69:88:5a:b7:60:dd:73:01:54:d0:b2:18:65
     b3:75:8a:5a:a7:ed:5e:ef:22:45:d4:07:bd:06:32:1e:b8:92:07:49:72:cc:7e:7a:63:fb:3f:e1:92:c3:9a:5a
     d9:10:d9:8a:96:0d:f2:89:d1:a0:87:d0:26:42:b8:51:f9:1d:72:fc:fb:ee:d4:32:14:e8:0c:0d:f3:7a:fa:63

   • Under Modules, check the following boxes: chansaver, keepnick, kickrejoin, nickserv, perform, simple_away → Click “Save and Return”

• Under Default Settings, change Channel Modes to +stn

• Under Default Settings, change Buffer Size to 1000

• Click Save and Return

http://nzedb.readthedocs.io/en/master/ircscraper.html#ircscraper