Differences

This shows you the differences between two versions of the page.

--- redhat:virtualization-redhat:start [2020/04/01 10:34] – michael
+++ redhat:virtualization-redhat:start [2020/09/02 12:37] (current) – michael
@@ Line 1: / Line 1: @@
-====== KVM Hypervisor unter Redhat / CentOS 7 ======
+====== KVM Hypervisor on Red Hat / CentOS 8.x ======
+**KVM** is an open source hardware virtualization solution which can be used to run several Linux-based or Windows-based systems in parallel on one host.
+KVM is known as a so-called "**Kernel based Virtual Machine**". This is because after installing the package, the KVM module is loaded with the kernel at the next boot, turning a //normal Linux server// into a bare metal hypervisor.
+Now I will describe how to set up such a [[https://de.wikipedia.org/wiki/Kernel-based_Virtual_Machine|KVM Hypervisor]] and how to manage it afterwards.
-**KVM** ist eine Open Source Hardware Virtualisierungslösung, welche dazu gebraucht werden kann; um mehrere Linux basierende oder auch Windows basierende Systeme parallel auf einem Host zu betreiben. KVM ist typischerweise bekannt als sogenannte "**Kernel based Virtual Machine**". Dies ist so, weil nach der Installation des Pakages das KVM Module beim nächsten Boot mit dem Kernel geladen wird und so einen //normalen Linux Server// in einen fast Baremetal-Hypervisor verwandelt. Nun werde ich beschreiben, wie genau so ein [[https://de.wikipedia.org/wiki/Kernel-based_Virtual_Machine|KVM Hypervisor]] aufgesetzt wird und wie man ihn anschliessend verwalten kann.
+  * [[redhat:virtualization-redhat:kvm-suspend-restore-vm-on-reboot|Configure KVM to suspend/restore virtual machines on host reboot]]
+  * [[redhat:virtualization-redhat:kvm-installation-centos7|OLD - KVM Installation on CentOS / Red Hat 7.x]]
+----
+<WRAP center round important 100%>
+<wrap em>Before we start, verify support for **Virtual Technology** of the CPU by issueing the following command:</wrap>
-<wrap em>Bevor mit der Installation des Hypervisors begonnen wird, solle immer zuerst überprüfen; ob die aktuelle Maschine überhaupt eine Virtualisierung unterstützt. Dies wird folgendermassen überprüft:</wrap>
 <code>
-# grep -E '(vmx|svm)' /proc/cpuinfo
+# lscpu | grep Virtualization
 </code>
+<sxh bash; gutter: false;>
+Virtualization:      VT-x
+Virtualization type: full
+</sxh>
-Wenn man nun den Output **vmx** oder **svn** bekommt ist alles in Ordnung. Wenn nichts kommt kann auf diesem Host nicht Visualisiert werden.
+The output of the above command shows that, our server kvm-virtualization-01.recipes.com supports Virtualization.
+However, if the above command returns no result on your server then,
-[[redhat:virtualization-redhat:kvm-installation-centos7|redhat:virtualization-redhat:kvm-installation-centos7]]
+  - ''In case of bare-metal machine, you have to enable the VT support from system BIOS.''
+  - ''In case of virtual machine, you have to enable the VT support from VM's CPU Settings.''
+</WRAP>
-===== Installation von KVM =====
-Jetzt kann die Installation von KVM durchgeführt werden. Als erstes werden dafür die benötigten Pakages installiert.
-<code>
- # yum install qemu-kvm qemu-img virt-manager libvirt libvirt-python libvirt-client virt-install virt-viewer bridge-utils
-</code>
-Nach erfolgreicher Installation müssen die Services gestartet & aktiviert werden.
+===== Installing KVM and QEMU =====
-<code>
-# systemctl start libvirtd
-# systemctl enable libvirtd
-</code>
-Nun sollte überprüft werden, ob KVM erfolgreich seine beiden Core-Module laden konnte.
+In CentOS 8 / RHEL 8, virtualization components including KVM and QEMU hypervisors are bundled in virt module. Therefore, it is really simple now to configure a KVM virtualization host in CentOS 8.
 <WRAP center box 100%>
+We are installing virt module using dnf command.
 <code>
-# lsmod | grep kvm
+# dnf install -y @virt
 </code>
+<sxh bash; gutter: false;>
-<sxh bash; gutter: false>
+...
-kvm_intel             162153  0
+================================================================================
-kvm                   525409  1 kvm_intel
+Installed:
+  libguestfs-1:1.38.4-11.1.module_el8.0.0+189+f9babebb.x86_64
+  libvirt-client-4.5.0-24.3.module_el8.0.0+189+f9babebb.x86_64
+  libvirt-daemon-config-network-4.5.0-24.3.module_el8.0.0+189+f9babebb.x86_64
+  libvirt-daemon-kvm-4.5.0-24.3.module_el8.0.0+189+f9babebb.x86_64
+  alsa-lib-1.1.6-3.el8.x86_64
+  autogen-libopts-5.18.12-7.el8.x86_64
+  boost-atomic-1.66.0-6.el8.x86_64
+  boost-chrono-1.66.0-6.el8.x86_64
+  boost-date-time-1.66.0-6.el8.x86_64
+  boost-iostreams-1.66.0-6.el8.x86_64
+  boost-program-options-1.66.0-6.el8.x86_64
+  boost-random-1.66.0-6.el8.x86_64
+  boost-regex-1.66.0-6.el8.x86_64
+  boost-system-1.66.0-6.el8.x86_64
+  boost-thread-1.66.0-6.el8.x86_64
+  cairo-1.15.12-3.el8.x86_64
+  celt051-0.5.1.3-15.el8.x86_64
+  dnsmasq-2.79-4.el8.x86_64
+  edk2-ovmf-20180508gitee3198e672e2-9.el8_0.1.noarch
+  fribidi-1.0.4-6.el8.x86_64
+  genisoimage-1.1.11-39.el8.x86_64
+  glusterfs-api-3.12.2-40.2.el8.x86_64
+  glusterfs-cli-3.12.2-40.2.el8.x86_64
+  gnutls-dane-3.6.5-2.el8.x86_64
+  gnutls-utils-3.6.5-2.el8.x86_64
+  graphite2-1.3.10-10.el8.x86_64
+  gstreamer1-1.14.0-3.el8.x86_64
+  gstreamer1-plugins-base-1.14.0-4.el8.x86_64
+  harfbuzz-1.7.5-3.el8.x86_64
+  hivex-1.3.15-7.module_el8.0.0+189+f9babebb.x86_64
+  ipxe-roms-qemu-20181214-1.git133f4c47.el8.noarch
+  iso-codes-3.79-2.el8.noarch
+  libX11-1.6.7-1.el8.x86_64
+  libX11-common-1.6.7-1.el8.noarch
+  libX11-xcb-1.6.7-1.el8.x86_64
+  libXau-1.0.8-13.el8.x86_64
+...
 </sxh>
 </WRAP>
-Stimmt die Ausgabe mit der oben aufgelisteten Ausgabe in etwa überein, so sollte nun schon alles parat sein zum Virtualisierungen.
+<WRAP center box 100%>
+We are also installing virt-install package, because it provides some very useful command line tools.
-<WRAP center round tip 100%>
-''**Wichtig zu Beachten:**
-Falls Redhat 7 oder CentOS 7 als minimal Installation installiert wurde, so fehlt nun noch das sogenannte x-window Pakage, welches jedoch zum anzeigen des Virt-managers unumgänglich ist. Zum nachinstallieren folgenden Befehl ausführen:''
 <code>
-# yum install "@X Window System" xorg-x11-xauth xorg-x11-fonts-* xorg-x11-utils -y
+# dnf install -y virt-install
 </code>
-</WRAP>
+<sxh bash; gutter: false;>
+...
+================================================================================
+Installing:
+ virt-install        noarch 2.0.0-5.1.el8                       AppStream 100 k
+Installing dependencies:
+ libosinfo           x86_64 1.2.0-5.el8                         AppStream 244 k
+ osinfo-db           noarch 20181011-8.el8_0.1                  AppStream 172 k
+ osinfo-db-tools     x86_64 1.2.0-1.el8                         AppStream  90 k
+ python3-libvirt     x86_64 4.5.0-2.module_el8.0.0+189+f9babebb AppStream 291 k
+ virt-manager-common noarch 2.0.0-5.1.el8                       AppStream 921 k
+ python3-chardet     noarch 3.0.4-7.el8                         BaseOS    195 k
+ python3-pysocks     noarch 1.6.8-3.el8                         BaseOS     34 k
+ python3-requests    noarch 2.20.0-1.el8                        BaseOS    123 k
+ python3-urllib3     noarch 1.23-5.el8                          BaseOS    178 k
-===== Vorbereitungen zu starten des Virt-Managers =====
+Transaction Summary
-Befindet man sich auf einem Server mit GUI und arbeitet auch direkt auf diesem, so kann man ganz einfach via Konsole den **virt-manager** aufrufen und das Programm wird gestartet.
+================================================================================
+Install  10 Packages
+...
+</sxh>
-Ist man jedoch so wie ich, per **Putty** von einem **Windows System** aus verbunden, so wird dies nicht ganz so einfach funktionieren. Es muss nämlich hierzu zuerst noch ein spezielles Programm Namens: "[[https://sourceforge.net/projects/xming/|Xming]]"
-auf dem Windows System installiert werden. **Xming** erlaubt es Grafikdarstellungen welche via SSH gesendet werden in Windows als eigenes Fenster Remote darzustellen. So können wir also dann später auch ganz einfach nur den **virt-manager** per Putty aufrufen und es sollte dann die besagte Konsole öffnen.
-<WRAP center box 100%>
-==== Xming Installation ====
-  - ''Als erstes sollte wenn nicht schon passiert Xming [[https://sourceforge.net/projects/xming/|HIER]] **heruntergeladen** und **installiert** werden.''
-  - ''Anschliessend soll Xming gestartet werden.'' {{ :redhat:virtualization-redhat:xming-systray.png?nolink&700 |}}
-  - ''<wrap hi><wrap em>WICHTIG:</wrap>Nun müssen alle bestehenden Sitzungen von Putty zum Hypervisor getrennt werden!</wrap>''
-  - ''Als nächstes müssen wir dann in Putty selber bei der Hypervisor-Verbindung das **X11 forwarding** zulassen.'' ''Dazu geht man beim richtigen System (Nach dem laden des Putty Profiles) Unter **Connection / SSH / X11** und setzt dort ein häcklein bei "**Enable X11 forwarding**."'' {{ :redhat:virtualization-redhat:putty-x11.png?nolink&400 |}}
-  - ''Nun kann man die veränderten **Einstellung in der Session speichern**; Und anschliessend die Putty Verbindung neu aufbauen.''
-''<wrap em>Hat nun alle geklappt kann, jetzt auch von der Putty Session aus ganz einfach mit dem Befehl: "**virt-manager**" die Management Oberfläche von KVM gestartet werden.</wrap>''
 </WRAP>
-===== Starten des Virt Managers =====
+<WRAP center box 100%>
+''Validate all the components on your KVM host can support virtualization.''
-Virt Manager is a graphical tool through which we can install and manage virtual machines. To start the virt manager type the '**virt-manager**' command from the terminal.
 <code>
-# virt-manager
+# virt-host-validate
 </code>
-{{:redhat:virtualization-redhat:virt-manger-1.jpg?nolink|}}
+<sxh bash; gutter: false;>
+  QEMU: Checking for hardware virtualization                                 : PASS
+  QEMU: Checking if device /dev/kvm exists                                   : PASS
+  QEMU: Checking if device /dev/kvm is accessible                            : PASS
+  QEMU: Checking if device /dev/vhost-net exists                             : PASS
+  QEMU: Checking if device /dev/net/tun exists                               : PASS
+  QEMU: Checking for cgroup 'memory' controller support                      : PASS
+  QEMU: Checking for cgroup 'memory' controller mount-point                  : PASS
+  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
+  QEMU: Checking for cgroup 'cpu' controller mount-point                     : PASS
+  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
+  QEMU: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
+  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
+  QEMU: Checking for cgroup 'cpuset' controller mount-point                  : PASS
+  QEMU: Checking for cgroup 'devices' controller support                     : PASS
+  QEMU: Checking for cgroup 'devices' controller mount-point                 : PASS
+  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
+  QEMU: Checking for cgroup 'blkio' controller mount-point                   : PASS
+  QEMU: Checking for device assignment IOMMU support                         : PASS
+  QEMU: Checking if IOMMU is enabled by kernel                               : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments)
+</sxh>
+It looks like ''**IOMMU** (input-output memory management unit)'' support is not yet enabled in the CentOS 8 Kernel.
-===== Configure Bridge Interface =====
+The solution is already suggested by the above command. Therefore, we are adding the same in the Kernel command line options.
-Before Start creating VMs , let’s first create the bridge interface. Bridge interface is required if you want to access virtual machines from outside of your hypervisor network.
-<code>
+<code># grub2-editenv - set "$(grub2-editenv - list | grep kernelopts) intel_iommu=on"</code>
-# cd /etc/sysconfig/network-scripts/
-# cp ifcfg-eno49 ifcfg-br0
-</code>
-<WRAP center round box 100%>
+<wrap em>To take effect, restart your machine!</wrap>
-**Edit** the **Interface file** and set followings:
-<code>
+After reboot, again run the ''virt-host-validate'' command.
-# vim ifcfg-eno49
-</code>
-  * ''TYPE=Ethernet''
+<sxh bash; gutter: false;>
-  * ''BOOTPROTO=static''
+  QEMU: Checking for hardware virtualization                                 : PASS
-  * ''DEVICE=eno49''
+  QEMU: Checking if device /dev/kvm exists                                   : PASS
-  * ''ONBOOT=yes''
+  QEMU: Checking if device /dev/kvm is accessible                            : PASS
-  * ''BRIDGE=br0''
+  QEMU: Checking if device /dev/vhost-net exists                             : PASS
+  QEMU: Checking if device /dev/net/tun exists                               : PASS
+  QEMU: Checking for cgroup 'memory' controller support                      : PASS
+  QEMU: Checking for cgroup 'memory' controller mount-point                  : PASS
+  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
+  QEMU: Checking for cgroup 'cpu' controller mount-point                     : PASS
+  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
+  QEMU: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
+  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
+  QEMU: Checking for cgroup 'cpuset' controller mount-point                  : PASS
+  QEMU: Checking for cgroup 'devices' controller support                     : PASS
+  QEMU: Checking for cgroup 'devices' controller mount-point                 : PASS
+  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
+  QEMU: Checking for cgroup 'blkio' controller mount-point                   : PASS
+  QEMU: Checking for device assignment IOMMU support                         : PASS
+  QEMU: Checking if IOMMU is enabled by kernel                               : PASS
+</sxh>
+''Everything is fine now.''
 </WRAP>
-<WRAP center round box 100%>
+KVM and QEMU hypervisors has been installed on CentOS 8.
-**Edit** the **Bridge file (ifcfg-br0)** and set the followings:
-<code>
-# vim ifcfg-br0
-</code>
-  * ''TYPE=Bridge''
+===== Configure KVM Environment =====
-  * ''BOOTPROTO=static''
-  * ''DEVICE=br0''
-  * ''ONBOOT=yes''
-  * ''IPADDR=192.168.1.21''
-  * ''NETMASK=255.255.255.0''
-  * ''GATEWAY=192.168.1.1''
-  * ''DNS1=192.168.1.1''
-<wrap em>Replace the IP address and DNS server details as per your setup.</wrap>
+<WRAP center box 100%>
-</WRAP>
+==== Allow QEMU/KVM Commands for personal user: ====
-Restart the network Service to enable the bridge interface.
+If you want to allow your login user (non-root) to run virsh command or other KVM/QEMU commands, or use these commands without sudo, (needed by cockpit) then add your login user to the libvirt group as follows:
 <code>
-# systemctl restart network
+# usermod -aG libvirt YOURUSERNAME
 </code>
+</WRAP>
-Check the Bridge interface configuration, with the command below:
+<WRAP center box 100%>
+==== Setup own VM-Storage / ISO-Storage configuration: ====
+**Create needed directories**:
 <code>
-# ip addr show br0
+# mkdir -p /data01/vm-storage
+# mkdir /data01/iso-images
 </code>
+----
-===== Start Creating Virtual Machines. =====
+**Make the virsh pool configuration changes**:
-Create Virtual Machine either from the command line using '**virt-install**' command or from GUI (**virt-manager**)
-<WRAP center box 100%>
+  - Listing current pools: <code># virsh pool-list</code><sxh bash; gutter: false;>
-''**Now let’s Create a virtual machine** of **Windows Server 2012 R2** using virt-manager.''
-  - Start the “virt-manager”
+Name                 State      Autostart
-  - Go to the File Option, click on ''New Virtual Machine'' <WRAP clear/> {{:redhat:virtualization-redhat:virt-manger-2.jpg?nolink|}}
+-------------------------------------------
-  - We will be using ISO file as installation media. In the next step Specify the path of ISO file.{{:redhat:virtualization-redhat:virt-manger-3.jpg?nolink|}}
+default              active     yes
-  - Click on Forward. And specify the Compute Resources : RAM and CPU as per your setup. {{:redhat:virtualization-redhat:virt-manger-4.jpg?nolink|}}
+</sxh>
-  - Click on Forward to proceed further. Then specify the storage Size of Virtual Machine, In my case I am using 25G.{{:redhat:virtualization-redhat:virt-manger-5.jpg?nolink|}}
+  - ''Destroying current default pool:''<code># virsh pool-destroy default</code><sxh bash; gutter: false;>Pool default destroyed</sxh>
-  - In the Next step Specify the Name of Virtual Machine and select network as **Bridge br0**{{:redhat:virtualization-redhat:virt-manger-6.jpg?nolink|}}
+  - ''Undefine current default pool:''<code># virsh pool-undefine default</code><sxh bash; gutter: false;>Pool default has been undefined</sxh>
-  - Click on Finish to start the installation. {{:redhat:virtualization-redhat:virt-manger-7.png?nolink|}}
+  - ''Defining a new pool with name "default":''<code># virsh pool-define-as --name default --type dir --target /data01/vm-storage</code><sxh bash; gutter: false;>Pool default defined</sxh>
+  - ''Set pool to be started when libvirt daemons starts:''<code># virsh pool-autostart default</code><sxh bash; gutter: false;>Pool default marked as autostarted</sxh>
-''<wrap em>Follow the screen instructions and complete the installation.</wrap>''
+  - ''Start pool:''<code># virsh pool-start default</code><sxh bash; gutter: false;>Pool default started</sxh>
+  - ''Checking pool state:''<code># virsh pool-list</code><sxh bash; gutter: false;>
+Name                 State      Autostart
+-------------------------------------------
+default              active     yes
+</sxh>
+From now, when creating virtual machines, Virtual Machine Manager will inform you that the *.img file (virtual disk of your VM), will be saved at ''/data01/vm-storage/''.
 </WRAP>
+===== Installing Cockpit Web Interface in CentOS 8 =====
-===== Creating a virtual Machine from Command Line: =====
+Although, KVM commandline-tools are quite sufficient for managing a Virtualization environment. But, we can also use the CentOS 8 native Web UI i.e. Cockpit to manage virtual machines via a graphical interface.
-Virtual Machines can be created from the console as well using the ''virt-install'' command. In the following example i'm going to install an virtual machine with Ubuntu 16.04 LTS.
+We are installing Cockpit using dnf command. To add support of managing virtual machines, we have to install cockpit-machines package as well.
+<WRAP center box 100%>
 <code>
-# virt-install --name=Ubuntu-16-04 --file=/var/lib/libvirt/images/ubuntu16-04.dsk --file-size=20 --nonsparse --graphics spice --vcpus=2 --ram=2048 --cdrom=ubuntu-16.04-server-amd64.iso --network bridge=br0 --os-type=linux --os-variant=generic
+# dnf install -y cockpit cockpit-machines
 </code>
-The above **virt-install** command consitst of following options :
+<sxh bash; gutter: false;>
+...
+================================================================================
+Installing:
+ cockpit                      x86_64 185.1-1.el8_0              BaseOS     68 k
+ cockpit-machines             noarch 184.1-1.el8                AppStream 669 k
+Installing dependencies:
+ PackageKit                   x86_64 1.1.12-2.el8               AppStream 600 k
+ PackageKit-glib              x86_64 1.1.12-2.el8               AppStream 141 k
+ cairo-gobject                x86_64 1.15.12-3.el8              AppStream  33 k
+ python3-cairo                x86_64 1.16.3-6.el8               AppStream  90 k
+ python3-gobject              x86_64 3.28.3-1.el8               AppStream  25 k
+ python3-systemd              x86_64 234-8.el8                  AppStream  81 k
+ setroubleshoot-plugins       noarch 3.3.10-1.el8               AppStream 365 k
+ checkpolicy                  x86_64 2.8-2.el8                  BaseOS    338 k
+ cockpit-bridge               x86_64 185.1-1.el8_0              BaseOS    596 k
+ cockpit-system               noarch 185.1-1.el8_0              BaseOS    1.6 M
+ cockpit-ws                   x86_64 185.1-1.el8_0              BaseOS    834 k
+ gdk-pixbuf2                  x86_64 2.36.12-2.el8              BaseOS    466 k
+ glib-networking              x86_64 2.56.1-1.1.el8             BaseOS    155 k
+ gsettings-desktop-schemas    x86_64 3.28.1-1.el8               BaseOS    619 k
+...
+</sxh>
+</WRAP>
-  * **–name** = <Name of the Virtual Machine>
+<WRAP center box 100%>
-  * **–file** = <Location where our virtual machine disk file will be stored >
+Enable and start Cockpit Unit.
-  * **–file-size** = < Size of the Virtual Machine, in my case it is 20GB >
+<code>
-  * **–nonsparse** = < Allocate the whole storage while creating>
+# systemctl enable --now cockpit.socket
-  * **–graphics** = < Specify the graphical tool for interactive installation, in above example I am using spice >
+</code>
-  * **–vcpu** = < Number of virtual CPU for the Machine >
-  * **–ram** = < RAM size for the virtual Machine >
-  * **–cdrom** = < Virtual CD ROM which specify the installation media like ISO file >
-  * **–network** = < it is used to specify which network we will use for the virtual machine, in this example I am bridge interface>
-  * **–os-type** = < Operating system type like linux and window>
-  * **–os-variant**= <KVM maintains the OS variants like ‘fedora18′, ‘rhel6’ and ‘winxp’ , this option is optional and if you not sure about OS variant you can mentioned it as generic>
+<sxh bash; gutter: false;>
+Created symlink /etc/systemd/system/sockets.target.wants/cockpit.socket at /usr/lib/systemd/system/cockpit.socket
+</sxh>
-Open now the Virt Viewer //(on Windoff)// and follow the instruction to complete the installation.
+''Cockpit service is by-default allowed in CentOS 8 firewall.''
-{{:redhat:virtualization-redhat:ubuntu-16-04-virt-install.jpg?nolink|}}
-Follow the instruction now and complete the installation.
-Once the Installation is completed we can access the Virtual Machine console from ‘virt-manager‘ as shown below.
-{{:redhat:virtualization-redhat:ubuntu-16-04-virt-install2.jpg?nolink|}}
-That’s it, basic installation and configuration of KVM hypervisor is completed.
-<WRAP center box 100%>
-===== KVM Manageing (web-Based): =====
-Solutions to manage and supervise virtual machines over a WebGUI.
-<WRAP center round tip 100%>
-  * ''https://www.virtualizor.com/''
-  * ''https://github.com/ArchipelProject/Archipel/wiki''
 </WRAP>
+Browse URL https://YOUR-SERVERS-IP:9090/ in a client's browser.
-''Alternativ, komplette KVM-Virtualisierungs-Plattformen:''
+The Cockpit uses a self-signed SSL certificate, therefore, you may see a Security warning.
-  * https://ovirt.org/
+Ignore the Security warning and continue to the website.
-  * https://www.proxmox.com/de/
-</WRAP>
-===== Redhat Dokumentation zum Thema =====
-<WRAP center round download 80%>
-''{{ :redhat:virtualization-redhat:red_hat_enterprise_linux-7-virtualization_deployment_and_administration_guide-en-us.pdf |}}''
-</WRAP>