====== PVESH ======
===== Add a pool and subpool =====
<code>
pvesh create /pools --poolid PROD
pvesh create /pools --poolid "PROD/db"
</code>


===== Add Resource Mappingse for PCI Devices =====
In this example we add a NVIDIA GPU.
  - Get PCI address <code>lspci |grep -e NVIDIA|grep VGA|cut -d. -f1
eb:00</code>
  - Get Subsystem ID from PCI address <code>lspci -vnn -s  eb:00|grep Subsystem|awk '{print $4}'| tr -d '[]'|head -n1</code>
  - Get IOMMU group from PCI address <code>find /sys/kernel/iommu_groups/ -type l | grep 3b:00|awk -F'/' '{print $5}'|head -n1</code>
  - Get Vendor ID from PCI address <code>lspci -nn -s 3b:00 |head -n1|awk '{print $11}'| tr -d '[]'</code>
  - Add Ressource Mapping <code>pvesh create /cluster/mapping/pci --id myGPU--description "My GPU" \
--map id=<Vendor ID>,iommugroup=<Group ID>,node=<nodename>,path=0000:<PCI address>,subsystem-id=<Subsystem ID> 
#example:
pvesh create /cluster/mapping/pci --id myGPU--description "GPU for XYZ" \
--map id=10de:25b2,iommugroup=91,node=f-c-pve01-2-224,path=0000:eb:00,subsystem-id=17aa:1879</code>


====== Other notes ======
==== Enable/Disable Maintence Mode ====
<code>
#Enable
ha-manager crm-command node-maintenance enable myServer
#Disable
ha-manager crm-command node-maintenance enable myServer
</code>

==== Create empty (dummy) VMs ====
<code>
qm create <new ID>
Example:
qm create 200
</code>

==== List VMs ====
<code>
qm list
</code>

==== Corosync  ====
Very important to have a low latency connection. A bond would add 100ms per failure, so it would work. It is better to have two separate cluster links. 

==== Windows VirtIO drivers ====
The Windows VirtIO drivers can be found at:
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/
The version to be chosen depends on Windows installation. Older Windows versions
need older driver packages.

==== Windows change disk type from SATA to SCSI ====
  * Before you can change the disk type, the drivers must be installed. 
  * Add a small SCSI disk and reboot the server. 
  * Check if there is a triangle in the device manager
  * shutdown the server
  * detach and remove the small disk
  * detach your OS disk
  * add your OS disk and change the device type to SCSI
  * also change the boot order

Now it should work, that you can boot from your disk attached as SCSI disk. SCSI is normally faster then VirtIO for Windows OS. 


==== Enter a LXC container from the Proxmox shell: ====
<code>
pct enter <container ID>
# Example: 
pct enter 101
</code>

==== Import disk to VM ==== 
<code>
qm disk import <VM-ID> </path/to/image.qcow2> <storage-name>
# Example: 
qm disk import 103 /mnt/pve/samba/import/Rocky-10-GenericCloud-Base.latest.x86_64.qcow2 zpool
</code>
After the import is DONE the this is marked as //unused disk// at the VM. With an double click it could be added. Also the boot order must be adjusted. 

==== Change the VM ID range ==== 
{{:myhelp:change_vm_id_range.png?600|}}
Datacenter -> Options -> Next Free VMID Range

===== Ceph =====
Install Ceph per node. 
==== Usage per disk and much more info ====
<code>ceph osd  df tree</code>
==== Usage of Ceph  ====
<code>ceph df</code>

==== OSD create on CLI  ====
<code>pveceph osd create /dev/sdX</code>


====== OT takeaways from the training ======
https://www.turnkeylinux.org/ -> Turnkey applications based on LXC

https://www.olivetin.app/ -> Small and simple automater

https://cloudinit.readthedocs.io/en/latest/reference/examples.html -> Cloud init examples

https://alerta.io/ -> Monitoring - Notification