TL;DR
KVM supports four main storage backends for virtual machines: iSCSI, NFS, GlusterFS, and Ceph, each suited to different scalability, performance, and reliability needs.
- iSCSI delivers block storage over existing IP networks with strong performance, but setup and security require more effort.
- NFS is the simplest to configure and works across most hypervisors, though it can become a single point of failure at scale.
- GlusterFS still runs in some existing environments but is now deprecated and not recommended for new deployments.
- Ceph combines block, object, and file storage in one system, built for hyper-converged and large-scale infrastructure at the cost of deployment complexity.
As we have seen, Libvirt is a tool that provides API management for multiple hypervisors, including Kernel-based Virtual Machine). Managing storage with a KVM virtual environment is important to the overall infrastructure. We will consider different storage types with KVM, such as iSCSI, NFS, GlusterFS, and Ceph. We will also look at virtual disk images and formats to use with KVM.
KVM Storage Overview
KVM can use many different storage backends to meet the storage needs of virtualized environments. The different storage types provide options for different use cases. Admins can choose the best storage solution for their environment depending on their needs and what kinds of workloads they are hosting.
Let’s look at:
- iSCSI (Internet Small Computer Systems Interface)
- NFS (Network File System)
- GlusterFS (now deprecated)
- Ceph
iSCSI
When you think of an enterprise storage technology for block storage, most will no doubt think about iSCSI. It has been around for quite some time. It is a storage protocol that sends SCSI commands over IP networks. It quickly began to replace Fiber channel storage networks since it can use the same type of network hardware and technologies as IP-based networks. It is often seen used in conjunction with virtualized environments and storage for VMs.
Advantages of iSCSI
Note the following advantages of iSCSI storage solutions:
- Flexibility: iSCSI can be used over existing IP networks, meaning you don’t need a dedicated storage network
- Scalability: It supports large storage pools, which makes it suitable for scaling your environment
- Performance: iSCSI can offer high performance using modern high-speed and high-bandwidth networks
Disadvantages of iSCSI
iSCSI storage may see the following disadvantages:
- Network Dependency: iSCSI performance can be significantly affected by network latency and congestion
- Complex Configuration: Setting up and managing iSCSI targets and initiators can be complex, especially in large environments
- Security Concerns: ISCSI traffic must be properly secured. It is vulnerable to interception and attacks without proper security measures
Configuring iSCSI with Libvirt
Install iSCSI Initiator:
sudo apt-get install open-iscsi
Discover and Login to iSCSI Target:
sudo iscsiadm -m discovery -t sendtargets -p
sudo iscsiadm -m node -T-p –login
Verify Connection:
sudo iscsiadm -m session -o show
Add a storage pool for iSCSI:
virsh pool-define-as iscsi_pool iscsi –source-host
–source-path
virsh pool-start iscsi_pool
virsh pool-autostart iscsi_pool
NFS
Like iSCSI, NFS is also very common. It is a distributed file system protocol allowing remote directories to be shared over a network. NFS provides file-based access to virtual machines with a hypervisor.
Advantages of NFS
Note the following advantages of NFS storage solutions:
- Ease of Use: NFS is easy to configure and manage
- Cost-Effective: It uses existing network infrastructure, so like iSCSI, you don’t need a dedicated environment
Disadvantages of NFS
NFS storage may see the following disadvantages:
- Performance Overhead: NFS can introduce additional overhead, impacting performance, especially in high I/O operations
- Single Point of Failure: NFS servers can become a single point of failure if not configured with high availability
- Scalability Issues: NFS may struggle with scalability in environments requiring large-scale, high-performance storage
Configuring NFS
Install NFS Client:
sudo apt-get install nfs-common
Mount NFS Share:
sudo mkdir -p /mnt/nfs
sudo mount -t nfs:/path/to/share /mnt/nfs
Add a storage pool for NFS:
virsh pool-define-as nfs_pool netfs –source-host
–source-path /path/to/share –target /mnt/nfs
virsh pool-start nfs_pool
virsh pool-autostart nfs_pool
GlusterFS
This type of storage is no longer recommended for greenfield installations. RedHat Gluster Storage will reach end-of-life in 2024. Using GlusterFS as a storage type for Storage Domains is now deprecated for use. However, it is worth mentioning as there are likely still deployments out there built on top of the solution.
Advantages of GlusterFS
Note the following advantages of GlusterFS:
- Scalability: GlusterFS can scale out by adding more storage nodes
- High Availability: It supports data replication and self-healing
- Performance: It is optimized for large-scale data storage and access
Disadvantages of GlusterFS
- Deprecated: GlusterFS is now deprecated and not recommended for use
Configuring GlusterFS
Install GlusterFS Client:
sudo apt-get install glusterfs-client
Mount GlusterFS Volume:
sudo mkdir -p /mnt/glusterfs
sudo mount -t glusterfs:/volname /mnt/glusterfs
Configure GlusterFS Storage in Libvirt:
virsh pool-define-as gluster_pool gluster –source-host
–source-path volname –target /mnt/glusterfs
virsh pool-start gluster_pool
virsh pool-autostart gluster_pool
Ceph
What is Ceph? It is a distributed file system. It provides an object store that allows organizations to run hyperconverged infrastructure (HCI). Ceph admins have a single solution containing block, object, and file storage. These features make it a versatile choice for businesses looking to satisfy their storage needs in virtualization environments.
Advantages of Ceph
Note the following advantages of Ceph
- Scalability: Ceph can scale out to thousands of nodes.
- Reliability: It ensures data redundancy and high availability.
- Performance: Ceph is optimized for high performance in large-scale deployments.
Disadvantages of Ceph
You may see the following disadvantages working with Ceph storage:
- Complex Deployment: Deploying and configuring a Ceph cluster can be complex and time-consuming.
- Resource Requirements: Ceph requires substantial hardware resources, including CPU, memory, and storage
- Maintenance Overhead: Regular maintenance and monitoring are required to ensure optimal performance and reliability.
Configuring Ceph with Libvirt
Install Ceph Client:
sudo apt-get install ceph
Configure Ceph Authentication:
sudo ceph auth get-key client.admin > /etc/ceph/ceph.client.admin.keyring
sudo chmod 600 /etc/ceph/ceph.client.admin.keyring
Configure Ceph Storage:
virsh secret-define –file secret.xml
virsh secret-set-value –secret–base64
virsh pool-define-as ceph_pool rbd –source-host–source-name pool –auth-username libvirt virsh pool-start ceph_pool
virsh pool-autostart ceph_pool
| Attribute | iSCSI | NFS | GlusterFS | Ceph |
|---|---|---|---|---|
| Storage type | Block (over IP) | File-based | Distributed file/scale-out | Distributed, block + object + file |
| Key advantage | High performance over existing IP networks | Easy to configure and manage | Scales out with replication and self-healing | Scales to thousands of nodes, built for HCI |
| Key disadvantage | Complex setup, security must be actively managed | Can be a single point of failure at scale | Deprecated | Complex deployment, high resource requirements |
| Current status | Actively used, mature | Actively used, common | Deprecated, RedHat Gluster Storage reached EOL in 2024 | Actively used, growing for HCI |
Virtual Disk Images and Formats for KVM
KVM supports various virtual disk image formats, each with its advantages and use cases. The most common formats include:
- QCOW2 (QEMU Copy On Write v2): It has features like snapshots, compression, and encryption. This format is the one seem most often with the KVM hypervisor
- RAW: The RAW format is simple and uncompressed. It has great performance but lacks advanced features of other disk formats
- VMDK (Virtual Machine Disk): This format is widely known as used by VMware but is supported by KVM for compatibility
- VHD (Virtual Hard Disk): This format is used with Microsoft Hyper-V but also supported by KVM
Choosing the Right Disk Format
Note the following when choosing which disk format to use in your virtualization environment. Each has its strengths:
- QCOW2 – Use this disk format in environments where snapshots and efficient storage is needed
- RAW – If you have the need for performance-sensitive applications with direct disk access, RAW is best
- VMDK/VHD – These formats provide the best format for mobility and interoperability of virtual machines between different hypervisors
Wrapping up
Libvirt provides many different storage solutions for your virtualization environment. Many viable KVM storage solutions exist, including iSCSI, NFS, GlusterFS, and Ceph. These allow admins to create storage solutions for their KVM-based virtual environments.
These have advantages and disadvantages, allowing admins to choose which solution fits their needs. The different virtual disk images and formats allow you to optimize your storage for guest operating systems that serve different purposes. With these different storage solutions, admins can create scalable and reliable storage for VMs and other workloads.
Related Posts:
KVM for Beginners – Understanding Libvirt Networking – Part 4
FAQ
1) What storage backends does KVM support?
KVM can use several storage backends, including iSCSI, NFS, GlusterFS, and Ceph, each managed through Libvirt storage pools. Admins choose a backend based on their environment’s network setup, scale, and workload requirements. GlusterFS is included for completeness but is now deprecated.
2) Is GlusterFS still supported for KVM storage?
GlusterFS is deprecated and not recommended for new, greenfield KVM deployments. RedHat Gluster Storage reached end-of-life in 2024. It is still worth knowing because existing deployments built on it remain in production.
3) What is the difference between iSCSI and NFS for KVM storage?
iSCSI provides block-level storage by sending SCSI commands over IP networks, which suits performance-sensitive workloads but requires more complex target and initiator configuration. NFS provides file-based storage and is easier to set up and manage, though it can introduce more overhead on high I/O operations.
4) What is Ceph used for in KVM virtualization?
Ceph is a distributed file system that provides block, object, and file storage in a single solution, making it a fit for hyper-converged infrastructure (HCI). It scales out to thousands of nodes and is built for reliability through data redundancy, though it requires substantial hardware resources and ongoing maintenance.
5) Which virtual disk format should I use with KVM, QCOW2 or RAW?
QCOW2 is the most commonly used format with KVM and supports snapshots, compression, and encryption, making it a fit for environments that need efficient storage management. RAW is simpler and uncompressed, offering better performance for direct-disk-access, performance-sensitive applications but without QCOW2’s advanced features.
6)What are the disadvantages of using NFS for KVM storage?
NFS can introduce performance overhead, particularly under high I/O operations, and an NFS server can become a single point of failure if it isn’t configured for high availability. It can also run into scalability limits in environments that need large-scale, high-performance storage.
7) How do you add an iSCSI storage pool in Libvirt?
After installing the open-iscsi initiator -> discovering -> and logging into the iSCSI target, you verify the connection with “iscsiadm -m session -o show”, then define and start the pool using “virsh pool-define-as, virsh pool-start, and virsh pool-autostart”.
8) Is Ceph a good fit for hyper-converged infrastructure?
Yes. Ceph provides a single solution for block, object, and file storage, which is what makes it a versatile choice for organizations running hyper-converged infrastructure. The trade-off is deployment complexity and the hardware resources needed to run it well.
Follow our Twitter and Facebook feeds for new releases, updates, insightful posts and more.
