Its Time for NVMeoF to let iSCSI begin its slow retirement journey

Why Its Time for NVMeoF

Its Time for NVMeoF to let iSCSI begin its slow retirement journey for networked shared block storage. Dating back to the late 1990s and early 2000s, iSCSI encapsulates the SCSI command set over a TCP/IP (IP)- based fabric network, or as an alternative to SCSI on Fibre Channel (SCSI_FCP aka FCP aka FC). The SCSI command set dates back to the 1980s, having been deployed on physical parallel cabling and later on other transports, including Fibre Channel, Serial Attached SCSI (SAS), and iSCSI.

The reason it’s time for NVMe over Fabric (NVMeoF), and specifically NVMeoF using TCP and RDMA, to let iSCSI begin its slow retirement journey, is the evolution of the NVMe protocol command set and the expansion of its ecosystem as a modern, ground-up, streamlined server storage I/O protocol command set.

Keep in mind that there are two aspects in play with NVMeoF, which are similar to iSCSI and Fibre Channel-based FCP: the upper layer or protocol command set (e.g., NVMe and SCSI), and media through which these command sets are implemented via various interfaces such as TCP or RDMA on Ethernet, via InfiniBand, Fibre Channel, or physical PCIe, among others.

Ethernet has established itself as the dominant network interface, supporting various upper layers, such as TCP, and increasing its speed. Likewise, InfiniBand and Fibre Channel continue to increase their speeds and maintain their deployment roles in some environments.

Besides supporting faster networks, the NVMe command set was architected and has been deployed to support the current and evolving generations of processors with multiple cores, enabling more concurrent I/O operations and compute threads with less CPU overhead due to a modern, streamlined I/O stack and associated native drivers.

Why NVMeoF (over TCP or RDMA)

Leverage the combined improved benefits of NVMe as a server storage I/O communications protocol, along with TCP networking and industry-standard Ethernet as a communications fabric. With the NVMe command protocol server storage, I/O processing is streamlined and optimized for modern high-speed, low-latency network fabrics, such as Ethernet, as the transport.

The NVMe command set, compared to the SCSI command set, enables faster I/O processing with lower latency, higher IOPs, greater bandwidth, and more transactions, while increasing concurrency and the number of threads and reducing CPU usage per I/O operation. Similar to the SCSI command set, which has been deployed on various industry-standard transport interfaces, the NVMe command set is also deployed across various transports. For example, SCSI has been supported on IP (iSCSI), Fibre Channel (SCSI_FCP, e.g., FCP), Serial Attached SCSI (SAS), and Parallel SCSI, among others.

Likewise, the NVMe command set has been implemented via PCIe, including U.2, as well as over fabrics (e.g., NVMe over Fabric, aka NVMeoF), most notably with TCP and RDMA, along with some NVMe over Fibre Channel. In addition to being a modern, streamlined command set, NVMe also supports more devices, queues, and sessions, and is optimized with host-server initiator and target-side stacks.
An example of how NVMe is being embraced from a software support standpoint is Microsoft’s production release of a new, re-architected, optimized I/O stack as part of Windows Server 2025 in December 2025. Recently, Microsoft released, as part of the latest vnext insiders preview build, an all-new NVMeoF (TCP & RDMA) initiator for testing and evaluation.

NVMe has proven itself as a command set using various transports for use as a target for various SSD and flash storage devices (e.g., back-end) in servers, clients, and storage systems, as well as a front-end alternative to SCSI_FCP (e.g., FCP or FC) and iSCSI among others on storage systems, and software-defined storage solutions. Also, keep in mind that while the SCSI command set dates back to at least the early 1990s for general-purpose access of Magnetic Tape, CD/DVD, HDD, and later SSD, NVMe developed in the last decade was created from the ground up for solid-state devices (SSDs), including flash, among other persistent storage mediums.

Why NVMeoF (TCP) vs iSCSI

It would be, and is, too easy to say the 2000s called and want iSCSI back; however, tongue-in-cheek, there is some reality to that. However, looking closer, iSCSI evolved in the early 2000s as a low-cost SCSI command set alternative to Fibre Channel (SCSI_FCP, e.g., FCP or FC), as well as the survivor of the block IP storage wars (e.g., iFCP, FCIP, iSCSI, etc.) for general-purpose IP-based SCSI storage access.

Fibre Channel-based SCSI (FCP) went on to dominate higher-performance fabric- or networked-storage, aka storage area networks (SANs), with iSCSI having some success in other areas alongside file-based IP storage access via NFS & SMB, among others.

Now that we are in the mid-2020s, the ecosystem around NVMe has matured and continues to do so, with full-stack NVMe solutions, hardware- and software-based initiators, native NVMe drivers, target-based storage systems, and storage devices using various transports. NVMe has evolved from direct-attached PCIe card slots to PCIe U.2 & U.3, M.2 and EDSFF slots, as well as NVMe over Fabrics including TCP and RDMA, among others.

The benefit of using NVMeoF vs iSCSI is lower latency, due to native NVMe commands over TCP being streamlined, vs older SCSI encapsulation and associated cumbersome driver stacks. Besides the streamlined drivers and transports, NVMeoF also provides massive parallelism and multi-threading to boost server storage I/O while leveraging today’s multi-core processors and offload capabilities.

The net result is more I/O operations supporting higher IOPs, bandwidth, and transactions per second, with lower latency and lower CPU usage, resulting in greater productivity and better economics for databases, virtualization, AI/ML, and other block-based workloads.

Who supports NVMe and NVMeoF?

Most host and client operating systems support NVMe or NVMeoF initiators (e.g., Linux, VMware, Microsoft Windows [in the current Insider build], among others). There are also hardware adapter-based NVMe initiators, such as those from Broadcom, Chelsio, and NVIDIA. There are software NVMe and NVMe target packages for many platforms, such as Linux-based NVMe, nvme-tcp, nvme-rdma, nvme-fc, nvmet, nvmet-tcp, and SPDK (Storage Performance Development Kit), among others.

Besides server- or client-side support, there are storage and server hardware chassis building blocks that support NVMe, as well as storage system solutions from Dell, HPE, Hitachi, NetApp, Pure, among others. Within large cloud service providers such as AWS, Microsoft Azure, and Google Cloud Platform (GCP), NVMe is also widely used across their platform ecosystems.

Why it’s time for NVMeoF to let iSCSI begin its slow retirement journey
This is not to say that iSCSI is dead or that it has no future; rather, NVMeoF and its ecosystem have evolved and are ready to do more. Likewise, the SCSI command set is not dead; it’s still used with SAS, Fibre Channel (FCP), iSCSI, etc. However, NVMe has evolved and is seeing broad adoption from server and client attachment to storage devices and storage systems over various transport interfaces and media (PCIe, U.2/U.3, M.2, EDSFF, Ethernet, and fabrics).

Storage systems have also evolved, adding front-end support for NVMeoF and back-end attachment of NVMe flash and SSDs. With the combination of server, client and other software platforms having or in public release review, new streamlined native NVMe drivers and I/O stacks along with NVMeoF initiators (or targets), now is the time to look at NVMeoF as a replacement or alternative to iSCSI, and perhaps even Fibre Channel in the future to support existing and legacy workloads as well as new and emerging application landscapes and their demanding server storage I/O and compute needs (e.g. AI among others).

Additional Resources Where to learn more

Microsoft Windows Server Insider Build NVMeoF Initiator (Blog post)
Installing new Insider’s build Windows Server vnext NVMeoF, host initiator (blog post)
NVMe Linux Driver and related info
Announcing Native NVMe in Windows Server 2025 (Microsoft Post)
Introducing the Windows NVMe-oF Initiator Preview in Windows Server Insiders Builds (Microsoft Post)
Windows Server Insider Builds (Microsoft Downloads)
ToE NVMeoF TCP Performance Line Boost Performance Reduce Costs (blog post)
TheNVMeplace.com (Various NVMe resources)
Azure Cloud Storage here (Microsoft Post)
Microsoft Azure Data Box (Blog Post)
Azure Elastic SAN from Cloud to On-Prem here (Blog Post)
Cloud and Software Defined Data Infrastructure topics here

Additional learning experiences along with common questions (and answers) pertaining to SCSI, Fibre Channel, NVMe, Cloud, Hardware, Software, Services, techniques, server, storage, I/O networking, data protection among other topics are found in my Software Defined Data Infrastructure Essentials book.

What this all means

Now is a good time to look into NVMe and, specifically, NVMe over Fabric (NVMeoF) if you have not recently done so, to refresh or learn what it can do. Likewise, if you are using iSCSI or Fibre Channel-based SCSI_FCP (e.g., not FICON), now is a good time to review or refresh your understanding of NVMeoF and where it fits into future plans for block storage. For those who say block storage is dead and everything is object storage, file storage, or even table storage, that’s fine, as those are all companions to block storage in a balanced ecosystem, each with its own uses, roles, and fits.

Ok, nuff said.

Cheers Gs

Greg Schulz – Ten time Microsoft MVP Cloud and Data Center Management and Azure Storage, along with previous ten-time VMware vExpert. Author of Software Defined Data Infrastructure Essentials (CRC Press), as well as Cloud and Virtual Data Storage Networking (CRC Press), The Green and Virtual Data Center (CRC Press), Resilient Storage Networks (Elsevier) and twitter @storageio. Courteous comments are welcome for consideration. First published on https://storageioblog.com any reproduction in whole, in part, with changes to content, without source attribution under title or without permission is forbidden.

All Comments, (C) and (TM) belong to their owners/posters, Other content (C) Copyright 2006-2026 Server StorageIO and UnlimitedIO. All Rights Reserved. StorageIO is a registered Trade Mark (TM) of UnlimitedIO LLC.