How to setup ZFS Storage Appliance iSCSI luns on Ubuntu Server

Both Ubuntu Server 20.10 and Ubuntu Server 21.04 come with the iSCSI initiator package open-iscsi
installed out of the box allowing for easy configuration for iSCSI attached storage.
This article describes how to integrate iSCSI LUNs presented by an ZFS Storage Appliance (ZFSSA) onto the Ubuntu Server environment using the command-line interface.
This article has been updated to show an installation on Ubuntu Server 21.04, thus your output may differ from that below.
Prerequisites
This article assumes the following:
- For the Ubuntu server:
- You know the
root
password.
- You know the
- For the ZFSSA:
- You know the
hostname
/IP address
. - You know and
root
password. - Pools are configured.
- The network has been configured to present LUNS.
- You know the
Installing the Ubuntu iSCSI Initiator
The open-iscsi
package is installed by default on both Ubuntu Server 20.10 and 21.04. However, if you have removed the package for some reason you can simply reinstall the iSCSI initiator package using the apt(8)
command. For example:
# apt install open-iscsi Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: libisns0 libopeniscsiusr0.2.0 The following NEW packages will be installed: libisns0 libopeniscsiusr0.2.0 open-iscsi 0 upgraded, 3 newly installed, 0 to remove and 15 not upgraded. Need to get 437 kB of archives. After this operation, 2176 kB of additional disk space will be used. Do you want to continue? [Y/n] y : Preconfiguring packages ... : : Preparing to unpack .../open-iscsi_2.1.2-1ubuntu1_amd64.deb ... Unpacking open-iscsi (2.1.2-1ubuntu1) ... : : Setting up open-iscsi (2.1.2-1ubuntu1) ... Created symlink /etc/systemd/system/sockets.target.wants/iscsid.socket → /lib/systemd/system/iscsid.socket. Created symlink /etc/systemd/system/iscsi.service → /lib/systemd/system/open-iscsi.service. Created symlink /etc/systemd/system/sysinit.target.wants/open-iscsi.service → /lib/systemd/system/open-iscsi.service. : : update-initramfs: Generating /boot/initrd.img-5.8.0-36-generic
iSCSI Initiator auto startup
The iSCSI Initiator should automatically start on each reboot of the Ubuntu Server. However, we can confirm here that this is the case.
- Check that the iSCSI Initiator will start when the system is rebooted:
# systemctl is-enabled open-iscsi.service enabled
- If the iSCSI Initiator service is disabled, we can simple enable it and start the service as follows:
# systemctl enable open-iscsi.service Synchronizing state of open-iscsi.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable open-iscsi Created symlink /etc/systemd/system/iscsi.service → /lib/systemd/system/open-iscsi.service. Created symlink /etc/systemd/system/sysinit.target.wants/open-iscsi.service → /lib/systemd/system/open-iscsi.service. # systemctl start open-iscsi
Ubuntu Server Configuration
Identifying the Host IQN
The iSCSI Qualified Name (IQN) of the Ubunutu server is used to identify the host to the ZFSSA. The IQN is automatically generated when the iSCSI initiator package is installed and is stored in a file in /etc/iscsi
. To find the IQN of your Ubuntu server, simply type:
# cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1993-08.org.debian:01:b96d30143d8e
The IQN for our Ubuntu Server is iqn.1993-08.org.debian:01:b96d30143d8e
.
CHAP Authentication
In this article, we are using the Challange Handshake Authentication Protocol (CHAP) to authenticate communication between the Ubuntu server and the ZFSSA. The IQN is used as the CHAP name for the iSCSI initiator and a secure password is used for the CHAP secret.
- To configure CHAP authentication, use your favourite text editor and edit the file
/etc/iscsi/iscsid.conf
- Remove the # character from the beginning of the following line:
#node.session.auth.authmethod = CHAP
to:node.session.auth.authmethod = CHAP
- Set the CHAP username and password, using the following steps:
- Edit the lines that set the CHAP username and password to remove the # character from the beginning of these lines:
#node.session.auth.username = username #node.session.auth.password = password
to:node.session.auth.username = username node.session.auth.password = password
- Change
username
to the IQN of the Ubuntu server. For example, the username is:node.session.auth.username = iqn.1993-08.org.debian:01:b96d30143d8e
- Change the password to a reasonably secure password to be used for authentication. For our example, the password
CHAPsecret11
is used. The line now look like:node.session.auth.password = CHAPsecret11
- Edit the lines that set the CHAP username and password to remove the # character from the beginning of these lines:
- To set the username and password for all targets, use the following:
- Removing the # character in front of the following lines:
#node.session.auth.username_in = username #node.session.auth.password_in = password
- and also set the username and password as shown:
node.session.auth.username_in = chapuser node.session.auth.password_in = CHAPsecret22
- Removing the # character in front of the following lines:
- Save the changes and quit your editor
Configure the ZFSSA using CLI
Access to the ZFSSA CLI is possibe using any SSH client. For example:
% ssh root@zfssa Password: ************ Last login: Sun May 10 19:39:23 2020 from ubuntu zfssa:>
Define the iSCSI Target Group
A target group is created on the ZFSSA to define the ports and the protocol by which the LUNs to be presented to the Ubuntu server is accessed.
In our example, an iSCSI target group is created that contains the LUN as an iSCSI target that is identified by the default IQN for the ZFSSA and presented over default appliance interfaces.
zfssa:> configuration san targets zfssa:configuration san targets> iscsi zfssa:configuration san targets iscsi> create zfssa:configuration san targets iscsi target (uncommitted)> set alias=iSCSI-Targets alias = iSCSI-Targets (uncommitted) zfssa:configuration san targets iscsi target-001 (uncommitted)> commit zfssa:configuration san targets iscsi> list TARGET ALIAS target-000 iSCSI-Targets | +-> IQN iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881 zfssa:configuration san targets iscsi> select target-000 zfssa:configuration san targets iscsi target-000> list Properties: alias = iscsi-Target iqn = iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881 auth = none targetchapuser = (unset) targetchapsecret = (unset) interfaces = nge0 zfssa:configuration san targets iscsi target-000> set targetchapuser=chapuser targetchapuser = iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881 (uncommitted) zfssa:configuration san targets iscsi target-000> set targetchapsecret=CHAPsecret22 targetchapsecret = ************ (uncommitted) zfssa:configuration san targets iscsi target-000> commit zfssa:configuration san targets iscsi target-000> list Properties: alias = iqn = auth = targetchapuser = \ iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881 targetchapsecret = ************ interfaces = zfssa:configuration san targets iscsi target-000> cd .. zfssa:configuration san targets iscsi> list TARGET ALIAS target-000 iSCSI-Targets | +-> IQN iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881 zfssa:configuration san targets iscsi> group zfssa:configuration san targets iscsi groups> create zfssa:configuration san targets iscsi group (uncommitted)> set name=iscsi-TG name = iscsi-TG (uncommitted) zfssa:configuration san targets iscsi group (uncommitted)> set targets=iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881 targets = iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881 (uncommitted) zfssa:configuration san targets iscsi group (uncommitted)> commit zfssa:configuration san targets iscsi groups> list GROUP NAME group-000 iscsi-TG | +-> TARGETS iqn.1986-03.com.sun:02:e8589be5-144d-c9b2-89d4-f7fe4e887881 zfssa:configuration san targets iscsi groups> cd / zfssa:>
Defining an iSCSI Initiator
An iSCSI initiator is defined to restrict which servers have access to a particular volume. If more than one host can write to a given volume concurrently, inconsistency in file system caching between hosts can cause corruption in the on-disk image. Typically, a single initiator is given access to a volume, unless a specialized cluster file system is being used.
The iSCSI initiator serves to define the “host” from the point of view of the ZFS Storage Appliance. To identify the Ubuntu server to the ZFS Storage Appliance, the Ubuntu iSCSI IQN must be registered with the ZFS Storage Appliance as shown in the following example.
zfssa:> configuration san initiators zfssa:configuration san initiators> iscsi zfssa:configuration san initiators iscsi> ls Children: groups => Manage groups zfssa:configuration san initiators iscsi> create zfssa:configuration san initiators iscsi initiator (uncommitted)> set alias=Ubuntu alias = Ubuntu (uncommitted) zfssa:configuration san initiators iscsi initiator (uncommitted)> set initiator= iqn.1993-08.org.debian:01:b96d30143d8e initiator = iqn.1993-08.org.debian:01:b96d30143d8e (uncommitted) zfssa:configuration san initiators iscsi initiator (uncommitted)> set chapuser= iqn.1993-08.org.debian:01:b96d30143d8e chapuser = iqn.1993-08.org.debian:01:b96d30143d8e (uncommitted) zfssa:configuration san initiators iscsi initiator (uncommitted)> set chapsecret=CHAPsecret11 chapsecret = ************ (uncommitted) zfssa:configuration san initiators iscsi initiator (uncommitted)> commit zfssa:configuration san initiators iscsi> ls Initiators:
NAME ALIAS initiator-000 Ubuntu | +-> INITIATOR iqn.1993-08.org.debian:01:b96d30143d8e
Children: groups => Manage groups
zfssa:configuration san initiators iscsi>
Defining an iSCSI Initiator Group
Related iSCSI initiators can be combined into logical groups to allow single commands to be executed on multiple iSCSI initiators, such as, for example, assigning LUN access to all iSCSI initiators in a group with one command. In this example, the iSCSI initiator group will contain one initiator, but in a cluster, where multiple servers are treated as a single logical entity, the initiator group may contain multiple initiators.
zfssa:configuration san initiators iscsi> groups zfssa:configuration san initiators iscsi groups> create zfssa:configuration san initiators iscsi group (uncommitted)> set name=ubuntu-servers name = ubuntu-servers (uncommitted) zfssa:configuration san initiators iscsi group (uncommitted)> set initiators=iqn.1993-08.org.debian:01:b96d30143d8e initiators = iqn.1993-08.org.debian:01:b96d30143d8e (uncommitted) zfssa:configuration san initiators iscsi group (uncommitted)> commit zfssa:configuration san initiators iscsi groups > ls Groups: GROUP NAME group-000 ubuntu-servers | +-> INITIATORS iqn.1988-12.com.oracle:5fa4af61c4a0 zfssa:configuration san initiators iscsi groups> cd / zfssa:>
Defining an ZFS Storage Appliance Project
A project can be defined in the ZFS Storage Appliance to group related volumes. A project allows inheritance of properties for file systems and LUNs presented from the project and also allows quotas and reservations to be applied.
zfssa:> shares zfssa:shares> project ubuntu-project zfssa:shares ubuntu-project (uncommitted)> commit
Defining an Oracle ZFS Storage Appliance LUN
A LUN must now be created from an existing pool of storage resources, which the Ubuntu server will access. For this example, a thin-provisioned 32 GB LUN called Demo will be created. The LUN will be set up for an EXT4 file system.
The target group will be the iSCSI target group created in the section "Defining an iSCSI Target Group" to ensure that this LUN can be accessed using iSCSI protocol. The initiator group defined in the section "Defining an iSCSI Initiator Group" will be used to ensure that only the server(s) defined in the ubuntu-servers
group can access this LUN. (In this example, this group contains only one server.)
zfssa:shares> select ubuntu-project zfssa:shares ubuntu-project> lun Demo zfssa:shares ubuntu-project/Demo (uncommitted)> set volsize=32G volsize = 32G (uncommitted) zfssa:shares ubuntu-project/Demo (uncommitted)> set targetgroup=iscsi-TG targetgroup = iscsi-TG (uncommitted) zfssa:shares ubuntu-project/Demo (uncommitted)> set initiatorgroup=ubuntu-servers initiatorgroup = ubuntu-servers (uncommitted) zfssa:shares ubuntu-project/Demo (uncommitted)> list Properties: checksum = fletcher4 (inherited) compression = off (inherited) dedup = false (inherited) copies = 1 (inherited) logbias = latency (inherited) secondarycache = all (inherited) volblocksize = 8K (default) volsize = 32G (uncommitted) sparse = false (default) exported = true (inherited) targetgroup = iscsi-TG (uncommitted) initiatorgroup = ubuntu-servers (uncommitted) lunumber = (default) assignednumber = (default) status = (default) fixednumber = (default) lunguid = (default) canonical_name = (default) nodestroy = (default) Children: snapshots => Manage snapshots replication => Manage remote replication zfssa:shares ubuntu-project/Demo (uncommitted)> set volblocksize=16k volblocksize = 16K (uncommitted) zfssa:shares ubuntu-project/Demo (uncommitted)> commit zfssa:shares ubuntu-project> cd / zfssa>
Configuring a LUN for Use by the Ubuntu Server
Now that the LUN is prepared and available to an authenticated iSCSI initiator, the LUN must be configured for use by the Ubuntu server by completing the following steps:
- Initiate an Ubuntu iSCSI session with the ZFS Storage Appliance. Since the LUN was created prior to initiating the iSCSI session, it will be automatically enabled.
# systemctl start open-iscsi # iscsiadm -m discovery -t sendtargets -p zfssa 192.168.0.73:3260,2 iqn.1986-03.com.sun:02:012213d6-826b-47b3-a438-f336cf0f8c01 # iscsiadm -m node -p zfssa -l Logging in to [iface: default, target: iqn.1986-03.com.sun:02:012213d6-826b-47b3-a438-f336cf0f8c01, portal: 192.168.0.73,3260] Login to [iface: default, target: iqn.1986-03.com.sun:02:012213d6-826b-47b3-a438-f336cf0f8c01, portal: 192.168.0.73,3260] successful.
- (optional) Review
/var/log/syslog
and check for any errors associated with iSCSI initiator and connectivity. For example:# tail /var/log/syslog May 10 12:36:16 ubuntu kernel: [70844.682726] scsi host3: iSCSI Initiator over TCP/IP May 10 12:36:16 ubuntu kernel: [70844.693372] scsi 3:0:0:0: Direct-Access SUN Sun Storage 7000 1.0 PQ: 0 ANSI: 5 May 10 12:36:16 ubuntu kernel: [70844.695880] sd 3:0:0:0: Attached scsi generic sg2 type 0 May 10 12:36:16 ubuntu kernel: [70844.696616] sd 3:0:0:0: Power-on or device reset occurred May 10 12:36:16 ubuntu kernel: [70844.700720] sd 3:0:0:0: [sdb] 67108864 512-byte logical blocks: (34.4 GB/32.0 GiB) May 10 12:36:16 ubuntu kernel: [70844.701208] sd 3:0:0:0: [sdb] Write Protect is off May 10 12:36:16 ubuntu kernel: [70844.701211] sd 3:0:0:0: [sdb] Mode Sense: 53 00 00 00 May 10 12:36:16 ubuntu kernel: [70844.702200] sd 3:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA May 10 12:36:16 ubuntu kernel: [70844.729760] sd 3:0:0:0: [sdb] Attached SCSI disk May 10 12:36:17 ubuntu iscsid: Connection1:0 to [target: iqn.1986-03.com.sun:02:012213d6-826b-47b3-a438-f336cf0f8c01, portal: 192.168.0.73,3260] through [iface: default] is operational now
- Locate the new LUN device on the Ubuntu server. The quickest method I find is to grep syslog for
Attached SCSI disk
. For example:# grep -i "attached scsi disk" /var/log/syslog May 10 11:27:39 ubuntu kernel: [66727.428119] sd 3:0:0:0: [sdb] Attached SCSI disk May 10 12:36:16 ubuntu kernel: [70844.729760] sd 3:0:0:0: [sdb] Attached SCSI disk
From the about output we can determine that our new lun issdb
. Let's confirm:# ls -l /dev/sdb brw-rw---- 1 root disk 8, 16 May 10 12:36 /dev/sdb
- Prepare the lun by creating a Linux partition table using the
fdisk(8)
command:# fdisk /dev/sdb Welcome to fdisk (util-linux 2.36.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognised partition table. Created a new DOS disklabel with disk identifier 0x1581ec2b. Command (m for help): p Disk /dev/sdb: 32 GiB, 34359738368 bytes, 67108864 sectors Disk model: Sun Storage 7000 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 32768 bytes / 32768 bytes Disklabel type: dos Disk identifier: 0x1581ec2b Command (m for help): n Partition type p primary (0 primary, 0 extended, 4 free) e extended (container for logical partitions) Select (default p): p Partition number (1-4, default 1): 1 First sector (2048-67108863, default 2048): 2048 Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-67108863, default 67108863): 67108863 Created a new partition 1 of type 'Linux' and of size 32 GiB. Command (m for help): w The partition table has been altered. Calling ioctl() to re-read partition table. Synching disks. #
- Configure the device as storage for a large EXT4 file system without any intervening virtualization as shown in the following example:
# mkfs.ext4 -L Demo /dev/sdb1 mke2fs 1.45.6 (20-Mar-2020) Creating filesystem with 8388352 4k blocks and 2097152 inodes Filesystem UUID: c103d253-5861-4800-9d6e-086fa8712b80 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624 Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done
- Setup the new filesystem for mounting at system boot time:
# echo "LABEL=Demo /demo ext4 defaults 2 2" >> /etc/fstab # mkdir /demo # mount /demo # df -h /demo Filesystem Size Used Avail Use% Mounted on /dev/sdb1 32G 49M 30G 1% /demo
The output from thedf(1)
command shows that we now have 32 GB of new space is now available.