Installing and Configuring LDOMs (Oracle VM for SPARC)

Logical Domains (LDoms or LDOM) is the server virtualization and partitioning technology for SPARC V9 processors. It was first released by Sun Microsystems in April 2007. After the Oracle acquisition of Sun in January 2010, the product has been re-branded as Oracle VM Server for SPARC from version 2.0 onwards.

Each domain is a full virtual machine with a reconfigurable subset of hardware resources. Domains can be securely live migrated between servers while running. Operating systems running inside Logical Domains can be started, stopped, and rebooted independently. A running domain can be dynamically reconfigured to add or remove CPUs, RAM, or I/O devices without requiring a reboot.

Before start configuring our first Oracle VM for SPARC let us understand types of ldoms, ldom services and virtual devices.

Types of logical domains

Guest No direct access to underlying hardware and does not provide virtual device or services to other ldoms. Uses virtual device.
I/O has direct access to underlying hardware in the server. It can be used in cases like oracle DB which wants direct/raw access to the storage devices.
Service provides virtualized devices and services to guest domains.
Control Service domain that also runs the ldoms manager software to control the configuration of hypervisor. This ldom manager is responsible for mapping between physical and virtual devices.

Virtual Services and Devices

VLDC virtual logical domain channel communication channel between logical domain and hypervisor
VCC Virtual console concentrator Acts as a virtual console for each logical domain
VSW Virtual switch service provides network access for guest ldoms to the physical network ports
VDS virtual disk service provides virtual storage service for guest ldoms
VCPU virtual CPU Each thread of a T series CPU acts as a virtual CPU
MAU Mathematical arithmetic unit Each core of T series CPU will have a MAU for accelerated RAS/DSA encryption
Memory Physical memory is mapped into virtual memory and assigned to ldoms
VCONS Virtual console a port in guest ldom that connects to the VCC service in control domain
VNET Virtual network network port in guest ldom which is connected to the VSW service in the control domain
VSDEV Virtual disk service device physical storage device that is virtualized by VDS service in control domain
VDISK Virtual disk VDISK in guest domain is connected to the VDS service in control domain/service domain

Installing the OVM software

To install the LDOM software simply unzip the software zip and run the install-ldm script with -s option in case you don't want to use the configuration assistant to configure the primary and guest ldoms.

primary# unzip
primary# ./install-ldm -s

Creating the default services

Create the essential services like vsw, vcc and vds required to serve the guest LDOMs.

primary# ldm add-vcc port-range=5000-5100 primary-vcc0 primary
primary# ldm add-vds primary-vds0 primary
primary# ldm add-vsw net-dev=nxge0 primary-vsw0 primary
primary# ldm list-services primary
    NAME             VOLUME         OPTIONS          DEVICE
    NAME             PORT-RANGE
    primary-vcc0     5000-5100
    NAME             MAC               NET-DEV   DEVICE     MODE
    primary-vsw0     02:04:4f:fb:9f:0d nxge0     switch@0   prog,promisc

Initial configuration of the control domain

By default all the VCPUs, Memory and MAUs are assigned to the primary domain which is the default domain created after installing the OVM for SPARC software. Primary or control domain is used to configure all the guest ldoms and provide necessary virtual services to them like vcc, vsw and vds. Logical domain manager is responsible to create, delete, modify and control ldoms, thus make sure the ldmd service is running before configuring the primary and guest domains. Use delayed reconfiguration in order to configure the primary ldom without rebooting for previous changes to take effect.

primary# svcs -a | grep ldmd
online 14:23:34 svc:/ldoms/ldmd:default
primary# ldm set-mau 1 primary
primary# ldm set-vcpu 8 primary
primary# ldm start-reconf primary
primary# ldm set-memory 4G primary
primary# ldm add-config new_config
primary# ldm list-config
new_config [current]

Reboot the primary domain for configuration settings to take effect

primary# shutdown -y -g0 -i6

Enable networking between primary and guest domains

By default communication between control domain and all the guest domains is disabled. To enable it, virtual switch has to be configured as the network device instead of nxge0.

primary# ifconfig nxge0 down unplumb
primary# ifconfig vsw0 plumb
primary# ifconfig vsw0 netmask + broadcast + up
primary# mv /etc/hostname.nxge0 /etc/hostname.vsw0

Enable virtual network terminal server daemon

The vntsd daemon is responsible to provide the virtual network terminal services to the guest ldoms. If this service is not running enable it with svcadm command.

primary# svcadm enable vntsd
primary# svcs vntsd
    STATE          STIME    FMRI
    online         Oct_12   svc:/ldoms/vntsd:default

Setting up the Guest Domain

We would assign 8 VCPUs, 2 GB of memory and 1 MAU to our first guest ldom. Also a virtual network vnet1 will be created and associated with the virtual switch vsw0.

primary# ldm add-domain ldom01
primary# ldm add-vcpu 8 ldom01
primary# ldm add-memory 2G ldom01
primary# ldm set-mau 1 ldom01
primary# ldm add-vnet vnet1 primary-vsw0 ldom01

Adding storage to the guest domain

Here we first need to specify the physical device that needs to be exported by vdsdev to the guest domain and then we actually add the virtual disk thus created to the guest domain. Now use any one of the 3 methods mentioned below.

  1. Adding physical disks
    primary# ldm add-vdsdev /dev/dsk/c2t1d0s2 vol1@primary-vds0
    primary# ldm add-vdisk vdisk1 vol1@primary-vds0 ldom01
  2. Adding file
    primary# mkfile 10g /ldoms/ldom01_boot
    primary# ldm add-vdsdev /ldoms/ldom01_boot vol1@primary-vds0
    primary# ldm add-vdisk vdisk1 vol1@primary-vds0 ldom01
  3. Adding a volume
    primary# zfs create -V 5gb pool/vol01
    primary# ldm add-vdsdev /dev/zvol/dsk/pool/vol01 vol1@primary-vds0
    primary# ldm add-vdisk vdisk1 vol1@primary-vds0 ldom01

Setting variables

Setup the boot environment variable for the guest ldom.

primary# ldm set-var auto-boot?=true ldom01
primary# ldm set-var boot-device=vdisk1 ldom01

Setting up the solaris ISO image for installing guest ldom

Now we can also do a jumpstart installation of the guest domain. But one of the easiest and most widely used method is add iso image as virtual disk to the guest ldom and install it from it. Here you can access the vdisk sol10_iso in the ok prompt and boot from it.

primary# ldm add-vdsdev options=ro /data/sol_10.iso iso@primary-vds0
primary# ldm add-vdisk sol10_iso iso@primary-vds0 ldom01

Bind and start installing the ldom

primary# ldm bind ldom01
primary# ldm start ldom01
LDom ldom01 started
ok> devalias
ok> boot sol10_iso

Connect the guest domain

Now check the port which is bound with the guest domain and connect the virtual console of the guest domain.

primary# ldm list
primary active -n-cv  SP   8    4G    0.3% 8h 46m 
ldom01  active -n--- 5000  8    2G     48% 1h 52m

Flag definitions Now you can see various flags in the ldm list command output. The flags represent the current state of the ldom.

1 s starting or stopping
2 n normal
t transition
3 d delayed reconfiguration
4 c control domain
5 v virtual I/O service domain
6 s source domain in migration
t target domain in migration
e error occurred in migration
primary# telnet localhost 5000
    Connected to localhost.
    Escape character is '^]'.
Connecting to console "ldom01" in group "ldom01" .... Press ~? for control options ..

Other useful commands

View current version of Oracle VM server for SPARC software

primary# ldm -V

Long listing of domains

primary# ldm list -l

List the resource for all LDOMs and per LDOM

primary# ldm list -o cpu primary# ldm list -o network,memory ldom01

List the boot variables

primary# ldm list-variable boot-device ldg1

List the bindings of all the LDOMs

primary# ldm list-bindings ldom

List all server resources, bound and unbound

primary# ldm list-devices -a
primary# ldm list-devices mem