Monthly Archives: August 2019

Modifying the destination SnapMirror Volume when using XDP

References: 
How to modify Destination SnapMirror volume size and other attributes

SnapMirror Configuration Best Practice ONTAP 9.1-9.2

NetApp has been using a replication technology called SnapMirror for a very long time. I have been around so long, I remember when when we called it BMM or Bare Metal Migrate. It was used to help customers replicate (or migrate) data from an old Network Appliance, Inc filer to a new model. It was barbaric! You actually had to create some floppies. The source and the destination had to boot from the special media. A special connection was made over IP (usually a point-to-point Gigabit link, but sometimes 100BaseT). The transfer started and finished. This was the beginning of SnapMirror. Over the next years it molded into a product that was available in the operating system (again, before ONTAP, before 7-mode!) and was dubbed SnapMirror.

The Types

Ever since that time, the underlying replication engine was known as DP or Data Protection. This intelligently moved blocks of data from the source to the destination. When SnapVault was introduced, a new engine was used to move data around. This engine was called XDP or eXtended Data Protection. This engine is more of a logical block transfer that was updated to be used in regular SnapMirror transfers as well. In ONTAP 9.1, it became the default for SnapMirror and in 9.3 it became the default for SVM-DR. In the next couple of releases, I Suspect DP will be removed completely from ONTAP.

Forwards and Backwards

One the fantastic benefits of an XDP mirror is the ability to replicate to releases on ONTAP forwards or backwards! Historically, a DP mirror can only mirror to the current version or newer and never backwards. As ONTAP rapidly advances, this affords the customer the ability to no longer have to keep source and destination ONTAP releases tight. They can let them change and still maintain the ability to migrate forwards and backwards.

Changes Stick…

When a DP SnapMirror was configured, the destination needed to be the same size or larger than the source. After the relationship was established, nearly any change on the source was made automatically on the destination. Resizing the source volume, adding more inodes, changing the Snapshot schedule or SnapShot reserve space…all of things and more triggered the same change to happen on the destination.

…But not with XDP

This is no longer the case with an XDP SnapMirror, especially in a SVM-DR relationship. In this setup, the relationship will mirror all volumes from one SVM to another and all volumes in the SVM are replicated using XDP. If a change is made on the source and want that change to be applied to the destination, the storage administrator needs to make that happen. This process cannot be done normally on the destination. Why? When an XDP mirror has been established, the destination becomes read-only and the storage admin is no longer able to run commands on the destination volume.

The Long Way

From experience, I can tell you there are two way to make changes to the destination. The first way is the long way. There are a few steps here:

  1. Update your mirror
  2. Quiesce your mirror
  3. Break your mirror (which now creates a RW destination)
  4. Make any changes to the volume
    1. vol size or vol modify commands to manipulate inodes, snapshot reserve space, etc.
  5. Re-sync the mirror
  6. Resume the mirror

This is a lot of work. Resyncing can take time. There is an alternative as I recently found out!

The Short Way

The short was is actually very simple and it uses a command I recently became aware of.

Note: This is a warning! Use of “diag” mode on the ONTAP CLI can be dangerous and result in loss of data.

The short way is also very quick. First access the CLI. Use SSH or the console or the Service-Processor(SP) or the Baseboard Management Controller (BMC) and access the ONTAP command line. Then, enter “diag” mode:

::> set -privilege diag
::*>

Your normal prompt will change and display the asterisk as indicated above. All thats left is to modify the destination volume(s):

::*> vserver config override -command "volume size -vserver <dest_SVM> -volume <dest_vol> -new-size <new_size>"
::*> vserver config override -command "volume modify -vserver <dest_SVM> -volume <dest_vol> -autosize-mode {off|grow|grow_shrink}"

Make the changes that are required (volume size, autogrow mode, autogrow thresholds, etc) and then just set the mode back to admin:

::*> set -privilege admin
::>

Ever wish you could SORT on the ONTAP CLI?

ONTAP has a very good command line interface (CLI). There is an online document titled ONTAP 9 System Administration Reference. This is a PDF documenting the ONTAP CLI. There are three privileges which include:

  • admin (default)
    • Most commands and parameters are available at this level. They are used for common or routine tasks
  • advanced
    • Commands and parameters at this level are used infrequently, require advanced knowledge, and can cause problems if used inappropriately. You use advanced commands or parameters only with the advice of support personnel
  • diagnostic
    • Diagnostic commands and parameters are potentially disruptive. They are used only by support personnel to diagnose and fix problems
Moving forward, this blog post will presume that the reader is knowledgeable about the ONTAP CLI. If you would like to know more about the ONTAP CLI, please refer to the link above.

Why Sort?

There are many times where I am looking through ONTAP CLI output and find it difficult to read as a human. I always wished there was a way I could sort the output on the CLI without have to script from linux and use sort. Some examples could be that I have a large number of data LIFs on multiple SVMs and would like to see easily what addresses are in use or looking through a large number of flexvols and would like the output in a particular order.

SSSSHHHHH!

To enable sorting on the ONTAP CLI requires the use of diagnostic mode as indicated earlier. Even more, the sort option is truly hidden and the argument itself will not obey the ONTAP CLI tab-completion! You must type out the argument completely.

How to make it work

From the ONTAP CLI, eneter into diagnostic mode:

cluster::> set -privilege diagnostic

Warning: These diagnostic commands are for use by NetApp personnel only.
Do you want to continue? {y|n}

cluster::*>

Notice the prompt changed to indicate you are no longer in the default/standard admin mode. Be extra careful in diagnostic (or even advanced) mode as entering the wrong command here could result in data loss. The sorting option I am about to show you is for you to piggy-back on to regular commands you already use.

Let’s Sort!

Now that we are in diagnostic mode, it is very easy to sort output. Anywhere in your command, simply add in -sort-by field1 <,field2,field3,etc>. The argument does not need to be at the end, it is usually most convenient there.

Tip: to get rid of the ONTAP CLI pagination, enter: rows 0
This turns off the per-screen paging and usually tightens up output

I was able to access a customers’ system for some sample output. The names have been changed to protect the innocent. Let’s look at some regular output, noting all the Network Addresses and how they are not sorted. The normal output is sorted by vserver then by LIF name. Note, the output has had only the right-most two field removed for readability!

            Logical    Status     Network            Current       
Vserver     Interface  Admin/Oper Address/Mask       Node          
----------- ---------- ---------- ------------------ ------------- 
Cluster
            CustCluster1_clus1 up/up 169.254.5.203/16 CustCluster1
            CustCluster1_clus2 up/up 169.254.91.160/16 CustCluster1
            CustCluster2_clus1 up/up 169.254.196.87/16 CustCluster2
            CustCluster2_clus2 up/up 169.254.18.84/16 CustCluster2
            CustCluster3_clus1 up/up 169.254.158.204/16 CustCluster3
            CustCluster3_clus2 up/up 169.254.243.163/16 CustCluster3
            CustCluster4_clus1 up/up 169.254.33.135/16 CustCluster4
            CustCluster4_clus2 up/up 169.254.142.203/16 CustCluster4
CustSVM01
            cifs_ediscovery up/up 192.172.76.192/24    CustCluster4 
            mgmt_svm     up/up    192.172.239.205/24   CustCluster4 
CustSVM02
            cifs_eaprofile up/up  192.172.76.207/24    CustCluster1 
            cifs_cust     up/up    192.172.76.193/24    CustCluster3 
            cifs_cust_COPS_G up/up 192.172.76.195/24    CustCluster3 
            cifs_cust_PST_archive up/up 192.172.76.196/24 CustCluster3
            cifs_cust_archive up/up 192.172.76.197/24   CustCluster3 
            cifs_cust_auditlogs up/up 192.172.76.198/24 CustCluster3 
            cifs_cust_citrix up/up 192.172.76.199/24    CustCluster3 
            cifs_cust_g_drive up/up 192.172.76.200/24   CustCluster3 
            cifs_cust_nya_public up/up 192.172.76.201/24 CustCluster3
            cifs_cust_ovw up/up    192.172.76.202/24    CustCluster3 
            cifs_cust_software up/up 192.172.76.203/24  CustCluster3 
            cifs_cust_sqlbackups up/up 192.172.76.204/24 CustCluster3
            cifs_cust_support up/up 192.172.76.205/24   CustCluster3 
            cifs_cust_utility up/up 192.172.76.194/24   CustCluster3 
            cifs_cust_wtr up/up    192.172.76.206/24    CustCluster3 
            mgmt_svm     up/up    192.172.239.206/24   CustCluster4 
            nfs_ISO      up/up    192.172.73.199/24    CustCluster3 
            tmp_nbu      up/up    192.172.76.209/24    CustCluster1 
CustZCluster
            mgmt_D1      up/up    192.172.239.193/24   CustCluster1 
            mgmt_D2      up/up    192.172.239.194/24   CustCluster2 
            mgmt_D3      up/up    192.172.239.195/24   CustCluster3 
            mgmt_D4      up/up    192.172.239.196/24   CustCluster4 
            mgmt_cluster up/up    192.172.239.192/24   CustCluster1 
            snapmirror_D1 up/up   192.172.239.240/24   CustCluster1 
            snapmirror_D2 up/up   192.172.239.241/24   CustCluster2 
            snapmirror_D3 up/up   192.172.239.242/24   CustCluster3 
            snapmirror_D4 up/up   192.172.239.243/24   CustCluster4 
CustZServSVM01
            cifs_AT_PAASDB_DC00 up/up 192.172.76.210/24 CustCluster4
            cifs_fsw_gp_evclu_dc01 up/up 192.172.76.208/24 CustCluster4
            cifs_fsw_jd1sw_ssawcl up/up 192.172.76.214/24 CustCluster4
            iscsi1_D1_e0e up/up   192.172.74.192/24    CustCluster1 
            iscsi1_D2_e0e up/up   192.172.74.193/24    CustCluster2 
            iscsi1_D3_e0e up/up   192.172.74.194/24    CustCluster3 
            iscsi1_D4_e0e up/up   192.172.74.195/24    CustCluster4 
            iscsi2_D1_e0f up/up   192.172.75.192/24    CustCluster1 
            iscsi2_D2_e0f up/up   192.172.75.193/24    CustCluster2 
            iscsi2_D3_e0f up/up   192.172.75.194/24    CustCluster3 
            iscsi2_D4_e0f up/up   192.172.75.195/24    CustCluster4 
            mgmt_svm     up/up    192.172.239.207/24   CustCluster4 
            nfs_D1       up/up    192.172.73.192/24    CustCluster1 
            nfs_D2       up/up    192.172.73.193/24    CustCluster2 
            nfs_D3       up/up    192.172.73.194/24    CustCluster3 
            nfs_D4       up/up    192.172.73.195/24    CustCluster4 
            nfs_vm_gp_ev_dc01_os_index up/up 192.172.73.203/24 CustCluster1
            nfs_vm_gp_ev_dc01_vault up/up 192.172.73.204/24 CustCluster4
            nfs_vm_gp_ev_dc02_os_index up/up 192.172.73.201/24 CustCluster1
            nfs_vm_gp_ev_dc02_vault up/up 192.172.73.207/24 CustCluster4
            nfs_vm_gp_ev_dc03_os_index up/up 192.172.73.202/24 CustCluster2
            nfs_vm_gp_ev_dc03_vault up/up 192.172.73.205/24 CustCluster4
            nfs_vm_gp_ev_dc04_os_index up/up 192.172.73.197/24 CustCluster2
            nfs_vm_gp_ev_dc04_vault up/up 192.172.73.206/24 CustCluster4
            nfs_vm_msdp_cc_nbd_02 up/up 192.172.73.198/24 CustCluster4
            nfs_vm_msdp_cc_nbd_03 up/up 192.172.73.200/24 CustCluster4
            nfs_vm_nfs_wa up/up   192.172.73.196/24    CustCluster2 
64 entries were displayed.

CustCluster::>

I would like to know across all vservers, what addresses are actually used. It would be great to sort this list by the address field. The command is simply this: net int show -sort-by address

CustCluster::*> network interface show -sort-by address

            Logical    Status     Network            Current       
Vserver     Interface  Admin/Oper Address/Mask       Node          
----------- ---------- ---------- ------------------ ------------- 
CustZServSVM01
            nfs_D1       up/up    192.172.73.192/24    CustCluster1 
            nfs_D2       up/up    192.172.73.193/24    CustCluster2 
            nfs_D3       up/up    192.172.73.194/24    CustCluster3 
            nfs_D4       up/up    192.172.73.195/24    CustCluster4 
            nfs_vm_nfs_wa up/up   192.172.73.196/24    CustCluster2 
            nfs_vm_gp_ev_dc04_os_index up/up 192.172.73.197/24 CustCluster2
            nfs_vm_msdp_cc_nbd_02 up/up 192.172.73.198/24 CustCluster4
CustSVM02
            nfs_ISO      up/up    192.172.73.199/24    CustCluster3 
CustZServSVM01
            nfs_vm_msdp_cc_nbd_03 up/up 192.172.73.200/24 CustCluster4
            nfs_vm_gp_ev_dc02_os_index up/up 192.172.73.201/24 CustCluster1
            nfs_vm_gp_ev_dc03_os_index up/up 192.172.73.202/24 CustCluster2
            nfs_vm_gp_ev_dc01_os_index up/up 192.172.73.203/24 CustCluster1
            nfs_vm_gp_ev_dc01_vault up/up 192.172.73.204/24 CustCluster4
            nfs_vm_gp_ev_dc03_vault up/up 192.172.73.205/24 CustCluster4
            nfs_vm_gp_ev_dc04_vault up/up 192.172.73.206/24 CustCluster4
            nfs_vm_gp_ev_dc02_vault up/up 192.172.73.207/24 CustCluster4
            iscsi1_D1_e0e up/up   192.172.74.192/24    CustCluster1 
            iscsi1_D2_e0e up/up   192.172.74.193/24    CustCluster2 
            iscsi1_D3_e0e up/up   192.172.74.194/24    CustCluster3 
            iscsi1_D4_e0e up/up   192.172.74.195/24    CustCluster4 
            iscsi2_D1_e0f up/up   192.172.75.192/24    CustCluster1 
            iscsi2_D2_e0f up/up   192.172.75.193/24    CustCluster2 
            iscsi2_D3_e0f up/up   192.172.75.194/24    CustCluster3 
            iscsi2_D4_e0f up/up   192.172.75.195/24    CustCluster4 
CustSVM01
            cifs_ediscovery up/up 192.172.76.192/24    CustCluster4 
CustSVM02
            cifs_cust     up/up    192.172.76.193/24    CustCluster3 
            cifs_cust_utility up/up 192.172.76.194/24   CustCluster3 
            cifs_cust_COPS_G up/up 192.172.76.195/24    CustCluster3 
            cifs_cust_PST_archive up/up 192.172.76.196/24 CustCluster3
            cifs_cust_archive up/up 192.172.76.197/24   CustCluster3 
            cifs_cust_auditlogs up/up 192.172.76.198/24 CustCluster3 
            cifs_cust_citrix up/up 192.172.76.199/24    CustCluster3 
            cifs_cust_g_drive up/up 192.172.76.200/24   CustCluster3 
            cifs_cust_nya_public up/up 192.172.76.201/24 CustCluster3
            cifs_cust_ovw up/up    192.172.76.202/24    CustCluster3 
            cifs_cust_software up/up 192.172.76.203/24  CustCluster3 
            cifs_cust_sqlbackups up/up 192.172.76.204/24 CustCluster3
            cifs_cust_support up/up 192.172.76.205/24   CustCluster3 
            cifs_cust_wtr up/up    192.172.76.206/24    CustCluster3 
            cifs_eaprofile up/up  192.172.76.207/24    CustCluster1 
CustZServSVM01
            cifs_fsw_gp_evclu_dc01 up/up 192.172.76.208/24 CustCluster4
CustSVM02
            tmp_nbu      up/up    192.172.76.209/24    CustCluster1 
CustZServSVM01
            cifs_AT_PAASDB_DC00 up/up 192.172.76.210/24 CustCluster4
            cifs_fsw_jd1sw_ssawcl up/up 192.172.76.214/24 CustCluster4
CustZCluster
            mgmt_cluster up/up    192.172.239.192/24   CustCluster1 
            mgmt_D1      up/up    192.172.239.193/24   CustCluster1 
            mgmt_D2      up/up    192.172.239.194/24   CustCluster2 
            mgmt_D3      up/up    192.172.239.195/24   CustCluster3 
            mgmt_D4      up/up    192.172.239.196/24   CustCluster4 
CustSVM01
            mgmt_svm     up/up    192.172.239.205/24   CustCluster4 
CustSVM02
            mgmt_svm     up/up    192.172.239.206/24   CustCluster4 
CustZServSVM01
            mgmt_svm     up/up    192.172.239.207/24   CustCluster4 
CustZCluster
            snapmirror_D1 up/up   192.172.239.240/24   CustCluster1 
            snapmirror_D2 up/up   192.172.239.241/24   CustCluster2 
            snapmirror_D3 up/up   192.172.239.242/24   CustCluster3 
            snapmirror_D4 up/up   192.172.239.243/24   CustCluster4 
Cluster
            CustCluster1_clus1 up/up 169.254.5.203/16 CustCluster1
            CustCluster2_clus2 up/up 169.254.18.84/16 CustCluster2
            CustCluster4_clus1 up/up 169.254.33.135/16 CustCluster4
            CustCluster1_clus2 up/up 169.254.91.160/16 CustCluster1
            CustCluster4_clus2 up/up 169.254.142.203/16 CustCluster4
            CustCluster3_clus1 up/up 169.254.158.204/16 CustCluster3
            CustCluster2_clus1 up/up 169.254.196.87/16 CustCluster2
            CustCluster3_clus2 up/up 169.254.243.163/16 CustCluster3
64 entries were displayed.

CustCluster::*>

Notice the LIFs are now sorted in Address order! SVM names may be listed multiple times as the list, sorted by address jockeys back and forth between SVMs. Well, lets’ clean this up a little more. I want the same data but sorted by SVM first, then by address. That command would be: network interface show -sort-by vserver,address . Here is the actual output with the same data:

CustCluster::*> network interface show -sort-by vserver,address

            Logical    Status     Network            Current       
Vserver     Interface  Admin/Oper Address/Mask       Node          
----------- ---------- ---------- ------------------ ------------- 
Cluster
            CustCluster1_clus1 up/up 169.254.5.203/16 CustCluster1
            CustCluster2_clus2 up/up 169.254.18.84/16 CustCluster2
            CustCluster4_clus1 up/up 169.254.33.135/16 CustCluster4
            CustCluster1_clus2 up/up 169.254.91.160/16 CustCluster1
            CustCluster4_clus2 up/up 169.254.142.203/16 CustCluster4
            CustCluster3_clus1 up/up 169.254.158.204/16 CustCluster3
            CustCluster2_clus1 up/up 169.254.196.87/16 CustCluster2
            CustCluster3_clus2 up/up 169.254.243.163/16 CustCluster3
CustSVM01
            cifs_ediscovery up/up 192.172.76.192/24    CustCluster4 
            mgmt_svm     up/up    192.172.239.205/24   CustCluster4 
CustSVM02
            nfs_ISO      up/up    192.172.73.199/24    CustCluster3 
            cifs_cust     up/up    192.172.76.193/24    CustCluster3 
            cifs_cust_utility up/up 192.172.76.194/24   CustCluster3 
            cifs_cust_COPS_G up/up 192.172.76.195/24    CustCluster3 
            cifs_cust_PST_archive up/up 192.172.76.196/24 CustCluster3
            cifs_cust_archive up/up 192.172.76.197/24   CustCluster3 
            cifs_cust_auditlogs up/up 192.172.76.198/24 CustCluster3 
            cifs_cust_citrix up/up 192.172.76.199/24    CustCluster3 
            cifs_cust_g_drive up/up 192.172.76.200/24   CustCluster3 
            cifs_cust_nya_public up/up 192.172.76.201/24 CustCluster3
            cifs_cust_ovw up/up    192.172.76.202/24    CustCluster3 
            cifs_cust_software up/up 192.172.76.203/24  CustCluster3 
            cifs_cust_sqlbackups up/up 192.172.76.204/24 CustCluster3
            cifs_cust_support up/up 192.172.76.205/24   CustCluster3 
            cifs_cust_wtr up/up    192.172.76.206/24    CustCluster3 
            cifs_eaprofile up/up  192.172.76.207/24    CustCluster1 
            tmp_nbu      up/up    192.172.76.209/24    CustCluster1 
            mgmt_svm     up/up    192.172.239.206/24   CustCluster4 
CustZCluster
            mgmt_cluster up/up    192.172.239.192/24   CustCluster1 
            mgmt_D1      up/up    192.172.239.193/24   CustCluster1 
            mgmt_D2      up/up    192.172.239.194/24   CustCluster2 
            mgmt_D3      up/up    192.172.239.195/24   CustCluster3 
            mgmt_D4      up/up    192.172.239.196/24   CustCluster4 
            snapmirror_D1 up/up   192.172.239.240/24   CustCluster1 
            snapmirror_D2 up/up   192.172.239.241/24   CustCluster2 
            snapmirror_D3 up/up   192.172.239.242/24   CustCluster3 
            snapmirror_D4 up/up   192.172.239.243/24   CustCluster4 
CustZServSVM01
            nfs_D1       up/up    192.172.73.192/24    CustCluster1 
            nfs_D2       up/up    192.172.73.193/24    CustCluster2 
            nfs_D3       up/up    192.172.73.194/24    CustCluster3 
            nfs_D4       up/up    192.172.73.195/24    CustCluster4 
            nfs_vm_nfs_wa up/up   192.172.73.196/24    CustCluster2 
            nfs_vm_gp_ev_dc04_os_index up/up 192.172.73.197/24 CustCluster2
            nfs_vm_msdp_cc_nbd_02 up/up 192.172.73.198/24 CustCluster4
            nfs_vm_msdp_cc_nbd_03 up/up 192.172.73.200/24 CustCluster4
            nfs_vm_gp_ev_dc02_os_index up/up 192.172.73.201/24 CustCluster1
            nfs_vm_gp_ev_dc03_os_index up/up 192.172.73.202/24 CustCluster2
            nfs_vm_gp_ev_dc01_os_index up/up 192.172.73.203/24 CustCluster1
            nfs_vm_gp_ev_dc01_vault up/up 192.172.73.204/24 CustCluster4
            nfs_vm_gp_ev_dc03_vault up/up 192.172.73.205/24 CustCluster4
            nfs_vm_gp_ev_dc04_vault up/up 192.172.73.206/24 CustCluster4
            nfs_vm_gp_ev_dc02_vault up/up 192.172.73.207/24 CustCluster4
            iscsi1_D1_e0e up/up   192.172.74.192/24    CustCluster1 
            iscsi1_D2_e0e up/up   192.172.74.193/24    CustCluster2 
            iscsi1_D3_e0e up/up   192.172.74.194/24    CustCluster3 
            iscsi1_D4_e0e up/up   192.172.74.195/24    CustCluster4 
            iscsi2_D1_e0f up/up   192.172.75.192/24    CustCluster1 
            iscsi2_D2_e0f up/up   192.172.75.193/24    CustCluster2 
            iscsi2_D3_e0f up/up   192.172.75.194/24    CustCluster3 
            iscsi2_D4_e0f up/up   192.172.75.195/24    CustCluster4 
            cifs_fsw_gp_evclu_dc01 up/up 192.172.76.208/24 CustCluster4
            cifs_AT_PAASDB_DC00 up/up 192.172.76.210/24 CustCluster4
            cifs_fsw_jd1sw_ssawcl up/up 192.172.76.214/24 CustCluster4
            mgmt_svm     up/up    192.172.239.207/24   CustCluster4  
64 entries were displayed.

CustCluster::*>

All done?

When you are finished, do not forget to either logout, or set your privilege mode back to admin: set -privilege admin