Open3

NVMe-oFのお勉強

Hiroyuki SatoHiroyuki Sato

Use a file to emulate nvme device を実行しただけ。

環境

  • OS: AlmaLinux 8
  • RPM
    • nvmetcli-0.7-3.el8.noarch
    • nvme-cli-1.16-5.el8.x86_64

準備

ダミーのディスクを作る

mkdir /var/nvmetest
dd if=/dev/zero of=/var/nvmetest/nvmedisk1 bs=1GB count=1
1+0 records in
1+0 records out
1000000000 bytes (1.0 GB, 954 MiB) copied, 0.323693 s, 3.1 GB/s
losetup

作成したディスクを /dev/loop0 にアタッチする

losetup /dev/loop0 /var/nvmetest/nvmedisk1

確認

losetup
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE               DIO LOG-SEC
/dev/loop0         0      0         0  0 /var/nvmetest/nvmedisk1   0     512

nvme関連のモジュールをカーネルに追加する

modprobe nvme_loop
modprobe nvmet
modprove nvme_fabrics

確認

lsmod | grep nvme
nvme_loop              20480  0
nvmet                 114688  1 nvme_loop
nvme_fabrics           24576  1 nvme_loop
nvme                   45056  4
nvme_core             139264  8 nvme,nvme_loop,nvme_fabrics
t10_pi                 16384  2 nvmet,nvme_core

設定ファイルを用意する。

{
  "hosts": [
    {
      "nqn": "hostnqn"
    }
  ], 
  "ports": [
    {
      "addr": {
        "adrfam": "", 
        "traddr": "", 
        "treq": "not specified", 
        "trsvcid": "", 
        "trtype": "loop"
      }, 
      "portid": 1, 
      "referrals": [], 
      "subsystems": [
        "testnqn"
      ]
    }
  ], 
  "subsystems": [
    {
      "allowed_hosts": [
        "hostnqn"
      ], 
      "attr": {
        "allow_any_host": "0"
      }, 
      "namespaces": [
        {
          "device": {
            "nguid": "7ee38388-53be-4cc7-82b1-3c797896e876", 
            "path": "/dev/loop0"
          }, 
          "enable": 1, 
          "nsid": 1
        }
      ], 
      "nqn": "testnqn"
    }
  ]
}

設定のロード

nvmetcli restore /tmp/nvme.json

確認

dmesg | tail
[1327458.905933] nvmet: adding nsid 1 to subsystem testnqn

NVMe-oF実行前の確認

lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0         7:0    0 953.7M  0 loop
nvme0n1     259:0    0   1.8T  0 disk
|-nvme0n1p1 259:1    0   200M  0 part /boot/efi
|-nvme0n1p2 259:2    0     1G  0 part /boot
|-nvme0n1p3 259:3    0     8G  0 part [SWAP]
`-nvme0n1p4 259:4    0   1.8T  0 part /

loopデバイスの、NVMeディスクを接続する。

nvme connect -t loop -n testnqn -q hostnqn

確認 nvme1n1 が見える

lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0         7:0    0 953.7M  0 loop
nvme0n1     259:0    0   1.8T  0 disk
|-nvme0n1p1 259:1    0   200M  0 part /boot/efi
|-nvme0n1p2 259:2    0     1G  0 part /boot
|-nvme0n1p3 259:3    0     8G  0 part [SWAP]
`-nvme0n1p4 259:4    0   1.8T  0 part /
nvme1n1     259:5    0 953.7M  0 disk

パーティションの作成

parted /dev/nvme1n1
GNU Parted 3.2
Using /dev/nvme1n1
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Error: /dev/nvme1n1: unrecognised disk label
Model: NVMe Device (nvme)
Disk /dev/nvme1n1: 1000MB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:
(parted) mklabel
New disk label type? gpt
(parted) mkpart
Partition name?  []?
File system type?  [ext2]? xfs
Start? 0%
End? 100%
(parted) p
Model: NVMe Device (nvme)
Disk /dev/nvme1n1: 1000MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End    Size   File system  Name  Flags
 1      1049kB  999MB  998MB  xfs

(parted) quit
Information: You may need to update /etc/fstab.

確認

lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0         7:0    0 953.7M  0 loop
nvme0n1     259:0    0   1.8T  0 disk
|-nvme0n1p1 259:1    0   200M  0 part /boot/efi
|-nvme0n1p2 259:2    0     1G  0 part /boot
|-nvme0n1p3 259:3    0     8G  0 part [SWAP]
`-nvme0n1p4 259:4    0   1.8T  0 part /
nvme1n1     259:5    0 953.7M  0 disk
`-nvme1n1p1 259:7    0   952M  0 part

ファイルシステムの作成

mkfs.xfs /dev/nvme1n1p1
meta-data=/dev/nvme1n1p1         isize=512    agcount=4, agsize=60928 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=243712, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=1566, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Discarding blocks...Done.

つづく

Hiroyuki SatoHiroyuki Sato

nvmetclinvme コマンド

nvmetcli

nvmetcli help
syntax: /usr/sbin/nvmetcli save [file_to_save_to]
        /usr/sbin/nvmetcli restore [file_to_restore_from]
        /usr/sbin/nvmetcli clear
        /usr/sbin/nvmetcli ls

nvme

nvme
nvme-1.16
usage: nvme <command> [<device>] [<args>]

The '<device>' may be either an NVMe character device (ex: /dev/nvme0) or an
nvme block device (ex: /dev/nvme0n1).

The following are all implemented sub-commands:
  list                      List all NVMe devices and namespaces on machine
  list-subsys               List nvme subsystems
  id-ctrl                   Send NVMe Identify Controller
  id-ns                     Send NVMe Identify Namespace, display structure
  id-ns-granularity         Send NVMe Identify Namespace Granularity List, display structure
  list-ns                   Send NVMe Identify List, display structure
  list-ctrl                 Send NVMe Identify Controller List, display structure
  nvm-id-ctrl               Send NVMe Identify Controller NVM Command Set, display structure
  primary-ctrl-caps         Send NVMe Identify Primary Controller Capabilities
  list-secondary            List Secondary Controllers associated with a Primary Controller
  cmdset-ind-id-ns          I/O Command Set Independent Identify Namespace
  ns-descs                  Send NVMe Namespace Descriptor List, display structure
  id-nvmset                 Send NVMe Identify NVM Set List, display structure
  id-uuid                   Send NVMe Identify UUID List, display structure
  id-iocs                   Send NVMe Identify I/O Command Set, display structure
  id-domain                 Send NVMe Identify Domain List, display structure
  list-endgrp               Send NVMe Identify Endurance Group List, display structure
  create-ns                 Creates a namespace with the provided parameters
  delete-ns                 Deletes a namespace from the controller
  attach-ns                 Attaches a namespace to requested controller(s)
  detach-ns                 Detaches a namespace from requested controller(s)
  get-ns-id                 Retrieve the namespace ID of opened block device
  get-log                   Generic NVMe get log, returns log in raw format
  telemetry-log             Retrieve FW Telemetry log write to file
  fw-log                    Retrieve FW Log, show it
  changed-ns-list-log       Retrieve Changed Namespace List, show it
  smart-log                 Retrieve SMART Log, show it
  ana-log                   Retrieve ANA Log, show it
  error-log                 Retrieve Error Log, show it
  effects-log               Retrieve Command Effects Log, show it
  endurance-log             Retrieve Endurance Group Log, show it
  predictable-lat-log       Retrieve Predictable Latency per Nvmset Log, show it
  pred-lat-event-agg-log    Retrieve Predictable Latency Event Aggregate Log, show it
  persistent-event-log      Retrieve Presistent Event Log, show it
  endurance-event-agg-log   Retrieve Endurance Group Event Aggregate Log, show it
  lba-status-log            Retrieve LBA Status Information Log, show it
  resv-notif-log            Retrieve Reservation Notification Log, show it
  boot-part-log             Retrieve Boot Partition Log, show it
  get-feature               Get feature and show the resulting value
  device-self-test          Perform the necessary tests to observe the performance
  self-test-log             Retrieve the SELF-TEST Log, show it
  supported-log-pages       Retrieve the Supported Log pages details, show it
  set-feature               Set a feature and show the resulting value
  set-property              Set a property and show the resulting value
  get-property              Get a property and show the resulting value
  format                    Format namespace with new block format
  fw-commit                 Verify and commit firmware to a specific slot (fw-activate in old version < 1.2)
  fw-download               Download new firmware
  admin-passthru            Submit an arbitrary admin command, return results
  io-passthru               Submit an arbitrary IO command, return results
  security-send             Submit a Security Send command, return results
  security-recv             Submit a Security Receive command, return results
  get-lba-status            Submit a Get LBA Status command, return results
  capacity-mgmt             Submit Capacity Management Command, return results
  resv-acquire              Submit a Reservation Acquire, return results
  resv-register             Submit a Reservation Register, return results
  resv-release              Submit a Reservation Release, return results
  resv-report               Submit a Reservation Report, return results
  dsm                       Submit a Data Set Management command, return results
  copy                      Submit a Simple Copy command, return results
  flush                     Submit a Flush command, return results
  compare                   Submit a Compare command, return results
  read                      Submit a read command, return results
  write                     Submit a write command, return results
  write-zeroes              Submit a write zeroes command, return results
  write-uncor               Submit a write uncorrectable command, return results
  verify                    Submit a verify command, return results
  sanitize                  Submit a sanitize command
  sanitize-log              Retrieve sanitize log, show it
  reset                     Resets the controller
  subsystem-reset           Resets the subsystem
  ns-rescan                 Rescans the NVME namespaces
  show-regs                 Shows the controller registers or properties. Requires character device
  discover                  Discover NVMeoF subsystems
  connect-all               Discover and Connect to NVMeoF subsystems
  connect                   Connect to NVMeoF subsystem
  disconnect                Disconnect from NVMeoF subsystem
  disconnect-all            Disconnect from all connected NVMeoF subsystems
  gen-hostnqn               Generate NVMeoF host NQN
  show-hostnqn              Show NVMeoF host NQN
  dir-receive               Submit a Directive Receive command, return results
  dir-send                  Submit a Directive Send command, return results
  virt-mgmt                 Manage Flexible Resources between Primary and Secondary Controller
  rpmb                      Replay Protection Memory Block commands
  fid-support-effects-log   Submit Feature ID Support and Effects Log, Return result
  lockdown                  Submit a Lockdown command,return result
  version                   Shows the program version
  help                      Display this help

See 'nvme help <command>' for more information on a specific command

The following are all installed plugin extensions:
  intel           Intel vendor specific extensions
  amzn            Amazon vendor specific extensions
  lnvm            LightNVM specific extensions
  memblaze        Memblaze vendor specific extensions
  wdc             Western Digital vendor specific extensions
  huawei          Huawei vendor specific extensions
  netapp          NetApp vendor specific extensions
  toshiba         Toshiba NVME plugin
  micron          Micron vendor specific extensions
  seagate         Seagate vendor specific extensions
  virtium         Virtium vendor specific extensions
  shannon         Shannon vendor specific extensions
  dera            Dera vendor specific extensions
  sfx             ScaleFlux vendor specific extensions
  transcend       Transcend vendor specific extensions
  zns             Zoned Namespace Command Set
  nvidia          NVIDIA vendor specific extensions
  ymtc            Ymtc vendor specific extensions

See 'nvme <plugin> help' for more information on a plugin
Hiroyuki SatoHiroyuki Sato

コマンドを動かしてみる(その1)

nvme list

nvme list
Node                  SN                   Model                                    Namespace Usage                      Format           FW Rev
--------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1                                                                        1           0.00   B /   0.00   B      1   B +  0 B
/dev/nvme1n1                                                                        1           0.00   B /   0.00   B      1   B +  0 B

nvme list-subsys

nvme list-subsys
nvme-subsys0 - NQN=nqn.2018-01.com.wdc:nguid:21393D35-F200-4D61-9A21-46ED3B84A226(dummy)
\
 +- nvme0 pcie 0000:01:00.0 live
nvme-subsys1 - NQN=testnqn
\
 +- nvme1 loop  live

nvme id-ctrl (要: root権限)

nvme id-ctrl /dev/nvme1n1
NVME Identify Controller:
vid       : 0
ssvid     : 0
sn        : dab33544bd7d67a07729
mn        : Linux
fr        : 4.18.0-4
rab       : 6
ieee      : 000000
cmic      : 0xb
mdts      : 0
cntlid    : 0x1
ver       : 0x10300
rtd3r     : 0
rtd3e     : 0
oaes      : 0x900
ctratt    : 0x41
rrls      : 0
cntrltype : 0
fguid     :
crdt1     : 0
crdt2     : 0
crdt3     : 0
nvmsr     : 0
vwci      : 0
mec       : 0
oacs      : 0
acl       : 3
aerl      : 3
frmw      : 0x3
lpa       : 0x7
elpe      : 127
npss      : 0
avscc     : 0
apsta     : 0
wctemp    : 0
cctemp    : 0
mtfa      : 0
hmpre     : 0
hmmin     : 0
tnvmcap   : 0
unvmcap   : 0
rpmbs     : 0
edstt     : 0
dsto      : 0
fwug      : 0
kas       : 10
hctma     : 0
mntmt     : 0
mxtmt     : 0
sanicap   : 0
hmminds   : 0
hmmaxd    : 0
nsetidmax : 0
endgidmax : 0
anatt     : 10
anacap    : 31
anagrpmax : 128
nanagrpid : 128
pels      : 0
domainid  : 0
megcap    : 0
sqes      : 0x66
cqes      : 0x44
maxcmd    : 1024
nn        : 1024
oncs      : 0xc
fuses     : 0
fna       : 0
vwc       : 0x1
awun      : 0
awupf     : 0
icsvscc     : 0
nwpc      : 1
acwu      : 0
ocfs      : 0
sgls      : 0x1
mnan      : 1024
maxdna    : 0
maxcna    : 0
subnqn    : testnqn
ioccsz    : 4
iorcsz    : 1
icdoff    : 0
fcatt     : 0
msdbd     : 0
ofcs      : 0
ps    0 : mp:25.00W operational enlat:16 exlat:4 rrt:0 rrl:0
          rwt:0 rwl:0 idle_power:- active_power:-

jsonでも出せる

nvme id-ctrl /dev/nvme1n1 -o json
{
  "vid" : 0,
  "ssvid" : 0,
  "sn" : "dab33544bd7d67a07729",
  "mn" : "Linux                                   ",
  "fr" : "4.18.0-4",
  "rab" : 6,
  "ieee" : 0,
  "cmic" : 11,
  "mdts" : 0,
  "cntlid" : 1,
  "ver" : 66304,
  "rtd3r" : 0,
  "rtd3e" : 0,
  "oaes" : 2304,
  "ctratt" : 65,
  "rrls" : 0,
  "crdt1" : 0,
  "crdt2" : 0,
  "crdt3" : 0,
  "nvmsr" : 0,
  "vwci" : 0,
  "mec" : 0,
  "oacs" : 0,
  "acl" : 3,
  "aerl" : 3,
  "frmw" : 3,
  "lpa" : 7,
  "elpe" : 127,
  "npss" : 0,
  "avscc" : 0,
  "apsta" : 0,
  "wctemp" : 0,
  "cctemp" : 0,
  "mtfa" : 0,
  "hmpre" : 0,
  "hmmin" : 0,
  "tnvmcap" : 0,
  "unvmcap" : 0,
  "rpmbs" : 0,
  "edstt" : 0,
  "dsto" : 0,
  "fwug" : 0,
  "kas" : 10,
  "hctma" : 0,
  "mntmt" : 0,
  "mxtmt" : 0,
  "sanicap" : 0,
  "hmminds" : 0,
  "hmmaxd" : 0,
  "nsetidmax" : 0,
  "anatt" : 10,
  "anacap" : 31,
  "anagrpmax" : 128,
  "nanagrpid" : 128,
  "domainid" : 0,
  "megcap" : 0,
  "sqes" : 102,
  "cqes" : 68,
  "maxcmd" : 1024,
  "nn" : 1024,
  "oncs" : 12,
  "fuses" : 0,
  "fna" : 0,
  "vwc" : 1,
  "awun" : 0,
  "awupf" : 0,
  "icsvscc" : 0,
  "nwpc" : 1,
  "acwu" : 0,
  "ocfs" : 0,
  "sgls" : 1,
  "maxdna" : 0,
  "maxcna" : 0,
  "subnqn" : "testnqn",
  "ioccsz" : 4,
  "iorcsz" : 1,
  "icdoff" : 0,
  "fcatt" : 0,
  "msdbd" : 0,
  "ofcs" : 0,
  "psds" : [
    {
      "max_power" : 2500,
      "flags" : 0,
      "entry_lat" : 16,
      "exit_lat" : 4,
      "read_tput" : 0,
      "read_lat" : 0,
      "write_tput" : 0,
      "write_lat" : 0,
      "idle_power" : 0,
      "idle_scale" : 0,
      "active_power" : 0,
      "active_work_scale" : 0
    }
  ]
}

nvme id-ns (要: root権限)

nvme id-ns /dev/nvme1n1
NVME Identify Namespace 1:
nsze    : 0x1dcd65
ncap    : 0x1dcd65
nuse    : 0x1dcd65
nsfeat  : 0x12
nlbaf   : 0
flbas   : 0
mc      : 0
dpc     : 0
dps     : 0
nmic    : 0x1
rescap  : 0
fpi     : 0
dlfeat  : 0
nawun   : 0
nawupf  : 0
nacwu   : 0
nabsn   : 0
nabo    : 0
nabspf  : 0
noiob   : 0
nvmcap  : 0
npwg    : 0
npwa    : 0
npdg    : 7
npda    : 7
nows    : 0
mssrl   : 0
mcl     : 0
msrc    : 0
anagrpid: 1
nsattr	: 0
nvmsetid: 0
endgid  : 0
nguid   : 7ee3838853be4cc782b13c797896e876
eui64   : 0000000000000000
lbaf  0 : ms:0   lbads:9  rp:0 (in use)

人間が読む時には、-Hオプションをつける

nvme id-ctrl /dev/nvme1n1 -H
NVME Identify Controller:
vid       : 0
ssvid     : 0
sn        : dab33544bd7d67a07729
mn        : Linux
fr        : 4.18.0-4
rab       : 6
ieee      : 000000
cmic      : 0xb
  [3:3] : 0x1	ANA supported
  [2:2] : 0	PCI
  [1:1] : 0x1	Multi Controller
  [0:0] : 0x1	Multi Port

mdts      : 0
cntlid    : 0x1
ver       : 0x10300
rtd3r     : 0
rtd3e     : 0
oaes      : 0x900
  [31:31] : 0	Discovery Log Change Notice Not Supported
  [27:27] : 0	Zone Descriptor Changed Notices Not Supported
  [15:15] : 0	Normal NSS Shutdown Event Not Supported
  [14:14] : 0	Endurance Group Event Aggregate Log Page Change Notice Not Supported
  [13:13] : 0	LBA Status Information Notices Not Supported
  [12:12] : 0	Predictable Latency Event Aggregate Log Change Notices Not Supported
  [11:11] : 0x1	Asymmetric Namespace Access Change Notices Supported
  [9:9] : 0	Firmware Activation Notices Not Supported
  [8:8] : 0x1	Namespace Attribute Changed Event Supported

ctratt    : 0x41
  [15:15] : 0	Extended LBA Formats Not Supported
  [14:14] : 0	Delete NVM Set Not Supported
  [13:13] : 0	Delete Endurance Group Not Supported
  [12:12] : 0	Variable Capacity Management Not Supported
  [11:11] : 0	Fixed Capacity Management Not Supported
  [10:10] : 0	Multi Domain Subsystem Not Supported
  [9:9] : 0	UUID List Not Supported
  [7:7] : 0	Namespace Granularity Not Supported
 [6:6] : 0x1	Reserved
  [5:5] : 0	Predictable Latency Mode Not Supported
  [4:4] : 0	Endurance Groups Not Supported
  [3:3] : 0	Read Recovery Levels Not Supported
  [2:2] : 0	NVM Sets Not Supported
  [1:1] : 0	Non-Operational Power State Permissive Not Supported
  [0:0] : 0x1	128-bit Host Identifier Supported

rrls      : 0
cntrltype : 0
  [7:2] : 0	Reserved
  [1:0] : 0	Controller type not reported
fguid     :
crdt1     : 0
crdt2     : 0
crdt3     : 0
nvmsr     : 0
  [1:1] : 0	NVM subsystem Not part of an Enclosure
  [0:0] : 0	NVM subsystem Not part of an Storage Device

vwci      : 0
  [7:7] : 0	VPD Write Cycles Remaining field is Not valid.
  [6:0] : 0	VPD Write Cycles Remaining

mec       : 0
  [1:1] : 0	NVM subsystem Not contains a Management Endpoint on a PCIe port
  [0:0] : 0	NVM subsystem Not contains a Management Endpoint on an SMBus/I2C port

oacs      : 0
  [10:10] : 0	Lockdown Command and Feature Not Supported
  [9:9] : 0	Get LBA Status Capability Not Supported
  [8:8] : 0	Doorbell Buffer Config Not Supported
  [7:7] : 0	Virtualization Management Not Supported
  [6:6] : 0	NVMe-MI Send and Receive Not Supported
  [5:5] : 0	Directives Not Supported
  [4:4] : 0	Device Self-test Not Supported
  [3:3] : 0	NS Management and Attachment Not Supported
  [2:2] : 0	FW Commit and Download Not Supported
  [1:1] : 0	Format NVM Not Supported
  [0:0] : 0	Security Send and Receive Not Supported

acl       : 3
aerl      : 3
frmw      : 0x3
  [5:5] : 0	Multiple FW or Boot Update Detection Not Supported
  [4:4] : 0	Firmware Activate Without Reset Not Supported
  [3:1] : 0x1	Number of Firmware Slots
  [0:0] : 0x1	Firmware Slot 1 Read-Only

lpa       : 0x7
  [6:6] : 0	Telemetry Log Data Area 4 Not Supported
  [5:5] : 0	LID 0x0, Scope of each command in LID 0x5, 0x12, 0x13 Not Supported
  [4:4] : 0	Persistent Event log Not Supported
  [3:3] : 0	Telemetry host/controller initiated log page Not Supported
  [2:2] : 0x1	Extended data for Get Log Page Supported
  [1:1] : 0x1	Command Effects Log Page Supported
  [0:0] : 0x1	SMART/Health Log Page per NS Supported

elpe      : 127
npss      : 0
avscc     : 0
  [0:0] : 0	Admin Vendor Specific Commands uses Vendor Specific Format

apsta     : 0
  [0:0] : 0	Autonomous Power State Transitions Not Supported

wctemp    : 0
 [16:0] : -273 C (0 Kelvin)	Warning temperature (WCTEMP)

cctemp    : 0
 [16:0] : -273 C (0 Kelvin)	Critical temperature (CCTEMP)

mtfa      : 0
hmpre     : 0
hmmin     : 0
tnvmcap   : 0
unvmcap   : 0
rpmbs     : 0
 [31:24]: 0	Access Size
 [23:16]: 0	Total Size
  [5:3] : 0	Authentication Method
  [2:0] : 0	Number of RPMB Units

edstt     : 0
dsto      : 0
fwug      : 0
kas       : 10
hctma     : 0
  [0:0] : 0	Host Controlled Thermal Management Not Supported

mntmt     : 0
mxtmt     : 0
sanicap   : 0
  [31:30] : 0	Additional media modification after sanitize operation completes successfully is not defined
  [29:29] : 0	No-Deallocate After Sanitize bit in Sanitize command Supported
    [2:2] : 0	Overwrite Sanitize Operation Not Supported
    [1:1] : 0	Block Erase Sanitize Operation Not Supported
    [0:0] : 0	Crypto Erase Sanitize Operation Not Supported

hmminds   : 0
hmmaxd    : 0
nsetidmax : 0
endgidmax : 0
anatt     : 10
anacap    : 31
  [7:7] : 0	Non-zero group ID Not Supported
  [6:6] : 0	Group ID does not change
  [4:4] : 0x1	ANA Change state Supported
  [3:3] : 0x1	ANA Persistent Loss state Supported
  [2:2] : 0x1	ANA Inaccessible state Supported
  [1:1] : 0x1	ANA Non-optimized state Supported
  [0:0] : 0x1	ANA Optimized state Supported

anagrpmax : 128
nanagrpid : 128
pels      : 0
domainid  : 0
megcap    : 0
sqes      : 0x66
  [7:4] : 0x6	Max SQ Entry Size (64)
  [3:0] : 0x6	Min SQ Entry Size (64)

cqes      : 0x44
  [7:4] : 0x4	Max CQ Entry Size (16)
  [3:0] : 0x4	Min CQ Entry Size (16)

maxcmd    : 1024
nn        : 1024
oncs      : 0xc
  [8:8] : 0	Copy Not Supported
  [7:7] : 0	Verify Not Supported
  [6:6] : 0	Timestamp Not Supported
  [5:5] : 0	Reservations Not Supported
  [4:4] : 0	Save and Select Not Supported
  [3:3] : 0x1	Write Zeroes Supported
  [2:2] : 0x1	Data Set Management Supported
  [1:1] : 0	Write Uncorrectable Not Supported
  [0:0] : 0	Compare Not Supported

fuses     : 0
  [0:0] : 0	Fused Compare and Write Not Supported

fna       : 0
  [3:3] : 0	FormatNVM Broadcast NSID (FFFFFFFFh) Supported
  [2:2] : 0	Crypto Erase Not Supported as part of Secure Erase
  [1:1] : 0	Crypto Erase Applies to Single Namespace(s)
  [0:0] : 0	Format Applies to Single Namespace(s)

vwc       : 0x1
  [2:1] : 0	Support for the NSID field set to FFFFFFFFh is not indicated
  [0:0] : 0x1	Volatile Write Cache Present

awun      : 0
awupf     : 0
icsvscc     : 0
  [0:0] : 0	NVM Vendor Specific Commands uses Vendor Specific Format

nwpc      : 1
  [2:2] : 0	Permanent Write Protect Not Supported
  [1:1] : 0	Write Protect Until Power Supply Not Supported
  [0:0] : 0x1	No Write Protect and Write Protect Namespace Supported

acwu      : 0
ocfs      : 0
  [1:1] : 0	Controller Copy Format 1h Not Supported
  [0:0] : 0	Controller Copy Format 0h Not Supported

sgls      : 0x1
 [21:21]: 0	Transport SGL Data Block Descriptor Not Supported
 [20:20]: 0	Address Offsets Not Supported
 [19:19]: 0	Metadata Pointer Containing SGL Descriptor is Not Supported
 [18:18]: 0	SGL Length Larger than Buffer Not Supported
 [17:17]: 0	Byte-Aligned Contig. MD Buffer Not Supported
 [16:16]: 0	SGL Bit-Bucket Not Supported
 [15:8] : 0	SGL Descriptor Threshold
  [2:2] : 0	Keyed SGL Data Block descriptor Not Supported
  [1:0] : 0x1	Scatter-Gather Lists Supported. No Dword alignment required.

mnan      : 1024
maxdna    : 0
maxcna    : 0
subnqn    : testnqn
ioccsz    : 4
iorcsz    : 1
icdoff    : 0
fcatt     : 0
  [0:0] : 0	Dynamic Controller Model

msdbd     : 0
ofcs      : 0
  [0:0] : 0	Disconnect command Not Supported

ps    0 : mp:25.00W operational enlat:16 exlat:4 rrt:0 rrl:0
          rwt:0 rwl:0 idle_power:- active_power:-

nvme list-ctrl

nvme list-ctrl /dev/nvme1n1
NVMe status: INVALID_FIELD: A reserved coded value or an unsupported value in a defined field(0x6002)