Oracle ASM day-to-day operations and failure scenarios

Day-to-day operation of ASM disk groups in an installed Oracle Restart or RAC environment, and what to do when something fails: capacity check (usable_file_mb), rebalance monitoring, adding/replacing/dropping disks, offline disk, new disk group, mount/dismount, ASM configuration backup and discovery string. It does not install ASM (installation: the Oracle Restart or RAC guide); if you search for an error code (e.g. ORA-15041, ORA-15042), the matching scenario opens. Every scenario follows the pattern diagnosis → safety net → execution → verification → follow-up; the database stays open (except for dismount).

Topology: Oracle Restart (single server) or 2–8 node RAC, Grid Infrastructure 19c (23ai/26ai differences in the notes)

Scenarios

• Capacity and disk group health check — Changes nothing: is a disk group full, how much space is left (the correct reading of usable_file_mb for each redundancy), offline disks, disk imbalance and compatibility attributes. The first card to pick for a space problem, an alert or a routine check.
• Monitor, speed up or slow down a rebalance — Progress and remaining time of an ongoing rebalance/resync (v$asm_operation, est_minutes) and changing the POWER. The database stays open; only the I/O load changes.
• Add a new disk to a disk group — The disk group is running low on space or needs to grow: prepare the new LUN on all nodes (multipath/udev/ASMLib/AFD), add it with ADD DISK and monitor the rebalance. The database stays open.
• Replace a disk with a new one (REPLACE DISK) — Replaces a failed (offline) disk or a disk due for renewal with a new one in a single operation; only that disk's data is copied (12.1+, NORMAL/HIGH). With EXTERNAL, add + drop is done in one command. The database stays open.
• Drop a disk from a disk group — To return a LUN or remove it from the storage: with DROP DISK the data is moved to the remaining disks; the disk is not detached until the rebalance has finished and the disk is FORMER. The database stays open.
• A disk is offline or failed (NORMAL/HIGH redundancy) — In a NORMAL/HIGH disk group a disk shows as OFFLINE, or the alert log has write/read errors: diagnosis, the disk_repair_time window, ONLINE DISK (fast mirror resync) if the path is fixed, extending the window if the repair will take longer. The database stays open.
• Create a new disk group — To create a new data, FRA or application disk group: redundancy, AU size, compatible.asm/rdbms attributes; on RAC, mount on all nodes and the Clusterware resource. Existing disk groups are not affected.
• Mount or dismount a disk group — To take a disk group down in a planned way (dismount) or to bring up a disk group that is down or will not open (mount): with srvctl start/stop diskgroup on all nodes. For a dismount no database may be using the disk group; if it will not mount, the scenario guides you by error code.
• ASM spfile, password file and metadata backup — No downtime: the ASM spfile, the password file, the disk group structure (md_backup) and, on RAC, the GPnP profile are backed up outside ASM, so that the ASM configuration can be rebuilt even if the disk groups are lost. Run it after the installation and after every structural change.
• Change the ASM disk discovery string (asm_diskstring) — To change the path pattern ASM uses to look for disks (e.g. moving to multipath, a new udev directory, ASMLib/AFD): first test with kfod, then asmcmd dsset. All existing disks (including OCR/voting) must also be found with the new string.

Before you start

• You have root access and a login shell as the grid user; the grid profile sets ORACLE_HOME=… and ORACLE_SID (+ASM…N…). The SQL steps run with sqlplus / as sysasm.
• The storage team has presented the new LUNs …to all nodes…; they are the same size as the existing disks; you have the WWIDs (or the VM disk serial numbers); the disks are empty (no file system, LVM or old ASM header).
• There is a current RMAN backup of the databases in the disk group; because the operation creates I/O load, an off-peak time has been chosen.
• No other rebalance is in progress on … (do not start two large operations at the same time).

Phases of the guide

• Capacity and health check — Show disk group usage · Check disk balance and state · Show compatibility and repair attributes · Usage per disk group and ongoing operations
• Rebalance monitoring and power setting — Show ongoing operations and the power setting · Change the rebalance power · Monitor the rebalance until it finishes
• Adding a disk: diagnosis — Show the state of the … disk group
• Preparing the new disks — Scan for and identify the new LUNs… (…)… · Scan for and identify the new LUNs… (…)… · Define the multipath aliases · Add the new disks to the udev rule · Create a partition on the disks and write the ASMLib label… (node 1 only)… · Read the labels on the other nodes (…) …
• Adding a disk: execution — Add the disks to the … disk group · Monitor the rebalance until it finishes
• Adding a disk: verification — Verify that the new disks are members · Follow-up
• Replacing a disk: diagnosis — Show the state of the … disk group · Show the state of the disk to replace
• Preparing the new disks — Scan for and identify the new LUNs… (…)… · Scan for and identify the new LUNs… (…)… · Define the multipath aliases · Add the new disks to the udev rule · Create a partition on the disks and write the ASMLib label… (node 1 only)… · Read the labels on the other nodes (…) …
• Replacing a disk: execution — If the old disk is still ONLINE, take it offline · Replace the disk with the new one · Add the new disk and drop the old one (EXTERNAL) · Monitor the rebalance until it finishes
• Replacing a disk: verification and follow-up — Verify the replacement · Check the old disk's header
• Dropping a disk: diagnosis — Show the state of the … disk group · Will there be enough space after the drop?
• Dropping a disk: execution — Drop the disks from … · Monitor the rebalance until it finishes
• Dropping a disk: verification and removal from the operating system — Verify that the dropped disks are FORMER · Remove the disks from the operating system
• Failed disk: diagnosis — Show the offline disks and the remaining time · Check the disk path in the operating system… (…)… · Check the disk path in the operating system · Search the ASM alert log for disk events
• Failed disk: bring the disk back (ONLINE) — Bring the disk online · Monitor the resync until it finishes · Verify that the disk is online
• Failed disk: extend the drop timer — Give the offline disk a new timer · After the repair
• New disk group: diagnosis — Show the existing disk groups and the candidate disks
• Preparing the new disks — Scan for and identify the new LUNs… (…)… · Scan for and identify the new LUNs… (…)… · Define the multipath aliases · Add the new disks to the udev rule · Create a partition on the disks and write the ASMLib label… (node 1 only)… · Read the labels on the other nodes (…) …
• New disk group: creation — Create the … disk group · Mount the disk group on all nodes · Verify the disk group resource
• New disk group: follow-up — Put the disk group into use
• Mount / dismount: diagnosis — Show the disk group's state and its users · Show the Clusterware resource
• ASM spfile, password file and metadata backup — Show the location of the ASM spfile and password file · Create the backup directory · Take the backups · Copy the backups off the server
• Discovery string: diagnosis — Show the current string and the paths of the member disks · Verify that the new string finds all member disks
• Discovery string: execution and verification — Change the discovery string · Verify on all ASM instances