Exadata: Usage of script with location and description

 Exadata: Usage of script with location and description

Below are details about exadata Storage Server, Compute Node, Infiband Switches, IB Switches,Network, Hardware detials, Imageinfo and Logs location details

Utility Path	Usage/Comments
Infiniband	Some of these tools may be found in /opt/oracle.SupportTools/ibdiagtools on cells or database servers. Also see the  Infiniband Triage wiki page.
/opt/oracle.SupportTools/ibdiagtools/infinicheck
/opt/oracle.SupportTools/ibdiagtools/verify-topology
ibquery errors
/usr/bin/ibdiagnet	Detecting fabric issues
/usr/sbin/ibaddr	Examining HCA state & guids
/usr/sbin/ibcheckerrors	Detecting fabric issues
/usr/sbin/ibcheckerrs	Detecting fabric issues
/usr/sbin/ibcheckstate	Detecting fabric issues
/usr/sbin/ibcheckwidth	Detecting fabric issues
/usr/sbin/ibclearcounters	Reset counters when detecting fabric issues
/usr/sbin/ibclearerrors	Reset counters when detecting fabric issues
/usr/sbin/ibdatacounters	Not directly used. perfquery is used instead
/usr/sbin/ibdatacounts	Not directly used. perfquery is used instead
/usr/sbin/ibhosts	Lising cells/db nodes
/usr/sbin/iblinkinfo.pl	Obtaining the fabric topology
/usr/sbin/ibnetdiscover	Obtaining the fabric topology
/usr/sbin/ibnodes	Lising cells/db nodes/switches
/usr/sbin/ibping	Checking IB level connectivity
/usr/sbin/ibportstate	Testing port failure/disabling bad links
/usr/sbin/ibqueryerrors.pl	Detecting fabric issues
/usr/sbin/ibstat	Examining HCA state & guids
/usr/sbin/ibstatus	Examining HCA state & guids
/usr/sbin/ibswitches	Listing IB switch names
/usr/sbin/ibtracert	Examining IB routes
/usr/sbin/perfquery	Computing throughput, detecting fabric errors
/usr/sbin/saquery	Not directly used
/usr/sbin/set_nodedesc.sh	Setting the HCA node description based on node type
/usr/sbin/sminfo	Determing location of master SM
/usr/sbin/smpdump	not directly used
/usr/sbin/smpquery	not directly used
/usr/sbin/vendstat	not directly used
/usr/bin/ibv_devices	listing local HCAs
/usr/bin/ibv_devinfo	listing details of local HCAs
/usr/bin/ibv_rc_pingpong	Determining working status of HCA
/usr/bin/ibv_srq_pingpong	Determining working status of HCA
/usr/bin/ibv_uc_pingpong	Determining working status of HCA
/usr/bin/ibv_ud_pingpong	Determining working status of HCA
/usr/bin/mstflint	Burning new HCA firmware/obtaining current firmware version
/usr/bin/ib_rdma_bw	Computing IB level stats for troubleshooting
/usr/bin/ib_rdma_lat	Computing IB level stats for troubleshooting
/usr/bin/ib_read_bw	Computing IB level stats for troubleshooting
/usr/bin/ib_read_lat	Computing IB level stats for troubleshooting
/usr/bin/ib_send_bw	Computing IB level stats for troubleshooting
/usr/bin/ib_send_lat	Computing IB level stats for troubleshooting
/usr/bin/ib_write_bw	Computing IB level stats for troubleshooting
/usr/bin/ib_write_lat	Computing IB level stats for troubleshooting
/usr/bin/qperf	Computing throughput for RDS/TCP/SDP protocols
/sbin/ifconfig	Determining configuration/status of network interfaces
/usr/bin/ib-bond	Determining active slave interface for bond0
/usr/bin/rds-gen	Not directly used
/usr/bin/rds-info	Examining RDS state
/usr/bin/rds-ping	Determining RDS connectivity
/usr/bin/rds-sink	Not directly used
/usr/bin/rds-stress	Profiling RDS performance
Imaging and versions	These tools are related to imaging status and info as well as versions installed
imagehistory
imageinfo	Only on database servers version >= 11.2.1.3
/opt/oracle.cellos/CheckHWnFWProfile	Only applicable on cells. With the -d option, it will display versions found. Without options, it will report any mismatches against known correct vaiues.
/opt/oracle.SupportTools/CheckSWProfile.sh	Only applicable on cells. Without options, displays any mismatch against known good configurations.
collectlogs.sh	for collecting logs from onecommand deployments
Networking
cat /proc/net/bonding/bond*
cat /sys/class/net/eth?/operstate
cat /sys/class/net/bond*/operstate
ifconfig
ethtool <interface_name>	reports information about the interface like link mode capabilities
Logfiles on both database server and cells
/var/log/messages	Older versions of this file will be automatically renamed as messages.<number> with number 1 being the most recent history.
dmesg (a command that displays log)
/var/log/cellos/validations.log
/var/log/cellos/validations/*log
Logfiles on cells
$ADR_BASE/diag/asm/cell/<hostname> /trace/alert.log	Cell’s alert log. Also will find cell’s trace files in the same directory as the alert.log
Logfiles on database servers
$ORACLE_BASE/diag/asm/+asm/<instname> /trace/ alert_<instname>.log	ASM alert logfile
$ORACLE_BASE/diag/rdbms/<dbname>/ <instname>/trace/alert_<instname>.log	DB alert log – one for each database running…may be more than one DB
/u01/app/11.2.0/grid/log/<hostname>/ alert<hostname>.log	Grid Infrastructure alert logfile. This log is relatively high-level and will often lead you to one of the logs mentioned in the entry just below this one.
/u01/app/11.2.0/grid/log/<hostname>/ [cssd,crsd,diskmon]/*.log	Logfiles for CSSD, CRSD, and diskmon processes. These processes are the most likely ones to have issues and will expose most issues.
Infiniband Switches
sminfo	shows the current subnet master switch in the fabric – there should be exactly one regardless of how many switches are present in the fabric
ibswitches	lists all IB switches in the fabric
showunhealthy	shows any unhealthy sensors
env_test	lists all the data from the environmental sensors in the switch
nm2version	shows the current versions – use this to determine what version the switch is running right now
getfanspeed	shows the speed of the internal fans in the switch – can be useful if showunhealthy indicates a problem with one of the fans
Cell software commands (cellcli and friends)	These commands may be run from within cellcli
list cell detail
list alerthistory
list celldisk detail
list griddisk detail
list lun detail
list physicaldisk detail
list flashcache detail
list griddisk attributes name,status,asmmodestatus,asmdeactivationoutcome
alter cell validate configuration
adrci	show incident
mdadm –misc –detail /dev/md*	for an overview of the state of the raid devices on the storage cell
cat /proc/mdstat	for a view of the status of the devices
/usr/local/bin/ipconf –verify
mdadm -Q –detail /dev/md?	state information on a particular meta device
<GRID_HOME>/bin/kfod disks=all	lists disks available from DB node for ASM use (run on DB node)
Hardware	These commands may be run to query hardware status. Unless otherwise noted, they apply to cells and database servers.
ipmitool sel list	Lists the system event logs – these logs sometimes show HW events that aren’t seen elsewhere.
ipmitool sunoem cli ‘show /SYS’	Shows system serial number, fault_state (overall fault state, not necessarily a rollup – may be a fault on a component-level)
/opt/MegaRAID/MegaCli/MegaCli64 -adpallinfo -a0	All adapter info
/opt/MegaRAID/MegaCli/MegaCli64 -FwTermLog -dsply -a0	Diplay controller’s log
/opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuStatus -a0	Get battery status
/opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuProperties -a0	Get battery properties
/opt/MegaRAID/MegaCli/MegaCli64 -LDinfo -Lall -aALL	Looking for WriteThrough? on the Current Cache Policy – if disabled, may affect performance; easier to get this information from cellcli -e list lun attributes name,lunWriteCacheMode,status
/opt/MegaRAID/MegaCli/MegaCli64 -LDPdInfo -aAll	Helpful to investigate predictive failure if necessary
/opt/MegaRAID/MegaCli/MegaCli64 -PDList -a0	The Inquiry Data will contain the drive firmware, but decoding the string to get the firmware requires special instructions – beyond what is here. Check list physicaldisk attributes physicalFirmware in cellcli for drive FW version.
lspci [-v [ -v [ -v ]]]	Listing PCI devices. The more -v arguments you add, the more information detail it provides
lsscsi	Especially helpful on cells. Flash cards will show up as MARVELL devices. There should be 16 flash devices listed. If not, there’s a card missing or not visible to the OS.
/opt/oracle.cellos/scripts_aura.sh	This script lists the flash disks as will be seen from the cell software
/opt/oracle.SupportTools/sundiag.sh	Gathers many diagnostic command outputs and important logfiles for analysis of storage cell and disk issues