Oracle Exadata Dba Solution

Wednesday, November 23, 2016

What is oracle Threshold

Threshold I meaning in English is exit door, door frame, doorsill . That means threshold is like a point where near to exit from where .

Like we are standing on threshhold for entering a new era .

oracle threshold like Tablspace threshold where you may set a threshold value when system reach
in the value em thrugh a alert that you reach a threshold level and now need to take action .
This threshold value and alert you can set and minotor from Enter prise manager email notification

1.Configuration Threshold Monitoring

a> Threshold Rules may define . (this rules vary for the type of resourses monitor also various event )

b> Define threshold rule sets from the rules you have already define.The rules set agreegate threshold rules for deployment to specific devices
c> Assign threshold rule sets to specific devices.
d> update the agent for putting the new rules sets into action

What Is Threshold Rules Sets

A threshold rule set is a collection of threshold rules. Threshold rules are defined separately and later combined under the umbrella of a threshold rule set.

Notification Options

Notification may take place by email and/or SNMP trap.

    Configure SNMP server information using the Administration task:

    Administration --> Server Configuration --> SNMP Administration

    Configure email server information using the Administration task:

    Administration --> Server Configuration --> Email Administration

    If you are notifying via email, you will enter the Configuration Change Console user (defined in the People screen) to receive the notification. The email accounts for individuals are created in:

    Administration --> People --> People

Friday, September 2, 2016

warning: RMAN-08591:Invalid archivelog deletion policy

RMAN-08591: WARNING: invalid archived log deletion policy

If getting error when taken backup of rman for archive log deletion policy dont upset your datagurd broker binding parameter value may be set optional . For remove this warning from backup follow below steps:

[oracle@drawdbadm01 ~]$ dgmgrl sys/***
DGMGRL for Linux: Version 11.2.0.4.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected.
DGMGRL> show configuration

Configuration - ORCL_PRIMARY

Protection Mode: MaxPerformance
Databases:
    ORCL    - Primary database
    ORCLSTBY - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> show configuration

check verbosely only stand by database [here my stand by db name orclstby]

Check the binding value for the standby database on DR side.

DGMGRL> show database verbose 'ORCLSTBY';

Database - AWCTGFO

Role:            PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag:   50 minutes 52 seconds (computed 1 second ago)
Apply Lag:       50 minutes 53 seconds (computed 0 seconds ago)
Apply Rate:      471.00 KByte/s
Real Time Query: ON
Instance(s):
    orclstby11 (apply instance)
    orclstby2

Properties:
    DGConnectIdentifier             = 'orclstby_dg'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '10'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '+DATA_DCAW, +DATA_DRAW, +RECO_DCAW, +RECO                                                                                        _DRAW'
    LogFileNameConvert              = '+DATA_DCAW, +DATA_DRAW, +RECO_DCAW, +RECO                                                                                        _DRAW'
    FastStartFailoverTarget         = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    ApplyLagThreshold               = '0'
    TransportLagThreshold           = '0'
    TransportDisconnectedThreshold = '30'
    SidName(*)
    StaticConnectIdentifier(*)
    StandbyArchiveLocation(*)
    AlternateLocation(*)
    LogArchiveTrace(*)
    LogArchiveFormat(*)
    TopWaitEvents(*)
    (*) - Please check specific instance for the property value

Database Status:
SUCCESS

DGMGRL>

Now just change the bindinf value from optional to mandatory with datguard broker.

DGMGRL> edit database 'ORCLSTBY' set property Binding='mandatory';
Property "binding" updated
DGMGRL> exit

dgmgrl sys/********

DGMGRL> show configuration

check verbosely only stand by database [here my stand by db name orclstby]

DGMGRL> show database verbose 'ORCLSTBY';

Database - ORCSTBY

Role:            PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag:   50 minutes 52 seconds (computed 1 second ago)
Apply Lag:       50 minutes 53 seconds (computed 0 seconds ago)
Apply Rate:      471.00 KByte/s
Real Time Query: ON
Instance(s):
    orclstby11 (apply instance)
    orclstby2

Properties:
    DGConnectIdentifier             = 'orclstby_dg'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '10'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '+DATA_DCAW, +DATA_DRAW, +RECO_DCAW, +RECO                                                                                        _DRAW'
    LogFileNameConvert              = '+DATA_DCAW, +DATA_DRAW, +RECO_DCAW, +RECO                                                                                        _DRAW'
    FastStartFailoverTarget         = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    ApplyLagThreshold               = '0'
    TransportLagThreshold           = '0'
    TransportDisconnectedThreshold = '30'
    SidName(*)
    StaticConnectIdentifier(*)
    StandbyArchiveLocation(*)
    AlternateLocation(*)
    LogArchiveTrace(*)
    LogArchiveFormat(*)
    TopWaitEvents(*)
    (*) - Please check specific instance for the property value

Database Status:
SUCCESS

DGMGRL>

Now just change the binding value from optional to mandatory with dataguard broker.

DGMGRL> edit database 'ORCLSTBY' set property Binding='mandatory';
Property "binding" updated

Now check the binbind parameter value changed mandatory .

DGMGRL> show database verbose 'AWCTGFO';

Database - AWCTGFO

Role:            PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag:   1 hour(s) 30 minutes (computed 1 second ago)
Apply Lag:       1 hour(s) 30 minutes 1 second (computed 1 second ago)
Apply Rate:      466.00 KByte/s
Real Time Query: ON
Instance(s):
    awctgfo1 (apply instance)
    awctgfo2

Properties:
    DGConnectIdentifier             = 'awctgfo_dg'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'mandatory'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '10'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '+DATA_DCAW, +DATA_DRAW, +RECO_DCAW, +RECO_DRAW'
    LogFileNameConvert              = '+DATA_DCAW, +DATA_DRAW, +RECO_DCAW, +RECO_DRAW'
    FastStartFailoverTarget         = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    ApplyLagThreshold               = '0'
    TransportLagThreshold           = '0'
    TransportDisconnectedThreshold = '30'
    SidName(*)
    StaticConnectIdentifier(*)
    StandbyArchiveLocation(*)
    AlternateLocation(*)
    LogArchiveTrace(*)
    LogArchiveFormat(*)
    TopWaitEvents(*)
    (*) - Please check specific instance for the property value

Database Status:
SUCCESS
second option to reset the value for binding

Sqlplus / as sysdba

SQL>alter system set log_archive_dest_1='location=USE_DB_RECOVERY_FILE_DEST mandatory';

System altered.

RMAN> show ARCHIVELOG DELETION POLICY ;

Reset the value using configure with clear option.

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY clear;

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY;

old RMAN configuration parameters:

CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY;

new RMAN configuration parameters:

CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY;

new RMAN configuration parameters are successfully stored

starting full resync of recovery catalog

full resync complete

Thursday, September 1, 2016

Static listener vs Dynamic listener differences

Difference between Static listener and Dynamic listener registrations

Database can be registered in two ways with a listener

1. Static Registration :- A static registration of database to a listener is used when we want to remotely start our database. Static registration is like hardcoding a instance details in listener.ora file. In a static registration an instance is registered with the listener whether its up or not. When a client request comes listener opens a dedicated connection , and server later find out that instance is not up then it gives error message as "Oracle not available".

2. Dynamic Registration :- In Dynamic registration , registration is performed by PMON process. Once a Database instance starts, its PMON process registers instance details with associated listener. Dynamic registration does not require any manual configuration in the listener.ora file where as Static configuration does.From Oracle 8i dynamic configuration introduced.

PFB listener.ora file having static configuration :-

SID_LIST_LISTENER =
(SID_LIST =
    (SID_DESC =
      (SID_NAME = DCPROD)
      (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1)
    )
    (SID_DESC =
      (SID_NAME = DCPROD)
      (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1)
    )
)

LISTENER =
(DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = 10.100.26.241)(PORT = 1521))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
)

ADR_BASE_LISTENER = /oracle_home/oracle

PFB listener.ora file having dynamic configuration :-

LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.100.26.242)(PORT = 1521))
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
)
)

ADR_BASE_LISTENER = /oracle_home/oracle

As we can see dynamic configuration is more simple then static configuration.

PFB listener status details when using Dynamic regitration :-

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 01-SEP-2016 23:19:29

Copyright (c) 1991, 2013, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.100.26.241)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                01-SEP-2016 23:07:46
Uptime                    0 days 0 hr. 11 min. 43 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/11.2.0/db_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/product/11.2.0/db_1/network/log/listener.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.100.26.241)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Services Summary...
Service "DCPROD" has 2 instance(s).
Instance "DCPROD", status UNKNOWN, has 2 handler(s) for this service...
Instance "dcprod", status READY, has 1 handler(s) for this service...
Service "dcprodXDB" has 1 instance(s).
Instance "dcprod", status READY, has 1 handler(s) for this service...
The command completed successfully

Above status as "READY" resembles that listener has checked instance details and found its status as up or running to handle client request.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=amit)(PORT=1521)))

STATUS of the LISTENER

------------------------

Alias LISTENER

Version TNSLSNR for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production

Start Date 01-DEC-2013 16:42:04

Uptime 0 days 0 hr. 0 min. 0 sec

Trace Level off

Security ON: Local OS Authentication

SNMP ON

Listener Parameter File /u01/app/oracle/product/11.2.0/db_1/network/admin/listener.ora

Listener Log File /oracle_home/oracle/diag/tnslsnr/new/listener/alert/log.xml

Listening Endpoints Summary...

(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.100.26.242)(PORT=1521)))

(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))

The listener supports no services

The command completed successfully

Above status resembles that either Instance is not up, or its not been registered with listener.

PFB commands to register database instance with listerner if its not automatically registering with listener :-

[oracle@dbrac1 admin]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Thu Sep 1 23:22:18 2016

Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> alter system register;

System altered.

SQL> exit

Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

[oracle@dbrac1 lsnrctl status

PFB listener status details when using static regitration also

Above status as "UNKNOWN" means that listener has not checked instance status details that whether its up or not, it has just registered the details of instance. So listener is not sure that whether instance can handle client request or not.

Sunday, August 7, 2016

Exadata Two Node Cluster VKTM Process Trace File Warning

Problem Summary
---------------------------------------------------
Warning: VKTM detected a time drift

Problem Description
---------------------------------------------------
We found below problem from dc and dr alert log file .
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more det.

Please assist us on above warning . Will it impact on business?

Solution :

The time drifts usually occurring less than 1sec and 5 sec for forward and backward respectively are permissible and OK.
If the traces are emitting time drifts of amount beyond these ranges, then it needs to be analyzed.
Most of the times, during high loads, there would be issues with underlying OS due to virtual memory, network time protocol improper configuration etc.

In general VKTM process need to be scheduled in every 10ms, if due to above reasons this is not happening we see the time drifts and to certain level (mentioned above) are permissible.

So please upload the VKTM trace files to analyze.

You can also set the below event to supress these alerts in alert log:
==========================================
Event 10795 is not set in the database.
Event 10795 suppress the VKTM warnings in alert log file. Hence the event need to be set.

$ sqlplus / as sysdba
alter system set events '10795 trace name context forever, level 2' scope=spfile;
shut immediate
startup
oradebug setmypid
oradebug eventdump system

The last command should show you the 10795 event set in your system.

exadata ibcheckerror report

Exadata ibcheckerror report

[root@drawdbadm02 ~]# ibcheckerrors

src/query_smp.c:196; umad (DR path slid 0; dlid 0; 0,1,13,29,36 Attr 0x11:0) bad status 110; Connection timed out
#warn: counter SymbolErrorCounter = 65532       (threshold 10) lid 19 port 255
#warn: counter LinkErrorRecoveryCounter = 16    (threshold 10) lid 19 port 255
#warn: counter PortRcvErrors = 260      (threshold 10) lid 19 port 255
#warn: counter PortXmitDiscards = 267   (threshold 100) lid 19 port 255
Error check on lid 19 (Infiniscale-IV Mellanox Technologies) port all: FAILED
#warn: counter PortXmitDiscards = 204   (threshold 100) lid 19 port 7
Error check on lid 19 (Infiniscale-IV Mellanox Technologies) port 7: FAILED
#warn: counter LinkErrorRecoveryCounter = 14    (threshold 10) lid 19 port 20
#warn: counter PortRcvErrors = 260      (threshold 10) lid 19 port 20
Error check on lid 19 (Infiniscale-IV Mellanox Technologies) port 20: FAILED
#warn: counter PortRcvSwitchRelayErrors = 300   (threshold 100) lid 1 port 255
Error check on lid 1 (SUN DCS 36P QDR drawsw-ibb0 10.100.25.22) port all: FAILED
#warn: counter PortRcvSwitchRelayErrors = 230   (threshold 100) lid 1 port 10
Error check on lid 1 (SUN DCS 36P QDR drawsw-ibb0 10.100.25.22) port 10: FAILED
#warn: counter LinkDownedCounter = 10   (threshold 10) lid 2 port 255
#warn: counter PortRcvSwitchRelayErrors = 749   (threshold 100) lid 2 port 255
#warn: counter PortXmitDiscards = 290   (threshold 100) lid 2 port 255
Error check on lid 2 (SUN DCS 36P QDR drawsw-iba0 10.100.25.21) port all: FAILED
#warn: counter PortRcvSwitchRelayErrors = 108   (threshold 100) lid 2 port 1
Error check on lid 2 (SUN DCS 36P QDR drawsw-iba0 10.100.25.21) port 1: FAILED
#warn: counter PortXmitDiscards = 180   (threshold 100) lid 2 port 2
Error check on lid 2 (SUN DCS 36P QDR drawsw-iba0 10.100.25.21) port 2: FAILED
#warn: counter PortRcvSwitchRelayErrors = 274   (threshold 100) lid 2 port 7
Error check on lid 2 (SUN DCS 36P QDR drawsw-iba0 10.100.25.21) port 7: FAILED
#warn: counter PortRcvSwitchRelayErrors = 168   (threshold 100) lid 2 port 10
Error check on lid 2 (SUN DCS 36P QDR drawsw-iba0 10.100.25.21) port 10: FAILED

## Summary: 11 nodes checked, 0 bad nodes found
##          46 ports checked, 7 ports have errors beyond threshold

How To collect Oracle EXADATA Sundiag

[root@drawceladm01 ~]# /opt/oracle.SupportTools/sundiag.sh

Oracle Exadata Database Machine - Diagnostics Collection Tool

Last alert date is beyond 7 days. Skipping OSW/Metrics collection

Gathering Linux information

src/query_smp.c:196; umad (DR path slid 0; dlid 0; 0,1,13,29,36 Attr 0x11:0) bad status 110; Connection timed out

Skipping collection of OSWatcher/ExaWatcher logs, Cell Metrics and Traces

Skipping ILOM collection. Use the ilom or snapshot options, or login to ILOM

over the network and run Snapshot separately if necessary.

/var/log/exadatatmp/sundiag_drawceladm01_1320FM501H_2016_08_07_14_55
Gathering Cell information

Generating diagnostics tarball and removing temp directory

==============================================================================
Done. The report files are bzip2 compressed in /var/log/exadatatmp/sundiag_drawceladm01_1320FM501H_2016_08_07_14_55.tar.bz2
==============================================================================
Collect report with filezilla

oracle sysman default password password set

How To Set Oracle SYSMAN User Default Password

SYSMAN is not an Oracle user. It is related to OMS. This user does not appear in DBA_USERS.
When ever the console asks for the password for the first time give the default password as OEM_TEMP. Then it prompts for new password.
Change it and use the new password in the subsequent logins.

Its very simple first you check below query for sysman user and then conn sys (put your desire password) click here

[oracle@awback01 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sun Aug 7 13:15:21 2016

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select username
    from dba_users
    where username like 'SYSM%'
    order by 1; 2    3    4

USERNAME
------------------------------
SYSMAN

SQL> conn sysman
Enter password:
Connected.
SQL> show user
USER is "SYSMAN"

When ever the console asks for the password for the first time give the defailt password as OEM_TEMP. Then it prompts for new password.
Change it and use the new password in the subsequent logins.