Introduction

Sometimes, it is necessary to rename the temporary tablespace. Unfortunately the  temporary tablespace is used by the processes of a sap system. This short blog shows you how to change the temporary tablespace online, without disturbing the sap-system.

Scenario

The system consists of one database server and 2 application servers. The transaction DB02OLD stated a temporary tablespace with a wrong name. PSAPTEMP1 is defined as default temporary tablespace and is stored under

/oracle/SID/sapdata1/temp1_1/temp1.data1

We want to rename the temporary tablespace to PSAPTEMP.

Solution

How is the procedure to rename PSAPTEMP to PSAPTEMP?

1. Create the new temporary tablespace.

2. Define the new temporary tablespace as default temporary tablespace.

3. Drop the old temporary tablespace.

1. Create the new temporary tablespace.

orasid> mkdir /oracle/SID/sapdata1/temp_1/

orasid> sqlplus "/as sysdba"

SQL> CREATE TEMPORARY TABLESPACE "PSAPTEMP" TEMPFILE '/oracle/SID/sapdata1/temp_1/temp.data1' SIZE 20000M EXTENT MANAGEMENT LOCAL UNIFORM SIZE 2M;

2. Define the new temporary tablespace as default temporary tablespace.

SQL> alter database default temporary tablespace PSAPTEMP;

3. Drop the old temporary tablespace.

SQL> drop tablespace PSAPTEMP1 including contents and datafiles;

Every new operation in ORACLE will now use the new temporary tablespace PSAPTEMP!

In all probability there are handles on the temporary tablespace by the workprocesses of the sap system. This causes the drop statement to hang! In order to be able to identify the processes that have a handle on PSAPTEMP1 enter in sqlplus:

SQL> SET linesize 200

SQL> COLUMN tablespace FORMAT a15;

SQL> COLUMN machine FORMAT a15;

SQL> SELECT se.sid sid, se.process process, se.status status, su.tablespace tablespace,

            su.segtype segtype, su.contents contents, se.machine machine, se.program program

     FROM v$session se, v$sort_usage su

     WHERE se.saddr=su.session_addr;

You get a list with handles. In this list you can identify the server (column MACHINE) and the process (column PROCESS)  of the handle holder. The program column shows the process name. The process name indicates that workprocesses are holding the handles. There are two types of handles: ACTIVE and INACTIVE ones. You have to wait until the ACTIVE handle disappear, because this handle is initiated by workprocess processing a report using PSAPTEMP1. Killing this workprocess could inflict inconsistencies. The INACTIVE handles can be cleared by restarting the corresponding workprocess.

Now you know the server and the pid.Lets logon to the server and check it:

sidadm> ps -efa | grep <process>

Yes it is the correct workprocess. In transaction SM50 of the corresponding application server you can identify the workprocess with the determined pid and restart the workprocess in order to clear the handle.

Repeat this procedure for each INACTIVE handle. After clearing the last handle the drop statement will proceed.

After dropping PSAPTEMP1 you can delete the directory of PSAPTEMP1

orasid> rmdir /oracle/SID/sapdata1/temp1_1/

After new db-checks the new temporary tablespace PSAPTEMP appear in the DB02OLS.

Please excuse the usage of the old transaction DB02OLD bit I don't like the "look and feel" of ALV-grids. I prefer the old list processing of sap.

I hope you enjoyed the blog.

I am sharing a situation that happened after an Oracle upgrade from 11.2.0.4 to 12.2.0.1, although I can imagine this can happen at any time.

This is on AIX.

After the upgrade, the oracle processes were running as sidadm, which could be a problem:

the permissions of the oracle executable at $ORACLE_HOME/bin were set correctly with the correct s-bits set:

As we read in SAP note 206916 , ps -ef does not show the effective user which is the user actually running the process, but only the real user who ran the process.

This can be checked by running the command "ps -ef -o comm=command,args=argument,ruser,user":

On the right you can see 2 users. The first one is RUSER column which is the real user, this is what normal ps -ef shows us.

The user next to that more to the right is the USERS column which is the effective user. This must always be orasid.

So this database should in theory run fine, but still something has happened to cause this. We assumed the problem was some permission setting somewhere was set to sidadm and not orasid, but we could find that anywhere.

A few test revealed that if we start the listener with sidadm, the above happens. So stopping the listener and starting again with orasid solved it. A restart will be required as the current open processes will have an established connections.

This blog is step by step guide to upgrade oracle  10g to Oracle 11g. We cannot do patch upgrade kind of things since 11g is not a patch set of oracle. We have to install 11g home as a separate ORACLE_HOME in parallel to 10g Oracle Home and follow the steps to upgrade. For  the SAP we have to download Oracle 11g from SAP service market place. Download the latest version from SMP.

Pre-requisites :

• Existing data base should be highest patch level  (oracle 10g  10.2.0.5.0) (OPatch not mandatory)
• Check the Memory ,swap and Disk Space.
• SYSAUX  and  SYSTEM table-space should have minimum 200 MB free space.
• Check the oracle home directory's  OS  level permission.
• Stop the SAP Instances and services(don't stop oracle services)
• Need 5- to 7 GB free space on new oracle directory (where software going to install)

Installing the Oracle 11g Server Software

As of now Oracle 11.2.0.3 is Oracle database patch sets are full installations of the Oracle database software. For more information about identifying and installing the correct Oracle database software version, see SAP note 1431799.

Make sure that you have enough space for the Oracle inventory and that you have full access to the directories (inventory, installation location, temp directory). Otherwise, the Oracle Universal Installer cannot perform installation prerequisite checks.

Logon to system as <DBSID>ADM.

Start the Oracle Universal Installer (OUI) by double-clicking the file sapserver.cmd.

If a command prompt window appears, specify the drive letter of the local disk where you want to install the Oracle software,and the <DBSID>. The command prompt window only appears if you perform a new installation (under a different user), or if the ORACLE_HOME and <DBSID> are not set.

In the Oracle Universal Installer, enter the information as shown in the following table:

Screen Shot of Installation:::

Installing Required Patches ( not necessary )

After the database software installation, you need to install all required Oracle database patches, which on Windows include one patch collections and one or more additional (generic) patches.

For Oracle 11.2.0.3, check SAP Note 1631931 for the patches to be installed.

You can download the patches from:http://service.sap.com/oracle-downloadOracle 11.2.0.<x>

Prerequisites

Use an up-to-date version of OPatch to install the patches. Check SAP Note 839182 for instructions on how to use OPatch.

Copying and Adapting the SQL*Net Files

• After the database software installation, you must copy and modify the old SQL*Net files, that is, listener.ora, sqlnet.ora, and tnsnames.ora.(from old 10g oracle home)

• Log on as user <dbsid>adm.

• copy <old_Oracle_home>/database/init<SID>.sap to <new_Oracle_home>/database/ init<SID>.sap

• copy <old_Oracle_home>/database/init<SID>.utl to <new_Oracle_home>/database/init<SID>.utl

• Copy the SQL*Net parameter files from the directory <old_ORACLE_HOME>\network\admin to the directory <new_ORACLE_HOME>\network\admin.

• Edit the TNSNAME.ora & listener.ora add the new oracle home path <new_ORACLE_HOME>\network\admin directory.

       Identify the parameter ORACLE_HOME that belongs to the database you want to upgrade (SID_NAME in section SID_LIST) 

• If not available, add the following entry to the file listener.ora:ADR_BASE_<listener_name> = <full_path_to_saptrace_directory>

Note that this is the same value as set in the database profile parameter diagnostic_dest.

Since your database is still running in the old environment, do not start the listener with the new listener.ora file at this point.

Backing Up the Oracle Database

You must have a complete and usable database backup otherwise you might lose data

Procedure

1. Create a complete database and operating system backup, and back up any archive logs from Oracle 10g.

1. Check that the backup is usable. If for any reason you have problems during the upgrade you must be able to restore the database from this backup.

Changing the Environment for User <dbsid>adm

• ORACLE_HOME( map to new oracle home)
• TNS_ADMIN (if it exists)

EXAMPLE

Rrename ORACLE_HOME into ORACLE_HOME_before_upgrade, and TNS_ADMIN into TNS_ADMIN_before_upgrade.

Run the Pre-upgrade script…. one by one as below...

Note 1431793 - Oracle 11.2.0: Upgrade Scripts

=============================================

A detailled description and instructions how to use the scripts can be found in SAP note 1431793.

https://service.sap.com/sap/support/notes/1431793

create_restore_point.sql

pre_upgrade_checks.sql

pre_upgrade_status.sql

pre_upgrade_tasks.sql

utlu112i_sap.sql

Performing the Database Upgrade

• To upgrade the database, use the Oracle Database Upgrade Assistant (DBUA).

• StartAll ProgramsOracle - <NEW_ORACLE_HOME_NAME>Configuration and Migration ToolsDatabase Upgrade Assistant

Note:

     Note that if you use this method, you do not automatically run the SAP-specific DBUA upgrade scripts, and you have to run them after the upgrade      manually.

• If not yet stopped, stop the SAP system.
• Do not stop the database instance. The database instance must be running in the current (old) environment.
• Change the registry entry of ORA_<DBSID>_AUTOSTART by entering the command: <new_Oracle_Home>\bin\oradim -edit –sid <dbsid> –startmode manual  Ignore the error message: Unable to start service, OS Error 1056.
• Alternatively, use regedit.exe to change the registry key to value FALSE. The registry key is available at HKLM:\Sofware\Oracle\KEY_<DBSID>11203

DBUA steps:

Screen shot of DBUA

If you started the DBUA from the Start menu, you must perform additional steps to perform the post-upgrade script (see below).

Additional Steps When Starting the DBUA from the Start Menu

If you started the DBUA from the Start menu, you have to check the upgraded database and perform the post-upgrade script manually.

To do this perform the following steps of SAP note 1431793:

“6.1 Checking the upgraded database ”

“6.2 Running post-upgrade scripts manually ”

Post-Upgrade Steps

1. You set the user environment variables
2. You create the Oracle TNSListener service 
3. You change the registry value of Oracle Autostart 
4. You check the start up type of the Oracle services
5. You perform post-upgrade tasks 
6. You delete the old Oracle software

After the Oracle database upgrade, you need to perform the steps described here.

You set the user environment variable.

       In the user environment and system environment <DBSID>adm, check the PATH variable and, if required, remove all parts referring to the old Oracle                 version. 

          When you modify the user environment, you must be logged on as the corresponding <DBSID>adm user.

• In the user environment of <DBSID>adm, set the ORACLE_HOME variable to the correct value for Oracle 11g, for example to: <DRIVE>:\ORACLE\<DBSID>\11203.

• Copy tnsnames.ora and sqlnet.ora from %ORACLE_HOME%\network\admin to \\<sapglobalhost>\sapmnt\<DBSID>\SYS\profile\oracle.

       (This information is required for all ABAP application servers you install on a separate host to find the correct location of these files.) 

• Set the user environment variable TNS_ADMIN to point to the directory \\<sapglobalhost>\sapmnt\<DBSID>\SYS\profile\oracle.

          Set TNS_ADMIN on all ABAP application servers.

• Log off and log on again to activate the environment changes in your user session.

Creating the Oracle TNSListener Service

• Stop the old Oracle listener before you create the Oracle TNSListener Service.( from ms service or command prompt: LSNRCTL STOP

          Create New listener service use command prompt or run commad:

          <complete_path_to_new_Oracle_Home>\bin\lsnrctl start

          This will start the listener from the new Oracle home. As the listener service does not yet exist, you will get the following error message: Failed to           open service <Oracle<SID>11202TNSListener>, error 1060. Then the service will be created and started.

Changing the Registry Value of Oracle Autostart

Change the registry value of ORA_<DBSID>_AUTOSTART. Open a command prompt and enter the following command:<new_Oracle_Home>\bin\oradim -edit -sid <dbsid> -startmode manual

Ignore the error message: Unable to start service, OS Error 1056.

Alternatively, use regedit.exe to change the registry key to value FALSE. The registry key is available at HKLM:\Sofware\Oracle\KEY_<DBSID>11202

Checking the Start Up Type of the Oracle Services

        Start the Oracle services (Database and Listener), if not yet started. 

Also, if not yet started, start the Distributed Transaction Coordinator Service.

In a standard configuration, set the startup type of OracleService<DBSID> and Oracle<ORACLE_HOME_NAME>TNSListener to automatic to ensure that they are automatically restarted after booting.

Performing Post-Upgrade Tasks (Follow the SAP NOTE 1431793)

Example :Open a command prompt and change to the directory <new_oracle_home>\sap\ora_upgrade\post_ugprade.

From this directory start sqlplus.exe and enter the following commands:SQL>connect / as sysdbaSQL>@post_upgrade_status.sql

This provides an overview of the status of the upgraded database.

Run the below scripts one by one

• Check the Oracle database parameters against SAP Note 1431798 and adjust them if necessary.

       You can find an automated script in SAP Note 1171650 to help you check whether your SAP system complies with the database parameter               recommendations at any given point in time. 

• Update the Oracle database statistics with BRCONNECT as follows:

brconnect -c -u / -f stats -t all –f collect -p 4

Deleting the Old Oracle Software

• Do not delete the old software, if it still has to be accessed by another database instance running with this version.
• Before you delete the old Oracle software, you can perform a backup of your database
• To deinstall the Oracle 10.2 software, you use the Oracle Universal Installer (OUI) from older home.
• To delete a service, open a command prompt and enter:sc delete <service_name>

Above Details will help only for the standalone SAP on Oracle upgrade . And I did the upgraded as per  above mentioned steps .For  Cluster environment   download the standard document from sap. Database Upgrade Guide

Thanks

Yoonus

Visit.. WrongCodes

You see following ORA-00600 error in ST22 dump or alert.log.

ORA-00600: internal error code, arguments: [qkaffsindex1],
[], [], [], [], [], [], [], [], [], [], []

When you look into the relevant dev trace of the dump or
incident trace file referred in alert.log, you see the error happens when
running SQL statement like

"delete from <table name> where ..".

Reason:

This issue is addressed as an oracle bug 19438874 which
happens in oracle release 11.2 to 12.1. It is fixed in future release 12.2.

Solution:

The bug fix has been delivered in following SBP version.

SBP 11.2.0.3.14 201505

SBP 11.2.0.4.6 201505

SBP 12.1.0.2.3 20505

Install the above SBP to fix the error.

As a workaround, you can set parameter "_fast_full_scan_enabled"
to FALSE. But this is not a recommended way.

I still see some people forget to execute catsbp.sql after upgrade database. In the SBP installation guide (readme file), it has below content:

The point 5 lets us know we do not need to execute 6, 7 during upgrade, but the guide does not mention whether we should do this after upgrade. After testing upgrading from 10g to 11g again, we definitely need to run catsbp.sql after upgrade! If we do not run this script, the oracle internal dictionary is not updated. For example, we cannot query SBP information from dba_registry_history.

So by following the oracle upgrade guide and SBP readme file, below should be the right steps:

1. install oracle software only

2. install SBP by referring the readme file (note: it tells us we don't need to execute catsbp.sql in this phase)
3. dbua
4. database upgrade post actions
5. back to SBP readme file and run catsbp.sql, utlrp.sql for target instance.

Best regards,
James

Introduction

The idea for this blog post started a few weeks ago when i had to troubleshoot some Oracle database / SQL performance issues at client site. The SQL itself included several views and so placing hints (for testing purpose) into the views was not possible, especially as the views were used widely and not only by the SQL with the performance issue. In consequence this blog post is about the difference between embedded global and local hints and how to use them.

I highly recommend to read Jonathan Lewis's article "Basics of the Cost Based Optimizer – Part 4" first, if you are not familiar with query blocks or query block names. It is also crucial to have a basic understanding of query transformations as we talk about hinting in case of views and so query transformations may happen. Please check out my blog article "[Oracle] DB Optimizer Part VIII - Looking under the hood of query transformation (done by CBO) with simple real life example", if you are not familiar with query transformations.

The example i will use in this blog post is based on a SAP SELECT statement and a SAP view definition. I have no SAP system at hand right now, but i still have the definition of the tables AFKO / AUFK (including corresponding indexes) and the corresponding view CAUFV in a SQLd360 report. SQLd360 is a free tool created and maintained by Mauro Pagano, that provides a 360-degree overview around a SQL statement (and its used objects, object statistics, runtime information, etc.). It is similar to the "SQL_ID DATA COLLECTOR for Oracle" by Martin Frauendorfer (SAP), but it includes a lot more information and it also supported with Oracle 12c. It does not require any installation and can be executed by any user that has access to dictionary views.

Embedded global and local hints - What are they and how do they differ

At first a general warning. In my opinion applying (manual) hints should be the last resort in case of fixing a cost based optimizer / execution plan issue, but sometimes you just have to due to bugs or cost based optimizer limitations. But even in these cases you still can use SQL patches (or some other techniques) for applying hints, which are much more flexible than hard coding hints in source code.

What are embedded hints in general?

Embedded hints are hints that are included in the SQL statement. Now you may ask - are there any other hints as well? Yes, they are called outline hints. Outline hints are stored in outlines and SQL plan baselines or can also be included in the SQL itself. They are flanked by the special key words BEGIN_OUTLINE_DATA and END_OUTLINE_DATA.

You may already have seen the hint "IGNORE_OPTIM_EMBEDDED_HINTS" in ST05 or DBMS_XPLAN and asked what this means. This special hint instructs the CBO to ignore most of all the other supplied hints (there are some exceptions to this, but let's disregard this right now) except the ones that are flanked by BEGIN_OUTLINE_DATA and END_OUTLINE_DATA (= outline hints). We will see an example of this later on.

What are (embedded) local and global hints?

Local (embedded) hints

Local (embedded) hints apply to the query block in which they are placed. So for example, the SAP DBSL generates a local embedded hint "SELECT /*+ FULL("T_00") * FROM TABLE T_00", if you use an ABAP code extension like this "%_HINTS ORACLE 'FULL( T_00 )'". Unfortunately real applications are not that simple.

The SQLs may use views and query transformations are applied (under the hood) and in consequence such local (embedded) hints may not work anymore. If you look closely into the SQL outline section in ST05 or DBMS_XPLAN, you won't see such hints neither.

Global (embedded) hints

Global (embedded) hints are starting with a "@" symbol. Hints in outlines are always global hints as they are stored in one common location and need to apply to any possible query block. Global hints can also be used as embedded hints and are essential from time to time.

When do we need global (embedded) hints?

• If you want to hint a query block based on an Oracle DDIC view (e.g. like SAP view CAUFV). Otherwise you need to alter the view itself with a local hint, if you want change some behavior in there. This may not be tolerable as this view is used by various SQLs.
• If you want to hint a query block that is based on a query transformation.
• If you have a very complex query and you want to consolidate all hints in one place (comparable to outline hints). Spreading hints in various query blocks (for large SQL statements) can be very confusing and sometimes some hints can become invalid as well.

Let's have a look at an example with a SAP SELECT and the SAP view CAUFV.

Demo for using (embedded) local and global hints

The following demo was run on an Oracle database (12.1.0.1) on OEL 6.4 (2.6.39-400.109.1.el6uek.x86_64).

The SAP table, index and view definition is deployed in a schema called "TEST". The database object definition can be downloaded from here to re-run the test cases.

The following code is the SAP SELECT and the SAP bind variable definition that i will use for the demos:

VAR A0 CHAR(32)

VAR A1 CHAR(128)

VAR A2 CHAR(32)

VAR A3 CHAR(32)

VAR A4 CHAR(32)

VAR A5 CHAR(32)

VAR A6 CHAR(32)

VAR A7 CHAR(32)

SQL> SELECT * FROM "CAUFV" WHERE "MANDT"=:A0 AND "PLNBEZ"=:A1 AND "GLTRP" BETWEEN :A2 AND :A3 AND "WERKS"=:A4 AND "GETRI"=:A5 AND "LOEKZ"=:A6 AND "IDAT2"=:A7;

Let's have a look at the default execution plan, the query blocks and the outline data.

Plan hash value: 438111849

----------------------------------------------------------------------------------------

| Id  | Operation                               | Name   | E-Rows |E-Bytes| Cost (%CPU)|

-------------------------------------------------------- -------------------------------

|   0 | SELECT STATEMENT                        |        |        |       |     1 (100)|

|*  1 |  FILTER                                 |        |        |       |            |

|   2 |   NESTED LOOPS                          |        |        |       |            |

|   3 |    NESTED LOOPS                         |        |      1 |  2799 |     0   (0)|

|*  4 |     TABLE ACCESS BY INDEX ROWID BATCHED | AFKO   |      1 |  1729 |     0   (0)|

|*  5 |      INDEX RANGE SCAN                   | AFKO~0 |      1 |       |     0   (0)|

|*  6 |     INDEX RANGE SCAN                    | AUFK~A |      1 |       |     0   (0)|

|*  7 |    TABLE ACCESS BY INDEX ROWID          | AUFK   |      1 |  1070 |     0   (0)|

----------------------------------------------------------------------------------------

Query Block Name / Object Alias (identified by operation id):

-------------------------------------------------------------

   1 - SEL$F5BB74E1

   4 - SEL$F5BB74E1 / T0001@SEL$2

   5 - SEL$F5BB74E1 / T0001@SEL$2

   6 - SEL$F5BB74E1 / T0002@SEL$2

   7 - SEL$F5BB74E1 / T0002@SEL$2

Outline Data

-------------

  /*+

      BEGIN_OUTLINE_DATA

      IGNORE_OPTIM_EMBEDDED_HINTS

      OPTIMIZER_FEATURES_ENABLE('12.1.0.1')

      DB_VERSION('12.1.0.1')

      ALL_ROWS

      OUTLINE_LEAF(@"SEL$F5BB74E1")

      MERGE(@"SEL$2")

      OUTLINE(@"SEL$1")

      OUTLINE(@"SEL$2")

      INDEX_RS_ASC(@"SEL$F5BB74E1" "T0001"@"SEL$2" ("AFKO"."MANDT" "AFKO"."AUFNR"))

      BATCH_TABLE_ACCESS_BY_ROWID(@"SEL$F5BB74E1" "T0001"@"SEL$2")

      INDEX(@"SEL$F5BB74E1" "T0002"@"SEL$2" ("AUFK"."MANDT" "AUFK"."KOKRS"

          "AUFK"."ABKRS" "AUFK"."AUART"))

      LEADING(@"SEL$F5BB74E1" "T0001"@"SEL$2" "T0002"@"SEL$2")

      USE_NL(@"SEL$F5BB74E1" "T0002"@"SEL$2")

      NLJ_BATCHING(@"SEL$F5BB74E1" "T0002"@"SEL$2")

      END_OUTLINE_DATA

  */

The first thing you may notice is that no VIEW operation is listed in the execution plan although our SELECT was simple as "SELECT * FROM <VIEW_NAME>". The only objects that we can see in the execution plan are the underlying table and index objects. You also notice that this is the result of a query transformations (SEL$F5BB74E1), if you look closely at the query block name.

What kind of query transformation does kick in here? A CBO (10053) trace reveals the secret of course:

…

CVM - complex view merging

SPJ - select-project-join

…

CVM:   Checking validity of merging in query block SEL$2 (#0)

CVM: Considering view merge in query block SEL$2 (#0)

OJE: Begin: find best directive for query block SEL$2 (#0)

OJE: End: finding best directive for query block SEL$2 (#0)

CVM:   Merging SPJ view SEL$2 (#0) into SEL$1 (#0)

Registered qb: SEL$F5BB74E1 0xea272578 (VIEW MERGE SEL$1; SEL$2)

…

The original query block "SEL$2" (= SAP view CAUFV itself) is merged into query block "SEL$1" ("SELECT * FROM") and the transformed query block is named "SEL$F5BB74E1". You can notice this transformation (simple view merging) by looking at the outline data section as well (the hint "MERGE(@"SEL$2")"). However you also see that all hints in the outline data section are global hints (leading @ symbol). You also spot the previously mentioned bracket hints BEGIN_OUTLINE_DATA / END_OUTLINE_DATA and the hint IGNORE_OPTIM_EMBEDDED_HINTS in there.

Finally let's imagine the following scenario to get the whole idea of this blog post. You execute a SAP SELECT like "SELECT * FROM "CAUFV" WHERE …" in your ABAP coding, but the cost based optimizer has chosen a wrong execution plan. The SQL runs forever and you want to change the execution plan by adding a hint to your SQL.

What do you specify in the ABAP extension ("%_HINTS ORACLE") in this case?

Let's say you want to perform a full table scan on table AFKO instead of the index range scan on AFKO~0 plus table access (this makes no sense in real life, but it is a pretty simple example for demonstration). Let's try the common hinting syntax (= local embedded hint) at first.

SQL> SELECT /*+ FULL("T0001") */ * FROM "CAUFV" WHERE "MANDT"=:A0 AND "PLNBEZ"=:A1 AND "GLTRP" BETWEEN :A2 AND :A3 AND "WERKS"=:A4 AND "GETRI"=:A5 AND "LOEKZ"=:A6 AND "IDAT2"=:A7;

Plan hash value: 438111849

-----------------------------------------------------------------------------------------

| Id  | Operation                               | Name    | E-Rows |E-Bytes| Cost (%CPU)|

-----------------------------------------------------------------------------------------

|   0 | SELECT STATEMENT                        |         |        |       |     1 (100)|

|*  1 |  FILTER                                 |         |        |       |            |

|   2 |   NESTED LOOPS                          |         |        |       |            |

|   3 |    NESTED LOOPS                         |         |      1 |  2799 |     0   (0)|

|*  4 |     TABLE ACCESS BY INDEX ROWID BATCHED | AFKO    |      1 |  1729 |     0   (0)|

|*  5 |      INDEX RANGE SCAN                   | AFKO~0  |      1 |       |     0   (0)|

|*  6 |     INDEX RANGE SCAN                    | AUFK~A  |      1 |       |     0   (0)|

|*  7 |    TABLE ACCESS BY INDEX ROWID          | AUFK    |      1 |  1070 |     0   (0)|

-----------------------------------------------------------------------------------------

Nothing changed. Let's verify, if the applied SQL hint was valid at all (for more details please check my blog post [Oracle] DB Optimizer Part IV - What the heck ... Troubleshooting why hints are not considered).

…

Dumping Hints

=============

  atom_hint=(@=0x7f9019773968 err=0 resol=1 used=1 token=448 org=1 lvl=3 txt=FULL ("T0001") )

…

The hint "FULL("T0001")" was detected, its syntax was ok and it was used, but the execution plan did not change at all, although it is a valid operation. The main issue here is that it is applied as a local embedded hint to query block SEL$1, but the table AFKO (alias T0001) is specified in query block SEL$2. Finally these two query blocks are transformed into query block SEL$F5BB74E1 due to (simple) view merging.

You basically have two options, if you want to hint this SQL / execution plan based on SAP view CAUFV:

• You need to alter the SAP view CAUFV (which is usually not possible)
• You use a global embedded hint

In this case you can implement the second option in 2 ways. Both ways are valid and result in the same execution plan:

1) Add a global hint that applies to query block SEL$2 (= SAP view CAUFV)

SQL> SELECT /*+ FULL(@"SEL$2" "T0001") */ * FROM "CAUFV" WHERE "MANDT"=:A0 AND "PLNBEZ"=:A1 AND "GLTRP" BETWEEN :A2 AND :A3 AND "WERKS"=:A4 AND "GETRI"=:A5 AND "LOEKZ"=:A6 AND "IDAT2"=:A7;

2) Add a global hint that applies to transformed query block SEL$F5BB74E1

SQL> SELECT /*+ FULL(@"SEL$F5BB74E1" "T0001") */ * FROM "CAUFV" WHERE "MANDT"=:A0 AND "PLNBEZ"=:A1 AND "GLTRP" BETWEEN :A2 AND :A3 AND "WERKS"=:A4 AND "GETRI"=:A5 AND "LOEKZ"=:A6 AND "IDAT2"=:A7;

Plan hash value: 3565488190

------------------------------------------------------------------------------------------

| Id  | Operation                      | Name   | E-Rows |E-Bytes| Cost (%CPU)| E-Time   |

------------------------------------------------------------------------------------------

|   0 | SELECT STATEMENT               |        |        |       |     2 (100)|          |

|*  1 |  FILTER                        |        |        |       |            |          |

|   2 |   NESTED LOOPS                 |        |        |       |            |          |

|   3 |    NESTED LOOPS                |        |      1 |  2799 |     2   (0)| 00:00:01 |

|*  4 |     TABLE ACCESS FULL          | AFKO   |      1 |  1729 |     2   (0)| 00:00:01 |

|*  5 |     INDEX RANGE SCAN           | AUFK~A |      1 |       |     0   (0)|          |

|*  6 |    TABLE ACCESS BY INDEX ROWID | AUFK   |      1 |  1070 |     0   (0)|          |

------------------------------------------------------------------------------------------

Query Block Name / Object Alias (identified by operation id):

-------------------------------------------------------------

   1 - SEL$F5BB74E1

   4 - SEL$F5BB74E1 / T0001@SEL$2

   5 - SEL$F5BB74E1 / T0002@SEL$2

   6 - SEL$F5BB74E1 / T0002@SEL$2

Outline Data

-------------

  /*+

      BEGIN_OUTLINE_DATA

      IGNORE_OPTIM_EMBEDDED_HINTS

      OPTIMIZER_FEATURES_ENABLE('12.1.0.1')

      DB_VERSION('12.1.0.1')

      ALL_ROWS

      OUTLINE_LEAF(@"SEL$F5BB74E1")

      MERGE(@"SEL$2")

      OUTLINE(@"SEL$1")

      OUTLINE(@"SEL$2")

      FULL(@"SEL$F5BB74E1" "T0001"@"SEL$2")

      INDEX(@"SEL$F5BB74E1" "T0002"@"SEL$2" ("AUFK"."MANDT" "AUFK"."KOKRS"

          "AUFK"."ABKRS" "AUFK"."AUART"))

      LEADING(@"SEL$F5BB74E1" "T0001"@"SEL$2" "T0002"@"SEL$2")

      USE_NL(@"SEL$F5BB74E1" "T0002"@"SEL$2")

      NLJ_BATCHING(@"SEL$F5BB74E1" "T0002"@"SEL$2")

      END_OUTLINE_DATA

  */

Summary

I hope you get the main difference between local and global embedded hints and how to use them with this blog post. There are many little quirks as well, but hopefully you can troubleshoot such issues the next time, if you try to add a (local) hint and it is not considered at all, although it would be a valid operation.

If you have any further questions - please feel free to ask or get in contact directly, if you need assistance by troubleshooting Oracle database (performance) issues. You can also follow or drop me a note on Twitter of course.

References

• Book "Expert Oracle SQL Optimization, Deployment, and Statistics" by Tony Hasler
• Oracle optimizer blog - Optimizer Transformations: View Merging part 1
• Oracle Documentation - Hints

Some customers contacting SAP asking for the entry in Audit log. They found in DBACOCKPIT->Diagnostics->Audit log there is no entry. It is a little bit confusing as this audit ONLY audit DBACOCKPIT database configuration change. In online help you can get the detail information: http://help.sap.com/saphelp_nw73/helpdata/en/86/b9650db8c140829b2a5479ecb97088/content.htm?frameset=/en/9d/dc5196ea63429894807646bad3ca9c/frameset.htm For example if you created external DB connection then in audit log you can find relevant entry.

Hello all,

I recently released a tool in a shell script that aims to help with corrupted blocks.

Of course it can only help with the recovery process itself, as it will still need a good backup of the database.

To identify corrupted blocks and recover them is both critical and time consuming. In addition, it’s not always clear what to do in such situations, for that reason, I've created the tool to give administrators a starting point when they face a database corruption.

The tool help with the following:

1. Full database consistency check

2. Identify affected objects

3. Catalog backups with RMAN (required for next step)

4. Execute block level recovery to clean up the database

So the main idea here is to facilitate the process of consistency check and recovery. You can find the tool in the KBA 2205309 and as of now there's only Linux/Unix version.

Any comments are welcome.

Cheers,

Thiago

Hi ,

Recently I tested rolling back SBP by following SBP readme guide, it surprises that the rolling back action will un-install all the SBP you ever installed. Which means that it will bring the database back to the base version without SBP. This has confirmed by SAP oracle colleague.

So the only way to revert back to the old SBP is to roll back SBP, then install the old SBP again.

We have to be careful before deciding to revert back to the old SBP. Because the old SBP package may not be available from SAP website.

By the way I tested in 11g database version (but I believe all the versions should be affected).

Best regards,
James

There is one case which took me even more than 1 hour to find correct way to fix it.

The symptom is quite simple:

- We found there is one datafile in v$datafile has name of 'MISSING9999'.

- DBA_DATA_FILES shows following:

- At the same time we can see the 'missing' datafile exists on disk:  (interesting !!!)

[3]ls -la /oracle/SID/sapdata48/sr3_40/

total 55138644

drwxr-xr-x   2 orasid   dba          512 Sep 16 13:59 .

drwxr-xr-x  12 orasid   dba          512 Sep 16 13:59 ..

-rw-r-----   1 orasid   dba      34358697984 Sep 24 15:59 sr3.data40

Remember that we have nice SAP note 19519 already for long time which described exactly what happened and how to fix it. What an easy job. But is it?

Let's go through the alert.log from very beginning to understand what happened. (I skip the non-important lines here to save space.)

|Tue Sep 15 16:54:56 AST 2015|

|alter tablespace PSAPSR3 add datafile '/oracle/SID/sapdata48/sr3_40/sr3.data40' size 32767M autoextend off|

|Tue Sep 15 17:10:37 AST 2015|

|Completed: alter tablespace PSAPSR3 add datafile '/oracle/SID/sapdata48/sr3_40/sr3.data40' size 32767M autoextend off|

...

|Tue Sep 15 17:58:53 AST 2015|

|ORA-19502 signalled during: alter tablespace PSAPSR3 add datafile '/oracle/SID/sapdata48/sr3_41/sr3.data41' size 32767M autoextend off...|

...

|Tue Sep 15 18:15:02 AST 2015|

|Errors in file /oracle/SID/saptrace/usertrace/SID_ora_15635.trc:|

|ORA-00600: internal error code, arguments: [kdtdelrow-2], [23], [23], [], [], [], [], []|

|ORA-01116: error in opening database file 54|

|ORA-01110: data file 54: '/oracle/SID/sapdata48/sr3_40/sr3.data40'|

|ORA-27041: unable to open file|

|SVR4 Error: 2: No such file or directory|

|Additional information: 3|

We have created one new datafile called sr3.data40. (file id #54) But unfortunately, something went wrong at disk side which causes the datafile can not be read/write at that moment. Which leads the datafile to be MISSING. So the original problem was disk issue, and there is nothing wrong with Oracle dictionary nor control file. As long as we take back the file system, which contains this datafile, everything will go back to normal.

As we do confirmed that the datafile itself exists currently, looks to me that the disk issue has been resolved. Why the datafile is still reported to be MISSING? (*1)

And the interesting story goes on.

|Wed Sep 16 10:14:16 AST 2015|

|/* BRSPACE */ ALTER DATABASE BACKUP CONTROLFILE TO '/oracle/SID/sapreorg/serkinmy/cntrlSID.old'|

Thanks God that we had a backup of control file soon after. So the complete structure change of datafiles are recorded in this control file at least. And can I use it for recovery if needed? (*2)

|Wed Sep 16 13:45:34 AST 2015|

|Completed: CREATE CONTROLFILE REUSE DATABASE "SID" RESETLOGS  NOARCHIVELOG|

We tried several times to create new control file to fix the problem. I do not know why we wanted to do so, but I believe somebody did not really understand what was exactly wrong here, and most likely the alert.log was not read carefully. It makes the things looks more complicated.

|Wed Sep 16 13:59:48 AST 2015|

|/* BRSPACE */ alter tablespace PSAPSR3 add datafile '/oracle/SID/sapdata48/sr3_40/sr3.data40' size 20000M autoextend off|

|Wed Sep 16 14:05:27 AST 2015|

|Completed: /* BRSPACE */ alter tablespace PSAPSR3 add datafile '/oracle/SID/sapdata48/sr3_40/sr3.data40' size 20000M autoextend off|

Much worse thing is that we created a new datafile which has the same name of the lost data file under the same location. This is very cool to make things more and more complicated.

Now we understand why the 'missing' datafile does exist on disk. (*1) If we look back to DBA_DATA_FILES, there is another line:

|Wed Sep 23 17:38:31 AST 2015|

|Completed: /* BRSPACE */ alter tablespace PSAPSR3 add datafile '/oracle/SID/sapdata45/sr3_45/sr3.data45' size 30000M autoextend off|

And we even added several new data files afterwards, which is currently in use. So this makes the good backup of control file to be useless for me any more. (*2)

Now everything is clear now by reading the whole story of alert.log file:

1. The missing datafile is caused by I/O problem at the beginning.
2. A new datafile which has the same name was created under the same location as the missing one.

What is the solution to fix this problem then?

I think we have to reorganize the complete tablespace PSAPSR3, which contains the missing datafile to fix this problem.

Hi Folks,

Since oracle 12c, SAP recommends to use "oracle" user instead of "orasid" for database. Now there is a hot question, what if I have multiple databases with different <DBSID> in the same host? How to let one "oracle" user to admin all the databases in the same host?

In the past, we can easily achieve this, because every different databases with different <DBSID> have their unique admin user "orasid". Also all the environment variables of the database are existing in login scripts. As soon as "orasid" log on the OS, the environment variables will automatically set from home profile. This is an permanent change.

However since "oracle" comes to use, we cannot use login script to set environment variables like "orasid" (we have multiple databases installed).

But we need to run ". oraenv" as mentioned in note

"1915317 - Migrating Software Owner to 'oracle'" where we can set all the oracle related environment variables for <DBSID>. Afterwards we can have unique environment variables for the < DBSID> we are interested in. By using this way, we can switch to different database without problem for "oracle" user.

So the differences on environment variable settings:

1. "orasid": uses login profile to set, different "orasid" users have different login profile in the its own home directory, this is a permanent change
2. "oracle": we have to manually run ". oraenv" where asks to input <DBSID> to adjust environment variables, this is a temporary change

By the way we need to use "source" command before oraenv, like ". oraenv".

Best regards,
James

Hi Folks,

Migrate to <oracle> user is seemingly a mandatory step in current version of SAP oracle 12c upgrade guide. Although it is recommended to use oracle user since oracle 12c, but still some people do not want to migrate to oracle user for reasons.

SAP note 1915323 clearly mentions <orasid> is still supported in oracle 12.1 version, it is not must to use <oracle> user.

SAP note 1915323:

1.Question #1: Is it supported not to migrate to software owner 'oracle' and continue using 'ora<dbsid>' as Oracle software owner?

Answer #1: You can choose between user concept ‘Oracle Standard’ and user concept ‘SAP Classic’.

Also SAP oracle upgrade guide will be updated later, in note 1915323, it says the version of using <orasid> guide is supposed to release by mid of September. Unfortunately I checked the guide today, it is not updated yet. So please be more patient if you do not want to migrate to <oracle> user for 12.1 oracle version.


By the way maybe some day it is mandatory to use <oracle> user, but not yet.

Best regards,
James

Hi Folks,

Recently I am testing and verifying the problems&questions regarding migrating to "oracle" user on Oracle 12c.

We have an SAP note "1888485 - Database Parameter for 12.1.0.2" which tells us the right database parameters should be set. Please verify those
"Database Parameter Settings for Specific Platforms" database parameters carefully.

use_large_pages is the one that adjusted for oracle owner, after migrating "oracle" user, you have to adjust it from OS level as well according to SAP note 1672954.

We have met a very bad performance issue after upgrading to oracle 12c, it took us days to dig the cause in wrong way. We were getting deeper and deeper into wrong road for days (reviewing sqls, io, etc.). Luckily we finally found this parameter use_large_pages needs to adjusted for oracle user. Keeping parameters right is the prerequisite for tuning database performance.

Best regards,
James

Hi all,

Have you recently searched for the right information/documentation to manage your Oracle database!? If so, did you find it quickly?

I've released a tool to find the right document to proper manage your Oracle database using BR*tools, it's a public tool released on SAP Help Portal.

There, we offer the same selection criteria as before plus some enhanced search capabilities. We managed to get rid of some redundancies. So now, you’ll find the relevant content faster.

Refer to following link and enjoy.

SAP On Oracle - Managing your Oracle database with BR*Tools

Hope this helps you.

Best regards,

Jeisson Leonel

Hi all,

There are some important aspects that should be verified up front in order to have a clean up Oracle database and consequently prevent data loss due to corruptions, consider following recommendation.

• Run a FULL database consistency check during a backup cycle (once at least);
• Always ensure to have a valid backup in place, by performing a consistency check of the database backups. This is VERY IMPORTANT and will for sure avoid unusable backups;
• You should schedule 'Validate Structure' and 'Verify database' via DB13 to perform such checks;
• Using RMAN is also a possible way to easily check the database backups and perform the consistency check, you can use brbackup option '-w only_rmw';
• The usage of BR*tools (brbackup, brconnect) is recommended for such checks, do not perform manual checks if possible;
• It is also very important to perform regular OS/Hardware checks as corruptions are mostly related to such areas.

All detailed information concerning consistency checks is described on SAP Note 23345.

Concerning cases that corruption is found, SAP recommendation is to read carefully SAP Note 365481 and proceed exactly as per solution provided.

There are some cases that unnecessary restore/recovery actions are taken and lead to big problems and data loss, so unless you are sure about the steps you are about to take otherwise contact SAP support.

Best regards,

Jeisson Leonel

• Database Upgrade Assistant (DBUA)
• Manual Upgrade
• Oracle Data Pump Export
• Standard Import / Original Export and Import
• Oracle Transportable Tablespaces
• Oracle Data Guard SQL Apply (Logical Standby)

Each method has its pros and cons, which are described in detail in the Oracle White paper 486336 (http://www.oracle.com/technetwork/database/upgrade/11gr2-upgrade-methods-wp-2011-486336.pdf).

For low and medium important Databases, the DBUA is a good option to consider using as the first choice. Data Pump is better than the old import / export because it is faster and it can be parallelized.

For critical Databases, Export / Import using Datapump is the safest and most conservative. A copy of the original DB will be exported and imported into a new oracle home.

	SAP on Oracle Key Topics Find resources for administering your Oracle database with an SAP system, refer to following topics: Key Notes for SAP on Oracle Oracle DBA Overview Backup and Recovery Performance Real Application Clusters (RAC) Security
	Further SAP on Oracle content To stay tuned with all SAP on Oracle related community, refer to following links: SCN Community SAP on Oracle Wiki SAP on Oracle Twitter

Do you have any comments or suggestions? Please post them below.

SAP has certified Oracle12c for Business Suite and present DB's now can be upgraded to Oracle12c.

To upgrade to Oracle12c in your environment you need to go through below phases for smooth upgrade . Documents have been published for each phase so that it's easy to identify all the phases and plan to upgrade to Oracle12c can be smooth

1. Prerequisites and Upgrade checklist

Please refer to document on SCN:

ORACLE 12C upgrade Prerequisites and Checklist

2. Migrate the Oracle software from user ora<dbsid> to user oracle

This step is optional but make sure that Oracle12c installation is done with user oracle

Please refer to document on SCN:

ORACLE 12c Upgrade Migrate the Oracle software to oracle from orasid

3.   Preupgrade steps and preparation on Source System and Database

Please refer to document on SCN:

ORACLE12C Upgrade: Preupgrade steps

4.  Database Upgrade steps

Please refer to document on SCN:

ORACLE12C Upgrade: Database Upgrade

5.  Post Upgrade Tasks

Please refer to document on SCN:

ORACLE12C Upgrade: Post Upgrade Tasks

6. Updating the Oracle Instant Client

Please refer to document on SCN:

ORACLE12C Upgrade: Updating the Oracle Instant Client

Introduction

This blog post is inspired by a question from an attendee of Sigrid Keydana's DOAG 2015 conference session called "Raising the fetchsize, good or bad? Exploring memory management in Oracle JDBC 12c". Basically it was a question about what the wait event "SQL*Net more data to client" represents and what it really measures. In general you may use the following steps, if you don't know what a particular wait event means:

1. Check the official Oracle documentation or My Oracle Support for an explanation
2. Google for trustful sources / websites
3. Do your own research

So let's do this procedure for wait event "SQL*Net more data to client" as an example:

1. Official Oracle documentation "SQL*Net more data to client": "The server process is sending more data/messages to the client. The previous operation to the client was also a send. Wait Time: The actual time it took for the send to complete"
2. Google for trustful sources / websites, e.g. blog post "SQL*Net message to client vs SQL*Net more data to client" by Tanel Poder: "I’ll reiterate that both SQL*Net message to client and SQL*Net more data to client waits only record the time it took to write the return data from Oracle’s userland SDU buffer to OS kernel-land TCP socket buffer. Thus the wait times of only microseconds. Thanks to that, all of the time a TCP packet spent flying towards the client is actually accounted in SQL*Net message from client wait statistic."

Do you see the slightly but important difference between the official documentation and Tanel Poder's blog content? The Oracle documentation states that it measures the time it took for the send to complete, but Tanel says that it only measures the time to write the return data from Oracle’s userland SDU buffer to OS kernel-land TCP socket buffer. In such cases i would trust Tanel much more, but let's verify this on our own as we would need to do it anyway, if we do not find any trustful source via Google. The following approach can be applied to any wait event, if you want to know what it really represents.

Basics of Oracle's code instrumentation

I will not repeat all of the basics here as there is already a great source of information about this in the internet, but i am referring to Cary Millsap's "The Method R Guide to MASTERING ORACLE TRACE DATA" to get a basic understanding of what we are talking about.

Database call instrumentation

System call instrumentation

Basically Oracle's code instrumentation is very simple and straight forward, but we need to know where the measurement starts and where it exactly ends in the Oracle code, if we want to get an understanding of the measured time of Oracle functions and system calls. For the beginning the only thing we need to know is the following:

1. Oracle function kslwtbctx represents the start of a wait event
2. Oracle function kslwtbctx calls the function kskthbwt to get/store the corresponding wait event (number)
3. Oracle function kslwtectx represents the end of a wait event
4. Oracle function kslwtectx calls the function kskthewt to get/store the corresponding wait event (number)

Let's see what Oracle really measures in case of wait event "SQL*Net more data to client".

Deep dive into Oracle's wait interface for wait event "SQL*Net more data to client" with DTrace

The following demo was run on an Oracle database (12.1.0.2) on Solaris 11.2 (SunOS SOL 5.11 11.2 i86pc i386 i86pc).

Create the base table, data set and SDU configuration

SQL> create table TEST (a varchar2(3000));

SQL> insert into TEST values (rpad('x',3000,'x'));

SQL> commit;

shell> cat $TNS_ADMIN/sqlnet.ora

DEFAULT_SDU_SIZE=2048

Wait event verification

SQL> select EVENT#, EVENT_ID, NAME from V$EVENT_NAME where NAME in

     ('SQL*Net more data to client','SQL*Net message to client','SQL*Net message from client');

    EVENT#   EVENT_ID NAME

---------- ---------- ----------------------------------------------------------------

       384 2067390145 SQL*Net message to client

       386  554161347 SQL*Net more data to client

       388 1421975091 SQL*Net message from client

SQL> select * from TEST;

Network package dumps

At first let's have a look at the SQL*Net trace on server side (parameter TRACE_LEVEL_SERVER = SUPPORT in sqlnet.ora) to verify the network packages and get an idea what we need to look at in the Oracle code.

...

2015-12-09 18:51:45.874331 : nsbasic_brc:entry: oln/tot=0

2015-12-09 18:51:45.874378 : nttfprd:entry

2015-12-09 18:52:04.455082 : nttfprd:socket 17 had bytes read=275

2015-12-09 18:52:04.455311 : nttfprd:exit

2015-12-09 18:52:04.455358 : nsbasic_brc:type=6, plen=275

2015-12-09 18:52:04.455428 : nsbasic_brc:what=1, tot =275

2015-12-09 18:52:04.455475 : nsbasic_brc:packet dump

2015-12-09 18:52:04.455519 : nsbasic_brc:01 13 00 00 06 00 00 00  |........|

…

2015-12-09 18:52:04.456573 : nsbasic_brc:00 00 00 00 12 73 65 6C  |.....sel|

2015-12-09 18:52:04.456612 : nsbasic_brc:65 63 74 20 2A 20 66 72  |ect.*.fr|

2015-12-09 18:52:04.456653 : nsbasic_brc:6F 6D 20 54 45 53 54 01  |om.TEST.|

…

2015-12-09 18:52:04.456973 : nsbasic_brc:exit: oln=0, dln=265, tot=275, rc=0

2015-12-09 18:52:04.457022 : nioqrc:exit

2015-12-09 18:52:04.457377 : nioqsn:entry

2015-12-09 18:52:04.457423 : nioqsn:exit

2015-12-09 18:52:04.457583 : nioqsn:entry

2015-12-09 18:52:04.457656 : nsbasic_bsd:entry

2015-12-09 18:52:04.457694 : nsbasic_bsd:tot=0, plen=1967.

2015-12-09 18:52:04.457733 : nttfpwr:entry

2015-12-09 18:52:04.457904 : nttfpwr:socket 17 had bytes written=1967

2015-12-09 18:52:04.457954 : nttfpwr:exit

2015-12-09 18:52:04.457996 : nsbasic_bsd:packet dump

…

2015-12-09 18:52:04.458909 : nsbasic_bsd:0B 78 78 78 78 78 78 78  |.xxxxxxx|

2015-12-09 18:52:04.458946 : nsbasic_bsd:78 78 78 78 78 78 78 78  |xxxxxxxx|

2015-12-09 18:52:04.458982 : nsbasic_bsd:78 78 78 78 78 78 78 78  |xxxxxxxx|

…

2015-12-09 18:52:04.468161 : nsbasic_bsd:exit (0)

2015-12-09 18:52:04.468201 : nioqsn:exit

2015-12-09 18:52:04.468312 : nioqrc:entry

2015-12-09 18:52:04.468356 : nsbasic_bsd:entry

2015-12-09 18:52:04.468397 : nsbasic_bsd:tot=0, plen=1317.

2015-12-09 18:52:04.468443 : nttfpwr:entry

2015-12-09 18:52:04.468554 : nttfpwr:socket 17 had bytes written=1317

2015-12-09 18:52:04.468601 : nttfpwr:exit

2015-12-09 18:52:04.468642 : nsbasic_bsd:packet dump

2015-12-09 18:52:04.468718 : nsbasic_bsd:05 25 00 00 06 00 00 00  |.%......|

2015-12-09 18:52:04.468765 : nsbasic_bsd:00 00 78 78 78 78 78 78  |..xxxxxx|

2015-12-09 18:52:04.468805 : nsbasic_bsd:78 78 78 78 78 78 78 78  |xxxxxxxx|

2015-12-09 18:52:04.468846 : nsbasic_bsd:78 78 78 78 78 78 78 78  |xxxxxxxx|

…

2015-12-09 18:52:04.475756 : nsbasic_bsd:exit (0)

2015-12-09 18:52:04.475799 : nsbasic_brc:entry: oln/tot=0

2015-12-09 18:52:04.475840 : nttfprd:entry

2015-12-09 18:52:04.475891 : nttfprd:socket 17 had bytes read=21

2015-12-09 18:52:04.475931 : nttfprd:exit

2015-12-09 18:52:04.475971 : nsbasic_brc:type=6, plen=21

2015-12-09 18:52:04.476011 : nsbasic_brc:what=1, tot =21

2015-12-09 18:52:04.476050 : nsbasic_brc:packet dump

2015-12-09 18:52:04.476091 : nsbasic_brc:00 15 00 00 06 00 00 00  |........|

2015-12-09 18:52:04.476130 : nsbasic_brc:00 00 03 05 1D 01 00 00  |........|

2015-12-09 18:52:04.476181 : nsbasic_brc:00 0F 00 00 00           |.....   |

2015-12-09 18:52:04.476228 : nsbasic_brc:exit: oln=0, dln=11, tot=21, rc=0

2015-12-09 18:52:04.476269 : nioqrc:exit

2015-12-09 18:52:04.476441 : nioqsn:entry

2015-12-09 18:52:04.476488 : nioqsn:exit

…

Oracle needs two SDU buffers (of 2048 bytes) to send my table data (a lot of x characters) to the client. The first SDU buffer is filled up to 1967 bytes and written to the network socket and the second one is filled up to 1317 bytes with the rest of data and also written to the network socket. Let's crosscheck how Oracle accounts this in the Oracle wait interface.

DTracing the wait interface

I used a slightly modified version of Brendan Gregg's DTrace wrapper script (print decimal instead of hex values) called dapptrace. The symbol "->" indicates the entry into a C function and the symbol "<-" indicates the return from a C function. I marked the previously mentioned code instrumentation parts as red and the code that is actually measured by this particular wait event as orange.

shell> ./dapptrace.sh -UFp 1267

…                                                                                                                                                                                                                                                                                                                                                                                                                                    -> oracle:kslwtbctx(-139639050531264, 1, 0)

-> libc.so.1:gethrtime%hrt(6966823560, 0, 255)

<- libc.so.1:gethrtime%hrt = 256

-> oracle:kskthbwt(0, 3268360526, 386)

-> oracle:kgskbwt(-139639066914912, 7241882880, 0)

-> oracle:kstmgethighresustick(-139639066914912, -139639085284392, 7025938216)

<- oracle:kstmgethighresustick = 117

-> oracle:kgskdecr(-139639066914912, 1, 7025943768)

<- oracle:kgskdecr = 311

-> oracle:kgskupdbalance(-139639066914912, 7241882880, 0)

-> oracle:slcpums(-139639050532496, 7241882880, 0)

-> libc.so.1:_lwp_info(-139639050533200, 7241882880, 0)

<- libc.so.1:_lwp_info = 27

<- oracle:slcpums = 141

-> oracle:ksklheld(60507156, 456459, -139639085284392)

<- oracle:ksklheld = 109

<- oracle:kgskupdbalance = 325

-> oracle:kgskrunnextint(-139639066914912, 7241882880, 0)

<- oracle:kgskrunnextint = 1736

<- oracle:kgskbwt = 937

-> oracle:ksdhng_cache_scpd_ver_get(0, 7241882880, 0)

<- oracle:ksdhng_cache_scpd_ver_get = 26

-> oracle:ksdhng_cache_scpd_ver_get(1, 7241882880, 0)

<- oracle:ksdhng_cache_scpd_ver_get = 26

<- oracle:kskthbwt = 526

-> libc.so.1:memcpy(7260698720, -139639050531160, 48)

-> oracle:kslwt_start_snapshot(7260698616, 7260698616, 1)

<- oracle:kslwt_start_snapshot = 266

<- oracle:kslwtbctx = 1321

-> oracle:nioqsn(-139639066788456, 0, 6402865975)

-> libc.so.1:memcpy(-139639084441095, 6402865975, 20)

-> oracle:nsbsend(-139639084713376, -139639084713048, 1)

-> oracle:nsbasic_bsd(-139639084713376, -139639084713048, 1)

-> oracle:nttfpwr(-139639084639792, -139639084443042, -139639050534232)

-> libc.so.1:write(17, -139639084443042, 1967)

-> libc.so.1:__write(17, -139639084443042, 1967)

<- libc.so.1:__write = 22

<- libc.so.1:write = 252

<- oracle:nttfpwr = 277

<- oracle:nsbasic_bsd = 525

-> libc.so.1:memcpy(-139639084443032, 6402865995, 235)

<- oracle:nioqsn = 550

-> oracle:kslwtectx(-139639050531264, 6402865995, 6402866230)

-> libc.so.1:gethrtime%hrt(-139639050531264, 6402865995, 6402866230)

<- libc.so.1:gethrtime%hrt = 256

-> oracle:kslwt_end_snapshot(7260698616, 7260698616, 1)

<- oracle:kslwt_end_snapshot = 586

-> oracle:kslwt_update_stats_int(7260698616, 7260698616, 30)

-> oracle:kews_update_wait_time(7, 1652, -139639066773184)

<- oracle:kews_update_wait_time = 45

<- oracle:kslwt_update_stats_int = 733

-> oracle:kskthewt(3268362178, 386, 10)

-> oracle:kgskewt(-139639066914912, 7241882880, 7241882880)

<- oracle:kgskewt = 87

-> oracle:kgskewtint(-139639066914912, 7241882880, 7241882880)

-> oracle:kstmgethighresustick(-139639066914912, 7241882992, 1610635576)

<- oracle:kstmgethighresustick = 117

-> oracle:kgskmodruncnt(-139639066914912, 7241882880, 0)

-> oracle:ksklheld(-139639066914912, 1, 0)

<- oracle:ksklheld = 109

<- oracle:kgskmodruncnt = 435

-> oracle:slcpums(-139639050532160, 2, 4)

-> libc.so.1:_lwp_info(-139639050532912, 2, 4)

<- libc.so.1:_lwp_info = 27

<- oracle:slcpums = 141

<- oracle:kgskewtint = 972

<- oracle:kskthewt = 636

<- oracle:kslwtectx = 994

…

-> oracle:kslwtbctx(-139639050487328, 33, 11)

-> libc.so.1:gethrtime%hrt(6966823560, 0, 255)

<- libc.so.1:gethrtime%hrt = 256

-> oracle:kskthbwt(0, 3268369142, 388)

-> oracle:kgskbwt(-139639066914912, 7241882880, 0)

-> oracle:kstmgethighresustick(-139639066914912, -139639085284392, 7025938216)

<- oracle:kstmgethighresustick = 117

-> oracle:kgskdecr(-139639066914912, 1, 7025943768)

<- oracle:kgskdecr = 311

-> oracle:kgskupdbalance(-139639066914912, 7241882880, 0)

-> oracle:slcpums(-139639050488560, 7241882880, 0)

-> libc.so.1:_lwp_info(-139639050489264, 7241882880, 0)

<- libc.so.1:_lwp_info = 27

<- oracle:slcpums = 141

-> oracle:ksklheld(60630990, 464768, -139639085284392)

<- oracle:ksklheld = 109

<- oracle:kgskupdbalance = 325

-> oracle:kgskrunnextint(-139639066914912, 7241882880, 0)

<- oracle:kgskrunnextint = 1736

<- oracle:kgskbwt = 937

-> oracle:ksdhng_cache_scpd_ver_get(0, 7241882880, 0)

<- oracle:ksdhng_cache_scpd_ver_get = 26

-> oracle:ksdhng_cache_scpd_ver_get(1, 7241882880, 0)

<- oracle:ksdhng_cache_scpd_ver_get = 26

<- oracle:kskthbwt = 526

-> libc.so.1:memcpy(7260698720, -139639050487224, 48)

-> oracle:kslwt_start_snapshot(7260698616, 7260698616, 1)

<- oracle:kslwt_start_snapshot = 266

<- oracle:kslwtbctx = 1321

-> oracle:nioqrc(-139639066788456, 0, -139639050486724)

-> oracle:nsbsend(-139639084713376, -139639084713048, 1)

-> oracle:nsbasic_bsd(-139639084713376, -139639084713048, 1)

-> oracle:nttfpwr(-139639084639792, -139639084443042, -139639050492280)

-> libc.so.1:write(17, -139639084443042, 1317)

-> libc.so.1:__write(17, -139639084443042, 1317)

<- libc.so.1:__write = 22

<- libc.so.1:write = 252

<- oracle:nttfpwr = 277

<- oracle:nsbasic_bsd = 525

-> oracle:nsbrecv(-139639084713376, -139639084712880, -139639050491220)

-> oracle:nsbasic_brc(-139639084713376, -139639084712880, -139639050491220)

-> oracle:nttfprd(-139639084639792, -139639084445250, 2064)

-> libc.so.1:read(17, -139639084445250, 2064)

-> libc.so.1:__read(17, -139639084445250, 2064)

<- libc.so.1:__read = 22

<- libc.so.1:read = 252

<- oracle:nttfprd = 381

<- oracle:nsbasic_brc = 837

-> libc.so.1:memcpy(-139639050486724, -139639084445240, 1)

<- oracle:nioqrc = 919

-> oracle:kslwtectx(-139639050487328, -139639084445240, -139639084445239)

-> libc.so.1:gethrtime%hrt(-139639050487328, -139639084445240, -139639084445239)

<- libc.so.1:gethrtime%hrt = 256

-> oracle:kslwt_end_snapshot(7260698616, 7260698616, 1)

<- oracle:kslwt_end_snapshot = 586

-> oracle:kslwt_update_stats_int(7260698616, 7260698616, 30)

-> oracle:kews_update_wait_time(6, 1773, -139639066773184)

<- oracle:kews_update_wait_time = 45

<- oracle:kslwt_update_stats_int = 733

-> oracle:kskthewt(3268370915, 388, 10)

-> oracle:kgskewt(-139639066914912, 7241882880, 7241882880)

<- oracle:kgskewt = 87

-> oracle:kgskewtint(-139639066914912, 7241882880, 7241882880)

-> oracle:kstmgethighresustick(-139639066914912, 7241882992, 1610635576)

<- oracle:kstmgethighresustick = 117

-> oracle:kgskmodruncnt(-139639066914912, 7241882880, 0)

-> oracle:ksklheld(-139639066914912, 1, 0)

<- oracle:ksklheld = 109

<- oracle:kgskmodruncnt = 435

-> oracle:slcpums(-139639050488224, 2, 4)

-> libc.so.1:_lwp_info(-139639050488976, 2, 4)

<- libc.so.1:_lwp_info = 27

<- oracle:slcpums = 141

<- oracle:kgskewtint = 972

<- oracle:kskthewt = 636

<- oracle:kslwtectx = 994

…

After dtracing the Oracle process it becomes obvious that Oracle instrumented the first SDU buffer system call write (+ the calling Oracle functions) of 1967 bytes as "SQL*Net more data to client" (wait event number 386) in this case. The Solaris write() system call is defined as the following "If fildes refers to a socket, write() is equivalent to send(3SOCKET) with no flags set", but this is just buffered and does not wait until it gets an acknowledge from the peer (afaik). The second SDU buffer system call write (+ the calling Oracle functions) of 1317 bytes is accounted as "SQL*Net message from client" (wait event number 388), but if you look closely Oracle also accounts the system call read from this socket as "SQL*Net message from client". So basically it represents the wait time to transfer the data until it gets a message back from the client.

You also can correlate each C function flow with the SQL*Net trace as the function names are also printed in the trace file - just as an example (function nttfprd() calls system call read()) .

SQL*Net trace

…

2015-12-09 18:52:04.475840 : nttfprd:entry

2015-12-09 18:52:04.475891 : nttfprd:socket 17 had bytes read=21

2015-12-09 18:52:04.475931 : nttfprd:exit

…

DTrace

…

-> oracle:nttfprd(-139639084639792, -139639084445250, 2064)

-> libc.so.1:read(17, -139639084445250, 2064)

-> libc.so.1:__read(17, -139639084445250, 2064)

<- libc.so.1:__read = 22

<- libc.so.1:read = 252

<- oracle:nttfprd = 381

…

Extended SQL trace and truss

I run an extended SQL trace (and truss on the process) as a last crosscheck to verify this research / findings.

shell> truss -p 1267

…

write(8, " P A R S E   # 1 8 4 4 6".., 105)     = 105

write(8, " E X E C   # 1 8 4 4 6 6".., 103)     = 103

write(8, " W A I T   # 1 8 4 4 6 6".., 126)     = 126

write(17, "07AF\0\006\0\0\0\0\01017".., 1967)   = 1967

write(8, " W A I T   # 1 8 4 4 6 6".., 131)     = 131

write(8, " F E T C H   # 1 8 4 4 6".., 106)     = 106

write(17, "051F\0\006\0\0\0\0\0 x x".., 1317)   = 1317

read(17, "\015\0\006\0\0\0\0\00305".., 2064)    = 21

write(8, " W A I T   # 1 8 4 4 6 6".., 128)     = 128

write(8, " F E T C H   # 1 8 4 4 6".., 104)     = 104

write(8, " S T A T   # 1 8 4 4 6 6".., 140)     = 140

write(8, " W A I T   # 1 8 4 4 6 6".., 126)     = 126

write(17, "\0 f\0\006\0\0\0\0\00401".., 102)    = 102

read(17, 0xFFFF80FFBDF991BE, 2064) (sleeping...)

…

This cross-check fits to everything before (aside from the byte difference of 1967 vs. 2191 bytes in the wait interface, which i can not explain), but what becomes obvious is that the wait event "SQL*Net message to client" also does not really send anything via network to the client (no corresponding write() system call) in this case.

Summary

The intention of this blog post was to demonstrate that you can get detailed insights into Oracle's code instrumentation, if you run DTrace on a process and check what is in-between the function flow of C function kslwtbctx and C function kslwtectx. This approach can be adapted to any wait event and may get very interesting in case of CPU driven waits (e.g. mutexes / latches, etc.) or just for educational purpose.

If you have any further questions - please feel free to ask or get in contact directly, if you need assistance by troubleshooting Oracle database (performance) issues. You can also follow or drop me a note on Twitter of course.

Hello All,

Recently while working on a system I ran into a situation where my main tablespace PSAPSR3 had ran into full utilization and all my sapdata's were utilized till 99%. Hence no way to extend my tablespace by adding space or extending TS (a typical quick fixes) or reorgs (long planed fix), as there were no space available.

This was my situation before

While researching I cam across notes

910389 - FAQ: Oracle Segment Shrinking

771929 - FAQ: Index fragmentation

And all my problem got solved in 30 mins itself.

To conclude, with all my experience, Segment shrinking, is indeed a very convenient way of recovering space from tabespace in oracle 10g onwards.

Do try it out yourself.

Below is a step by step procedure,

Prerequisite :-

1. Ensure your tablespace is SSM Auto

2. Table doesn't have LOB columns.

3. prepare a script file like attached. Which i have attached in the place of <table name> put table to be shrinked & in place of <owner> put TS owner. Keep it in any location as its just for monitoring the shrink of table.

Activity :-

1. Execute the script to get current status

     And check free space on the table

2. Enable row movement.

3. Trigger shrink

4. Disable Row Movement

it took 5 seconds for a 10 MB table & its done.

5. Check status now

6. I receovered 7 MB space for the tablespace.

Closing Point :

Do try it out yourself and comment if I have missed something.

Reference Notes :-

771929 - FAQ: Index fragmentation

910389 - FAQ: Oracle Segment Shrinking

Thanks

Abhijeet