DBA_SCHEDULER_JOBs

This query will give you the names and the status of the jobs scheduled by DBMS_SCHEDULER:

SELECT job_name,enabled FROM DBA_SCHEDULER_JOBs;

sql> select job, what from dba_jobs where broken = 'Y' or failures > 0;

sql> select d.job_name,d.job_action from dba_scheduler_jobs d, sys.scheduler$_jobs where d.job_action = s.program_action and s.obj# = &myjobnum;

Then use "exec dbms_scheduler.disable" to disable each one as below:

I created a script "disablejobs.sql" as :

exec dbms_scheduler.disable('SYS.XMLDB_NFS_CLEANUP_JOB');
exec dbms_scheduler.disable('SYS.SM$CLEAN_AUTO_SPLIT_MERGE');
exec dbms_scheduler.disable('SYS.RSE$CLEAN_RECOVERABLE_SCRIPT');
exec dbms_scheduler.disable('SYS.FGR$AUTOPURGE_JOB');
exec dbms_scheduler.disable('SYS.BSLN_MAINTAIN_STATS_JOB');
exec dbms_scheduler.disable('SYS.DRA_REEVALUATE_OPEN_FAILURES');
exec dbms_scheduler.disable('SYS.HM_CREATE_OFFLINE_DICTIONARY');
exec dbms_scheduler.disable('SYS.ORA$AUTOTASK_CLEAN');
exec dbms_scheduler.disable('SYS.FILE_WATCHER');
exec dbms_scheduler.disable('SYS.PURGE_LOG');
exec dbms_scheduler.disable('SYS.MGMT_STATS_CONFIG_JOB');
exec dbms_scheduler.disable('SYS.MGMT_CONFIG_JOB');
exec dbms_scheduler.disable('SYS.RLM$SCHDNEGACTION');
exec dbms_scheduler.disable('SYS.RLM$EVTCLEANUP');



and then executed:

SQL> @disablejobs.sql

PL/SQL procedure successfully completed.

inbound connection timed out (ORA-3136)

WARNING: inbound connection timed out (ORA-3136)



The "WARNING: inbound connection timed out (ORA-3136)" in the alert log indicates that the client was not able to complete it's authentication within the period of time specified by parameter SQLNET.INBOUND_CONNECT_TIMEOUT.

You may also witness ORA-12170 without timeout error on the database server sqlnet.log file.
This entry would also have the clinet address which failed to get authenticated. Some applications or JDBC thin driver applications may not have these details.

From 10.2 onwards the default value of this parameter is 60 seconds, hence if the client is not able authenticate within 60 secs , the warning would appear in the alert log and the client connection will be terminated.

This timeout restriction was introduced to combat Denial of Service (DoS) attack whereby malicious clients attempt to flood database servers with connect requests that consumes resources.

There can be three main reasons for this error
Server gets a connection request from a malicious client which is not supposed to connect to the database , in which case the error thrown is the correct behavior. You can get the client address for which the error was thrown via sqlnet log file.
The server receives a valid client connection request but the client takes a long time to authenticate more than the default 60 seconds.
The DB server is heavily loaded due to which it cannot finish the client logon within the timeout specified.

The default value of 60 seconds is good enough in most conditions for the database server to authenticate a client connection. If its taking longer period, then its worth checking all the below points before going for the workadound:

1. Check whether local connection on the database server is sucessful & quick.

2. If local connections are quick ,then check for underlying network delay with the help of your network administrator.

3. Check whether your Database performance has degraded by anyway.

4. Check alert log for any critical errors for eg, ORA-600 or ORA-7445 and get them resolved first.
These critical errors might have triggered the slowness of the database server.



As a workaround to avoid only this warning messages, you can set the parameters SQLNET.INBOUND_CONNECT_TIMEOUT and INBOUND_CONNECT_TIMEOUT_listenername
to the value more than 60.

In server side sqlnet.ora file add SQLNET.INBOUND_CONNECT_TIMEOUT


SQLNET.INBOUND_CONNECT_TIMEOUT = 120In listener.ora file INBOUND_CONNECT_TIMEOUT_listenername


INBOUND_CONNECT_TIMEOUT_LISTENER = 110



From Oracle version 10.2.0.3 onwards the default value of INBOUND_CONNECT_TIMEOUT_
is 60 seconds. For previous releases it is zero by default.

Resizing / Recreating Online Redo Log Files

One of the best ways I have found to resize or recreate online redo log files and keep the current sequence is to perform it online. In this example, we will resize all online redo logs from 100MB to 250MB while the database is running and use SQL*Plus to drop/recreate them in stages.
Before looking at the tasks involved to perform the resize, let's look at the current online redo log groups and their sizes:

SQL> SELECT a.group#, a.member, b.bytes 2 FROM v$logfile a, v$log b WHERE a.group# = b.group#; GROUP# MEMBER BYTES ---------- ---------------------------------------- ------------ 1 /u03/app/oradata/ORA920/redo_g01a.log 104,857,600 1 /u04/app/oradata/ORA920/redo_g01b.log 104,857,600 1 /u05/app/oradata/ORA920/redo_g01c.log 104,857,600 2 /u03/app/oradata/ORA920/redo_g02a.log 104,857,600 2 /u04/app/oradata/ORA920/redo_g02b.log 104,857,600 2 /u05/app/oradata/ORA920/redo_g02c.log 104,857,600 3 /u03/app/oradata/ORA920/redo_g03a.log 104,857,600 3 /u04/app/oradata/ORA920/redo_g03b.log 104,857,600 3 /u05/app/oradata/ORA920/redo_g03c.log 104,857,600 9 rows selected.

Now let's take a look at the steps involved to resize / recreate all online redo log groups:

1. Make the last redo log CURRENT
   Force a log switch until the last redo log is marked "CURRENT" by issuing the following command:
   

   SQL> select group#, status from v$log; GROUP# STATUS ---------- ---------------- 1 CURRENT 2 INACTIVE 3 INACTIVE SQL> alter system switch logfile; SQL> alter system switch logfile; SQL> select group#, status from v$log; GROUP# STATUS ---------- ---------------- 1 INACTIVE 2 INACTIVE 3 CURRENT

2. Drop first redo log
   After making the last online redo log file the CURRENT one, drop the first online redo log:
   

   SQL> alter database drop logfile group 1; Database altered.

   As a DBA, you should already be aware that if you are going to drop a logfile group, it cannot be the current logfile group. I have run into instances; however, where attempting to drop the logfile group resulted in the following error as a result of the logfile group having an active status: SQL> ALTER DATABASE DROP LOGFILE GROUP 1; ALTER DATABASE DROP LOGFILE GROUP 1 * ERROR at line 1: ORA-01624: log 1 needed for crash recovery of instance ORA920 (thread 1) ORA-00312: online log 1 thread 1: '<file_name>' Easy problem to resolve. Simply perform a checkpoint on the database: SQL> ALTER SYSTEM CHECKPOINT GLOBAL; System altered. SQL> ALTER DATABASE DROP LOGFILE GROUP 1; Database altered.

3. Re-create dropped online redo log group
   Re-create the dropped redo log group with different size (if desired):
   

   SQL> alter database add logfile group 1 ( 2 '/u03/app/oradata/ORA920/redo_g01a.log', 3 '/u04/app/oradata/ORA920/redo_g01b.log', 4 '/u05/app/oradata/ORA920/redo_g01c.log') size 250m reuse; Database altered.

4. Force another log switch
   After re-creating the online redo log group, force a log switch. The online redo log group just created should become the "CURRENT" one:
   

   SQL> select group#, status from v$log; GROUP# STATUS ---------- ---------------- 1 UNUSED 2 INACTIVE 3 CURRENT SQL> alter system switch logfile; SQL> select group#, status from v$log; GROUP# STATUS ---------- ---------------- 1 CURRENT 2 INACTIVE 3 ACTIVE

5. Loop back to Step 2 until all logs are rebuilt
   After re-creating an online redo log group, continue to re-create (or resize) all online redo log groups until all of them are rebuilt.

After rebuilding (resizing) all online redo log groups, here is a snapshot of all physical files:

SQL> SELECT a.group#, a.member, b.bytes 2 FROM v$logfile a, v$log b WHERE a.group# = b.group#; GROUP# MEMBER BYTES ---------- ---------------------------------------- ------------ 1 /u03/app/oradata/ORA920/redo_g01a.log 262,144,000 1 /u04/app/oradata/ORA920/redo_g01b.log 262,144,000 1 /u05/app/oradata/ORA920/redo_g01c.log 262,144,000 2 /u03/app/oradata/ORA920/redo_g02a.log 262,144,000 2 /u04/app/oradata/ORA920/redo_g02b.log 262,144,000 2 /u05/app/oradata/ORA920/redo_g02c.log 262,144,000 3 /u03/app/oradata/ORA920/redo_g03a.log 262,144,000 3 /u04/app/oradata/ORA920/redo_g03b.log 262,144,000 3 /u05/app/oradata/ORA920/redo_g03c.log 262,144,000 9 rows selected.

Configuring Kernel Parameters For Oracle 10g Installation

This section documents the checks and modifications to the Linux kernel that should be made by the DBA to support Oracle Database 10g. Before detailing these individual kernel parameters, it is important to fully understand the key kernel components that are used to support the Oracle Database environment.



The kernel parameters and shell limits presented in this section are recommended values only as documented by Oracle. For production database systems, Oracle recommends that we tune these values to optimize the performance of the system.


Verify that the kernel parameters shown in this section are set to values greater than or equal to the recommended values.


Shared Memory : Shared memory allows processes to access common structures and data by placing them in a shared memory segment. This is the fastest form of Inter-Process Communications (IPC) available - mainly due to the fact that no kernel involvement occurs when data is being passed between the processes. Data does not need to be copied between processes .


Oracle makes use of shared memory for its Shared Global Area (SGA) which is an area of memory that is shared by all Oracle backup and foreground processes. Adequate sizing of the SGA is critical to Oracle performance since it is responsible for holding the database buffer cache, shared SQL, access paths, and so much more.


To determine all current shared memory limits, use the following :


# ipcs -lm
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 4194303
max total shared memory (kbytes) = 1073741824
min seg size (bytes) = 1


The following list describes the kernel parameters that can be used to change the shared memory configuration for the server:


1.) shmmax - Defines the maximum size (in bytes) for a shared memory segment. The Oracle SGA is comprised of shared memory and it is possible that incorrectly setting shmmax could limit the size of the SGA. When setting shmmax, keep in mind that the size of the SGA should fit within one shared memory segment. An inadequate shmmax setting could result in the following:
ORA-27123: unable to attach to shared memory segment


We can determine the value of shmmax by performing the following :


# cat /proc/sys/kernel/shmmax
4294967295


For most Linux systems, the default value for shmmax is 32MB. This size is often too small to configure the Oracle SGA. The default value for shmmax in CentOS 5 is 4GB which is more than enough for the Oracle configuration. Note that this value of 4GB is not the "normal" default value for shmmax in a Linux environment inserts the following two entries in the file /etc/sysctl.conf:


# Controls the maximum shared segment size, in bytes
kernel.shmmax = 4294967295


2.) shmmni : This kernel parameter is used to set the maximum number of shared memory segments system wide. The default value for this parameter is 4096. This value is sufficient and typically does not need to be changed. We can determine the value of shmmni by performing the following:


# cat /proc/sys/kernel/shmmni
4096


3.) shmall : This parameter controls the total amount of shared memory (in pages) that can be used at one time on the system. The value of this parameter should always be at least: We can determine the value of shmall by performing the following :


# cat /proc/sys/kernel/shmall
268435456


For most Linux systems, the default value for shmall is 2097152 and is adequate for most configurations. The default value for shmall in CentOS 5 is 268435456 (see above) which is more than enough for the Oracle configuration described in this article. Note that this value of 268435456 is not the "normal" default value for shmall in a Linux environment , inserts the following two entries in the file /etc/sysctl.conf:


# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 268435456


4.) shmmin : This parameter controls the minimum size (in bytes) for a shared memory segment. The default value for shmmin is 1 and is adequate for the Oracle configuration described in this article.We can determine the value of shmmin by performing the following:


# ipcs -lm | grep "min seg size"
min seg size (bytes) = 1


Semaphores :
After the DBA has configured the shared memory settings, it is time to take care of configuring the semaphores. The best way to describe a semaphore is as a counter that is used to provide synchronization between processes (or threads within a process) for shared resources like shared memory. Semaphore sets are supported in System V where each one is a counting semaphore. When an application requests semaphores, it does so using "sets". To determine all current semaphore limits, use the following:


# ipcs -ls
------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767


We can also use the following command:
# cat /proc/sys/kernel/sem
250 32000 32 128


The following list describes the kernel parameters that can be used to change the semaphore configuration for the server:


i.) semmsl - This kernel parameter is used to control the maximum number of semaphores per semaphore set. Oracle recommends setting semmsl to the largest PROCESS instance parameter setting in the init.ora file for all databases on the Linux system plus 10. Also, Oracle recommends setting the semmsl to a value of no less than 100.


ii.) semmni - This kernel parameter is used to control the maximum number of semaphore sets in the entire Linux system. Oracle recommends setting semmni to a value of no less than 100.


iii.) semmns - This kernel parameter is used to control the maximum number of semaphores (not semaphore sets) in the entire Linux system. Oracle recommends setting the semmns to the sum of the PROCESSES instance parameter setting for each database on the system, adding the largest PROCESSES twice, and then finally adding 10 for each Oracle database on the system. Use the following calculation to determine the maximum number of semaphores that can be allocated on a Linux system. It will be the lesser of:
SEMMNS -or- (SEMMSL * SEMMNI)


iv.) semopm - This kernel parameter is used to control the number of semaphore operations that can be performed per semop system call. The semop system call (function) provides the ability to do operations for multiple semaphores with one semop system call. A semaphore set can have the maximum number of semmslsemaphores per semaphore set and is therefore recommended to set semopm equal to semmsl in some situations. Oracle recommends setting the semopm to a value of no less than 100.


File Handles :
When configuring the Linux server, it is critical to ensure that the maximum number of file handles is large enough. The setting for file handles denotes the number of open files that you can have on the Linux system. Use the following command to determine the maximum number of file handles for the entire system:


# cat /proc/sys/fs/file-max
102312


Oracle recommends that the file handles for the entire system be set to at least 65536. We can query the current usage of file handles by using the following :


# cat /proc/sys/fs/file-nr
3072 0 102312


The file-nr file displays three parameters:
• Total allocated file handles
• Currently used file handles
• Maximum file handles that can be allocated


If we need to increase the value in /proc/sys/fs/file-max, then make sure that the ulimit is set properly. Usually for Linux 2.4 and 2.6 it is set to unlimited. Verify theulimit setting my issuing the ulimit command :


# ulimit
unlimited


IP Local Port Range :
Oracle strongly recommends to set the local port range ip_local_port_range for outgoing messages to "1024 65000" which is needed for systems with high-usage. This kernel parameter defines the local port range for TCP and UDP traffic to choose from.
The default value for ip_local_port_range is ports 32768 through 61000 which is inadequate for a successful Oracle configuration. Use the following command to determine the value of ip_local_port_range:


# cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000


Networking Settings :
With Oracle 9.2.0.1 and later, Oracle makes use of UDP as the default protocol on Linux for inter-process communication (IPC), such as Cache Fusion and Cluster Manager buffer transfers between instances within the RAC cluster.


Oracle strongly suggests to adjust the default and maximum receive buffer size (SO_RCVBUF socket option) to 1MB and the default and maximum send buffer size (SO_SNDBUF socket option) to 256KB.The receive buffers are used by TCP and UDP to hold received data until it is read by the application. The receive buffer cannot overflow because the peer is not allowed to send data beyond the buffer size window.


This means that datagrams will be discarded if they don't fit in the socket receive buffer, potentially causing the sender to overwhelm the receiver. Use the following commands to determine the current buffer size (in bytes) of each of the IPC networking parameters:


# cat /proc/sys/net/core/rmem_default
109568


# cat /proc/sys/net/core/rmem_max
131071


# cat /proc/sys/net/core/wmem_default
109568


# cat /proc/sys/net/core/wmem_max
131071


Setting Kernel Parameters for Oracle
If the value of any kernel parameter is different to the recommended value, they will need to be modified. For this article, I identified and provide the following values that will need to be added to the /etc/sysctl.conf file which is used during the boot process.


kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 1048576
net.core.rmem_max = 1048576
net.core.wmem_default = 262144
net.core.wmem_max = 262144


After adding the above lines to the /etc/sysctl.conf file, they persist each time the system reboots. If we would like to make these kernel parameter value changes to the current system without having to first reboot, enter the following command:


# /sbin/sysctl –p

SHMMAX and SHMALL for Oracle in Linux

SHMMAX and SHMALL are two key shared memory parameters that directly impact’s the way by which Oracle creates an SGA. Shared memory is nothing but part of Unix IPC System (Inter Process Communication) maintained by kernel where multiple processes share a single chunk of memory to communicate with each other.

While trying to create an SGA during a database startup, Oracle chooses from one of the 3 memory management models a) one-segment or b) contiguous-multi segment or c) non-contiguous multi segment. Adoption of any of these models is dependent on the size of SGA and values defined for the shared memory parameters in the linux kernel, most importantly SHMMAX.

So what are these parameters - SHMMAX and SHMALL?


SHMMAX is the maximum size of a single shared memory segment set in “bytes”.

silicon:~ #  cat /proc/sys/kernel/shmmax

536870912

SHMALL is the total size of Shared Memory Segments System wide set in “pages”.

silicon:~ #  cat /proc/sys/kernel/shmall

1415577

The key thing to note here is the value of SHMMAX is set in "bytes" but the value of SHMMALL is set in "pages".

What’s the optimal value for SHMALL?

As SHMALL is the total size of Shard Memory Segments System wide, it should always be less than the Physical Memory on the System and should also be less than sum of SGA’s of all the oracle databases on the server. Once this value (sum of SGA’s) hit the limit, i.e. the value of shmall, then any attempt to start a new database (or even an existing database with a resized SGA) will result in an “out of memory” error (below). This is because there won’t be any more shared memory segments that Linux can allocate for SGA.

ORA-27102: out of memory

Linux-x86_64 Error: 28: No space left on device.

So above can happen for two reasons. Either the value of shmall is not set to an optimal value or you have reached the threshold on this server.

Setting the value for SHMALL to optimal is straight forward. All you want to know is how much “Physical Memory” (excluding Cache/Swap) you have on the system and how much of it should be set aside for Linux Kernel and to be dedicated to Oracle Databases.

For e.g. Let say the Physical Memory of a system is 6GB, out of which you want to set aside 1GB for Linux Kernel for OS Operations and dedicate the rest of 5GB to Oracle Databases. Then here’s how you will get the value for SHMALL.

Convert this 5GB to bytes and divide by page size. Remember SHMALL should be set in “pages” not “bytes”.

So here goes the calculation.

Determine Page Size first, can be done in two ways. In my case it’s 4096 and that’s the recommended and default in most cases which you can keep the same. 

silicon:~ # getconf PAGE_SIZE

4096

or

silicon:~ # cat /proc/sys/kernel/shmmni

4096

Convert 5GB into bytes and divide by page size, I used the linux calc to do the math.

silicon:~ # echo "( 5 * 1024 * 1024 * 1024 ) / 4096 " | bc -l

1310720.00000000000000000000

Reset shmall and load it dynamically into kernel

silicon:~ # echo "1310720" > /proc/sys/kernel/shmall

silicon:~ # sysctl –p

Verify if the value has been taken into effect.

silicon:~ # sysctl -a | grep shmall

kernel.shmall = 1310720

Another way to look this up is

silicon:~ # ipcs -lm

------ Shared Memory Limits --------

max number of segments = 4096                          /* SHMMNI  */

max seg size (kbytes) = 524288                  /* SHMMAX  */

max total shared memory (kbytes) = 5242880      /* SHMALL  */

min seg size (bytes) = 1

To keep the value effective after every reboot, add the following line to /etc/sysctl.conf

echo “kernel.shmall = 1310720” >> /etc/sysctl.conf

Also verify if sysctl.conf is enabled or will be read during boot.

silicon:~ # chkconfig boot.sysctl

boot.sysctl  on

If returns “off”, means it’s disabled. Turn it on by running

silicon:~ # chkconfig boot.sysctl on

boot.sysctl  on

What’s the optimal value for SHMMAX?

Oracle makes use of one of the 3 memory management models to create the SGA during database startup and it does this in following sequence. First Oracle attempts to use the one-segment model and if this fails, it proceeds with the next one which's the contiguous multi-segment model and if that fails too, it goes with the last option which is the non-contiguous multi-segment model.

So during startup it looks for shmmax parameter and compares it with the initialization parameter *.sga_target. If shmmax > *.sga_target, then oracle goes with one-segment model approach where the entire SGA is created within a single shared memory segment.

But the above attempt (one-segment) fails if SGA size otherwise *.sga_target  > shmmax, then Oracle proceeds with the 2^nd option – contiguous multi-segment model. Contiguous allocations, as the name indicates are a set of shared memory segments which are contiguous within the memory and if it can find such a set of segments then entire SGA is created to fit in within this set. 


But if cannot find a set of contiguous allocations then last of the 3 option’s is chosen – non-contiguous multi-segment allocation and in this Oracle has to grab the free memory segments fragmented between used spaces.

So let’s say if you know the max size of SGA of any database on the server stays below 1GB, you can set shmmax to 1 GB. But say if you have SGA sizes for different databases spread between 512MB to 2GB, then set shmmax to 2Gigs and so on.

Like SHMALL, SHMMAX can be defined by one of these methods..

Dynamically reset and reload it to the kernel..

silicon:~ #  echo "536870912" >  /proc/sys/kernel/shmmax

silicon:~ #  sysctl –p           -- Dynamically reload the parameters.

Or use sysctl to reload and reset ..

silicon:~ #  sysctl -w kernel.shmmax=536870912

To permanently set so it’s effective in reboots…

silicon:~ #  echo "kernel.shmmax=536870912" >>  /etc/systctl.conf

Install doc for 11g recommends the value of shmmax to be set to "4GB – 1byte" or half the size of physical memory whichever is lower. I believe “4GB – 1byte” is related to the limitation on the 32 bit (x86) systems where the virtual address space for a user process can only be little less than 4GB. As there’s no such limitation for 64bit (x86_64) bit systems, you can define SGA’s larger than 4 Gig’s. But idea here is to let Oracle use the efficient one-segment model and for this shmmax should stay higher than SGA size of any individual database on the system.

mknod operation

Implement the mknod operation using the following example :

File Name: exppipe.sh
#!/bin/sh
. $HOME/.bash_profile
cd /home/oracle/mknode/
mknod exp_pipe p
gzip -cNf exp_data.dmp.gz &
exp demo/demo file=exp_pipe log=exp_data.log owner=demo statistics=none
rm -f exp_pipe p
fi

_kgl_large_heap_warning_threshold -- oracle 10.2 hidden parameter

Today in my environment i faced the below error message

Memory Notification: Library Cache Object loaded into SGA
Heap size 9940K
exceeds notification threshold (8192K)

for that i found the solution

In 10.2.0.2, the threshold was increased to 50MB after regression tests, so this should be a reasonable and recommended value.

So, you may try to increase the parameter  _kgl_large_heap_warning_threshold= 52428800

SQL> alter system set "_kgl_large_heap_warning_threshold"=52428800 scope=spfile ;
SQL> shutdown immediate
SQL> startup

Strongly recommend you to upgrade the database to the latest patchset which is 10.2.0.5,
where you can avoid such problems.