Oracle Database 12c -- Database Hung with ORA-12751: cpu time or run time policy violation

Problem

An Oracle Database 12c version might get hung during high load hours for a considerable amount of time, like 30-60 minutes, with messages in the alert log like:

2022-10-24T05:37:59.996625+03:00
Thread 1 cannot allocate new log, sequence 1955502
Checkpoint not complete
  Current log# 101 seq# 1955501 mem# 0: +ORADATA1/<db>/ONLINELOG/group_101.1289.1115574715
  Current log# 101 seq# 1955501 mem# 1: +ORAFRA1/<db>/ONLINELOG/group_101.6109.1115574719
2022-10-24T05:43:34.633626+03:00
minact-scn: useg scan erroring out with error e:12751
2022-10-24T05:48:40.778478+03:00
minact-scn: useg scan erroring out with error e:12751
2022-10-24T05:58:53.097701+03:00
minact-scn: useg scan erroring out with error e:12751
Suspending MMON action 'Block Cleanout Optim, Undo Segment Scan' for 104400 seconds
2022-10-24T06:18:46.715311+03:00
AWR automatic snapshot ID (44334) has failed: DBID = 1695286113, CON_DBID = 1695286113, ERROR = 13509.
2022-10-24T06:23:52.620523+03:00
Suspending MMON action 'undo usage' for 104400 seconds
2022-10-24T06:28:53.602085+03:00
Suspending MMON slave action kerpiMmonSlvActFlush_ for 104400 seconds
2022-10-24T06:35:32.717140+03:00
Thread 1 advanced to log sequence 1955502 (LGWR switch)
  Current log# 102 seq# 1955502 mem# 0: +ORADATA1/<db>/ONLINELOG/group_102.1288.1115574727
  Current log# 102 seq# 1955502 mem# 1: +ORAFRA1/<db>/ONLINELOG/group_102.6108.1115574731

On the alert log extract the database halted for almost 1 hour

Other symptoms might be:

• AWR snapshots were not generated during this period
• new sessions could not be established 
  
• MMON trace file has the following error messages:

---------------------------------------------------
----- END DDE Action: 'ORA_12751_DUMP' (SUCCESS, 1 csec) -----
----- END DDE Actions Dump (total 1 csec) -----
KEBM: MMON action policy violation. 'undo usage' viol=1; err=12751
 
*** 2022-10-24T06:29:00.065494+03:00
DDE: Problem Key 'ORA 12751' was flood controlled (0x6) (no incident)
ORA-12751: cpu time or run time policy violation
*** KEWRAFLD: Error=12751 encountered by Auto Flush Local DB.
*** KEWRAFM1: Error=12751 encountered by Auto Flush Main.
KEBM: MMON action policy violation. 'AWR Auto Flush Task' viol=1; err=12751
*** KEWAPF: Other Flushing Failed, return Code: 2.
DDE: Problem Key 'ORA 12751' was flood controlled (0x6) (no incident)
 
*** 2022-10-24T06:34:08.225529+03:00
ORA-12751: cpu time or run time policy violation
*** KEWRAFLD: Error=12751 encountered by Auto Flush Local DB.
*** KEWRAFM1: Error=12751 encountered by Auto Flush Main.
KEBM: MMON action policy violation. 'AWR Auto Flush Task' viol=1; err=12751

Solution

During peak hours, when there are long and large transactions in a database, the MMON process starts scanning UNDO tablespaces aggressively which might cause database hang and errors. As the result database might seem hung and AWR snapshots not generated during this period 

To disable this MMON behavior, set the hidden parameter:

SQL> alter system set "_smu_debug_mode"=134217728 scope=both;

Reference:

• Alert Log Shows 'ORA-12751: cpu time or run time policy violation' and Associated MMON Trace Shows 'KEBM: MMON action policy violation. 'Block Cleanout Optim, Undo Segment Scan' viol=1; err=12751' (Doc ID 1671412.1)
• Bug 9040676   MMON ACTION POLICY VIOLATION. 'BLOCK CLEANOUT OPTIM, UNDO SEGMENT SCAN'