Rman DDboost issue when starting the backup

Issue:
we have installed DDboost for rman  backup configuration and the setup completed successfully,However when testing the backup got the below error.


allocated channel: c1
channel c1: SID=77 device type=SBT_TAPE
channel c1: Data Domain Boost API

Starting backup at 02-OCT-18
channel c1: starting full datafile backup set
channel c1: specifying datafile(s) in backup set
including current control file in backup set
channel c1: starting piece 1 at 02-OCT-18
released channel: c1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on c1 channel at 10/02/2018 11:38:37
ORA-19506: failed to create sequential file, name="03telatp_1_1", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
   sbtbackup: dd_rman_connect_to_backup_host failed

Root cause:
looks like some connection credential issStarting Control File and SPFILE Autobackup at 02-OCT-18

piece handle=LPMDMD_cf_c-292763313-20181002-01 comment=API Version 2.0,MMS Version 1.2.1.0

Solution:

make sure you have the  feed the variable like the below.

RUN {

ALLOCATE CHANNEL t1 TYPE SBT_TAPE PARMS 'BLKSIZE=1048576, SBT_LIBRARY=$ORACLE_HOME/lib/libddobk.so,ENV=(STORAGE_UNIT=$STORAGE_UNIT,BACKUP_HOST=$DATADOMAIN_HOST,ORACLE_HOME=$ORACLE_HOME)' FORMAT '%U-%d';

send 'set username $DDBOOST_USER password $PASSWORD servername $DATADOMAIN_HOST';

RELEASE CHANNEL t1;

}

Starting Control File and SPFILE Autobackup at 02-OCT-18

piece handle=DB_cf_c-292763313-20181002-01 comment=API Version 2.0,MMS Version 1.2.1.0

Finished Control File and SPFILE Autobackup at 02-OCT-18

released channel: c1

oracle golden gate 12.3 silent installation on Linux server

Here is the step by step to install the oracle golden gate using silent method


1)export the below variable
export JAVA_HOME=/usr/java6

  export PATH=$JAVA_HOME/bin:$PATH
   export LIBPATH=$JAVA_HOME/bin:$LIBPATH


export JAVA_HOME=/usr/bin/java

  export PATH=$JAVA_HOME/bin:$PATH
   export LIBPATH=$JAVA_HOME/bin:$LIBPATH

oggusr@<server_name>:/appshr/oggusrQL/9.5> uname -a
Linux <server_name> 2.6.32-573.7.1.el6.x86_64 #1 SMP Tue Sep 22 14:04:58 CDT 2015 x86_64 x86_64 x86_64 GNU/Linux


./runInstaller -silent -responseFile /usr/local/ogg/sw/fbo_ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp


2)edit the response file

cat oggcore.rsp

####################################################################
## Copyright(c) Oracle Corporation 2017. All rights reserved.     ##
##                                                                ##
## Specify values for the variables listed below to customize     ##
## your installation.                                             ##
##                                                                ##
## Each variable is associated with a comment. The comment        ##
## can help to populate the variables with the appropriate        ##
## values.                                                        ##
##                                                                ##
## IMPORTANT NOTE: This file should be secured to have read       ##
## permission only by the oracle user or an administrator who     ##
## own this installation to protect any sensitive input values.   ##
##                                                                ##
####################################################################

#-------------------------------------------------------------------------------
# Do not change the following system generated value.
#-------------------------------------------------------------------------------
oracle.install.responseFileVersion=/oracle/install/rspfmt_ogginstall_response_schema_v12_1_2


################################################################################
##                                                                            ##
## Oracle GoldenGate installation option and details                          ##
##                                                                            ##
################################################################################

#-------------------------------------------------------------------------------
# Specify the installation option.
# Specify ORA12c for installing Oracle GoldenGate for Oracle Database 12c and
#         ORA11g for installing Oracle GoldenGate for Oracle Database 11g
#-------------------------------------------------------------------------------
INSTALL_OPTION=ORA11g

#-------------------------------------------------------------------------------
# Specify a location to install Oracle GoldenGate
#-------------------------------------------------------------------------------
SOFTWARE_LOCATION=/ogg/ggs_home

#-------------------------------------------------------------------------------
# Specify true to start the manager after installation.
#-------------------------------------------------------------------------------
#START_MANAGER=

#-------------------------------------------------------------------------------
# Specify a free port within the valid range for the manager process.
# Required only if START_MANAGER is true.
#-------------------------------------------------------------------------------
#MANAGER_PORT=

#-------------------------------------------------------------------------------
# Specify the location of the Oracle Database.
# Required only if START_MANAGER is true.
#-------------------------------------------------------------------------------
#DATABASE_LOCATION=


################################################################################
##                                                                            ##
## Specify details to Create inventory for Oracle installs                    ##
## Required only for the first Oracle product install on a system.            ##
##                                                                            ##
################################################################################

#-------------------------------------------------------------------------------
# Specify the location which holds the install inventory files.
# This is an optional parameter if installing on
# Windows based Operating System.
#-------------------------------------------------------------------------------
#INVENTORY_LOCATION=

#-------------------------------------------------------------------------------
# Unix group to be set for the inventory directory.
# This parameter is not applicable if installing on
# Windows based Operating System.
#-------------------------------------------------------------------------------


SOFTWARE_LOCATION=/ogg/ggs_home
3)invoke the  runinstaller
./runInstaller -silent -responseFile /usr/local/ogg/sw/fbo_ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp

oggusr@<server_name>:/usr/local/ogg/sw/fbo_ggs_Linux_x64_shiphome/Disk1> ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp                                               <
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 3685 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 3642 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2018-09-07_01-07-05PM. Please wait ...oggusr@<server_name>:/usr/local/ogg/sw/fbo_ggs_Linux_x64_shiphome/Disk1> X11 connection rejected because of wrong authentication.
You can find the log of this install session at:
 /appshr/oraInventory/logs/installActions2018-09-07_01-07-05PM.log
The installation of Oracle GoldenGate Core was successful.
Please check '/appshr/oraInventory/logs/silentInstall2018-09-07_01-07-05PM.log' for more details.
Successfully Setup Software.

4)make sure the golden gate installation successful and check the logs

oracle golden gate installation error on IBM AIX

Issue:
when I tried to install oracle golden gate 12.3 with IBM AIX server got the below  error.
$ ./runInstaller
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB. Actual 2024 MB Passed
Checking swap space: must be greater than 150 MB. Actual 8192 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2018-07-10_02-44-46PM. Please wait ...cvidntcrkdwh1 {lpndwd} /appndwd/oradata/lpndwd.export/ogg/fbo_ggs_AIX_ppc_shiphome/Disk1>
$ Error: Port Library failed to initialize: -125 
Error: Could not create the Java Virtual Machine. 
Error: A fatal exception has occurred. Program will exit. 

Solution:
looks like Java environment was not set properly and set the Java  & other settings and issue will resolve.


root cause:
Jave & LIBPATH seeting has to be configured properly.

export JAVA_HOME=/usr/java6
export PATH=$JAVA_HOME/bin:$PATH



export LIBPATH=$JAVA_HOME/bin:$LIBPATH

How to get the DDL for the group in Netezza database

Issue:
How to get the DDL for the group in Netezza database

Solution:
Netezza we have utility to get the  DDL for the group.

nz_ddl_group <group_name>


/export/home/nz> nz_ddl_group TESTGRP

\echo
\echo *****  Creating group:  "TESTGRP"
CREATE GROUP TESTGRP WITH RESOURCE MINIMUM 1 RESOURCE MAXIMUM 5 ;

oracle DB crashed when opened after the full restore

Issue:
we have got the below issue  after opening the database from restore and it crashed the instance and throw the below error.
Tue Aug 07 22:47:12 2018
SMON: enabling cache recovery
Errors in file /apporadb1/oracle/product/11.2.0/log/diag/rdbms/oradb1/oradb1/trace/oradb1_ora_25952386.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Errors in file /apporadb1/oracle/product/11.2.0/log/diag/rdbms/oradb1/oradb1/trace/oradb1_ora_25952386.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Error 704 happened during db open, shutting down database
USER (ospid: 25952386): terminating the instance due to error 704
Instance terminated by USER, pid = 25952386
ORA-1092 signalled during: alter database open resetlogs...
opiodr aborting process unknown ospid (25952386) as a result of ORA-1092

Root cause:after checking this error and looks like  Db asking for upgrade and  we noticed the source database was 11.2.0 and target would be 11.2.0.4 .so in order to open the database ,we  just need to upgrade the database  to 11.2.0.4.

Solution:

1)startup the DB in upgrade mode

2)run the catupgrd.sql

3)run the utlrp.sql

4)do the sanity check on the database again.

what are the situations golden gate replicate process will abend?

There are many reasons behind when the golden gate replicat process abending,However I am listing few reasons for that and you can take action against it.

1)respective table-space 100% full
2)map table does not have right partition
3)discard file  size fully used and space exhusetd
4) trail file location (File system) 100% used
5)Whenever we get bad data and OGG unable to push the data to table
6)whenever replicat process unable to read the trail file

Solution:
look for the report file error & ggs error log an take action accordingally
.

How to redirect the oracle golden gate output to another files

Here is the simple code to redirect the  OGG output to new text file
you need to add the command on the text file and need to pass on this script.

cat obey_cmds.txt

info all

echo obey /appods/ggshp/v11110078/obey_cmds.txt | $GGS_HOME/ggsci > Replication_info_all.log

Nz_migrate throws the error

Issue:
when I tried to migrate the date from source to target ,got the below error and looks like all syntax and everything correct,even though got the below error.

nz@<netezza_server_name>:nz@NPS_HOST:/export/home/nz> nz_migrate -shost NPS_HOST -thost NPS_HOST -sdb DBA_DB -tdb NPS_DB_OLD  -tableFile  /export/home/nz/NPS_DB_OLD_tbl.log  -suser admin -spassword  ****** -tuser admin -tpassword ****** -CreateTargetTable YES -TruncateTargetTable NO -format binary  -threads 4

-bash: !0: event not found

Solution:
just use single quote on the password character,it will  run without any issues.double  quote wont work.

nz_migrate -shost NPS_HOST -thost NPS_HOST -sdb DBA_DB -tdb NPS_DB_OLD  -tableFile  /export/home/nz/NPS_DB_OLD_tbl.log  -suser admin -spassword  '****** '-tuser admin -tpassword '******' -CreateTargetTable YES -TruncateTargetTable NO -format binary  -threads 4

Step by step troubleshooting for Netezza database down

Here is the steps  troubleshoot for Netezza database down issue

1.look for the  contents of sysmgr.log under/nz/kit/log/sysmgr
2.look for the content of the pg.log under /nz/ki/log/postgres/
3.grep the timeline of the DB down from the pg.log


nz@<server_name>:/export/home/nz> grep -i "shut down" /nz/kit/log/postgres/pg.log.1
2018-06-05 05:07:16.858990 EDT [30134]  NOTICE:  database system was shut down at 2018-06-05 05:07:16 EDT
2018-06-05 14:00:39.323065 EDT [1983]   NOTICE:  database system was shut down at 2018-06-05 14:00:39 EDT
2018-06-05 16:29:28.631563 EDT [17920]  NOTICE:  database system was shut down at 2018-06-05 16:29:28 EDT
2018-06-05 17:21:01.303872 EDT [27842]  NOTICE:  database system was shut down at 2018-06-05 17:21:01 EDT

2018-06-05 18:35:51.314193 EDT [2921]   NOTICE:  database system was shut down at 2018-06-05 18:35:51 EDT

4.look for the core files and convert the core files to readable format.


  456  2018-06-05 23:56:17 nzgdb -core /nz/data/base/1/core.1528233612.21424 -bt > 21424.bt
  457  2018-06-05 23:56:51 nzgdb -core /nz/data/base/1/core.1528230523.11816 -bt > 11816.bt
  458  2018-06-05 23:57:29 nzgdb -core /nz/data/base/1/core.1528221590.28144 -bt > 28144.bt
  459  2018-06-05 23:58:03 nzgdb -core /nz/data/base/1/core.1528189585.23904 -bt > 23904.bt

  460  2018-06-05 23:58:46 nzgdb -core /nz/data/base/1/core.1528185575.5528 -bt > 5528.bt

5.grep the PID with pg.log find out what went wrong on the system during the core dump generated time.


  466  2018-06-06 00:05:22 grep '\[27208\]' /nz/kit/log/postgres/pg.log

  467  2018-06-06 00:09:44 2018-06-05 18:34:36.874070 EDT [27208]  DEBUG:  connection: host=10.128.72.44 user=NZKOMUSER database=CVKOMNZP remotepid=5177762 fetype=1

  468  2018-06-06 00:09:44 2018-06-05 18:34:36.874088 EDT [27208]  DEBUG:  Session id is 16279

  469  2018-06-06 00:09:44 2018-06-05 18:34:36.877957 EDT [27208]  DEBUG:  QUERY: select current_catalog, current_user

  470  2018-06-06 00:09:44 2018-06-05 18:34:36.879496 EDT [27208]  DEBUG:  QUERY: exec dta_box_inv_hist_incr_load(' TARGET_TABLE==DTA_STG_nz_tbl_HISTORY><TARGET_FINAL_TABLE==nz_tbl_HISTORY><SRC_LAST_MODIFIED_DATE_COL==dtm_last_updated><SRC_PULL_DATE_COL==BOXINVTRY_ID><SRC_PARALLEL_TASK_COL==><SOURCE_TABLE==nz_tbl_HISTORY><SOURCE_SCHEMA==KOMUSER><SOURCE_FILTER==><SRC_PK_COL==BOXINVTRY_ID><DTA_JOB_ID==503><TPPIL_KLMDATE==04-jun-2018 06:49:29 AM ')

  471  2018-06-06 00:09:44 2018-06-05 18:34:36.897353 EDT [27208]  DEBUG:  UpdateStatsInsertDummy called on table 'TEMPFUNC13279476'

  472  2018-06-06 00:09:44 2018-06-05 18:34:36.906701 EDT [27208]  DEBUG:  STORED PROCEDURE EXEC: SELECT  dta_nz_parse_parameter_str(ltrim( $1 )) - Variables ( TARGET_TABLE==DTA_STG_nz_tbl_HISTORY><TARGET_FINAL_TABLE==nz_tbl_HISTORY><SRC_LAST_MODIFIED_DATE_COL==dtm_last_updated><SRC_PULL_DATE_COL==BOXINVTRY_ID><SRC_PARALLEL_TASK_COL==><SOURCE_TABLE==nz_tbl_HISTORY><SOURCE_SCHEMA==KOMUSER><SOURCE_FILTER==><SRC_PK_COL==BOXINVTRY_ID><DTA_JOB_ID==503><TPPIL_KLMDATE==04-jun-2018 06:49:29 AM )

  473  2018-06-06 00:09:44 2018-06-05 18:34:36.914496 EDT [27208]  DEBUG:  STORED PROCEDURE EXEC: SELECT  * from table(REGEXP_SPLIT_ROWS( $1 ,'><')) - Variables (TARGET_TABLE==DTA_STG_nz_tbl_HISTORY><TARGET_FINAL_TABLE==nz_tbl_HISTORY><SRC_LAST_MODIFIED_DATE_COL==dtm_last_updated><SRC_PULL_DATE_COL==BOXINVTRY_ID><SRC_PARALLEL_TASK_COL==><SOURCE_TABLE==nz_tbl_HISTORY><SOURCE_SCHEMA==KOMUSER><SOURCE_FILTER==><SRC_PK_COL==BOXINVTRY_ID><DTA_JOB_ID==503><TPPIL_KLMDATE==04-jun-2018 06:49:29 AM )

  474  2018-06-06 00:09:44 2018-06-05 18:34:36.927852 EDT [27208]  DEBUG:  STORED PROCEDURE EXEC: insert into TEMPFUNC13279476 values( 'TARGET_TABLE','DTA_STG_nz_tbl_HISTORY')

  475  2018-06-06 00:09:44 2018-06-05 18:34:41.076371 EDT [27208]  DEBUG:  1 rows inserted

  476  2018-06-06 00:09:44 2018-06-05 18:34:41.084061 EDT [27208]  DEBUG:  STORED PROCEDURE EXEC: insert into TEMPFUNC13279476 values( 'TARGET_FINAL_TABLE','nz_tbl_HISTORY')

  477  2018-06-06 00:09:44 2018-06-05 18:34:44.982932 EDT [27208]  DEBUG:  1 rows inserted

  478  2018-06-06 00:09:44 2018-06-05 18:34:44.990653 EDT [27208]  DEBUG:  STORED PROCEDURE EXEC: insert into TEMPFUNC13279476 values( 'SRC_LAST_MODIFIED_DATE_COL','dtm_last_updated')

  479  2018-06-06 00:09:44 2018-06-05 18:34:47.670894 EDT [27208]  DEBUG:  1 rows inserted

  480  2018-06-06 00:09:44 2018-06-05 18:34:47.678735 EDT [27208]  DEBUG:  STORED PROCEDURE EXEC: insert into TEMPFUNC13279476 values( 'SRC_PULL_DATE_COL','BOXINVTRY_ID')

  481  2018-06-06 00:09:44 2018-06-05 18:34:52.229912 EDT [27208]  DEBUG:  1 rows inserted

  482  2018-06-06 00:09:44 2018-06-05 18:34:52.237441 EDT [27208]  DEBUG:  STORED PROCEDURE EXEC: insert into TEMPFUNC13279476 values( 'SRC_PARALLEL_TASK_COL','')

  483  2018-06-06 00:09:44 2018-06-05 18:34:53.913637 EDT [27208]  DEBUG:  1 rows inserted

  484  2018-06-06 00:09:44 2018-06-05 18:34:53.926314 EDT [27208]  DEBUG:  STORED PROCEDURE EXEC: insert into TEMPFUNC13279476 values( 'SOURCE_TABLE','nz_tbl_HISTORY')

  485  2018-06-06 00:09:44 2018-06-05 18:34:55.508722 EDT [27208]  DEBUG:  1 rows inserted

  486  2018-06-06 00:09:44 2018-06-05 18:34:55.518331 EDT [27208]  DEBUG:  STORED PROCEDURE EXEC: insert into TEMPFUNC13279476 values( 'SOURCE_SCHEMA','KOMUSER')

  487  2018-06-06 00:09:44 2018-06-05 18:34:56.975557 EDT [27208]  DEBUG:  1 rows inserted

  488  2018-06-06 00:09:44 2018-06-05 18:34:56.983190 EDT [27208]  DEBUG:  STORED PROCEDURE EXEC: insert into TEMPFUNC13279476 values( 'SOURCE_FILTER','')

  489  2018-06-06 00:09:44 2018-06-05 18:34:59.367628 EDT [27208]  DEBUG:  1 rows inserted

  490  2018-06-06 00:09:44 2018-06-05 18:34:59.375228 EDT [27208]  DEBUG:  STORED PROCEDURE EXEC: insert into TEMPFUNC13279476 values( 'SRC_PK_COL','BOXINVTRY_ID')

  491  2018-06-06 00:09:44 2018-06-05 18:35:00.948244 EDT [27208]  DEBUG:  1 rows inserted

  492  2018-06-06 00:09:44 2018-06-05 18:35:00.955864 EDT [27208]  DEBUG:  STORED PROCEDURE EXEC: insert into TEMPFUNC13279476 values( 'DTA_JOB_ID','503')

  493  2018-06-06 00:09:44 2018-06-05 18:35:04.072782 EDT [27208]  DEBUG:  1 rows inserted

  494  2018-06-06 00:09:44 2018-06-05 18:35:04.080479 EDT [27208]  DEBUG:  STORED PROCEDURE EXEC: insert into TEMPFUNC13279476 values( 'TPPIL_KLMDATE','04-jun-2018 06:49:29 AM ')

  495  2018-06-06 00:09:44 2018-06-05 18:35:06.168680 EDT [27208]  DEBUG:  1 rows inserted

  496  2018-06-06 00:09:44 2018-06-05 18:35:06.401591 EDT [27208]  DEBUG:  STORED PROCEDURE EXEC: delete from nz_tbl_

we had checked all the coredump generated during the time all the time NPS executing the same query and crashing the NPS and finally we asked user to stop executing the query and later IBM found that it was bug on the 7.0 version system

work around for this bug:
1)upgrade the NPS to recent version
2)clean the versioning of the table

we cleared the version of the table and NPs stable after that.

unable to start the Netezza database

issue:
we have done some  maintanance on Netezza 7.0 and after maintenance we are unable to start the Netezza database,it throws the below error,while starting the NPS.

nz@<server_name>:/export/home/nz> nzstart
ERROR:  System limit for 'nofile' is too low (1024); must be 9600
ERROR:  System limit for 'nice' is too low (0); must be 20
nz@<server_name>:/export/home/nz> nzstate
System state is 'Stopped'.

Solution:
when we check the log,we dont have anything on sysmgr.log,because ,it not even started HW components like SPU & DISK,Then we relaised the ulimit  value would be small to startup the system and increased the ulimit and issue resolved.

However to increase the ulimit on the Netezza server you may need root privilege to update the new value on the server.

This is happened because our NPS got failover to HA2 and we never  had HA2 as active server and this server does not have sufficient ulimit value and fixed later.

ulimit -n 9600