Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Oracle Data Guard over VCS 6.0

Created: 25 Nov 2013 • Updated: 26 Nov 2013 | 16 comments
This issue has been solved. See solution.

Hello experts.

It's being almost a month testing and digging on the VCS documentation regarding "Data Guard and Oracle agents supported by Veritas".

The first I can say, documentation is incomplete, confusing and it assumes many things which is a REAL PAIN for a newbie. 

Anyhow, after weeks of issues and spending hours trying to understand who the whole thing works together, I still have many questions.

Scenario:

  • 2 Linux RH6 physical nodes
  • Oracle 11.2.0.4 and Data Guard configured. This is all SINGLE INSTANCE.
  • NO SHARED storage!

I read the full documentation regarding Oracle Data Guard Agents for Veritas which was pretty much a real challenge.

Questions I have:

  1. Why Netlsnr agent shuts down my listener? Switchover fails because of that. I had to remove the Netlsnr resource and monitor.
  2. How can I perform a smooth switchover from VCS using OraDG and OraDGBroker agents? It's pretty amazing why this information is not provided on the document.
  3. Is it there any other complete documentation regarding OraDG and OraDGBroker operation? It says "agent supports this or that" but it wont tells you the details "how" or "how to make sure this or that happens", or exactly "when". It's really vague.

There is an example of what "vague" means. This is a portion of the document:

Note: For activateStandby, deactivateStandby, flashbackRecover, SnapshotOn, and SnapshotOff action functions, you must increase the values of the ActionTimeout and MonitorInterval attributes for OraDG and OraDGSnap types before running.

To "increase" them. Right. How much??? I found some values on another document but not sure why I have to do this or how to confirm the values are good. There are many other examples like this which makes VCS and Oracle implementation a pain.

Thanks for any help you can give me. Ask me for any information and I'll provide it.

Alex.

Operating Systems:
Discussion Filed Under:

Comments 16 CommentsJump to latest comment

mikebounds's picture

Can you copy and paste or attach your main.cf file.

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

actitud's picture

Here you have.

Keep in mind that I have remove the listener resource from here because of the "issue" I had during switchover.

include "OracleASMTypes.cf"
include "types.cf"
include "Db2udbTypes.cf"
include "OracleTypes.cf"
include "OraDGTypes.cf"
include "SybaseTypes.cf"
 
cluster FWT_CLUSTER (
        UserNames = { admin = lkjfgoijtrhsDsfDffg, mov = asFAdsfjaseSADas }
        ClusterAddress = "10.1.1.10"
        Administrators = { admin, mov }
        HacliUserLevel = COMMANDROOT
        )
 
system node1 (
        )
 
system node2 (
        )
 
group ClusterService (
        SystemList = { node1 = 0, node2 = 1 }
        AutoStartList = { node1, node2 }
        OnlineRetryLimit = 3
        OnlineRetryInterval = 120
        )
 
        IP webip (
                Device = a1b1
                Address = "10.1.1.10"
                NetMask = "255.255.255.192"
                )
 
        NIC csgnic (
                Device = a1b1
                )
 
        webip requires csgnic
 
 
        // IP webip
        //   {
        // NIC csgnic
        //      }
 
 
group mov_db_grp (
        SystemList = { node1 = 0, node2 = 1 }
        Parallel = 2
        SystemZones = { node1 = 0, node2 = 1 }
        )
 
        Oracle mov_db (
                Sid @node1 = movp
                Sid @node2 = movp
                Owner = oracle
                Home = "/u01/oracle/product/11.2.0.4"
                StartUpOpt = CUSTOM
                EnvFile = "/home/oracle/.profile"
                )
 
        // resource dependency tree
        //
        //      group mov_db_grp
        //      {
        //      Oracle mov_db
        //      }
 
 
group oradg_broker_grp (
        SystemList = { node1 = 0, node2 = 1 }
        )
 
        OraDGBroker oradg_broker (
                Sid = movp
                Owner = oracle
                Home = "/u01/oracle/product/11.2.0.4"
                AgentDebug = 1
                LinkRes = mov_db
                )
 
        requires group mov_db_grp online local soft
 
 
        // resource dependency tree
        //
        //      group oradg_broker_grp
        //      {
        //      OraDGBroker oradg_broker
        //      }
 
 
group oradg_grp (
        SystemList = { node1 = 0, node2 = 1 }
        )
 
        OraDG oradguard (
                LinkRes = mov_db
                Flashback = 1
                )
 
        requires group mov_db_grp online local soft
 
 
        // resource dependency tree
        //
        //      group oradg_grp
        //      {
        //      OraDG oradguard
        //      }
mikebounds's picture

I configured the Oracle DG agent about 3 years ago and it worked fine, but I agree the documentation is not great.  I don't know that much about Oracle, but my understanding from the "About the agent for Oracle Data Guard" is that:

  • Data Guard agent is used for single instance Oracle for all versions and this uses SQL *Plus to control Oracle DG.
  • Data Guard Broker agent is used for RAC but can also be used for single instance Oracle since 11gR2
 
So if this understanding is correct, as you are using single instance 11gR2, then you have choice of using EITHER Data Guard agent or Data Guard Broker agent, but you should not use both, as you have done.
As to which agent you should use, I guess this is an Oracle question, but it would be useful if the agent guide gave some direction on this.
 
When you had your listener resource configured, did you make Oracle resource depends on the Listener resource as in the agent guide.
 
Have you tested Oracle DG outside of VCS control - it it doesn't work outside of VCS control, then VCS won't be able to get it to work either.  If it works outside of VCS control, but not in VCS, then look at engine_A.log and OraDG_A.log and post any errors.
 
As an aside, I would not set EnvFile = "/home/oracle/.profile" - this is bad HA practice as if someone changed the .profile this could break VCS (so for instance I have seen a DBA add a menu to .profile to choose what dabase they wanted to log into and this change would break VCS if they had used .profile in VCS).
 
Mike
 

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

SOLUTION
actitud's picture

Hello Mike. First of all, thanks a lot for your time on this. Believe me, I spent nights on all these issues and I'm not newbie on Oracle's technologies.

Anyhow, if what you say is correct, then it makes perfect sense and it would be another disappointed surprise on the documentation as it does't mention the option of one OR another agent. So, it's a FACT that I can not use them both? Just checking.

Yes,

  • Data Guard Agent uses Sqlplus to perform the data guard operations (such as failovers, switchovers, etc)
  • DG Broker uses the database broker to do the same

I'll remove DataGuard and see if by using hagrp -switch on the Broker, I can make it work.

YES. When I had it on the main, I have configured the database resource to depend on the listener, and that's the main reason on why I can't understand why the Netlsnr shut it down when the database comes down :(. Maybe, If I only have the broker and the listener, everything will start to work magically.

There is nothing like Oracle DG outside VCS. What VCS does (with its agent) is to perform the same tasks you would perform from SqlPlus to do a switchover for example.

Broker (dgmgrl) works fine from outside VCS. What I'm doing right now is to freeze the agents and then run the switchover but it's clearly a dumb practice just for testing.

I'll check the profile (which I don't believe is causing any issues).

Again, thanks for your inputs. I'll test it and let you know the results.

BTW, the good side on all this is I've faced MANY (if not all) the possible issues so I've learned a lot.

Thanks,

Alex.

mikebounds's picture

Note, if you use BDG roker agent then you the need theStartUpOpt = STARTUP for Oracle resource , not CUSTOM as it needs to be for DG agent.

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

actitud's picture

Ok. That makes all this more understandable as one of the agents uses

"StartUpOpt = CUSTOM"

and the other one

"StartUpOpt = STARTUP"

Now, how can I perform the switchover from SVC using the OraDGBroker instead of the dgmgrl?

Should I perform the switchover manually from dgmgrl?

mikebounds's picture

In VCS just switch oradg_broker_grp to the other node - this will cause VCS to call the online function for the Oracle Broker DG resource which will run dgmgrl failover or switchover.

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

actitud's picture

I've tried that but nothing happens. Maybe I have to wait a little longer? Trying again right now.

This is what I've got last time from Engine log:

2013/11/25 14:54:54 VCS NOTICE V-16-1-10301 Initiating Online of Resource oradg_broker (Owner: Unspecified, Group: oradg_broker_grp) on System node2

2013/11/25 14:54:56 VCS INFO V-16-1-10298 Resource oradg_broker (Owner: Unspecified, Group: oradg_broker_grp) is online on node2 (VCS initiated)
2013/11/25 14:54:56 VCS NOTICE V-16-1-10447 Group oradg_broker_grp is online on system node2
 
Nothing came up from the OraDGBroker_A.log log.
 
This is what I have on hastatus after the switch
 
...
...
                mov_db           node1                ONLINE              
                mov_db           node2                ONLINE              
                oradg_broker         node1                OFFLINE             
                oradg_broker         node2                ONLINE
...
...
 
5 minutes and still waiting. Anything to do with the Listener resource? (not currently present on the main.cf)
 
mikebounds's picture

I've read the agent guide again, and as you say it is confusing as it is not clear which statements relate to both agents and which refer to just the "Oracle Data Guard agent"  as oppose to the "Oracle Data Guard broker agent"  - so I think the statement in the introdution:

The Oracle Data Guard agent supports replicated data clusters on Linux and Solaris operating systems
 
means ONLY the Oracle Data Guard agent supports replicated data clusters and the "Oracle Data Guard broker agent"  does not and if it doesn't, then by implication the "Oracle Data Guard broker agent"  must support Global clusters with the global cluster option (GCO).  It also appears the the Oracle Data Guard agent supports GCO as well as RDS, but this is not stated in the introduction.
Just to be clear, as you are a newbie - a replicated data cluster (RDC) is when you are replicating within a single cluster and a global cluster is where you are replicating between 2 clusters.
 
The Oracle Data Guard agent gives example for RDC and GCO, but the Oracle Data Guard broker agent ony gives an example for GCO.
 
Also in the "Configuring the agent manually in a global cluster" section it says:
 
Add a new group with at least one resource of type OraDG or OraDGBroker for VCS
and for "Configuring the agent manually in a replicated data cluster"
 
Create an OraDG service group and configure the attributes of the OraDG resource that you added.
 
So as you have an RDC, then I think you must use the "Oracle Data Guard agent", not the "Oracle Data Guard broker agent" 
 
Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

actitud's picture

In all this mess,  your help is obviously priceless. Thanks for that again Mike.

The main problem when using Data Guard Agent to do the switch (hagrp -switch oradg_grp -to nodeX) is that the actual BROKER configuration (managed by Oracle's dgmgrl) never gets aware of what happened in the backend and it gets "corrupted" showing wrong information. The only solution is to recreate the configuration as it will keep believing the primary database is the one from before the switch happened which is clearly wrong.

So i guess the only two options are:

  1. Perform the switch(over) from Data Guard Agent using VCS and THEN, recreate dgmgrl configuration
  2. Freeze the Data Guard Agent Group, perform the switchover from dgmgrl, switch the group and unfreeze.

I guess in case of number 2, I rather should not use dgmgrl at all.

What do you think?

Alex.

mikebounds's picture

Its seems a bit poor that if you use SQL Plus to switch Oracle Dataguard that Oracle doesn't update Oracle's dgmgrl BROKER configuration, so you could perhaps query this with Oracle, unless using dgmgrl is the official supported way from 11g and SQL Plus method is only available for backward compatibilty and if this is the case then the Symantec VCS agent really needs updating to allow Oracle Data Guard broker agent to be used in an RDC (if it can't already as the documenation is not exactly clear)

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

Kimberley's picture

Hi Alex and Mike,

I just wanted to let you know that I saw this thread and sent a link over to our Documentation team so that they could investigate the docs that you refer to here, and it looks like there's an opportunity for revision/updating. Someone from that team may ping you for additional information.

Best,

Kimberley

~ Community admin

Thanks for participating in the community!

actitud's picture

I'm nobody on Veritas's world but I'm open to provide any information needed to improve the documentation.

Thanks,

Alex.

actitud's picture

Kimberley.

I haven't got any information from the documentation team and besides I've marked this as RESOLVED, I still have lots of questions regarding how to properly set this up.

Thanks,

Alex.

Nayana Dighe's picture

Hello Alex,

I am from the Agent Pack documentation team. Can you send me an email addressed to nayana_dighe@symantec.com with the questions you have? With the help of the development and QA teams, I will try and get you the information you are looking for and will also update the agent guide at the earliest.

Best regards,

Nayana Dighe

Kimberley's picture

Hi Alex,

Thanks for letting me know. I'd touched base with our Docs team and believe they saw they thread, but will remind them that no one has reached out to you directly yet <<correction, looks like Nayana above did>>  - this seems like really interesting and valuable information for us at Symantec to collect. 

Thanks again for your help!

Best,

Kimberley

Thanks for participating in the community!