OPERATION            OPTIONS                 OBJECT# NAME                        
-------------------- -------------------- ---------- -----------------------------------
SELECT STATEMENT                                     .                          
SORT                 ORDER BY                        .                          
TABLE ACCESS         BY INDEX ROWID           120713 XXX.TABLE_WITH_MANY_COLUMNS
INDEX                RANGE SCAN               121558 XXX.INDEX_WITH_FEW_COLUMNS 


In a lot of cases we deal with a query for example which is returning fewer columns than exists in the involved table like four, from an involved table having twenty columns. Although the execution plan looks already pretty “OK”, it still results in a query which is in top three most resource taking queries.

To enhance the query we can use “Fat Indexes” or nowadays better known “Tapio Indexes” (I call them like that nowadays). What are these kind of indexes…. In principle nothing really new, but for me a couple of years ago an (re)eye opener by Tapio Lahdenmaki. This Finish independent database performance consultant and instructor gave a presentation at Miracle Open World 2007 in Denmark, I attended. Tapio can present this topic in a very special way, and in my opinion “his message” should be known by all developers and dba’s.

The most important thing of “his message” is the fact that we should create an index which is highly selective and contains all the columns asked in the query. So in principle we create an index with for example two selective columns, with the most selective one in front and behind these two columns the other selected columns. When creating such an index an execution plan would look like:

OPERATION            OPTIONS                 OBJECT# NAME                       
-------------------- -------------------- ---------- -----------------------------------
SELECT STATEMENT                                     .                           
SORT                 ORDER BY                        .                           
INDEX                RANGE SCAN               128140 XXX.INDEX_WITH_ALL_SELECTED_COLUMNS

The big advantage is that only the index is accessed and not the table anymore, what can result in a decrease of elapsed time for the query with a factor 4 or even more. Why this is much better? Let we go back how indexes and tables are stored on the physical storage.

An index is stored in an ordered way on the storage, so an index range scan will mean that data is retrieved from the storage in a sequential read from a storage point of view and so a read-head from a spindle can stay on almost the same place. This eliminates seek times and latency on the physical storage spindles.

A Table is stored in a non ordered way on the storage, so accessing table rows will almost always result in accessing the data in a scattered read from a storage point of view. This implies that a lot of seek and read times are wasted.

Of course the existence of caches makes it a bit different but still the basic principles are valid, this means also for striping. 

So when only an index is accessed instead of index and table, we don’t waste read times for columns we don’t need (table rows), and we eliminate a lot of seek times for retrieving the different table rows.

If you want to know more about this kind of index design, you can read the book “Relational Database Index Design and the Optimizers” from Tapio Lahdenmaki.

"GAPP" means General Approach Performance Profiling and can be used to find out where in your architecture the most wait time variance can be explained from your business process. "GAPP" makes it possible with very little data, in higly complex technical infrastructures, still be able to find the performance bottlenecks for a specific business process. The nice thing about the method is that it is not only able to pinpoint a bottleneck which is already there, it is also able to pinpoint a future bottleneck in a normal running system. This is something what only "GAPP" can do.

What makes “GAPP” special:

• The method can analyse the full infrastructure, so from front-end to back-end
• The method is not focussing on one piece of the infrastructure, like only the database
• The method is able to predict how the response time of a business process will react on changes in involved factors
• The method is able to predict when a certain bottleneck will evolve to a real problem

Further big advantages of the method are the gaps which can be jumped of:

• not allowed to trace
• not allowed to change anything in the application (eg. to get log data)
• not having a representable test environment
• a too complex technical infrastructure (eg. SOA environment, environments with more than 2 involved servers)

In the past a lot of times I saw in organization that at the end of the week the
system administrators sent an email to their management with a response time
graph from a business process (eg. load runner) and some graphs from the cpu,
memory and I/O from the database servers. When I looked at such emails I was not
sure what to think about it. The only thing I could say was that most of the
time spikes in the response time graph were not explained by the other graphs in
the email. This made me wonder if it would be possible to explain the spikes in
the response time based on data from the background architecture. This brought
me to the idea of creating a method, which could do this job. After a lot of
customer cases and investigation, the method is worth having the name “GAPP”.
The method is now so powerful that it works even if you have very little
information, as mentioned in the bullets above. In a lot of situations
architectures are getting just too complicated or organizations have not ways to
reproduce problems in their test environments. The method makes it possible with
available data to pinpoint where in the architecture a certain performance
bottleneck resides.

The method can further be said to be really general, because of the fact that any architecture can be analyzed by the method. Within the method four important steps can be determined:

1) Data Collection

• Which parts of the technical infrastructure are involved in the business process
• What data is available and in which time granularity

2) Data Synchronization

• Aggregation on lowest granularity of all used data
• Decision should be made on if data granularity for a factor is high enough

3) Data Mining

• Factorial Analysis
• Model Creation
• Prediction

4) Data Interpretation

• Mapping results with architecture knowledge
• Compare of result with measured data
• Conclusions

The used data in the analysis can be from any source. It is important to have data from the response times of the involved business process through time. Further is data of each involved server in the chain important, this can be sar, vmstat, nmon, etc. For the involved database(s) for example AWR, statspack, etc can be used. All this data will be brought to one place and at that time able to be data mined. Here it gets really interesting. On the moment the method is using Oracle Data Mining (ODM), although there are open source data minings engines, the ODM engine is really easy to be used and is really very powerful.

When I presented at Sunday I invited Charlie Berger, the Senior Product Manager from Oracle Data Mining. After the presentation he was nicely surprised by the usage of Oracle Data Mining for my performance profiling method. We had a very nice chat and we later on that week met again on the Oracle Demo Grounds here at Oracle Open World in San Francisco. 

Charlie Berger and me on the Oracle Demo Ground for Oracle Data Mining

Oracle Data Mining is very powerful and "GAPP" is more and more using the great algorithms which are in ODM. Within the market it is still not very clear how powerful this software actually is. To give you an idea why ODM is so powerful you have to keep in mind the following:

• ODM is as powerful as products like SAS and SPSS.

• ODM has the big advantage to be part of the database, so it is not necessary to reallocate the to be analyzed data to another place.
• So no security issues, the data stays where it is.
• The GUI from ODM is getting better and better, it can easily be obtained at OTN.
• ODM is also very powerful from SQL, eg. DBMS_PREDICTIVE_ANALYTICS which I personal used much for “GAPP”
• Models are stored in the database and can be used right away in the queries
• 50 of the statistical functions are free of charge to use
• ODM itself has a license fee which is pretty cheap compared with SAS or SPSS.

Just look at the picture below, to give an impression how the GUI tool Oracle Data Miner looks like:

I personally have used "GAPP" for a pretty long time now (more than two years), although first in experimental state, I now mention the method by name, when I use it for a customer. Just to give you an impression how powerful the method can be, I use an example of a very recent case (September 2008) I investigated with “GAPP”.

At the customer the screen for “Employee information” in their application was working with a very high variance in response time. Sometimes the response times were so bad that it was really unacceptable to work with. Based on this issue it was not acceptable to add more users to the system. First of all there had to be done an investigation due to what the big variance in response time was caused. Due to the high amount of components in the technical infrastructure and the fact that the infrastructure was shared by many other applications, made it for the customer very hard to pinpoint how to go further. Also normal tracing was very difficult to do. The technical infrastructure at the customer site looked (simplified) like the following:

The process involved went through a lot of steps like:

 

• Request from client
• Going through a firewall
• Going through a webserver
• Going through a portal server (including a portal database call)
• Going through single sign on via Oracle Internet Directory (including repository database call)
• Going through ADF application
• Retrieving information from database
• Etc, etc.

Although I will not put the complete report for the customer in this blog, I like to show the following factorial analysis I did based on the data gathered from all the involved systems and the response time of the business process gathered from the ADF application.

In the graph are just the important factors on the x-axe, the other factors, which are in total approximate 180 are not really important. Based on the factorial analysis the following response time graph and its prediction can be created:

The model shows that some of the spikes cannot be explained by the analyzed factors. This was due the fact that some factors which have had influence, were not part of the analysis. Although this is the case, in most of the time the prediction is pretty good. What makes it possible to have matrices measured to be used to predict the end user performance experience. So basically “GAPP” has given you an instrument to see how the end user experience is by only looking at the matrices.

In the customer case I was able to give them advise, what to do in their environment  to be able to cope with the extra users they wanted to serve. In this case the model showed that problems with memory on GH200 and GH300 would lead to unexpected performance drops of their application. The model can also predict how much these impacts are.

If you even want to go a step further with “GAPP”, you could also start to change factors to other values, to predict how the response time from the business process will react on this. This makes the use of “GAPP” really powerful and puts it above a level of other methods.
 

Currently I am very busy to finish my white paper on "GAPP" and to make the approach easier to be used in short performance tune assignments. Recently I have been using "GAPP’ very successful at a customer in the Nehterlands. The whole analysis took three days, including a stress test. I really hope that people start to see the big advantage of the method and understand that with the increasing complexity "GAPP" will be the right method to go.

If you have any request regarding the method, please sent me an email on: gerwin.hendriksen@amis.nl

With the traditional way of query, the following sequence of events can be identified:

• Client sends query to database server
• Database server will execute the query, to accomplish that the following has to be done:
• create execution plan
• ask for the different datasources to be retrieved from the storage
• apply the query criteria to reduce the datasources retrieved from the storage
• Database server sends the query result to the client

 

With Smart Scan in place, the following sequence of events can be identified:

• Client sends query to database server
• Database server will execute the query, to accomplish that the following has to be done:
• create execution plan
• apply the query criteria and sent Smart Scan constructed request
• Smart Scan identifies rows and columns direct on the storage
• process the much smaller datasources
• Database server sents the query results to the client

 

 
The big advantage of the Smart Scan principle is that there is less data transfer between physical disks and the database server. Currently the "Oracle(R) Exadata Storage Server" makes use of the infiniband protocol to talk to the underlaying disks, this gives together with a lot of hardware tuning an extreme performance with high end, big amounts of data transfer. Currently the machine is only usable for the Oracle database, so other things like software storage has to be done on a separate storage device.

To read further about the machine you can use the following link:
http://www.oracle.com/solutions/business_intelligence/database-machine.html

At 10:00 in the morning I attended the first presentation. From the two to be chosen tracks, I followed the presentation from Bernhard de Cock Buning, which was titled “Inside Oracle Clusterware”. This presentation showed a lot of background information regarding Oracle Clusterware, and the places to check when something goes wrong inside this product. Bernhard showed with confidence his knowledge of all the background processes which are part of Oracle Clusterware. His presentation and live examples in his vmware environment on his laptop showed that there are a lot of differences between Oracle versions in the way Oracle Clusterware is logging in the log files for the different background processes.

At 11:30 I attended the presentation of Toon Koppelaars, titled “Data Integrity: How it could be”. As usual I was really impressed by the way Toon is able to change “business” rules into plain SQL. Toon presented different integrity issues, and how they could be translated into reliable and performing pieces of code. In the presentation Toon showed also some complex synchronization issues and how they could be solved. For example a department should always have one manager and at least two admins. When in this example a department has a manager and two admins, and the two admins are deleted by two different sessions, the business rules could be violated. I personal think that it would be very well for everybody maintaining and developing in high end systems to have knowledge about this. Toon referred in his presentation to his book, which he wrote together with Lex de Haan called “Applied Mathematics for Database Professionals”.

After the midday break (13:30) I gave my presentation called “Jumping The GAPP”. In this presentation I showed the new method “GAPP” (introduced at HOTSOS 2008), and how it can be used to find performance bottlenecks in complex architectures without code change or tracing. In practice a lot of customers don’t allow you to start tracing in production system to find bottlenecks in a system. Under normal circumstances you would use a test environment, but in a lot of cases this test environment is not representing the production situation. With GAPP you can jump this problem. The basic principle what is used with GAPP is data mining which is already used for a long time in business. The input for GAPP can be any available data (System metrics, ASH, Statspack, Network data, SAN data, etc), which can compared with each other via the same time stamp. The method can show where in a complex architecture certain wait time has gone and where tunings effort should be focused. Also can GAPP do predictions of how certain changes will impact the performance. Currently I am busy to create a whitepaper, for now you can read more in a previous blog post.

At 15:00 I attended the presentation from Lonneke Dikmans, titled “OAS and SOA Suite: Work for DBA’s ?”. Lonneke presented how the Service Oriented Architecture would influence the work of a DBA. In her presentation it became very obvious that the SOA architecture will make the work of a dba more broad and different than it is from a traditional point of view. For the DBA it will be more and more important to know how the different components are connected and how they work together as one application. Although Lonneke was referring to Toon Koppelaars, with his vision of putting business rules into the database, it was from the SOA perspective, that the database is just storage of data. Lonneke successful showed that SOA can be used in such a way that this picture can be combined. I personal think that this is a real future challenge for all of us.

At 16:30 I attended the last presentation of the DBA Symposium, from Jacco Landlust, titled “Oracle Application Server; No Magic, but structured analysis”. In this presentation Jacco showed how a lot of trouble at Application Server level can be solved by a very flowchart a like error trapping method. He showed where log files can be found and how they should be read. It was impressive to see how easy Jacco was navigating in the different log files and what could be found by his drilldown method.

Overall I have to say that the DBA Symposium was a big success, and I think starting this new symposium was a very good idea.

Thanks and keeps jumping GAPP’s,

Gerwin Hendriksen

The presentation went well and a lot of people attended my presentation. I got a lot of nice criticism and was questioned a lot by Dr. Neil Gunther. Before and after the presentation I had a lot of nice discussion with him and he even offered to help me make this approach even bigger. For me this was a real honor and I really was very happy with his input.

Also a lot of other “important” people attended the presentation, like Anjo Kolk, James Morle, Cary Millsap, Jeroen Evers, Toon Koppelaars and many others. I personally was very happy to have the chance to present for such an audience, and was very happy with their reactions and criticism. I personal think that the HOTSOS Symposium is “The Place” for performance in the world.

Dr. Neil Gunther sitting with me:

To develop “GAPP”, it took a long time to come up with a way to be able to have all kind of metrics (CPU, I/O, etc.) able to say something about a business process response time. I already started to think about this when I was still working for Oracle, and later on when I was working for IBM. After a lot of hours, from which my wife said: “What you are doing again behind your laptop?” I have now the feeling that the approach is worth having a name.

To give an impression of where “GAPP” is all about, I will give a small introduction to the approach I created. First of all the approach can be used to performance profile business processes on high level, but depending on the metrics you put in, even on detailed level. If you look at the below slide from the presentation it will become clearer:

In this slide it becomes obvious that “GAPP” for now (maybe in future I can go further), can say something about the fluctuation of “Wait Time” for each resource in a chain for a business process. The “R” and the “Rslow” in the slide are only different due to the “Wait Time” for the different resources (CPU, I/O, Memory, etc.). Further I use the fact that utilization of a resource is related to “Wait Time”.

By having system metrics from all machines in a chain of an architecture, and having the response time of the business process on the same timestamps as the system metrics, the approach makes it possible to clarify which system metric is influencing the business process response time the most (Factorial Analyses). To accomplish this job on the moment I used a package called “DBMS_PREDICTIVE_ANALYTICS”, which is part of Oracle Data Mining (ODM), but also another data mining tool could be used (open source maybe?).

After knowing which system metric is influencing the business process the most, you could start to focus on that part of the architecture and do your optimization on the best spot to start with. To give an impression the following graph gives an impression of the “Factorial Analyses”. In the presentation I used an architecture with two machines (“E” and “Q”):

 In this graph it becomes clear that “Wait%” and “Disk Busy%” of logical hard disk 2 and 3 from machine “Q” explains the most response time fluctuation of the business process in the months mentioned in the slide. Further investigation should be done on machine “Q”. Later in the presentation I have showed that in other periods the “E” machine should be investigated.

An important remark is that the approach “GAPP” can work completely independent from Oracle, it can also use completely different metrics like “SQL statement” elapsed times. This makes it even possible to find out if a certain SQL statement is highly involved in the response time of a business process.

Also future response time prediction can be made by “GAPP”. The effect can be predicted, how much changes in certain metrics will impact the response time of the business process.

I would love to clarify more, but I think I will need to create an article to show how powerful this approach “GAPP” actually is.  In the end I want to say that for me I really “Jumped over the GAPP” I had in my data to analyze the performance problem I was facing at the customer. The longer I look at the approach, the more I find out that this approach is really very powerful, although still a lot of research should be done.

I thank everybody who attended my presentation at the HOTSOS Symposium 2008 and of course at the AMIS Query in the Netherlands. If you have any questions about the approach I would be glad to answer any question about it (gerwin.hendriksen@amis.nl). Further I would like to thank Cary Millsap for giving me the chance to present at the HOTSOS Symposium 2008.

Of course I will keep you posted about the progress of the approach… “GAPP”

With kind regards, Gerwin

When you encounter such a situation you should start to look into a 10046 level 8 or higher trace from the statement involved. In this trace you should first check which object is read in the "db file sequential read" very often, for example:

WAIT #1: nam=’db file sequential read’ ela= 159 p1=14 p2=137858 p3=1

In this line "ela" is telling the time it took to do the read in millionst of a second, "p1" tells the file_id, "p2" the block number in the file and "p3" the number of blocks read by the read. The block number together with the file_id and a query on dba_extents can bring up the object read.

The object you find here is what is causing the long wait. Another way can be by online checking the session running the query and check via v$session_wait and v$session what the object is the session is waiting for.

After finding which object is involved in the "db file sequential read", you can start to checkout the execution plan rowsource data of the query, and see where the table is in the execution plan. Most of the time the involved object is a table read via sequential reads after an index was accessed. You can find something like:

 148243                  TABLE ACCESS BY INDEX ROWID <FOUND TABLE>
 998934                   INDEX RANGE SCAN INDEX1 (object id 29469)

The above two lines have been extracted from a tkprof output or can be obtained in the STATS lines in a raw trace. The two lines above say that via the index "INDEX1" 998934 rows are retrieved from the table, but in real for the outcome of the query and the existing criteria on the query this number of retrieved rows should not be more than 148243 rows. So in this case we actually are jugling around in the temp file with almost 7 times more rows than actually needed for the outcome, so this means 7 times more sequential I/O as needed, much more used processing power of the CPU, etc, etc…..

So as we can see in the above example, we have indeed used an index to retrieve rows from the table, but the used index is not specific enough to prevent reading a lot of rows unneccesary. So it is highly important to checkout if such situations are around in a system. The thing to look for is a high number of rows in front of a index and a low number of rows (in compare) in front of a table.

So now the big question rises, how we can get around such an issue. The first thing we should checkout is how many indexes are on the table and on which columns. The second thing is that we should know what selection criteria are not part of the index. After this analyses we have some options to choose from and not all are very simple to do, it is very depending on the situation you are in:
- Change the optimizer behaviour: Regather schema statistics, use histograms, use hints, etc.
- Add extra index
- Change the selection criteria in such a way that the functional result stays the same
- Etc.

In the example above the solution was to change a selection criteria in the query (Business Objects Query change). This resulted in the below situation:

 148243                  TABLE ACCESS BY INDEX ROWID <FOUND TABLE>
 152241                   INDEX RANGE SCAN INDEX2 (object id 29478)

So the end result is now that the table is accessed via another index (INDEX2) and that around 4000 rows too much are read from the table. This is much better and the total performance of the query went up with a factor 8-9. This because everything was much less stressed.

What we can see in the above example is that not always high "db file sequential read" is a good thing, it also does not mean that much full table scans are well…… The most important thing is that the amount of I/O and the real needed amount of I/O is very close. How to find this out is sometimes a challenge, a 10046 level 8 can help with this.

As a last note use one important principe, never do I/O if it is not needed, and do it only it there is no other option.

Regards, Gerwin

Start:

whenever sqlerror exit failure rollback;

set heading off
REM define un = &username
define deftrace=’begin fnd_ctl.fnd_sess_ctl(””””,””””,””TRUE””,””TRUE””,””LOG””, ””ALTER SESSION SET EVENTS=””||””””””””||””10046 TRACE NAME CONTEXT FOREVER, LEVEL 12””||””””””””);end;’

select ‘Inserting profile option value’
from dual;

insert into fnd_profile_option_values
select 0,profile_option_id, 10004, user_id,sysdate,0,sysdate,0,0, ‘&deftrace’, null
from fnd_profile_options, fnd_user
where profile_option_name = ‘FND_INIT_SQL’
and user_name = ‘&APPS_USER’
/

commit;

Stop:

select ‘Press ENTER to disable trace for’, ‘&un’
from dual;

select ‘&ENTER’
from dual;

delete fnd_profile_option_values
where application_id=0 and level_id=10004
and level_value = (select user_id
                   from fnd_user
                   where user_name = ‘&APPS_USER’)
and profile_option_id = (select profile_option_id
                         from fnd_profile_options
                         where profile_option_name = ‘FND_INIT_SQL’)
/

commit:

Just to use the scripts the following steps and instructions should be used:

• run the start script in sqlplus and provide the application username to be traced in Self Service and give an Enter (this session stays open now)
• Login with the application user, and reproduce the to be traced functions.
• Logout with your application user.
• run the stop script in the sqlplus session for the traced Oracle Applications user.

* Important for the script:

• Check if define deftrace is one line ‘…..’
• Level 12 (waits&binds) can be changed to 8(waits), 4(binds) and 1(SQL_TRACE)

If you might have any questions, don’t hesitate to contact me.

Looking at the above comparison Data Guard has some very nice advantages above RAC, because of the fact that in Data Guard not one database is involved, but more. The Data Guard solution is using a standby database principle, what makes it possible to have a delay between the databases and in that way data safety for user errors. For example a truncate of a table in the production database can be stopped being applied on the standby. Further more, the Data Guard solution works with an “active”, “passive” principle and switching the database function is easy and very fast to be accomplished (a few minutes is possible on database level). So in principle production can be running on site A and in case of failure of site A, the passive (standby) database is brought up as production on site B. After Site A has been brought up, site A can just be working as passive (standby) database. So in the end it is not important on which side the production is running.

In practice the above is overshadowed with the fact that most of the time the passive side has not the same amount of resources available, like CPU and memory. So in that case the standby site is much slower as the normal production, or resources have to be increased before able to run production on the standby site. The other thing is that if the active site and the passive site are equally sized the amount of license fee for Oracle software and OS is the double of the production system capacity. In that case there is a possibility to have the standby database opened for reporting purposes (management reporting), in that way the standby hardware capacity can be used. Another option is to run development and test systems on the standby hardware and in case of failover only the production is running.

When looking at RAC, the most important difference is that it is in the end just one database. This has as advantage that just one dataset is involved and different memory structures (instances), running on different machines are using the same database (same dataset/storage), so also able to use the capacity of all involved servers for production. The big disadvantage is that protection against user errors like a truncate is not there (with Data Guard it is depending on the configuration and set delay).

In the end it is really important to define “High Availability”. In the Query from 14 december 2006 it was positioned to be the time of unplanned down time. I personal would like to add the statement and definition of “Political High Availability”. Although Data Guard needs very little time to failover between the “active” and the “passive” site, it still needs a few minutes of real down time (nobody can work). In case of RAC and one server dies we have still end users able to use the system. In general the need to reconnect, as a result of a node/instance failure when RAC is in place, is not considered to be downtime.

For performance reasons it is good in RAC to separate the functions of the different nodes running an instance (database servers), this to prevent I/O contention on the interconnect between the nodes. So the overhead of RAC will be decreased. This principle is simple to understand by knowing that using the same datasets on different RAC nodes will mean that database blocks will be transferred (block pinging) via the interconnect, because different rows, used on different nodes are physically in the same database block. The bigger the database block size the bigger the impact will be.

Based on this performance principle a function separation is very attractive in RAC, but has a coin side. We could think of a situation of a company operating in two countries, like the Netherlands and the UK, based on the performance principle we could choose to have both countries using their own server (two way RAC). In case of a failure of one of the two nodes, a complete country will have an outage. This is from an SLA (Service Level Agreement) not a very nice situation. For this I introduce the term “Political High Availability”, so it is better to have the two countries mixed and in that way just loose half of both countries, and not one whole country. From management point of view in the second situation we will not loose a country and in that case it will not be called an outage, but a disruption.

Another important thing is the fact that with increasing number of nodes in RAC we will have a lower percentage of users having an issue when one of the servers dies. On the other hand the more nodes in RAC we will have, the more RAC overhead we will get. This will be more severe in cases the datasets used on the different servers are more similar. In Oracle Applications for example a table like FND_CONCURRENT_REQUESTS can give a lot of overhead.

Also the complexity between solutions is important. The more complex the configuration the bigger the chance that something goes wrong due to user maintenance error. This means also that a more complex solution needs more skilled personnel, more expensive resources and due to that a more expensive solution.

For high availability over big distances it is recommended to use Data Guard, but for smaller distances like several kilometers it is possible to create an extended RAC cluster. When considering such a solution it is important to think of a triple site solution and not going for a twin data center solution, because of data consistency.

From a definition point of view I think the following reasons should be to use RAC instead of Data Guard or in combination with Data Guard (Oracle Maximum Availability Architecture):

• The “Political High Availability” is really important for the business.
• There is not enough capacity in one server to handle the load.
• Scalability is a requirement.