Weblog for the AMIS Technology corner
KC IT Architecture
I don’t like a reclaimable database disk copy
Jun 1st
As of Oracle 10g it’s possible to recover (or wind forward) a RMAN disk copy of the database to a specific time in the past. This type of backup is quite suitable for big databases (> 100GB), because you have to perform a time consuming full backup only once. All other backups to disk will be incremental ones, and with block change tracking on, these backups are done in a fraction of the time a full backup takes.
A setup like this needs a bit of disk space. Using the FRA (Flash Recovery Area, but from 11gR2 on it stands for the more appropriate Fast Recovery Area ), you may need to size it to 2 times the size of the database, or even more, depending on the number of database transactions.
However, till now I have been able to convince most clients I worked for of the benefits of this kind of backup regime, and to provide or even buy the extra disks necessary.
My favorite configuration of a RMAN database backup is as follows:
In the first week, on a Sunday, a onetime full (level 0) disk backup (copy of datafiles) to the FRA, and a subsequent backup of the database copy to tape. From then on, every day an incremental level 1 disk backup, and the backup to tape of all archives and incremental disk backups, not yet back upped.
Next Sunday, because I like to have a recovery window of 1 week if possible, the incremental merge of the database disk copy kicks in, winding it forward to make the equivalent of a new database image copy. An incremental merge of the database is executed every day from now on. On this Sunday (and every next Sunday), backup the database disk copy to tape, after it’s incrementally updated.
Why backup the database copy to tape on Sunday only…. well, you can save yourself a lot of time and a lot of tape this way, especially when your database is a big one. If you need to restore/recover from tape instead of from disk you lose the benefit of Fast Recovery anyway, so the extra time needed to apply some extra incremental backups to a Sunday’s full backup to roll forward to a specific point in time, won’t make that much a difference in restore/recovery time.
You yourself don’t have to delete old (older than 7 days!) backup sets, copy controlfiles and archives from disk, if you stick to this backup regime. Oracle labels every recovery file that is successfully back upped to tape to ‘reclaimable’, meaning that it will be deleted from disk if FRA disk space is needed for more recent recovery files. A nice feature, but it also means that the database copy on disk is set to reclaimable, every Sunday after it’s back upped to tape.
I really think this is a problem, because you want to make sure the database copy is intact in case you have to do a switch from one or more datafiles from the original database to datafiles of the copy. By eliminating restore time, you can be up and running again in minutes instead of hours, even if this is a big database. So, how do you prevent Oracle to label the database copy as ‘reclaimable’?
I checked some forums, and found this solution/workaround: set redundancy to 2.
Well, although this prevents the database copy to be set to reclaimable, it also prevents this nice automatic delete of disk backup sets, copy controlfiles, and archives, because they too will now never be labeled as reclaimable. So I adjusted the backup script, and added a delete from disk of all backup sets, copy controlfiles and archives older than 7 days, to be executed every day at the end of the database backup.
But the fact that this way the automatic administration of the FRA by Oracle is in effect disabled bothered me. It’s quite comforting to know that a sudden and unforeseen burst in database transactions can be alleviated by ‘reclaimable’ space within the FRA, and can provide some extra time to prevent the database from stalling. And I came up with the following solution: yes indeed, set redundancy to 2, but also backup all recovery files (except the database copy!) at least 2 times to tape.
This really solved my problem.
At the expense of some extra tape usage for double back upped archives and incremental disk backups, I can now be sure of the all time existence of a full copy of the database on disk that will never be labeled as ‘reclaimable’. I can leave the FRA administration to Oracle now, because recovery files back upped 3 times (1* disk + 2* tape) will be deleted from the FRA, if more recent recovery files need disk space.
Compliments; Instant productivity improvement for software teams, with a small effort….
Apr 25th
Hello, you project manager/team leader. I expected this title to grasp your attention. Would you like to know how to improve the performance of your team members? This can be done without massive statistics or an expensive performance improvement program. This magic pill is called positive feedback. Just give your team members the credits for their work and compliment them for their achievements.
As a project manager we are aimed on the end result. In our day to day job we are focused on the things that are not yet done and the things that could go wrong. This focus on future result and possible impediments make us forget the past achievements of our team.
Automatic testing Oracle Service Bus using Hudson, maven and SoapUI
Feb 23rd
A lot of current projects are implementing some sort of service based architecture. Testing in this architecture becomes more complex. When implementing an OSB project with Scrum you test-automation is imperative. Scrum will require more frequent testing of your system. This is only feasible (in time and money) when you automate as much as possible.
Using soapUI you are able to create visually SOAP tests on your OSB implementation and running them against the defined infrastructure (develop, test, acceptance). SoapUI enables with easy tools to implements verification and validation of the responses of your OSB implementation. When running the test you are also able to set limits in SLA response times on all the calls. This way you are able to monitor depreciation of performance in older parts of your OSB implementation when adding new services.
You can record and edit your SOAP test easy with the soapUI interface and edit it later. When you maven-enable your project it is quite easy running your tests when you implement the “maven-soapui-plugin” (see my other posting http://technology.amis.nl/blog/3061/automated-soap-testing-with-maven). In the meantime version 3.0 of this plugin is released.
When implementing this with Hudson you do not have to convert the results.xml into a Surefire report. Hudson will manage this for you. Hudson will also enable you with an historical overview of all your test results.
Agile software development, the principles. Principle 11: The best architectures, requirements, and designs emerge from self-organizing teams.
Dec 14th
This is the eleventh of 12 posts about the principles of agile software development. Purpose is to go back to the start of the agile manifesto (http://agilemanifesto.org/principles.html) and discuss the implementation of the 12 principles in real life software engineering. Goals of agility are to go deliver software of higher quality, faster, with a higher acceptance to end-users and able to follow the changing business requirements to strive for competitive advantage.
The question is: is this going to work in practice or is this only based on a nice marketing and sales story.
Principle 11: The best architectures, requirements, and designs emerge from self-organizing teams.
For a long time the engineering expertise (and also software engineering) was based upon the condition that you worked with specialists. These specialists emerged from the principle of division of labour and made it possible for these specialists to focus their attention on their specialism and create the best solution within their field of expertise. The Interaction designer designed a user interface, the architect created a global systems model, developers created code and infrastructure specialists created the necessary environment to run the application on.
Everyone was specialized and delivered the best solution within their capabilities. However when all these components where put together, noting worked. It is an illusion that specialists can design and foresee everything beforehand.
Within agile projects the solution is to use a self organizing team to perform these tasks. This team may consist of specialist, but this is not a requirement. The requirement of this team is that they work together and self-organize all aspects of the systems to be delivered. This team is permitted to make errors and invent their solutions, provided that they deliver and evaluate frequently (retrospective) and learn from their successes (and errors).
Oracle Enhances XBRL Reporting Capabilities with UBmatrix, Inc.
Jun 27th
Oracle posted a press release about their involvement and commitment for UBMatrix XBRL technologies last Wednesday saying (among others):
"To help publicly held companies facilitate the preparation, publishing and automatic exchange of financial statements in XBRL (eXtensible Business Reporting Language), Oracle will embed UBmatrix, Inc.’s leading XBRL technology into Oracle’s Enterprise Performance Management (EPM) System, Oracle announced today."
"Oracle and UBmatrix are also working together to provide enterprise class storage and management for XBRL-based information in Oracle® Database 11g. Using Oracle Database’s XML DB feature, organizations can easily and quickly store, access and query their XBRL data."
A lot of you might ask themselves: "Who are or what is UBMatrix?" and "How does it all fit in…?"
UBMatrix is one of those small but cool companies that have an idea and drive that idea to bigger heights. Their ideas about how to deal with XBRL and how to make it more succes full are very innovative. XBRL is an open standard to exchange and/or embed business and financial related data via XML. "Nothing more and nothing less…", but applying it has a great impact to other systems, sometimes called "the ripple effect".
XBRL is becoming the standard for financial information exchange and reporting demonstrated by the XBRL programs being driven by the SEC, FDIC, and the European Central Bank Supervisors. My Dutch Tax Office has embraced it and among others setup a taxonomy for XBRL (The Dutch Taxonomy Project). This "central hub" (the Dutch Tax Office) will eventually cause more and more organizations to exchange their data via XBRL (therefore the "ripple effect").
The setup for the Dutch Taxonomy project background is simple…
Enhancing fast queries using “Tapio-Indexes”
Jun 4th
During the years I have a lot of times encountered performance problems that ended up to be fast queries (less than 10ms) which are executed very much. In these situations the execution plans, from such queries can look like:
OPERATION OPTIONS OBJECT# NAME
-------------------- -------------------- ---------- -----------------------------------
SELECT STATEMENT .
SORT ORDER BY .
TABLE ACCESS BY INDEX ROWID 120713 XXX.TABLE_WITH_MANY_COLUMNS
INDEX RANGE SCAN 121558 XXX.INDEX_WITH_FEW_COLUMNS
In a lot of cases we deal with a query for example which is returning fewer columns than exists in the involved table like four, from an involved table having twenty columns. Although the execution plan looks already pretty “OK”, it still results in a query which is in top three most resource taking queries.
To enhance the query we can use “Fat Indexes” or nowadays better known “Tapio Indexes” (I call them like that nowadays). What are these kind of indexes…. In principle nothing really new, but for me a couple of years ago an (re)eye opener by Tapio Lahdenmaki. This Finish independent database performance consultant and instructor gave a presentation at Miracle Open World 2007 in Denmark, I attended. Tapio can present this topic in a very special way, and in my opinion “his message” should be known by all developers and dba’s.
Schrijf je in voor de ODTUG Sneak Preview op 15 juni 2009!
May 26th
Op maandag 15 juni 2009, een week voordat in Monterey de ODTUG 2009 plaats vindt, heb je de mogelijkheid om alvast te horen wat negen sprekers in de VS gaan presenteren, onder hen twee Oracle Aces en twee Oracle ACE directors.
- Aino Andriessen (AMIS Services) – ADF Development: More tales from the Trenches
- Lonneke Dikmans (Approach Alliance) – Top Ten Tips: Best Practices for Designing Services, Events, and Business Processes
- Olivier Dupont (iAdvise) – APEX at the Belgium airport
- Dimitri Gielis (APEX Evangelists) – Mastering an APEX page
- Roel Hartman (Logica) – How to integrate APEX and Oracle Forms?
- Lucas Jellema (AMIS Services) - Truth and Dare—The Story of How an Oracle Classic Stronghold Successfully Embraced SOA
- Toon Koppelaars (Rulegen) – Fat databases: A layered approach
- Ronald van Luttikhuizen (Approach Alliance) – Customer case: Implementing SOA in a database-centric environment
- Alex Nuijten (AMIS Services) – SQL Holmes – The case of the missing performance
In drie rondes kan steeds een keuze worden gemaakt uit drie presentaties over een aantal uiteenlopende onderwerpen, onder andere APEX, ADF, SOA en de database.
Dit jaar staat er zelfs één presentatie geprogrammeerd (van Dimitri Gielis) die exclusief op de ODTUG Sneak Preview bij te wonen is en uiteindelijk niet op de ODTUG te zien zal zijn.
Het is mogelijk om gratis deel te nemen aan deze unieke ODTUG Sneak Preview (aanmeldingsformulier)!!! Maar schrijf je snel in, want vol is vol!!!
De locatie van de Sneak Preview is het AMIS kantoor in Nieuwegein. De toegang tot de ODTUG Sneak Preview, het diner en de verfrissingen zijn gratis. Meer informatie over de presentaties vind je in de activiteitenagenda.
AMIS Query 18 Mei – Een Terugblik op Hotsos 2009
Apr 22nd
Begin maart 2009 werd in Irving, Texas, het internationale Oracle Performance Symposium Hotsos gehouden. Ook dit jaar stond het symposium garant voor inhoudelijke hoogstaande presentaties en discussies over performance. Vier Nederlanders, Jeroen Evers (Fameus), Toon Koppelaars (RuleGen), Gerwin Hendriksen (AMIS) en Marco Gralike (AMIS), hadden het genoegen om ter plekke te zijn en in de gelegenheid om al deze, over de aanwezigen uitgestorte (Oracle) performance verbeterende informatie, in zich op te nemen. Dit jaar was het Symposium sowieso bijzonder vanwege de 2 presentaties van Chris Date en een volledige in-depth training dag met Jonathan Lewis.
De AMIS Query “Een terugblik op Hotsos 2009”, op 18 mei aanstaande, zal een terugblik geven over het Hotsos Symposium van dit jaar en in gaan op vier, door de presentatoren, uitgekozen presentaties, onderwerpen, die op het Hotsos Symposium werden gegeven en die door ons Nederlanders als opmerkelijk werden ondervonden en/of een erg interessant onderwerp is om (ook) over te presenteren.
Jeroen Evers, een van de weinige Oracle Certified Masters in Nederland en OakTable lid, zal inhoud geven op deze avond aan de door hem bijgewoonde presentatie van Amit Poddar over “One Pass Distinct Sampling”. Een pittig onderwerp. Het is echter aan Jeroen toevertrouwd dat hij ons, netzo als in zijn dagelijkse werkzaamheden, een helder beeld kan schetsen m.b.t. de “in’s and out’s” over dit onderwerp.
Toon Koppelaars, zelf dit jaar ook weer presentator op Hotsos, mede auteur van het door velen geroemde boek “Applied Mathematics for Database Professionals” en OakTable lid, gaat deze avond presenteren over zijn visie t.a.v. de presentatie van “Using the PL/SQL Hierarchical Performance Profiler” van Bryn LLewellyn (Oracle) over een van de nieuwe performance profiler tools van de firma Oracle.
Gerwin Hendriksen, Principal Database Consultant en een van de Expertise Managers van AMIS, zal in de rebound, zijn presentatie “Practical Use of Method GAPP to Find Performance Bottlenecks for Business Processes in Complex Architectures” ten gehore geven. Dit jaar werd Gerwin voor de 2e maal gevraagd om te presenteren op Hotsos, maar door het ter plekke kapot gaan van hardware, was hij niet instaat om zijn presentatie kracht bij te zetten en in kleur en geur uit de doeken te doen. Nu eenmalig op herhaling.
Marco Gralike, AMIS, Principal Database Consultant en Oracle ACE voor het specialisme Oracle XMLDB, zal deze avond verhalen over het Symposium zelf door middel van een korte introductie over het fenomeen “Hotsos” en, later op de dag, over Christian Antognini’s presentatie over “Bloom Filters”.
Zie voor meer informatie over de Hotsos sprekers en de gegeven presentaties: Hotsos 2009
Zoals altijd is iedereen van harte uitgenodigd om gratis de AMIS Query “Een terugblik op Hotsos” bij te wonen en aan het begin van de avond zal er eten aanwezig zijn om het navolgende performance presentatie geweld snel en stabiel te kunnen doorstaan.
Schrijf je in via de: AMIS Activiteiten pagina.
Marco
Session announcement: Oracle Service Bus (25 feb 2009)
Feb 23rd
Just wanted to draw your attention to a "Knowledge Center" session at AMIS this wednessday (25 feb 2009, starting at 16:30). It concerns the new Oracle Service Bus (previously the BEA Aqualogic Service Bus), and will be presented by former collegue Jeroen van Wilgenburg, now working at Xebia. Jeroen has worked with the OSB on his current project, so he will be able to give us real-life experiences, not just a "summary of features"! For more information, and information on how to register, take a look at the AMIS Agenda: http://www.amis.nl/activiteiten.php?id=681&preview=&draft=
Extreme performance introduced by patchset 11.1.0.7 smart scan
Sep 25th
Larry yesterday did his keynote here at Oracle Open World introducing the "Oracle(R) Exadata Storage Server". This morning I attended a session about the technical details behind the server. In principal the server works with 8 nodes running RAC on Oracle Enterprise Linux 5. For the storage management ASM is used and since 11.1.0.7 smart scan has been introduced. Smart scan has been designed for very big data sources used in queries, like in Data Warehouses, where tables from tera bytes are common. In principle the following is accomplished using smart scan:
Read the rest of this entry »
