Running the OBIEE admin tool on Unix

Mucho kudos to Borkur Steingrimsson for getting the OBIEE admin tool working on Unix!

CAF

Very interesting post by Kevin McGinley about CAF here:  CAF = Migration Utility? Use Caution!

It articulates better than I ever could reasons against using CAF particularly in a production environment.

Since the tool came out I’d been struggling to get my head around it, convinced I was missing something. I still don’t profess to understand it properly, but Kevin’s article reassures me that I shouldn’t be losing too much sleep over it, especially given that it’s unsupported and won’t work with 11g.

Troubleshooting Presentation Services / analytics connectivity

Short but sweet this one – a way of troubleshooting connectivity problems between analytics (the Presentation Services Plug-in, either j2ee servlet or ISAPI, a.k.a. SAWBridge) and sawserver (Presentation Services).

For a recap on the services & flow please see the first few paragraphs of this post.

Problems in connectivity between analytics and sawserver normally manifest themselves through this error message:

500 Internal Server Error
Servlet error: An exception occurred. The current application deployment descriptors do not allow for including it in this response. Please consult the application log for details.

Which IE and Firefox render something like this:

At this stage all this means is the analytics plugin, i.e. the J2EE or ISAPI servlet, has thrown an error. That is all. Now, 95% of the time this will be because Presentation Services isn’t running, either by design (i.e. you forgot to start it) or because it’s barfed (in which case you need to check its log files etc and fix the problem).

Analytics logfile

Best practice demands a logical approach, so rather than rushing off to Presentation Services, take moment to examine the analytics logfile. For OAS or OC4J you’ll normally find this in $J2EE_HOME/home/application-deployments/analytics/home_default_group_1/application.log (where $J2EE_HOME will be the j2ee directory underneath your OAS or OC4J installation folder). Open up the logfile and navigate to the bottom of it, and work up it backwards until you get a date and timestamp and a message like this:

09/12/09 09:38:30.885 analytics: Servlet error

The next line(s) will tell you what the problem is, followed by a bunch of generic java gibberish and stack. Ignore the latter and pick out the action error, which will often be:

java.net.ConnectException: Connection refused

or sometimes:

java.net.ConnectException: Connection timed out

(Does anyone have additional errors to add in here?)

Troubleshooting

The errors are often self-explanatory (so long as you understand the architecture); “Connection refused” means that analytics tried to connect to sawserver and couldn’t. Once the problem is established then it’s a case of working through in a logical manner to determine the cause.
Connection refused is 95% of the time simply that Presentation Services (sawserver) isn’t running. Or maybe it is running, but on a different host or port than analytics is looking for.

To check where analytics is going to be looking for sawserver, examine the analytics configuration file $J2EE_home/applications/analytics/analytics/WEB-INF/web.xml (different for ISAPI, see last paragraph here).
There’ll be configuration lines matching one of these two examples. The default is this:

<init-param>
<param-name>oracle.bi.presentation.sawserver.Host</param-name>
<param-value>localhost</param-value>
</init-param>
<init-param>
<param-name>oracle.bi.presentation.sawserver.Port</param-name>
<param-value>9710</param-value>
</init-param>

A customised (e.g. for <a href="clustered resilience) entry may look like this:

<init-param>
<param-name>oracle.bi.presentation.sawservers</param-name>
<param-value>BISandbox01:9710;BISandbox02:9710</param-value>
</init-param>

sawserver

Let’s check the connectivity from both sides. First off, is Presentation Services (sawserver) running on the server we’re expecting it to be and listening on the correct port? In unix we can check this quite simply using the ps command and filtering it with the grep command. On the host that we’re expecting sawserver to be, run this:

$ ps -ef|grep sawserver
oracle   14827     1  0 09:58 pts/0    00:00:00 /bin/sh /app/oracle/product/obiee/setup/sawserver.sh
oracle   14842 14827 35 09:58 pts/0    00:00:01 /app/oracle/product/obiee/web/bin/sawserver

If there’s no output from this (or only the grep itself) then sawserver’s not running, and you need to fix that before proceeding.
On Windows check the Services window (services.msc) and task manager for sawserver.exe.

Assuming sawserver is running, now check that it is listening on the port specific in the analytics configuration file (see above). In this example, I’m checking for the default port, 9710:

$ netstat -a|grep 9710
tcp        0      0 *:9710                      *:*                         LISTEN

If there’s no output from the command then it means that port 9710 is not in use, i.e. sawserver is not listening on it. N.B. at this point it is theoretically possible that another application is using port 9710 – all we’re proving is that something is using it. But unless you’ve changed sawserver’s port (in instanceconfig.xml) then the fact it’s started up means that it is it using 9710 because it won’t start if another application is using its port.
In Windows you can use netstat -a but there’s no grep by default so you need to scroll down the output to look for the port.

So – sawserver is running on the expected host, and listening on the correct port.

analytics

Now let’s examine connectivity from the point of view of the analytics plugin (which is flow of the traffic too, i.e. connecting TO sawserver).
On the server hosting your application server (OAS/OC4J/IIS, etc) -which may or may not be the same as your Presentation Services – we want to test if Presentation Services can be connected to at the network layer. To do this we’re going to prod the port and host that it’s configured on (according to web.xml, see above).

The following is on OEL 4, which is a based on RedHat so I’d expect that to behave the same.
First get a “control” output for connecting to a port that most definitely is not open to traffic. Find a port on your sawserver host (which may or may not be local) that’s unused:

$ netstat -a|grep 9999
$

If you get output from the netstat then pick another port until you don’t
Now let’s try connecting to it to see what happens when we connect to a closed port:

$ telnet localhost 9999
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

So – our control output for a closed port is this telnet: connect to address 127.0.0.1: Connection refused.

Recall what host and port we determined analytics was trying to connect to (from web.xml, see above), and run the test for it. In this example I’ll check for the default – localhost and 9710.
If we get something like this:

$ telnet localhost 9710
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

then it shows the port and host is accepting connections. You can’t do much more from here that I’m aware of, but it proves the port is open.

However if we get this:

$ telnet localhost 9710
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

then it would tell us that the port we’re expecting to be open isn’t – and you have a problem! See below for further suggestions.

On Windows you’ll get similar behaviour for a failed connection:

C:\>telnet localhost 9999
Connecting To localhost...Could not open connection to the host, on port 9999: Connect failed

For a successful connection you will normally find the command window clears and you get a flashing cursor. Enter a few random characters or hit Ctrl-C to return to the command prompt.

Further troubleshooting

If you get a connection error when you telnet to the host and port that you think sawserver should be on then you have identified the problem, and now need to diagnose the cause.
Starting points for this are:

• If you’re not using an IP then check if the hostname resolves correctly. Try pinging it. If it doesn’t ping then you have general connectivity problems outside of OBIEE and need to speak with your network team to resolve them.
• If the host pings but the port still is not accessible is it being blocked by a firewall?

There’s an interesting case study around this problem here, and an unsolved one here.

Summary

This is a fairly low-level way of methodically picking your way through problems between two of the OBIEE components.
As I’ve said, 95% of the time it’s a simple thing, that Presentation Services isn’t running. However hopefully this article gives you more of a basis on which to diagnose and solve the remaining 5% of issues.
If you can’t telnet to sawserver’s host and port from the machine that your application service is running on then your problem lies in connectivity and you need to fix that before trying to fix anything else.

Footnote – sawping

Just after writing this article I remembered a utility called sawping that I first saw mentioned by Srinivas Malyala here. In essence it does the same as what I documented using telnet above – it tests for sawserver on a given hostname and port.
I’d be interested to know if it does any more than check for the open port (i.e. does it interrogate the application on the end of the port to check it is sawserver). Watching saw.rpc entries to the sawserver.log it doesn’t look like it, or if it does it’s not logged.

To use it in unix you need to dot-source $OBIEE_HOME/setup/sa-init.sh (or sa-init64.sh) first to set your environment variables and paths:

$ . ./sa-init.sh
$

Test the default hostname and port (I don’t think this parses analytics’ web.xml):

$sawping
Server alive and well

Add the -v flag for more verbose output if you get an error:

$ sawping
Unable to connect to server. The server may be down or may be too busy to accept additional connections.

$ sawping -v
Unable to connect to server. The server may be down or may be too busy to accept additional connections.
An error occurred during execution of "connect". Connection refused [Socket:3]
Error Codes: ETI2U8FA

Test for sawserver on a different host:

$ sawping -s bisandbox02
Server alive and well

Note the message tells you what the problem is if there is an error (in this example, “Unable to resolve address”)

$ sawping -s bisandboxxxxx02 -v
An error occured during process. Run in verbose mode to see error details.
Unable to resolve the address for bisandboxxxxx02.
Error Codes: AXSBMN8D:

TRY_AGAIN

Test for sawserver on a different host and port:

$ sawping -s bisandbox02 -p 9711 -v
Server alive and well

To use the utility in Windows either add $OBIEE_Home/web/bin to your PATH environment variable, or reference it directly. The argument syntax remains the same:

C:\>c:\OracleBI\web\bin\sawping.exe
Server alive and well

C:\>c:\OracleBI\web\bin\sawping.exe  -s bisandbox02
Server alive and well

C:\>c:\OracleBI\web\bin\sawping.exe -p 9711 -v
Unable to connect to server. The server may be down or may be too busy to accept additional connections.
An error occurred during execution of "connect". No connection could be made because the target machine actively refused it.
 [Socket:1808]
Error Codes: ETI2U8FA

UKOUG TEBS 2009

This was my first UKOUG TEBS, in fact my first conference I’d ever attended! I was quite unsure what to expect, but three days later I can safely say it was invaluable.

The variety of presentations and expertise being shared was impressive, and it was great to hear people sharing and discussing their ideas and opinions around the subjects I work with each day.

Working in isolation is not a good idea, one can develop a blinkered or bunker mentality. Blogging is good for breaking out of this, as is, I’m now convinced, twittering. However, nothing beats meeting up with folk and discussing problems face to face, listening and learning from their experience.

For a long time I’ve struggled with training courses, finding them too simplistic and slow paced (there are occasional exceptions, as always). Now I know for sure – I’d happily forgo every single one for a conference any day!

My highlights of the conference included:

• Graham Wood talking about ASH
• Tim Reeves talking about WADER and WHAM, two in-house OBIEE metadata tools developed by Hitachi. There’s obviously a lot of work gone into them and therefore IP to be protected but it struck me that in other areas of technology the tools would be proudly shared as open-source (in one form or another) rather than in effect sales-demo’d.
• Listening to Dan Norris talk about Exadata, at a Roundtable session chaired by Joel Goodman. I felt a bit of a fraud at the “Roundtable” knowing very little about the subject but it was fascinating to listen to and brilliant to get it from the “horses mouth” instead of through the BS screen of salesmen.
• Doug Burns‘ entertaining, slick and informative demo of performance screens in EM. Bravo for ditching the slides and going live!
• An interesting BI best-practice session from Vicky Minnis. Good to see that it’s not unique for organisations to get their BI wrong. Frustrating to know that it’s so widespread and people don’t/won’t learn!
• Will O’Shea and Jason Perkins did a useful talk entitled “Healthy Datawarehouse” with some good detail and discussion in it. I felt sorry for the guys at the end when they got in-effect heckled by someone in the audience who was one of their users. A shame, because a lot of the audience got up and left at this point, so the Q&A was a bit spoilt. Even if the user had a valid point to make, it should have been made 1:1 after the session.
• And last but by no means least – meeting the guys from RittmanMead, and getting name-checked in one of Mark’s presentations!

The conference itself seemed very slickly run, I was impressed. The ICC and UKOUG staff were very helpful too. A few things for next time:

• The WiFi was very intermittent, and poor coverage
• The twitter hashtag for the conference was a great idea, but session tags seemed a step too far that several session chairs openly mocked …
• Twittering for cancelled sessions would have been a useful way of communicating it
• The free pen in the conference bag was rubbish!   The Informatica one from their stand was much better
• It would be fantastic if the sessions were available on a Google Calendar or iCal format, to save having to type them all in for storing on a PDA/phone etc.
• Great idea providing water bottle to refill instead of having lots of waste, kudos, but it tasted so plastiky, yuk!
• The venue for the roundtables was too open and subject to noise, both people accidentally wandering through and ICC staff moving tea & coffee setup around.
• Ironic that at a hi-tech conference there are bits of paper floating around for speaker evaluation forms that will presumably have to be entered into computer by hand. Paper’s convenient for some, but please can we have an online version too?

I think this summarises everything.

Why did this make me think of the OBIA upgrade documentation??

via I think this summarizes everything..

SoOotW and sweep

(Sorry for the lame title, but it gives me an excuse to put this picture in )

I was really pleased by the response I got from my posting about The state of OBIEE on the web, knowing that it’s not just me goes a long way to keeping my blood pressure down.

The OTN forum is still an unbalanced mix of tosh (is this guy for real?) with the occasional insightful and interesting post or idea such as this really neat one from Joe Bertram about using multiple presentation services on top of the same RPD to give different interfaces to different end devices.

That there is still some good content on OTN being posted, and with OBIEE 11g hopefully around the corner (and thus a whole lot more to discuss!), makes me think that Andriy Yakushyn’s kind offer of hosting a new, moderated, forum could be a good idea.
I discovered Oracle-L the other day which has what I think is a good approach to moderation:

New subscribers to this list are not able to post messages at first. To get this privilege turned on, you must email me. If you are using an anonymous email address, you must identify yourself. I also like to know what city you live in and what company you work for. Please be sure to send the request from the address for which you want the privilege, not from some other address. Please avoid being rude or impatient. Please subscribe to the list before asking for the posting privilege.

This way casual [lazy] fire-and-forget posters with their UGRENT!!! questions are dissuaded, the community becomes more of a community of knowns instead of user2371623 answering user8923713’s question – and important for making everything work, there is no burden on moderator(s) having to vet all posts. Anyone misbehaving can easily be warned or booted, but generally the forum would run itself.
To Joel Garry’s point against moderation, I disagree – to “gangfaq” someone on OTN would simply add to the clutter in the group.

OBIEE application servers, now and future

Oracle have published an interesting doc 968223.1, entitled “Enterprise Deployment of Oracle BI EE on OC4J and App Servers”.

It details the differences between OC4J and OAS which is useful for the current versions of OBIEE. It then also gives a useful heads-up — that WebLogic becomes the App server of choice in the next version of OBIEE

All of this changes in OBI EE 11g where several projects will become absolutely dependent upon an App Server. Within Oracle Fusion Middleware, WebLogic Server will be the application server and OBI EE 11g will be deployed and certified with WebLogic Server.

Resolved: sawserver : Error loading security privilege /system/privs/catalog/ChangePermissionsPrivilege

Whilst installing OBIA 7.9.6.1 I hit this problem when firing up Presentation Services (sawserver):

Error loading security privilege /system/privs/catalog/ChangePermissionsPrivilege.

A quick search on the forums threw up two posts suggesting a corrupted WebCat.

Since I’d got this webcat fresh out of the box I was puzzled how it could be corrupted.

I did a bit more tinkering (including nosying around in the sawserver log), before realising it was indeed corrupt, and that I was indeed a muppet.

Here’s what happened after copying EnterpriseBusinessAnalytics.zip to my (unix) Presentation Services box:

$unzip EnterpriseBusinessAnalytics.zip
Archive:  EnterpriseBusinessAnalytics.zip
   creating: root/
   creating: root/
   creating: root/shared/
   creating: root/shared/automotive/
   creating: root/shared/automotive/prompts/
  inflating: root/shared/automotive/prompts/gf_model+model+year+trim
  inflating: root/shared/automotive/prompts/gf_model+model+year+trim.atr
  inflating: root/shared/automotive/prompts.atr
[...]
[ lots of files here ]
[...]
  inflating: root.atr                

$ls -l
total 110494
-rw-------   1 user   group    37655058 Oct  7 03:44 EnterpriseBusinessAnalytics.zip
drwx------   5 user   group       1024 Sep 18 01:06 root
-rw-------   1 user   group         60 Dec  6  2006 root.atr

Huh? What gives? Where’s my EnterpriseBusinessAnalytics web cat folder?
Well, quite obviously it’s unpacked it without a parent directory name.
That’s easily solved:

$mkdir EnterpriseBusinessAnalytics
$mv root EnterpriseBusinessAnalytics

Then I started up Presentation Services and got the error “Error loading security privilege /system/privs/catalog/ChangePermissionsPrivilege.”

If you can spot my snafu at this point the my only defence is that there was quite a lot of other gumf in the catalog folder, not just the files illustrated above

The solution

Whilst I’d moved the root folder into my webcat folder, I’d neglected to move root.atr – in effect corrupting the web catalog.

So simple, but so frustating!

The solution in this case was to move root.atr into the webcat folder, alongside root.
It’s worth noting that this may not be the solution in all occurrences of this error, it depends on where the corruption has occurred.

Footnote

The silver lining being a good chance to poke around inside sawserver a bit more and discover gems like this in the logging:

The Oracle BI Presentation Server is proudly running under user: TODO_implement_this

It’s nice that it takes pride in its work, although shame we never get to find out the user’s name

Deploying Oracle Business Intelligence Enterprise Edition on Sun Systems

A very interesting new PDF from Sun on deploying OBIEE has been published, with discussions on architecture, performance and best practice.

This Sun BluePrints article describes an enterprise deployment architecture for Oracle Business Intelligence Enterprise Edition using Sun servers running the Solaris Operating System and Sun Storage 7000 Unified Storage systems. Designed to empower employees in organizations in any industry—from customer service, shipping, and finance to manufacturing, human resources, and more—to become potential decision makers, the architecture brings fault tolerance, security, resiliency, and performance to enterprise deployments. Taking advantage of key virtualization technologies, the architecture can be used to consolidate multiple tiers onto a single system to help reduce cost and complexity. A short discussion of the performance characteristics of the architecture using a realistic workload also is included.

The paper’s by Maqsood Alam, Luojia Chen, Chaitanya Jadaru, Ron Graham and Giri Mandalika.

Direct download: Deploying Oracle Business Intelligence Enterprise Edition on Sun Systems
Main link (requires Sun registration to download): https://www.sun.com/offers/details/821-0698.xml

#Fail: My Oracle Support

Metalink was retired this weekend and made way for the new My Oracle Support system. It didn’t go as smoothly as it could have done.

This post is going to be a bit of a rambling rant.

Ultimately people, including me, don’t like their cheese being moved (not unless there’s a really runny piece of Camembert at the end of it). That makes it a bit more difficult to discuss because some of people’s complaints will just be geeks being stubborn (and boy, can geeks be stubborn). Arguments descend into minutiae of detail and flash vs DHTML – whilst the bigger picture gets lost.

People especially don’t like their cheese being moved (okay okay enough of the cheese) change to systems that they depend on to do their job. If it were the migration of a blogging website or somesuch then it’d be a bummer, there’d be grumbling about it, but ultimately people would probably be quite sanguine about it. When it comes to a support website though, it has to be available.

If this were a system that we delivered to our users then we’d (hopefully) get laughed out the building and/or strung up. It stinks, and there’s no denying it. Maybe once upon a time the concept was a good one, but somewhere along the line looks overtook functionality and someone in charge forgot that this wasn’t a beauty contest but a support website relied on by many many people for doing their jobs. Some of the new functionality (and it is there) in MOS is quite neat — but I only discover it by accident because most of the time I’m waiting for the s#dding thing to load or respond to a mouse click.

I can understand a marketing agency designing some krazy kool website to sell junk food to kidz using lots of flash and clever code, and the benefit (whizzy effects impressing target audience) outweighs the disadvantage (lower spec’d PCs can’t display it properly or at reasonable speed). But a support website? C’mon! It’s a support website! It should work in Lynx (maybe not quite). It was apparently tested on a 2 gig / 3Ghz PC – I’d suggest that’s hardly standard fare yet.

I want to go to the Oracle support website and get support. I shouldn’t have to attend training or webinars to use a website. If I do, then the website’s badly designed. Seriously. And enough with the rambling waffly emails already. I get enough emails everyday that any communication about Metalink/MOS needs to be clear and concise. It doesn’t need BS in it about a “Leveraging the personalized, proactive, and collaborative support capabilities [...] reduce the time you spend maintaining Oracle solutions” (literal quotation).

As an OBIEE user I’ve already been using My Oracle Support after Metalink3 was discontinued a few months ago. After that migration I raised several non-technical SRs reporting various problems, and almost always got a response with the implication that I was doing something wrong or needed helping, rather than the impression that I’d reported a bug which needed fixing.
Somehow, and I’d have hoped this would come from within the organisation, bugs reported by customers need to go straight to Dev, rather than the customer fobbed off. And I was fobbed off without a doubt. Next time I shall not bother reporting problems because it’s not worth the time I spend on it.

Sr. Customer Support Manager Chris Warticki at Oracle has blogged about the cutover:

• Classic MetaLink:Na,na,na,Nah,na,na. Hey hey Goodbye
• My Oracle Support – Migration Update

There’s another blog from Support here.

OUG survey

OUG are running a survey until 19th Nov:

Last weekend, Oracle closed the current Metalink service and migrated the users to My Oracle Support.
UKOUG has had reports from its membership and from across EMEA of a number of problems in this migration.
In order to enter into dialogue with Oracle on this, we would appreciate it if you could complete the following very short survey.

You can find the survey here

Footnote – non-flash My Oracle Support

There is a non-flash version of My Oracle Support at http://supporthtml.oracle.com. However from where I am I can’t login directly (see errors below)

You might be able to get in indirectly on this link.

Clicking the Home link when going in on this link or trying to login from http://supporthtml.oracle.com gives 500 Internal Server error on IE and “Recursive error in error-page calling for /secure/error.jspx, see the application log for details. ” in FireFox.

Looking at http://supporthtml.oracle.com and having used the flash version for a while now the non-flash version looks pretty similar. More effort’s gone into its appearance than I’d expect for a site that’s been knocked out in HTML as a purely-functional alternative to the main flash site.
It’s evidently not fully functional yet but I wonder if someone’s taken the wise idea to do the rewrite in non-flash and will ditch the flash version at some point in the future?

Follow up

It looks like things are stabilising a bit, although I still get inconsistent results when using supporthtml.oracle.com.

Some more blogs about the problems:

• http://oracledoug.com/serendipity/index.php?%2Farchives%2F1541-My-Oracle-Support.html
• http://jkstill.blogspot.com/2009/11/metalink-we-barely-knew-ye.html
• http://optimaldba.blogspot.com/2009/11/its-messbut-its-no-surprise.html
• http://dbasrus.blogspot.com/2009/11/quite-frankly-warticki-you-should.html