rmoff

December 18, 2009

libnnz10.so: cannot restore segment prot after reloc: Permission denied

Filed under: obiee, OEL — rmoff @ 19:18

Quick post as the snow’s coming down and I wanna go home …

I’ve been working on building a VM based on OEL5.4 and OBIEE 10.1.3.4.1. After installing XE 10.2 I tried to fire my RPD up, but hit this:

/usr/lib/oracle/xe/app/oracle/product/10.2.0/server/lib/libnnz10.so: cannot restore segment prot after reloc: Permission denied
[nQSError: 46029] Failed to load the DLL /app/oracle/product/obiee/server/Bin/libnqsdbgatewayoci10g.so. Check if ‘Oracle OCI 10G’ database client is installed.

If you trace the ‘stack’ back you find that it parses down to this nub of an error:

libnnz10.so: cannot restore segment prot after reloc: Permission denied

Google throws up a few hits, but not terribly detailed.

The error’s to do with SELinux security. I’ll hold my hand up now and say that I don’t understand SELinux nor the implications of these commands except that it fixes my problem and I’m working in a sandbox environment. If you are working anywhere that this kind of thing matters then please consult with someone who knows what they’re talking about first!

  1. As root, generate a file of blocked activity (tail gets and follows the file, tee echoes it to stdout -the console- as well as writing it to the file oracle.log)
    tail -f /var/log/audit/audit.log | tee oracle.log
  2. Do something in OBIEE to generate the error (run a report etc), you should see more output appear from the root session, like this:
    type=AVC msg=audit(1261159266.277:119): avc:  denied  { execmod } for  pid=9396 comm="nqsserver" path="/usr/lib/oracle/xe/app/oracle/product/10.2.0/server/lib/libnnz10.so" dev=dm-0 ino=3834074 scontext=user_u:system_r:unconfined_t:s0 tcontext=system_u:object_r:lib_t:s0 tclass=file
    type=SYSCALL msg=audit(1261159266.277:119): arch=40000003 syscall=125 success=no exit=-13 a0=3611000 a1=1dd000 a2=5 a3=b73313c0 items=0 ppid=1 pid=9396 auid=500 uid=500 gid=500 euid=500 suid=500 fsuid=500 egid=500 sgid=500 fsgid=500 tty=pts1 ses=1 comm="nqsserver" exe="/app/oracle/product/obiee/server/Bin/nqsserver" subj=user_u:system_r:unconfined_t:s0 key=(null)
  3. You can see snippets in there that are recognisable – the nqsserver program name and path. Press Ctrl-C to halt the tail command and return to the prompt.
  4. Use the audit2allow program to add rules to permit the blocked behaviour:
    audit2allow -M nqsserver -i oracle.log

    Where -M defines a new module name (i’ve called it after nqsserver), and -i is the input file of blocked audit

  5. You should be prompted to run semodule to activate the change:
    semodule -i nqsserver.pp
  6. Now tail (no need to tee this time) the /var/log/audit/audit.log and repeat the activity you did before (run a report etc), and hopefully you won’t get any entries this time. Doublecheck NQServer.log to make sure.
  7. ? If a malicious program could do its blocked activity whilst you were ‘recording’ the audit file for blocked nqsserver might you end up allowing the malicious program inadvertently?

(Based on information from here: http://forums.oracle.com/forums/thread.jspa?messageID=1285844)

Advertisements

December 14, 2009

Running the OBIEE admin tool on Unix

Filed under: obiee — rmoff @ 09:43

Mucho kudos to Borkur Steingrimsson for getting the OBIEE admin tool working on Unix!

December 11, 2009

CAF

Filed under: caf, obiee — rmoff @ 12:39

Very interesting post by Kevin McGinley about CAF here:  CAF = Migration Utility? Use Caution!

It articulates better than I ever could reasons against using CAF particularly in a production environment.

Since the tool came out I’d been struggling to get my head around it, convinced I was missing something. I still don’t profess to understand it properly, but Kevin’s article reassures me that I shouldn’t be losing too much sleep over it, especially given that it’s unsupported and won’t work with 11g.

December 9, 2009

Troubleshooting Presentation Services / analytics connectivity

Filed under: obiee, sawping, sawserver — rmoff @ 11:56

Short but sweet this one – a way of troubleshooting connectivity problems between analytics (the Presentation Services Plug-in, either j2ee servlet or ISAPI, a.k.a. SAWBridge) and sawserver (Presentation Services).

For a recap on the services & flow please see the first few paragraphs of this post.

Problems in connectivity between analytics and sawserver normally manifest themselves through this error message:

500 Internal Server Error
Servlet error: An exception occurred. The current application deployment descriptors do not allow for including it in this response. Please consult the application log for details.

Which IE and Firefox render something like this:

At this stage all this means is the analytics plugin, i.e. the J2EE or ISAPI servlet, has thrown an error. That is all. Now, 95% of the time this will be because Presentation Services isn’t running, either by design (i.e. you forgot to start it) or because it’s barfed (in which case you need to check its log files etc and fix the problem).

Analytics logfile

Best practice demands a logical approach, so rather than rushing off to Presentation Services, take moment to examine the analytics logfile. For OAS or OC4J you’ll normally find this in $J2EE_HOME/home/application-deployments/analytics/home_default_group_1/application.log (where $J2EE_HOME will be the j2ee directory underneath your OAS or OC4J installation folder). Open up the logfile and navigate to the bottom of it, and work up it backwards until you get a date and timestamp and a message like this:

09/12/09 09:38:30.885 analytics: Servlet error

The next line(s) will tell you what the problem is, followed by a bunch of generic java gibberish and stack. Ignore the latter and pick out the action error, which will often be:

java.net.ConnectException: Connection refused

or sometimes:

java.net.ConnectException: Connection timed out

(Does anyone have additional errors to add in here?)

Troubleshooting

The errors are often self-explanatory (so long as you understand the architecture); “Connection refused” means that analytics tried to connect to sawserver and couldn’t. Once the problem is established then it’s a case of working through in a logical manner to determine the cause.
Connection refused is 95% of the time simply that Presentation Services (sawserver) isn’t running. Or maybe it is running, but on a different host or port than analytics is looking for.

To check where analytics is going to be looking for sawserver, examine the analytics configuration file $J2EE_home/applications/analytics/analytics/WEB-INF/web.xml (different for ISAPI, see last paragraph here).
There’ll be configuration lines matching one of these two examples. The default is this:

<init-param>
<param-name>oracle.bi.presentation.sawserver.Host</param-name>
<param-value>localhost</param-value>
</init-param>
<init-param>
<param-name>oracle.bi.presentation.sawserver.Port</param-name>
<param-value>9710</param-value>
</init-param>

A customised (e.g. for <a href="clustered resilience) entry may look like this:

<init-param>
<param-name>oracle.bi.presentation.sawservers</param-name>
<param-value>BISandbox01:9710;BISandbox02:9710</param-value>
</init-param>

sawserver

Let’s check the connectivity from both sides. First off, is Presentation Services (sawserver) running on the server we’re expecting it to be and listening on the correct port? In unix we can check this quite simply using the ps command and filtering it with the grep command. On the host that we’re expecting sawserver to be, run this:

$ ps -ef|grep sawserver
oracle   14827     1  0 09:58 pts/0    00:00:00 /bin/sh /app/oracle/product/obiee/setup/sawserver.sh
oracle   14842 14827 35 09:58 pts/0    00:00:01 /app/oracle/product/obiee/web/bin/sawserver

If there’s no output from this (or only the grep itself) then sawserver’s not running, and you need to fix that before proceeding.
On Windows check the Services window (services.msc) and task manager for sawserver.exe.

Assuming sawserver is running, now check that it is listening on the port specific in the analytics configuration file (see above). In this example, I’m checking for the default port, 9710:

$ netstat -a|grep 9710
tcp        0      0 *:9710                      *:*                         LISTEN

If there’s no output from the command then it means that port 9710 is not in use, i.e. sawserver is not listening on it. N.B. at this point it is theoretically possible that another application is using port 9710 – all we’re proving is that something is using it. But unless you’ve changed sawserver’s port (in instanceconfig.xml) then the fact it’s started up means that it is it using 9710 because it won’t start if another application is using its port.
In Windows you can use netstat -a but there’s no grep by default so you need to scroll down the output to look for the port.

So – sawserver is running on the expected host, and listening on the correct port.

analytics

Now let’s examine connectivity from the point of view of the analytics plugin (which is flow of the traffic too, i.e. connecting TO sawserver).
On the server hosting your application server (OAS/OC4J/IIS, etc) -which may or may not be the same as your Presentation Services – we want to test if Presentation Services can be connected to at the network layer. To do this we’re going to prod the port and host that it’s configured on (according to web.xml, see above).

The following is on OEL 4, which is a based on RedHat so I’d expect that to behave the same.
First get a “control” output for connecting to a port that most definitely is not open to traffic. Find a port on your sawserver host (which may or may not be local) that’s unused:

$ netstat -a|grep 9999
$

If you get output from the netstat then pick another port until you don’t
Now let’s try connecting to it to see what happens when we connect to a closed port:

$ telnet localhost 9999
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

So – our control output for a closed port is this telnet: connect to address 127.0.0.1: Connection refused.

Recall what host and port we determined analytics was trying to connect to (from web.xml, see above), and run the test for it. In this example I’ll check for the default – localhost and 9710.
If we get something like this:

$ telnet localhost 9710
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

then it shows the port and host is accepting connections. You can’t do much more from here that I’m aware of, but it proves the port is open.

However if we get this:

$ telnet localhost 9710
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

then it would tell us that the port we’re expecting to be open isn’t – and you have a problem! See below for further suggestions.

On Windows you’ll get similar behaviour for a failed connection:

C:\>telnet localhost 9999
Connecting To localhost...Could not open connection to the host, on port 9999: Connect failed

For a successful connection you will normally find the command window clears and you get a flashing cursor. Enter a few random characters or hit Ctrl-C to return to the command prompt.

Further troubleshooting

If you get a connection error when you telnet to the host and port that you think sawserver should be on then you have identified the problem, and now need to diagnose the cause.
Starting points for this are:

  • If you’re not using an IP then check if the hostname resolves correctly. Try pinging it. If it doesn’t ping then you have general connectivity problems outside of OBIEE and need to speak with your network team to resolve them.
  • If the host pings but the port still is not accessible is it being blocked by a firewall?

There’s an interesting case study around this problem here, and an unsolved one here.

Summary

This is a fairly low-level way of methodically picking your way through problems between two of the OBIEE components.
As I’ve said, 95% of the time it’s a simple thing, that Presentation Services isn’t running. However hopefully this article gives you more of a basis on which to diagnose and solve the remaining 5% of issues.
If you can’t telnet to sawserver’s host and port from the machine that your application service is running on then your problem lies in connectivity and you need to fix that before trying to fix anything else.

Footnote – sawping

Just after writing this article I remembered a utility called sawping that I first saw mentioned by Srinivas Malyala here. In essence it does the same as what I documented using telnet above – it tests for sawserver on a given hostname and port.
I’d be interested to know if it does any more than check for the open port (i.e. does it interrogate the application on the end of the port to check it is sawserver). Watching saw.rpc entries to the sawserver.log it doesn’t look like it, or if it does it’s not logged.

To use it in unix you need to dot-source $OBIEE_HOME/setup/sa-init.sh (or sa-init64.sh) first to set your environment variables and paths:

$ . ./sa-init.sh
$

Test the default hostname and port (I don’t think this parses analytics’ web.xml):

$sawping
Server alive and well

Add the -v flag for more verbose output if you get an error:

$ sawping
Unable to connect to server. The server may be down or may be too busy to accept additional connections.

$ sawping -v
Unable to connect to server. The server may be down or may be too busy to accept additional connections.
An error occurred during execution of "connect". Connection refused [Socket:3]
Error Codes: ETI2U8FA

Test for sawserver on a different host:

$ sawping -s bisandbox02
Server alive and well

Note the message tells you what the problem is if there is an error (in this example, “Unable to resolve address”)

$ sawping -s bisandboxxxxx02 -v
An error occured during process. Run in verbose mode to see error details.
Unable to resolve the address for bisandboxxxxx02.
Error Codes: AXSBMN8D:

TRY_AGAIN

Test for sawserver on a different host and port:

$ sawping -s bisandbox02 -p 9711 -v
Server alive and well

To use the utility in Windows either add $OBIEE_Home/web/bin to your PATH environment variable, or reference it directly. The argument syntax remains the same:

C:\>c:\OracleBI\web\bin\sawping.exe
Server alive and well

C:\>c:\OracleBI\web\bin\sawping.exe  -s bisandbox02
Server alive and well

C:\>c:\OracleBI\web\bin\sawping.exe -p 9711 -v
Unable to connect to server. The server may be down or may be too busy to accept additional connections.
An error occurred during execution of "connect". No connection could be made because the target machine actively refused it.
 [Socket:1808]
Error Codes: ETI2U8FA

December 3, 2009

UKOUG TEBS 2009

Filed under: obiee — rmoff @ 14:22

This was my first UKOUG TEBS, in fact my first conference I’d ever attended! I was quite unsure what to expect, but three days later I can safely say it was invaluable.

The variety of presentations and expertise being shared was impressive, and it was great to hear people sharing and discussing their ideas and opinions around the subjects I work with each day.

Working in isolation is not a good idea, one can develop a blinkered or bunker mentality. Blogging is good for breaking out of this, as is, I’m now convinced, twittering. However, nothing beats meeting up with folk and discussing problems face to face, listening and learning from their experience.

For a long time I’ve struggled with training courses, finding them too simplistic and slow paced (there are occasional exceptions, as always). Now I know for sure – I’d happily forgo every single one for a conference any day!

My highlights of the conference included:

  • Graham Wood talking about ASH
  • Tim Reeves talking about WADER and WHAM, two in-house OBIEE metadata tools developed by Hitachi. There’s obviously a lot of work gone into them and therefore IP to be protected but it struck me that in other areas of technology the tools would be proudly shared as open-source (in one form or another) rather than in effect sales-demo’d.
  • Listening to Dan Norris talk about Exadata, at a Roundtable session chaired by Joel Goodman. I felt a bit of a fraud at the “Roundtable” knowing very little about the subject but it was fascinating to listen to and brilliant to get it from the “horses mouth” instead of through the BS screen of salesmen.
  • Doug Burns‘ entertaining, slick and informative demo of performance screens in EM. Bravo for ditching the slides and going live!
  • An interesting BI best-practice session from Vicky Minnis. Good to see that it’s not unique for organisations to get their BI wrong. Frustrating to know that it’s so widespread and people don’t/won’t learn!
  • Will O’Shea and Jason Perkins did a useful talk entitled “Healthy Datawarehouse” with some good detail and discussion in it. I felt sorry for the guys at the end when they got in-effect heckled by someone in the audience who was one of their users. A shame, because a lot of the audience got up and left at this point, so the Q&A was a bit spoilt. Even if the user had a valid point to make, it should have been made 1:1 after the session.
  • And last but by no means least – meeting the guys from RittmanMead, and getting name-checked in one of Mark’s presentations! 🙂

The conference itself seemed very slickly run, I was impressed. The ICC and UKOUG staff were very helpful too. A few things for next time:

  • The WiFi was very intermittent, and poor coverage
  • The twitter hashtag for the conference was a great idea, but session tags seemed a step too far that several session chairs openly mocked …
  • Twittering for cancelled sessions would have been a useful way of communicating it
  • The free pen in the conference bag was rubbish! 😛  The Informatica one from their stand was much better
  • It would be fantastic if the sessions were available on a Google Calendar or iCal format, to save having to type them all in for storing on a PDA/phone etc.
  • Great idea providing water bottle to refill instead of having lots of waste, kudos, but it tasted so plastiky, yuk!
  • The venue for the roundtables was too open and subject to noise, both people accidentally wandering through and ICC staff moving tea & coffee setup around.
  • Ironic that at a hi-tech conference there are bits of paper floating around for speaker evaluation forms that will presumably have to be entered into computer by hand. Paper’s convenient for some, but please can we have an online version too?

Create a free website or blog at WordPress.com.