rmoff

December 6, 2010

Adding OBIEE monitoring graphs into OAS

Filed under: hack, jmx, monitoring, OAS, obiee, unix — rmoff @ 21:30

Introduction

This is the third part of three detailed articles making up a mini-series about OBIEE monitoring. It demonstrates how to capture OBIEE performance information, and optionally graph it out and serve it through an auto-updating webpage.

This final article describes how to bolt on to OAS a simple web page hosting the graphs that you created in part 2, plotting data from OBIEE collected in part 1.

The webpage

This is just an old-school basic HTML page, with a meta-refresh tag (which note that Chrome doesn’t work with) and img tags:

<html>
<meta http-equiv="refresh" content="60">
<head>
<title>OBIEE Servers</title>
</head>
<body>
<img src="server01.png"></br>
<img src="server02.png"></br>
</body>
</html>

I shan’t patronise you nor embarrass myself with my rusty HTML skills any further – I’ll leave you how to build your dashboard how you want it.

OAS

This is hack! I am not an expert at Apache, so please don’t take my word for it that this is the best way to do it. It worked for me, but mightn’t for you.

If you’ve got OAS installed for your OBIEE installation, you can tweak it to serve up your new graphs too. If you’re using OC4J, IIS, or another webserver, then you’ll have to figure this bit out yourself.

Assuming that your OBIEE JMX graphs and HTML files are in /tmp/obieejmx, make sure that they’re readable by all:

chmod -R o+rx /tmp/obieejmx

Now go to your OAS folder, navigate to Apache/Apache/conf, and MAKE A BACKUP of httpd.conf

cd /your/path/to/OAS/here
cd Apache/Apache/conf
cp httpd.conf httpd.conf.bak

Open httpd.conf in vi (or if you’re not a real man then FTP the file to Windows and open it in Notepad 😉 )

  1. Search for
    <IfModule mod_alias.c>
  2. Add the following beneath it:
        Alias /obieejmx/ "/tmp/obieejmx/"
        <Directory "/obieejmx/">
            AllowOverride None
            Options None
            Order allow,deny
            Allow from all
        </Directory>
    
    • Here’s where you’d change the location of your graphs and HTML file if you needed to
  3. Save httpd.conf
  4. Restart Apache
    opmnctl restartproc ias-component=HTTP_Server

    or if that doesn’t work restart OAS

    opmnctl shutdown
    opmnctl startall

Assuming you normally access OBIEE through http://myserver:7777/analytics/ then you should now be able to go to http://myserver:7777/obieejmx/ and view the fruits of your hard-earned work.

What next

Obviously, the gnuplot/OAS hack is a bit crude, but for me was the quickest way to get “to market” the power of the OBIEE systems management metric collection by jmx that is possible for anyone with some basic *nix skills and some time to put it together.

The BI Management Pack for Enterprise Manager probably provides some if not all of this functionality but isn’t always available to use (and also has licensing implications).

Whether you collect metrics for day-to-day monitoring of OBIEE, capacity planning, or investigative work, I’ve hopefully demonstrated how easy it is to work with once you’ve got the basics mastered. And the beauty of doing it with shell scripts is that you can customise it to your heart’s content.

There’s a whole bunch of analysis that I’d like to do now, around things like our registered user count vs logged on users vs active users (to determine what actually is our concurrent user rate), as well as profiling BI Server load against database load.

It would also be fun to develop the HTML just a little bit further to create a mock drill-down on the graphs, although if you’re anything like me be aware of “just tweaking for a minute” turning into far too long given then throwaway nature of the solution.

Finally, bear in mind this is now dated technology – some of it may be on the junk heap for OBI11g.

January 21, 2010

Hardening OAS

Filed under: Apache, OAS, security — rmoff @ 11:18

Oracle Application Server (OAS) is the Web and Application server typically deployed with OBIEE. There are several settings which by default may be viewed as security weaknesses. Whether realistically a target or not, it’s good practice to always be considering security and lock down your servers as much as reasonably possible. I adopt the default stance of having to find a reason to leave something less secure, rather than justify why it needs doing.

There are various tools and companies out there that will help you scan your deployments for weaknesses. In reading about this I found Nikto which runs on all platforms. In essence it takes a hostname and port and scans for known vulnerabilities in web servers (not just OAS).

Listed below are some of the simple things you can do to secure your default deployment of OAS.

Almost all of this is derived from the very excellent Securing Oracle Application Server by Caleb Sima

In the text below I refer to $OAS_HOME which may not be an actual environment variable, but is the home directory of your OAS installation.

Don’t forget to backup config files before you change them, and take backups of deleted files and directories.

After making the changes bounce OAS (opmnctl stopall; opmnctl startall).

As well as the specifics below you should always keep an eye on Oracle’s Critical Patch Updates.

Web server version and details

By default OAS will report its version in both HTTP headers and on error pages (such as those returned on a 404 Not Found which is easy to obtain by entering a non-existent URL):


Apply these two changes to $OAS_HOME/Apache/Apache/conf/httpd.conf:

  1. Search for ServerSignature and change it from On to Off
    This removes the server information from error pages
    Ref: http://httpd.apache.org/docs/2.2/mod/core.html#serversignature
  2. Add this on the next line:
    ServerTokens ProductOnly

    This removes some server version info from the HTTP header, and is the least possible data to reveal in Apache.
    Ref: http://httpd.apache.org/docs/2.2/mod/core.html#servertokens

After the changes have been made:


TRACE method

Read Apache Tips: Disable the HTTP TRACE method/ for information on how to see if HTTP TRACE is enabled. It is by default in OAS, and most security scanners will pick it up as a problem.

To disable it, add to $OAS_HOME/Apache/Apache/conf/httpd.conf:
TraceEnable Off

Default content

Most web and application servers come with default content such as example pages or “Welcome” pages, and OAS is no exception.
The reason for getting rid of this content is to give potential attackers one less thing to work with. Static content might give them information about software versions or paths. Dynamic content (JSPs etc) may be exploitable. Either way – what is to be gained from leaving it in place?

Apache default content

In $OAS_HOME/Apache/Apache:

mv htdocs/ htdocs.old
mkdir htdocs
vi htdocs/index.html
# enter:
<HTML><HEAD><TITLE>Nothing to see here</TITLE></HEAD><BODY>Nothing to see here, move along.</BODY></HTML>


rm $OAS_Home/Apache/Apache/icons/README
rm $OAS_Home/Apache/Apache/fcgi-bin/*

j2ee

cd $OAS_HOME/j2ee/home/default-web-app
rm -r WEB-INF/classes
rm -r examples/
echo "Nothing to see here" > index.html 

Pre-populated username in OAS login form

This could help an attacker as they are given a username to start trying to login as.

However, I can’t work out how to disable it. I opened a thread on OTN here:
http://forums.oracle.com/forums/thread.jspa?threadID=1010227&tstart=0

If you know, please leave a comment!

Weak ciphers / SSL version 2 supported

Disable the weak SSL ciphers & disable version 2 of the protocol

Add to httpd.conf after the TraceEnable statement from above:

SSLProtocol ALL -SSLv2
SSLCipherSuite HIGH:!SSLv2:!ADH:!aNULL:!eNULL:!NULL

Ref: http://httpd.apache.org/docs/2.0/mod/mod_ssl.html#sslciphersuite

Ref: http://adamyoung.net/Disable-SSLv2-System-Wide

Restarting OAS

When I started implementing this I used opmnctl restartproc, but found that HTTP_Server came back as status “Stop” or “Bounce”. I also got errors like: “time out while waiting for a managed process to restart”.

What I think happened was that the httpd (Apache) processes didn’t come down properly, and so couldn’t restart.

Therefore I resorted to opmnctl shutdown, then search for any remaining httpd processes (ps -ef|grep httpd) and kill any (kill -9 xxxx), and then restart OAS (opmnctl startall)

November 25, 2009

OBIEE application servers, now and future

Filed under: OAS, obiee, obiee 11g, weblogic — rmoff @ 12:24

Oracle have published an interesting doc 968223.1, entitled “Enterprise Deployment of Oracle BI EE on OC4J and App Servers”.

It details the differences between OC4J and OAS which is useful for the current versions of OBIEE. It then also gives a useful heads-up — that WebLogic becomes the App server of choice in the next version of OBIEE

All of this changes in OBI EE 11g where several projects will become absolutely dependent upon an App Server. Within Oracle Fusion Middleware, WebLogic Server will be the application server and OBI EE 11g will be deployed and certified with WebLogic Server.

November 6, 2009

OBIEE clustering – specifying multiple Presentation Services from Presentation Services Plug-in

Filed under: load balancing, OAS, obiee, sawserver, unix — rmoff @ 12:00

Introduction

Whilst the BI Cluster Controller takes care nicely of clustering and failover for BI Server (nqsserver), we have to do more to ensure further resilience of the stack.

A diagram I come back to again and again when working out configuration or connectivity problems is the one on P16 of the Deployment Guide. With this you can work out most issues for yourself through simple reasoning. Print it out, pin it to your wall, and read it!

As a reminder, when a user calls up the address for Answers or Dashboards the flow goes :

  1. web browser
  2. web serve r (eg OAS – Apache)
  3. app server (eg OAS – OC4J) -> BI Presentation Services Plug-in (“analytics”)
  4. BI Presentation Services
  5. (BI Server)
  6. (Database)

With clustering we are aiming to spread the load as much as possible. This gives us resilience if a component fails and capacity as the work is shared out.

This posting examines how to configure step 3 on the above list (BI Presentation Services Plug-in) to work with multiple BI Presentation Services.

From the Deployment Guide:

BI Presentation Services Plug-ins route session requests to BI Presentation Services instances using native protocol. The connections are load balanced using native load balancing capability.

BI Presentation Services receives requests from BI Presentation Services Plug-in […]. Although an initial user session request can go to any BI Presentation Services in the cluster, each user is then bound to a specific BI Presentation Services instance.

Be aware that “BI Presentation Services” is not the same as “BI Presentation Services Plug-in”:

  • “BI Presentation Services” is sawserver, a service in its own right.
  • “BI Presentation Services Plug-in” is a java servlet called analytics deployed within a J2ee application server.
    • There is also a version for IIS using ISAPI. This article is only about the j2ee version. The configuration principles should remain the same for the ISAPI plugin though.

Configuration

To configure the j2ee plug-in, do the following:

  1. Locate web.xml found in $J2EE_home/applications/analytics/analytics/WEB-INF
    • See note below regarding this path as it is contrary to that given in the Deployment Guide on p35
  2. Create a backup of the web.xml file
  3. By default the file will have two sets of init-params. Remove these:
    <init-param>
    <param-name>oracle.bi.presentation.sawserver.Host</param-name>
    <param-value>localhost</param-value>
    </init-param>
    <init-param>
    <param-name>oracle.bi.presentation.sawserver.Port</param-name>
    <param-value>9710</param-value>
    </init-param>
    
  4. Add in a new init-param in place of the two you removed, specifying your Presentation Services hosts and ports (syntax is host:port) in a semi-colon delimited list
    <init-param>
    <param-name>oracle.bi.presentation.sawservers</param-name>
    <param-value>BISandbox01:9710;BISandbox02:9710</param-value>
    </init-param>
  5. Save your modified web.xml file
  6. Restart your application server
    • In OAS you can use opmnctl restartproc
  7. Login to Answers and test that it works
  8. Stop one of your Presentation Services (sawserver)
  9. Refresh Answers. You’ll probably get a 500 Internal Server Error.
    • If you check the application.log it shows that it can’t connect to the Presentation Services (because you’ve just stopped it, duh!)
  10. Refresh Answers again in a minute or two. You should get Presentation Services back, but from a different instance.
    • Does anyone know where this period is defined, eg is it a timeout setting, multiple failed connection attempts?
  11. Work through all your Presentation Services servers, stopping and starting the service on each to ensure each is being picked up

How do you know which Presentation Services you’re using?

This is where it can get a bit confusing!

The images that you see rendered on the page are local to the BI Presentation Services Plug-in. So if you muck around with the files in /res you can tag the login page with the server that analytics plugin is running on. If you’re not using web server load balancing then this will always be the web server that you’re connecting to.

The web catalog is specified by the BI Presentation Services instance. Once your clustering is setup then obviously you must share or replicate your web catalog. However whilst setting up the plugin->presentation services connectivity it might be an idea to have separate instances. Set up the default dashboard on login simply to show the Presentation Sevices server name as a text box (hardcode it). Do this for each server. You can go and check the actual Request in the web catalog on each server’s file system to make sure you’re on the right one.

Logfiles

  • BI Presentation Services Plug-in:
    •  $J2EE_home/application-deployments/analytics/home_default_group_1/application.log
    • Also available through OAS’s Enterprise Manager, click Logs link top right and navigate to the analytics Application
  • BI Presentation Services:
    • $OracleBIData/web/log/sawlog0.log

Common errors

500 Internal Server Error

Servlet error: An exception occurred. The current application deployment descriptors do not allow for including it in this response. Please consult the application log for details.

BI Presentation Services Plug-in has thrown an error, and you should check its logfile (see below).

analytics: Servlet error java.net.ConnectException: Connection refused

The BI Presentation Services Plug-in is trying to connect to a Presentation Services and can’t. Either you’ve specified the wrong host or port details in the web.xml, or Presentation Services (sawserver) is not running.

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

This typically means that the BI Presentation Services Plug-in is not running. Check in OAS that the analytics application is started

Bonus – shared config

In researching this I found an interesting point in the 10.1.3.4.1 release notes. You can specify the analytics configuration in a shared config file using the oracle.bi.presentation.sawbridge.configFilePath param-name.

On a clustered setup with shared filesystem you can therefore have one file listing the Presentation Services servers to use, and reference this from each analytics config.

Ref: Configuring Oracle BI EE Using an EAR File

web.xml location

The Deployment Guide p35 states that the web.xml for java servlet is $OracleBI_HOME/web/app/WEB-INF. However, in my experience this should actually be $J2EE_home/applications/analytics/analytics/WEB-INF.

The table on p97 in the Infrastructure Installation and Configuration Guide concurs with this, and shows different locations for web.xml. The difference is whether your installation using IIS or OAS/OC4J.

So for OAS/OC4J web.xml is $J2EE_home/applications/analytics/analytics/WEB-INF, and for IIS’s ISAPI plugin it is $OracleBI_HOME/web/app/WEB-INF

October 16, 2009

Heads up – Critical Patch Update affecting OBIEE

Filed under: bug, OAS, obiee, security — rmoff @ 09:40

The Critical Patch Update Pre-Release Announcement for October has been published. The pre-release is advance notice of the affected software prior to release of the quarterly Critical Patch Update. It is published on the Thursday prior to the patch releases (which was postponed by a week because of OOW).

It looks like if you’re running OBIEE 10.1.3.4.0 or 10.1.3.4.1 through OAS 10.1.2.3.0/10.1.3.4.0/10.1.3.5.0 then you should check back next Tuesday 20th for details.

Paraphrasing the announcement:

Security vulnerabilities addressed by this Critical Patch Update affect the following products:
[…]
• Oracle Application Server 10g Release 3 (10.1.3), versions 10.1.3.4.0, 10.1.3.5.0
• Oracle Application Server 10g Release 2 (10.1.2), version 10.1.2.3.0
[…]
• Oracle Business Intelligence Enterprise Edition, versions 10.1.3.4.0, 10.1.3.4.1
[…]
Oracle Application Server Executive Summary

This Critical Patch Update contains 3 new security fixes for the Oracle Application Server. 2 of these vulnerabilities may be remotely exploitable without authentication, i.e., may be exploited over a network without the need for a username and password. None of these fixes are applicable to client-only installations, i.e., installations that do not have an Oracle Application Server installed.
[…]
The highest CVSS base score of vulnerabilities affecting Oracle Application Server products is 4.3.

The Oracle Application Server components affected by vulnerabilities that are fixed in this Critical Patch Update are:

* Oracle Business Intelligence Enterprise Edition
[…]

More details from the Oracle Critical Patch Updates and Security Alerts page.

[update 21st October]
Details here, patch is for BI Server so presumably the application server is irrelevant

August 6, 2009

Clean install of OAS – Enterprise Manager not available

Filed under: OAS — rmoff @ 08:29

I successfully installed OAS 10.1.3.3 and patched to 10.1.3.4. http://localhost:7777 gave the OAS welcome page, but going to http://localhost:7777/em gave 404 Not Found.

In [OASHome]/j2ee/home/config/servers.xml search for ascontrol, you should get:

<application name="ascontrol" path="../../home/applications/ascontrol.ear" parent="system" start="false" />

change the start attribute to true

<application name="ascontrol" path="../../home/applications/ascontrol.ear" parent="system" start="true" />

Restart OAS ([OAShome]/opmn/bin/opmnctl restartproc) and Enterprise Manager should now be available

July 28, 2009

How to find out what web application server is in use

Filed under: OAS, obiee, sawserver — rmoff @ 12:06

If, for some reason, you need to check what web application server is in use for Presentation Services (as this chap needed to), you can use an add-in for FireFox called HttpFox to inspect the HTTP headers.

1. Install HttpFox (and obviously Firefox if you don’t have it already!)
2. Open the HttpFox window (Tools -> HttpFox -> Toggle HttpFox)
3. Click the Start button in the HttpFox window
4. Navigate to your OBIEE home page
5. Click the Stop button in the HttpFox window
6. Click on the first entry in the list, URL should be http://yourserver:7777/analytics/saw.dll?Dashboard
7. In the right-hand pane of the Headers tab you should see Server listed. In this instance, it’s Oracle-Application-Server-10g/10.1.3.1.0 Oracle-HTTP-Server

May 18, 2009

Custom HTTP error page in OBIEE / OAS

Filed under: Apache, OAS, obiee — rmoff @ 13:52

It’s possible to change the error pages served up by OAS/Apache by using the ErrorDocument directive. This is widely documented.

However, to get this to take effect in an oc4j application (such as analytics) you need to change mod_oc4j.conf too.
(I found this out from this post here)

Take backups of httpd.conf and mod_oc4j.conf, and then edit them as follows:

In httpd.conf add:
ErrorDocument 500 /500.html
where /500.html is a relative path to your custom document

In mod_oc4j.conf add to the end of the file:
Oc4jUseOHSErrors on

This will make any HTTP 500 (Internal Server error) errors show the page 500.html, instead of the default Apache one.

Don’t forget to bounce OHS after making this change:
opmnctl restartproc ias-component=HTTP_Server

April 15, 2009

OBIEE and F5 BIG-IP

Filed under: Apache, load balancing, OAS, obiee — rmoff @ 13:44

We’ve got a setup of two OAS/Presentation Services boxes and two BI Server boxes, with load balancing/failover throughout.
The Load Balancing of the web requests is being done through a separate bit of kit, an F5 BIG-IP load balancer. This directs the requests at the two OAS servers.

The problem we have is that by default OAS serves HTTP on port 7777, but the F5 is using port 80. A request for our load balanced URL: http://bi.mycompany.com/analytics/ barfs out with

Internet Explorer cannot display the webpage

Most likely causes:
-You are not connected to the Internet.
-The website is encountering problems.
-There might be a typing error in the address.

or in FireFox:

Failed to Connect The connection was refused when attempting to contact bi.mycompany.com:7777. Though the site seems valid, the browser was unable to establish a connection.

Using the excellent HttpFox add-in for Firefox I could see the HTTP requests/responses:

  1. http://bi.mycompany.com/analytics/ goes via the loadbalancer on the default HTTP port 80 to OAS
  2. OAS responds with HTTP/1.1 302 Moved Temporarily to http://bi.mycompany.com:7777/analytics/saw.dll?Dashboard
  3. The web client requests this URL (http://bi.mycompany.com:7777/analytics/saw.dll?Dashboard) from the load balancer but because it’s port 7777 F5 rejects the request (NS_ERROR_CONNECTION_REFUSED)

We could also just use the direct URL http://bi.mycompany.com/analytics/saw.dll?Dashboard but this is hardly user friendly (and also means that if they typo when entering it they’ll get an unhelpful error as above)

Looking at the httpd.conf for Apache to find the port config made me think of the UseCanonicalName setting which I also encountered recently. This setting is to do with how Apache deals with the server name in the URL being requested and the hostname of the server configured in Apache.
When I got the behaviour described above UseCanonicalName was set to Off, which I think means Apache does not rewrite the URL at all, so the redirect was to http://bi.mycompany.com:7777/analytics/saw.dll?Dashboard which is the F5 Load Balancer address.
If I changed UseCanonicalName to On then the F5 load balancing starts to work, as this happens instead:

  1. http://bi.mycompany.com/analytics/ goes via the loadbalancer on the default HTTP port 80 to OAS
  2. OAS responds with HTTP/1.1 302 Moved Temporarily to http://oasserver_1.mycompany.com:7777/analytics/saw.dll?Dashboard

i.e. the request goes directly to one of the load balanced servers, and correctly on port 7777.
The disadvantage of this is that the URL used by the web client then becomes http://oasserver_1.mycompany.com which means the user is no longer hitting the load balancer so any failover wouldn’t get picked up. It also means that users might start bookmarking OAS servers directly instead of the load balancer, again meaning that they don’t hit the load balancer so a server failover wouldn’t get picked up.

Eventually I got this resolved, with a bit of help from a very helpful chap at Oracle. By changing the httpd.conf to set Port 80, when Apache rewrites URLs it now uses Port 80.
Listen remains as 7777.
Traffic from web client now hits the LB on port 80, which forwards to 7777 on one of the OAS servers, which if necessary rewrite the URL and use port 80 in the rewrite.
Because Listen remains as 7777 there is no need to run Apache as root.
You can also set ServerName to the load balancer address (bi.mycompany.com) and UseCanonicalName to On. If you do this then I don’t think it’s possible to access web pages on a specific OAS server (eg oasserver_1) because entering http://oasserver_1.mycompany.com:7777/analytics just redirects to bi.mycompany.com/analytics.

Ref: Deploying F5 with Oracle Application Server 10g
Ref: Oracle HTTP Server – Port setting
Ref: Metalink 301755.1 – What Is the Difference Between Port & Listen In Httpd.Conf

April 2, 2009

OAS makes you log in twice

Filed under: Apache, OAS — rmoff @ 12:27

A very minor irritation, but an irritation nonetheless, is when I go to Application Server Control in OAS I have to login twice.

Reading around I found this is an Apache feature, and is actually designed behaviour.

For reasons I’ve not explored our servers have several different hostnames which resolve to the same IP, e.g.:
myserver
myserver-app
myserver-data

When you request a page from Apache using a hostname other than that configured as ServerName in Apache’s httpd.conf, it redirects you to the version of the page using the ServerName.

If I go to http://myserver.company.fqdn.net:7777/em/ I get the login page as expected. Having typed the password I get sent to the login page again – but this time at http://myserver-app.company.fqdn.net:7777/em/, and this ties in with httpd.conf in which ServerName is myserver-app.

Per the Apache FAQ this can be resolved in several ways. ServerName can either be altered (not sure if this would impact other functions), or change UseCanonicalName from On to Off. Obviously you could also just login at a URL which corresponded with the ServerName in httpd.conf 🙂

Ref: Apache UseCanonicalName documentation

After changing httpd.conf, don’t forget to bounce Apache:

opmnctl restartproc ias-component=HTTP_Server
Older Posts »

Blog at WordPress.com.