Apache | rmoff

January 21, 2010

Hardening OAS

Filed under: Apache, OAS, security — rmoff @ 11:18

Oracle Application Server (OAS) is the Web and Application server typically deployed with OBIEE. There are several settings which by default may be viewed as security weaknesses. Whether realistically a target or not, it’s good practice to always be considering security and lock down your servers as much as reasonably possible. I adopt the default stance of having to find a reason to leave something less secure, rather than justify why it needs doing.

There are various tools and companies out there that will help you scan your deployments for weaknesses. In reading about this I found Nikto which runs on all platforms. In essence it takes a hostname and port and scans for known vulnerabilities in web servers (not just OAS).

Listed below are some of the simple things you can do to secure your default deployment of OAS.

Almost all of this is derived from the very excellent Securing Oracle Application Server by Caleb Sima

In the text below I refer to $OAS_HOME which may not be an actual environment variable, but is the home directory of your OAS installation.

Don’t forget to backup config files before you change them, and take backups of deleted files and directories.

After making the changes bounce OAS (opmnctl stopall; opmnctl startall).

As well as the specifics below you should always keep an eye on Oracle’s Critical Patch Updates.

Web server version and details

By default OAS will report its version in both HTTP headers and on error pages (such as those returned on a 404 Not Found which is easy to obtain by entering a non-existent URL):

Apply these two changes to $OAS_HOME/Apache/Apache/conf/httpd.conf:

Search for ServerSignature and change it from On to Off
This removes the server information from error pages
Ref: http://httpd.apache.org/docs/2.2/mod/core.html#serversignature
Add this on the next line:
ServerTokens ProductOnly

This removes some server version info from the HTTP header, and is the least possible data to reveal in Apache.
Ref: http://httpd.apache.org/docs/2.2/mod/core.html#servertokens

After the changes have been made:

TRACE method

Read Apache Tips: Disable the HTTP TRACE method/ for information on how to see if HTTP TRACE is enabled. It is by default in OAS, and most security scanners will pick it up as a problem.

To disable it, add to $OAS_HOME/Apache/Apache/conf/httpd.conf:
TraceEnable Off

Default content

Most web and application servers come with default content such as example pages or “Welcome” pages, and OAS is no exception.
The reason for getting rid of this content is to give potential attackers one less thing to work with. Static content might give them information about software versions or paths. Dynamic content (JSPs etc) may be exploitable. Either way – what is to be gained from leaving it in place?

Apache default content

In $OAS_HOME/Apache/Apache:

mv htdocs/ htdocs.old
mkdir htdocs
vi htdocs/index.html
# enter:
<HTML><HEAD><TITLE>Nothing to see here</TITLE></HEAD><BODY>Nothing to see here, move along.</BODY></HTML>

rm $OAS_Home/Apache/Apache/icons/README
rm $OAS_Home/Apache/Apache/fcgi-bin/*

j2ee

cd $OAS_HOME/j2ee/home/default-web-app
rm -r WEB-INF/classes
rm -r examples/
echo "Nothing to see here" > index.html

Pre-populated username in OAS login form

This could help an attacker as they are given a username to start trying to login as.

However, I can’t work out how to disable it. I opened a thread on OTN here:
http://forums.oracle.com/forums/thread.jspa?threadID=1010227&tstart=0

If you know, please leave a comment!

Weak ciphers / SSL version 2 supported

Disable the weak SSL ciphers & disable version 2 of the protocol

Add to httpd.conf after the TraceEnable statement from above:

SSLProtocol ALL -SSLv2
SSLCipherSuite HIGH:!SSLv2:!ADH:!aNULL:!eNULL:!NULL

Ref: http://httpd.apache.org/docs/2.0/mod/mod_ssl.html#sslciphersuite

Ref: http://adamyoung.net/Disable-SSLv2-System-Wide

Restarting OAS

When I started implementing this I used opmnctl restartproc, but found that HTTP_Server came back as status “Stop” or “Bounce”. I also got errors like: “time out while waiting for a managed process to restart”.

What I think happened was that the httpd (Apache) processes didn’t come down properly, and so couldn’t restart.

Therefore I resorted to opmnctl shutdown, then search for any remaining httpd processes (ps -ef|grep httpd) and kill any (kill -9 xxxx), and then restart OAS (opmnctl startall)

Comments Off

August 14, 2009

Unix script to report on OBIEE and OBIA processes state

Filed under: Apache, cluster, dac, obia, obiee, sawserver, unix — rmoff @ 07:22

Here’s a set of scripts that I use on our servers as a quick way to check if the various BI components are up and running.

areservicesrunning

Because we split the stack across servers, there are different scripts called in combination. On our dev boxes we have everything and so the script calls all three sub-scripts, whereas on Production each server will run one of:

BI Server
Presentation Server & OAS
Informatica & DAC

The scripts source another script called process_check.sh which I based on the common.sh script that comes with OBIEE.

The BI Server script includes logic to only check for the cluster controller if it’s running on a known clustered machine. This is because in our development environment we don’t cluster the BI Server.

Each script details where the log files and config files can be found, obviously for your installation these will vary. I should have used variables for these, but hey, what’s a hacky script if not imperfect 🙂

The script was written and tested on HP-UX.

Installation

Copy each of these onto your server in the same folder.

You might need to add that folder to your PATH.

Edit are_processes_running.sh so that it calls the appropriate scripts for the components you have installed.

You shouldn’t need to edit any of the other scripts except to update log and config paths.

The scripts

are_processes_running.sh

# are_processes_running.sh
# RNM 2009-04-21
# https://rnm1978.wordpress.com

clear
echo "=-=-=-=-=-=-=-=-=-=-=- "
echo " "

# Comment out the scripts that are not required
# For example if there is no ETL on this server then only
# run the first two scripts
_are_BI_processes_running.sh
_are_PS_processes_running.sh
_are_INF_processes_running.sh

echo " "
echo "=-=-=-=-=-=-=-=-=-=-=- "

_are_BI_processes_running.sh

# _are_BI_processes_running.sh
# RNM 2009-04-21
# https://rnm1978.wordpress.com

. process_check.sh

########## BI Server #################
echo "====="
if [ "$(is_process_running nqsserver)" = yes ]; then
  tput bold
  echo "nqsserver (BI Server) is running"
  tput rmso
else
  tput rev
  echo "nqsserver (BI Server) is not running"
  tput rmso
  echo "  To start it enter:"
  echo "    run-sa.sh start64"
fi
echo "  Log files:"
echo "    tail -n 50 -f /app/oracle/product/obiee/server/Log/NQServer.log"
echo "    tail -n 50 -f /app/oracle/product/obiee/server/Log/nqsserver.out.log"
echo "    tail -n 50 -f /app/oracle/product/obiee/server/Log/NQQuery.log"
echo "  Config file:"
echo "    view /app/oracle/product/obiee/server/Config/NQSConfig.INI"

echo "====="
if [ "$(is_process_running nqscheduler)" = yes ]; then
  tput bold
  echo "nqscheduler (BI Scheduler) is running"
  tput rmso
else
  tput rev
  echo "nqscheduler (BI Scheduler) is not running"
  tput rmso
  echo "  To start it enter:"
  echo "    run-sch.sh start64"
fi
echo "  Log files:"
echo "    tail -n 50 -f /app/oracle/product/obiee/server/Log/NQScheduler.log"
echo "    tail -n 50 -f /app/oracle/product/obiee/server/Log/nqscheduler.out.log"
echo "    ls -l /app/oracle/product/obiee/server/Log/iBots/"
echo "  Config file:"
echo "    view /data/bi/scheduler/config/instanceconfig.xml"

echo "====="
echo "$hostname"
if [ "$(hostname)" = "BICluster1" -o "$(hostname)" = "BICluster2" ]; then
  if [ "$(is_process_running nqsclustercontroller)" = yes ]; then
    tput bold
    echo "BI Cluster Controller is running"
    tput rmso
  else
    tput rev
    echo "BI Cluster Controller is not running"
    tput rmso
    echo "  To start it enter:"
    echo "    run-ccs.sh start64"
  fi
    echo "  Log files:"
  echo "    tail -n 50 -f /app/oracle/product/obiee/server/Log/NQCluster.log"
  echo "    tail -n 50 -f /app/oracle/product/obiee/server/Log/nqsclustercontroller.out.log"
  echo "  Config file:"
  echo "    view /app/oracle/product/obiee/server/Config/NQClusterConfig.INI"
else
  echo "(Not checked for Cluster Controller because not running on BICluster1 or BICluster2)"
fi

_are_PS_processes_running.sh

# _are_PS_processes_running.sh
# RNM 2009-04-21
# https://rnm1978.wordpress.com

. process_check.sh

########## OAS  #################
echo "====="
if [ "$(is_process_running httpd)" = yes ]; then
  tput bold
  echo "Apache (HTTP server) is running"
  tput rmso
else
  tput rev
  echo "Apache (HTTP server) is not running"
  tput rmso
  echo "  It should have been started as part of OAS. Check that opmn (Oracle Process Manager and Notification) is running"
  echo "  If opmn is running then run this command to check the status of the components:"
  echo "    opmnctl status -l"
  echo "  If opmn is not running then start it with this command:"
  echo "    opmnctl startall"
fi
echo "  Log files:"
echo "    ls -lrt /app/oracle/product/OAS_1013/Apache/Apache/logs"
echo "  Config file:"
echo "    view /app/oracle/product/OAS_1013/Apache/Apache/conf/httpd.conf"

echo "====="
if [ "$(is_process_running opmn)" = yes ]; then
  tput bold
  echo "opmn (OAS - Oracle Process Manager and Notification) is running"
  tput rmso
else
  tput rev
  echo "opmn (OAS - Oracle Process Manager and Notification) is not running"
  tput rmso
  echo "  To start it use this command:"
  echo "    opmnctl startall"
fi
echo "  Log files:"
echo "    ls -lrt /app/oracle/product/OAS_1013/opmn/logs"
echo "    ls -lrt /app/oracle/product/OAS_1013/j2ee/home/log"
echo "  Config file:"
echo "    view /app/oracle/product/OAS_1013/opmn/conf/opmn.xml"
echo "    view /app/oracle/product/OAS_1013/j2ee/home/config/server.xml"

########## Presentation Services #################
echo "====="
if [ "$(is_process_running javahost)" = yes ]; then
  tput bold
  echo "javahost is running"
  tput rmso
else
  tput rev
  echo "javahost is not running"
  tput rmso
  echo "  It is started as part of the sawserver startup script"
  echo "  To start it run this command:"
  echo "    run-saw.sh start64"
    echo "  To start it independently run this command:"
  echo "    /app/oracle/product/obiee/web/javahost/bin/run.sh"
  fi
echo "  Log files:"
echo "    ls -lrt /data/web/web/log/javahost/"
echo "  Config file:"
echo "    view /app/oracle/product/obiee/web/javahost/config/config.xml"

echo "====="
if [ "$(is_process_running sawserver)" = yes ]; then
  tput bold
  echo "sawserver (Presentation Services) is running"
  tput rmso
else
  tput rev
  echo "sawserver (Presentation Services) is not running"
  tput rmso
  echo "  To start it enter:"
  echo "    run-saw.sh start64"
fi
echo "  Log files:"
echo "    tail -n 50 -f /data/web/web/log/sawserver.out.log"
echo "    tail -n 50 -f /data/web/web/log/sawlog0.log"

echo "  Config file:"
echo "    view /data/web/web/config/instanceconfig.xml"
echo "    ls -l /data/web/web/config/"

_are_INF_processes_running.sh

# _are_INF_processes_running.sh
# RNM 2009-04-22
# https://rnm1978.wordpress.com

. process_check.sh

########## Informatica #################
echo "====="
inf_running=1
if [ "$(is_process_running server/bin/pmrepagent)" = yes ]; then
  tput bold
  echo "pmrepagent (Informatica Repository Server) is running"
  tput rmso
else
  tput rev
  echo "pmrepagent (Informatica Repository Server) is not running"
  tput rmso
  inf_running=0
fi
if [ "$(is_process_running server/bin/pmserver)" = yes ]; then
  tput bold
  echo "pmserver (Informatica Server) is running"
  tput rmso
else
  tput rev
  echo "pmserver (Informatica Server) is not running"
  tput rmso
  inf_running=0
fi
if [ "$inf_running" -eq 0 ]; then
  echo " "
  echo "  To start PowerCenter:"
  echo "    cd /app/oracle/product/informatica/server/tomcat/bin"
  echo "    infaservice.sh startup"
fi
echo " "
echo "  Log files (PowerCenter):"
echo "    ls -lrt /app/oracle/product/informatica/server/tomcat/logs"
echo " "
echo "  Log files (ETL jobs):"
echo "    ls -lrt /app/oracle/product/informatica/server/infa_shared/SessLogs"
echo "    ls -lrt /app/oracle/product/informatica/server/infa_shared/WorkflowLogs"

########## DAC #################

echo "====="
if [ "$(is_process_running com.siebel.etl.net.QServer)" = yes ]; then
  tput bold
  echo "DAC is running"
  tput rmso
else
  tput rev
  echo "DAC is not running"
  tput rmso
  echo " "
  echo "  To start the DAC server:"
  echo "    cd /app/oracle/product/informatica/DAC_Server/"
  echo "    nohup startserver.sh &"
  echo " "
fi
echo "  Log files:"
echo "    ls -lrt /app/oracle/product/informatica/DAC_Server/log"

process_check.sh

</pre>
# process_check.sh
# get_pid plagiarised from OBIEE common.sh
# RNM 2009-04-03
# RNM 2009-04-30 Exclude root processes (getting false positive from OpenView polling with process name)

get_pid ()
{
 echo `ps -ef| grep $1 | grep -v grep | grep -v "    root " | awk '{print $1}'` # the second grep excludes the grep process from matching itself, the third one is a hacky way to avoid root processes
}

is_process_running ()
{
process=$1
#echo $process
procid=`get_pid $process`
#echo $procid
if test "$procid" ; then
 echo "yes"
else
 echo "no"
fi
}

Comments Off

May 18, 2009

Custom HTTP error page in OBIEE / OAS

Filed under: Apache, OAS, obiee — rmoff @ 13:52

It’s possible to change the error pages served up by OAS/Apache by using the ErrorDocument directive. This is widely documented.

However, to get this to take effect in an oc4j application (such as analytics) you need to change mod_oc4j.conf too.
(I found this out from this post here)

Take backups of httpd.conf and mod_oc4j.conf, and then edit them as follows:

In httpd.conf add:
ErrorDocument 500 /500.html
where /500.html is a relative path to your custom document

In mod_oc4j.conf add to the end of the file:
Oc4jUseOHSErrors on

This will make any HTTP 500 (Internal Server error) errors show the page 500.html, instead of the default Apache one.

Don’t forget to bounce OHS after making this change:
opmnctl restartproc ias-component=HTTP_Server

Comments Off

April 15, 2009

OBIEE and F5 BIG-IP

Filed under: Apache, load balancing, OAS, obiee — rmoff @ 13:44

We’ve got a setup of two OAS/Presentation Services boxes and two BI Server boxes, with load balancing/failover throughout.
The Load Balancing of the web requests is being done through a separate bit of kit, an F5 BIG-IP load balancer. This directs the requests at the two OAS servers.

The problem we have is that by default OAS serves HTTP on port 7777, but the F5 is using port 80. A request for our load balanced URL: http://bi.mycompany.com/analytics/ barfs out with

Internet Explorer cannot display the webpage

Most likely causes:
-You are not connected to the Internet.
-The website is encountering problems.
-There might be a typing error in the address.

or in FireFox:

Failed to Connect The connection was refused when attempting to contact bi.mycompany.com:7777. Though the site seems valid, the browser was unable to establish a connection.

Using the excellent HttpFox add-in for Firefox I could see the HTTP requests/responses:

http://bi.mycompany.com/analytics/ goes via the loadbalancer on the default HTTP port 80 to OAS
OAS responds with HTTP/1.1 302 Moved Temporarily to http://bi.mycompany.com:7777/analytics/saw.dll?Dashboard
The web client requests this URL (http://bi.mycompany.com:7777/analytics/saw.dll?Dashboard) from the load balancer but because it’s port 7777 F5 rejects the request (NS_ERROR_CONNECTION_REFUSED)

We could also just use the direct URL http://bi.mycompany.com/analytics/saw.dll?Dashboard but this is hardly user friendly (and also means that if they typo when entering it they’ll get an unhelpful error as above)

Looking at the httpd.conf for Apache to find the port config made me think of the UseCanonicalName setting which I also encountered recently. This setting is to do with how Apache deals with the server name in the URL being requested and the hostname of the server configured in Apache.
When I got the behaviour described above UseCanonicalName was set to Off, which I think means Apache does not rewrite the URL at all, so the redirect was to http://bi.mycompany.com:7777/analytics/saw.dll?Dashboard which is the F5 Load Balancer address.
If I changed UseCanonicalName to On then the F5 load balancing starts to work, as this happens instead:

http://bi.mycompany.com/analytics/ goes via the loadbalancer on the default HTTP port 80 to OAS
OAS responds with HTTP/1.1 302 Moved Temporarily to http://oasserver_1.mycompany.com:7777/analytics/saw.dll?Dashboard

i.e. the request goes directly to one of the load balanced servers, and correctly on port 7777.
The disadvantage of this is that the URL used by the web client then becomes http://oasserver_1.mycompany.com which means the user is no longer hitting the load balancer so any failover wouldn’t get picked up. It also means that users might start bookmarking OAS servers directly instead of the load balancer, again meaning that they don’t hit the load balancer so a server failover wouldn’t get picked up.

Eventually I got this resolved, with a bit of help from a very helpful chap at Oracle. By changing the httpd.conf to set Port 80, when Apache rewrites URLs it now uses Port 80.
Listen remains as 7777.
Traffic from web client now hits the LB on port 80, which forwards to 7777 on one of the OAS servers, which if necessary rewrite the URL and use port 80 in the rewrite.
Because Listen remains as 7777 there is no need to run Apache as root.
You can also set ServerName to the load balancer address (bi.mycompany.com) and UseCanonicalName to On. If you do this then I don’t think it’s possible to access web pages on a specific OAS server (eg oasserver_1) because entering http://oasserver_1.mycompany.com:7777/analytics just redirects to bi.mycompany.com/analytics.

Ref: Deploying F5 with Oracle Application Server 10g
Ref: Oracle HTTP Server – Port setting
Ref: Metalink 301755.1 – What Is the Difference Between Port & Listen In Httpd.Conf

Comments Off

April 2, 2009

OAS makes you log in twice

Filed under: Apache, OAS — rmoff @ 12:27

A very minor irritation, but an irritation nonetheless, is when I go to Application Server Control in OAS I have to login twice.

Reading around I found this is an Apache feature, and is actually designed behaviour.

For reasons I’ve not explored our servers have several different hostnames which resolve to the same IP, e.g.:
myserver
myserver-app
myserver-data

When you request a page from Apache using a hostname other than that configured as ServerName in Apache’s httpd.conf, it redirects you to the version of the page using the ServerName.

If I go to http://myserver.company.fqdn.net:7777/em/ I get the login page as expected. Having typed the password I get sent to the login page again – but this time at http://myserver-app.company.fqdn.net:7777/em/, and this ties in with httpd.conf in which ServerName is myserver-app.

Per the Apache FAQ this can be resolved in several ways. ServerName can either be altered (not sure if this would impact other functions), or change UseCanonicalName from On to Off. Obviously you could also just login at a URL which corresponded with the ServerName in httpd.conf 🙂

Ref: Apache UseCanonicalName documentation

After changing httpd.conf, don’t forget to bounce Apache:

opmnctl restartproc ias-component=HTTP_Server

Comments (1)

rmoff

January 21, 2010

Hardening OAS

Web server version and details

TRACE method

Default content

Apache default content

j2ee

Pre-populated username in OAS login form

Weak ciphers / SSL version 2 supported

Restarting OAS

August 14, 2009

Unix script to report on OBIEE and OBIA processes state

Installation

The scripts

are_processes_running.sh

_are_BI_processes_running.sh

_are_PS_processes_running.sh

_are_INF_processes_running.sh

process_check.sh

May 18, 2009

Custom HTTP error page in OBIEE / OAS

April 15, 2009

OBIEE and F5 BIG-IP

April 2, 2009

OAS makes you log in twice

For more up-to-date writings…

Search blog

Follow me on twitter!

Recent Posts

Top Posts

Latest Tunes

Archives