Hi,

Problem is in Solaris environment. Application works fine in Windows environment with connection pool size of 10. But in Solaris even pool size set to 100, this error occurs.

WebSphere Version: 6.1.0.15

OS : Solaris 10

Database: Oracle 10g

J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource

com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 180004

at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1499)

at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2287)

at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:896)

at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:596 )

at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java: 439)

at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java: 408)

....................................

Anybody else have this problem? Any solutions? Suggestions?

Thank You.

This is even more exhaustive:
http://www.redbooks.ibm.com/redpapers/pdfs/redp4080.pdf.

The error is that despite waiting 3 minutes the pool manager couldn't
allocate a connection because all are in use.

As such, OS won't make a difference.

So, there is something weird and this should be helpful:
http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21217062.

You could also take a thread dump (kill -3 PID) and see what the threads are
doing (using the Thread Analyzer).

acherian@bottomline.com wrote:

Database side is already checked. DB is allowing more connections and connection pool has those connection. But all of the connections in the pool is in use. When pool size reaches maximum application fails.

I have seen the documents you mentioned. Using that information I was able to find the problem. But I couldn't find a solution.

So, what you know at this stage is that all connections are busy when
running on a given server but not on another.

What is a very possible cause is that there is some sort of misconfiguration
or unexpected condition when running on the Solaris 10 box.

This causes uncaught exception and the application runs out of the "normal"
code path and doesn't close the connection correctly (typically, no finally
clause or no connection.close() in the finally clause). This results in a
connection leak.

When on the "good" server, there is no exception and thus no leak.

Code review is the way to go. I would also review a thread dump to see
what's happening in the application.

acherian@bottomline.com wrote:

What happens when you increase the number of max connections in the pool?
One thing you might look at in TPV or another tool is how long each
connection is in use, there may be a programming issue here.

Ken

acherian@bottomline.com wrote:

I'm not sure if you mean the stack trace you already posted or something else.
That stack trace is just the symptoms, not the problem, right?

I mean, were you able to determine from the thread dump where all the threads
were that were currently holding the connections from the pool? Although, I'd
think that 100 connections would be larger than your WAS thread pool... (just
thinking out loud)

A few thoughts. If you're not using a pure-java JDBC driver, I could see the
Windows and Solaris drivers behaving differently. i.e. one of them having a bug.

Also, are your Windows and Solaris machines in the same network? We've seen
problems with Oracle connection pools connecting through a firewall, although
the symptom we've seen was a bit different.

Have you enabled the PMI monitoring of the connection pool and looked at the
trends?

Doug,

I mean, as far as my application is concerned the ConnectionWaitTimeoutException is the problem. But you are right; it is just the symptoms. I am assuming the problem is somewhere in WebSphere.

The JDBC drivers in use came with Oracle install. Both Windows and Solaris machines in the same network.

From the traces of Connection Manager, I can see connections are in use for more than 100 seconds.

Alex