Hi,

I use Berkeley-DB to manage more than 250 dbs and every db has the
same structure (16 bytes key and 8 bytes data) and almost the same
acount of records (about 500,000). These dbs are opened in the same
environment with transaction enabled. When initialization, I have 4
threads to retrieve data from them through cursor simultaneously but
no two thread will operate on the same db (thread 1 reads db 0,
4, ...; thread 2 reads db 1, 5, ...; etc.). After a while, the cursor
read process becomes very slow, it need more than 20 seconds to finish
the read from a db.

My cache size is 4G and I use the default page size which is 4K.
Following is part of my DB_CONFIG:

set_cachesize 4 0 4
mutex_set_max 1572864
set_lk_max_lockers 4096
set_lk_max_objects 2048
set_lk_max_locks 8388608

Hello,

I don't know valgrind either, but I know quite a bit about BDB. You
might want to ask
your question on the Oracle forums for BDB, in the meantime, here are
some thoughts.

You say 'After a while, the cursor read process becomes very slow'. I
guess that means
it is running fast until then? It sounds like you are doing all read
operations during this time?
If not, there may be some write operations holding locks - possible
contention. If you
are doing writing, are you using transactions?

Hi Don,

Thanks for your help! It's during the initialization period of my
program so there is no writing operations. I put detailed answers
below.

On 8 31 , 2 58 , Don Anderson gmail.com> wrote:

It becomes slow very quickly, after reading couple of dbs.
Yes, all read operations.

Hi,

I've got another better profiling result:

436,042,669,130 * ???:__dbc_get_pp [libdb-4.6.so]
3,340,463,958 > ???:__dbt_userfree (128479383x) [libdb-4.6.so]
412,916,377,218 > ???:__dbc_get (128479383x) [libdb-4.6.so]
1,430 > ???:_dl_runtime_resolve (2x) [/lib/ld-2.3.6.so]
11,434,668,171 > ???:__dbc_get_arg (128479383x) [libdb-4.6.so]

412,916,377,218 < ???:__dbc_get_pp (128479383x) [libdb-4.6.so]
412,916,377,218 * ???:__dbc_get [libdb-4.6.so]
142,030,630,667 > ???:__dbc_cleanup (128479383x) [libdb-4.6.so]
153,069,782,408 > ???:__bamc_get (128479380x) [libdb-4.6.so]
33,918,583,319 > ???:__db_ret (256958252x) [libdb-4.6.so]
58,972,371,752 > ???:__dbc_idup (128479383x) [libdb-4.6.so]
1,379 > ???:_dl_runtime_resolve (2x) [/lib/ld-2.3.6.so]
22,554 > ???:__hamc_get (3x) [libdb-4.6.so]

153,069,782,408 < ???:__dbc_get (128479380x) [libdb-4.6.so]
153,069,782,408 * ???:__bamc_get [libdb-4.6.so]
143,816,284,749 > ???:__bamc_next (128479124x) [libdb-4.6.so]
2,987,931 > ???:__bamc_search (256x) [libdb-4.6.so]

Very strange! I simply read through the data file (use std::ifstream)
before I retieve data through cursor, it becomes very fast! Seems it
depends on the file system's cache heavily!

Hello Rabbit,

Some comments inline.

On Aug 30, 10:12 pm, Rabbit gmail.com> wrote:

Hi Don,

Thank you very much again! ^_^

On 9 1 , 11 31 , Don Anderson gmail.com> wrote: