Jeremy Chadwick wrote:

I remember someone on the -fs list explained that there's
currently a hard limit for kmem at 2 GB minus epsilon.
So I suggest trying to set it to slightly less than 2 GB.

And yes, I agree that is unfortunate, because there are
high ZFS workloads where more than 2 GB kmem would be
useful. I hope this design limitation can be alleviated
somehow in the future.

Best regards
Oliver

Gerrit Kühn wrote:

That rings a bell ...

I remember reports of riser cards that apparently changed
the timing on the PCI bus so they were only marginally
compliant with the spec, or maybe not even that anymore.

If you try to remove the controller, please also remove
the riser card. It could well be that it's causing
problems, especially if it's on the same PCI bus as the
onboard re(4) interfaces.

Since getting my cable upgraded to 24Mbit down and 10Mbit up and
getting excited with torrents, I've started experiencing weird errors
with my 3Com NIC whenever it's being used at high speeds. It'll
basically work fine for a while until it suddenly stops sending any
packets and the connection to the switch is dropped, combined with
this error message:

xl1: transmission error: 90
xl1: tx underrun, increasing tx start threshold to 120 bytes

I can "fix" it by running ifconfig xl1 down; ifconfig xl1 up, but then
it'll bork out again later on when it feels the traffic's too
congested (which happens around 2MB/s torrent downloads, I think).

The machine in question is FreeBSD 6.0-STABLE and works as a router,
with two 3Coms, like so:

xl0: <3Com 3c905B-TX Fast Etherlink XL> port 0x1000-0x107f mem
0xe8000000-0xe800007f irq 11 at device 14.0 on pci0
xl1: <3Com 3c905-TX Fast Etherlink XL> port 0x1080-0x10bf irq 10 at
device 15.0 on pci0

On Thu, 29 May 2008, Robert Blayzor wrote:

As has been pointed out on wikis, mailing lists, and even on IRC, ZFS
requires a bit of tuning -- specifically in regards to vm.kmem_size and
vm.kmem_size_max. The opinion is: ZFS is memory-hungry.

On my home RELENG_7 amd64 box (2GB RAM), I could panic the system with
heavy I/O due to kmem_size being too small, until I used the following
values:

vm.kmem_size="1536M"
vm.kmem_size_max="1536M"

I decided to upgrade the box to 4GB of RAM, since I was worried about
memory exhaustion under even higher loads (during heavy I/O with ZFS,
I'd often see the "Wired" value in top reach 1.3-1.4GB).

I received the RAM today, installed it, works fine. I then chose to
adjust the vm.kmem_size and kmem_size_max settings to something larger,
which seemed like the logical choice. I went with:

vm.kmem_size="3584M"
vm.kmem_size_max="3584M"

Upon reboot, the kernel immediately panic'd with the following message:

kmem_suballoc(): bad status return of 3.

I then chose smaller values (going with 2048M); same panic.

Gerrit Kühn wrote:

In that case I would suspect that the one piece of hardware
that is misbehaving is broken and needs to be replaced.

I think it's unlikely. Do they share interrupts? (The
output of "vmstat -i" will tell you.)

In theory it could also be a power supply problem. I
assume that you use rather small (thus possibly weak)
power supplies for your ITX machines. Maybe the SATA
controller in that problematic machine drives the power
supply to its limit, and the re(4) interfaces suffer.
You could check whether removing the SATA controller
improves things. Or try to connect a stronger power
supply if you have one available.

I got really poor performance when I try to upload files to the box (via
pureFTP or Samba) when using jumbo frames somewhat above 2k.

uname -a :
FreeBSD carenath.tzim.net 7.0-STABLE FreeBSD 7.0-STABLE #4: Wed May 28
17:45:14 CEST 2008
tzim@carenath.tzim.net:/usr/obj/usr/src/sys/CARENATH amd64

It seems that TCP acks are delayed for about a second, so the upload
rate nevers goes above 60kB/s. I can provide wireshark captures done on
the box uploading the files if necessary.
The client is a Windows XP64 box, but I do not recall having the same
problem on 6.2. I did got about 100 MB/s for similar file transfers, but
I had to disable jumbo frames when I upgraded to 7.0-Release because of
some bug on txcsum/rxcsum on re(4) so I only noticed the issue recently.

The txcsum bug doesn't seem to occur anymore, but now I have this
strange rate issue, with or without checksum offloading disabled.
I also tried with tso disabled. Same results. Is it related to the re(4)
driver ? Or to the TCP stack ?

Arnaud Houdelette

I have a rather busy Apache 2.2 server; tons of small & some large
requests. It's a standard Dell 2650 server using the bge (broadcom)
network driver.

I seem to have a rather strange problem where after just a day or so
Apache just stops processing new connections. You can connect to port
80, but trying to get Apache to process any data just hangs. There is
nothing strange in dmesg or in /var/log/messages.

The server has plenty free available physical RAM, swap is untouched,
CPU load is low, etc. Apache is setup to handle a max of 100 clients
using prefork model.

If I stop and restart Apache, it does not help.

What I do notice is 1000's of sockets stuck in "FIN_WAIT_1" in netstat:

[web0:~] netstat -an | grep FIN_WAIT | wc -l
1827

Hello,
i have configured lagg interface on two Broacom (bce0 bce1).

I have tried with laggproto lacp (supported by the Nortel switch), with fce
and failover, but they all shows the same symptom:

Everything works fine until i unplug the cable of the first interface
(bce0), it will
show status: no carrier even on lagg0 interface, while bce0 shows no
carrier (correct) and
bce1 is active.

Any ideas?

Thanks!

Daniel

--

WBR,
Cordiali Saluti,

Daniel Ponticello, VP of Engineering

Network Coordination Centre of Skytek

On Wednesday 28 May 2008 01:15:18 Boris Samorodov wrote: