INFODOC ID: 11755 SYNOPSIS: Out of mbufs: How to determine the souce of the leak DETAIL DESCRIPTION: The following procedure explains how to determine if an "out of mbuf" message on a 4.x system is resulting from a kernel memory leak or an application memory leak. 1. Boot to single user mode. 2. Run netstat -m : check the mbufs in use. This will give an indication of how many mbufs the system uses for the kernel (should be around 5 %) 3. Go into multi user mode (ctrl d). 4. Run netstat -m: check the mfubs in use. This will give an indication of how much the minimal network + kernel will use(should be around 30 - 35%). 5. Allow logins. At this point, run the system until you get the "out of mbufs" message. 6. Once "out of mubfs" has been posted, run netstat -m. You should see 100% (or near) utilization of mubfs. 7. Bring the system down into single user mode. 8. Run netstat -m. If you see the number of mbufs in use decrease to between 5 - 35% (the first 2 data points you took) the mbuf memory leak can be attributed to applications. If the mbuf in use count stays high (90+%)then this is a kernel mbuf memory leak. SOLUTION SUMMARY: PRODUCT AREA: Kernel PRODUCT: Config SUNOS RELEASE: Solaris 1.x UNBUNDLED RELEASE: 4.x HARDWARE: any SRDB ID: 4839 SYNOPSIS: system gives "out of mbufs" error message DETAIL DESCRIPTION: What is the correct response to an "out of mbufs" error message? SOLUTION SUMMARY: Run the netstat -m command to see the current status of mbufs in use and which ones are failing. To increase the mbuf value, increase the value of MAXUSERS in the kernel config file or change the mbuf blockvalue in /sys/conf.common/param.c. There may be a possibility that the error is caused by a bug requiring patchid 100126. An example follows. Example output of netstat -m (In this example, we see that the 4096 data block is failing): 317/896 mbufs in use: /4 mbufs allocated to data 2 mbufs allocated to packet headers 89 mbufs allocated to socket structures 112 mbufs allocated to protocol control blocks 105 mbufs allocated to routing table entries 2 mbufs allocated to socket names and addresses 3 mbufs allocated to interface addresses 0/16 mapped pages in use 128 Kbytes allocated to network (30% in use) 0 requests for memory denied streams allocation: cumulative allocation current maximum total failures streams 6 7 40 0 queues 21 25 155 0 blks 9 95 14779222 0 total dblks 9 95 14779222 5 size 4 dblks 0 12 8942 0 size 16 dblks 1 6 916 0 size 64 dblks 1 86 462756 0 size 128 dblks 7 10 1327690 0 size 256 dblks 0 1 1426921 0 size 512 dblks 0 2 212706 0 size 1024 dblks 0 2 11047414 0 size 2048 dblks 0 2 291877 0 size 4096 dblks 0 0 0 5 Portions of param.c can be modified (change the corresponding block value in /sys/conf.common/param.c): #define NBLK4096 0 #define NBLK2048 32 #define NBLK1024 12 #define NBLK512 8 #define NBLK256 (24 + 2 * MAXUSERS) #define NBLK128 (72 + 6 * MAXUSERS) #defineNBLK64 (200 + 10 * MAXUSERS) #define NBLK16 (300 + 25 * MAXUSERS) #define NBLK4 (96 + 8 * MAXUSERS) For example, change: #define NBLK4096 0 to #define NBLK4096 8 BUG REPORT ID: n/a PATCH ID: 100126 PRODUCT AREA: n/a PRODUCT: Ethernet SUNOS RELEASE: 4.0.3, 4.0.3c UNBUNDLED RELEASE: n/a HARDWARE: All SRDB ID: 4837 SYNOPSIS: mbuf allocation DETAIL DESCRIPTION: How has mbuf allocation changed from OS 4.0.x to OS 4.1.x? Are there any known problems of running out of mbufs? SOLUTION SUMMARY: mbufs are now dynamically allocated. In the past, users were able to increase the values by editing /sys/conf.common/param.c. However, a few known bugs exist in which some systems run out of mbufs and report the following type of error: out of mbufs output out of mbufs mbuf map full Patchid 100126 addresses this problem. BUG REPORT ID: n/a PATCH ID: 100126 PRODUCT AREA: n/a PRODUCT: Ethernet SUNOS RELEASE: 4.1, 4.1 PSR A, 4.1.1 UNBUNDLED RELEASE: n/a HARDWARE: All SRDB ID: 3390 SYNOPSIS: Increasing available mblks DETAIL DESCRIPTION: How do you increase the number of available mblks? SOLUTION SUMMARY: NOTE: In both SunOS 4.1.X and Solaris 2.X mblks are allocated dynamically If you are running something earlier than 4.0.3, increasing MAXUSERS won't help. If you do so, you will run out of streams message blocks, probably due to lots of serial line activity. Stream drivers, such as the tty driver, grab message buffers out of a fixed-sized pool of blocks. Blocks are available in sizes that are powers of 4: 4,16,64.... 4096 bytes. If a block is not available in the size requested, the streams blockallocator tries to get one of the next larger size. Most serial line activity consumes 4-byte blocks, and a system with some ALM-2s or a few modems can run out quickly. This can be easily worked around by increasing the number of blocks, but it is also worthwhile to try to find out why the modems are running out -- is there a modem that's jiggling a noisy tty line? Is there something running on the machine that impairs its ability to process character i/o from terminals? Did some terminal devices just get added? The number of blocks available is defined in /sys/conf.common/param.c in pre-4.0.3 releases; the number of blocks is based on MAXUSERS, although the multiplicand of MAXUSERS can be insufficient. In 4.0.1 and earlier releases, the number of blocks is a constant independent of any other parameter. The #defines setting the number of message blocks are called NBLK4, NBLK16, NBLK64, etc. To determine how much they should be increased, examine the number of allocation failures using netstat -m. The last column of the buffer allocation statistics shows how many failures have occurred for each size. The failure counts are cumulative, that is, if it is not possible to get a 4-byte or a 16-byte block, then the counts for 4- and 16-byte blocks are both incremented. Try doubling the number of mbufs that are being consumed by editing the #define NBLKxxxx XXX line in param.c for the appropriate mbuf size and regeneration of the kernel. BUG REPORT ID: n/a PATCH ID: n/a PRODUCT AREA: Kernel PRODUCT: Config SUNOS RELEASE: SunOS 4.0 UNBUNDLED RELEASE: n/a HARDWARE: Sun3, Sun4