Linux memory overcommit

Last week I learned something very interesting about the way Linux allocates and manages memory by default out of the box.

In a way, Linux allocates memory the way an airline sells plane tickets. An airline will sell more tickets than they have actual seats, in the hopes that some of the passengers don't show up. Memory in Linux is managed in a similar way, but actually to a much more serious degree.

Under the default memory management strategy, malloc() essentially always succeeds, with the kenrel assuming you're not _really_ going to use all of the memory you just asked for. The malloc()'s will continue to succeed, but not until you actually try to use the memory you allocated will the kernel 'really' allocate it. This leads to severe pathology in low memory conditions, because the application has already allocated the memory, it thinks it can use it free and clear, but when the system is in a low memory condition and an application is trying to use additional memory it has already allocated, the memory access takes a very long time as the kernel hunts around for memory to give.

In an extremely low memory condition, the kernel will start firing off the "OOM Killer" routine. Processes are given 'OOM Scores' and the process with the highest score, win^H^H^Hloses. This leads to random processes on a machine being killed by the kernel. Keeping in the airline analogies, I found this entertaining post.

I found some interesting information about the Linux memory manager here in section 9.6. This section has three small C programs to test memory allocation. The second and third program produced pretty similar results for me so I'm omitting the third:

Here are the results of the test on an 8GB debian Linux box:

demo1:  malloc memory and do not use it:       Allocated 1.4TB, killed by OOM killer
demo2:  malloc memory and use it right away:   Allocated 7.8GB, killed by OOM killer

Here are the results on an 8GB Nexenta/Opensolaris machine:

demo1:  malloc memory and do not use it:       Allocated 6.6GB, malloc() fails
demo2:  malloc memory and use it right away:   Allocated 6.5GB, malloc() fails

Apparently, a big reason linux manages memory this way out of the box is to optimize memory usage on fork()'ed processes; fork() creates a full copy of the process space, but in this instance, with overcommitted memory, only pages which have been written to actually need to be allocated by the kernel. This might work very well for a shell server, a desktop, or perhaps a server with a large memory footprint that forks an actual PID rather than a thread, but in our situation, this is very undesirable.

We run a pretty java-heavy environment, with multiple large JVMs configured per host. The problem is that the heap sizes have been getting larger, and we were running in an overcommitted situation and did not realize it. The JVMs would all start up and malloc() their large heaps, and then at some later time once enough of the heaps were actually used, the OOM killer would kick in and more or less randomly off one of our JVMs.

I found that linux can be brought more in line with traditional/expected memory management by setting the sysctls: (Apparently these are available only 2.6 kernels)

vm.overcommit_memory (0=default, 1=malloc always succeeds(?!?), 2=strict overcommit)
vm.overcommit_ratio (50=default, I used 100)

The ratio appears to be the percentage off the system's total VM that can be allocated via malloc() before malloc() fails. This MIGHT be on a per-pid basis (need to research). This number can be greater than 100%, presumably to allow for some slop in the copy-on-write fork()'s. When I set this to 100 on a 8GB system, I was able to malloc() about 7.5G of stuff, which seemed about right since I had normal multi-user processes running and no swap configured. I don't know why you'd want to use a number much less than 100, unless it were a per-process limit, or you wanted to force some saved room for fscache.

The big benefit here is that malloc() can actually fail in a low memory condition. This means that the error can be caught and handled by the application. In my case, it means that JVMs fail at STARTUP time, with an obvious memory shortage related error in the logs, rather than having the process have the rug yanked out from under it hours or days later with no message in the application log, and no opportunity to clean up what it was doing.

Here are the demo programs with a linux machine set to strict overcommit/100 ratio:

demo1:    malloc memory and do not use it:        Allocated 7.3GB, malloc fails.
demo2:    malloc memory and use it right away:    Allocated 7.3GB, malloc fails.

Technorati Tags: linux, memory, OOM

Debugging mysql5 on Nexenta

Due to some very favorable benchmarking results, I am planning to migrate some of our production databases to Myqsl5 on Nexenta. Previously the database was running on a Debian Linux server, and the I/O subsystem performed much worse on that system.

I ran into a very strange problem with mysql5 under Nexenta, however. After a certain number of clients connected, sometimes the server would begin to refuse connections in a very strange way. It would accept the connection to the mysql port, and then immediately close the connection.

Fortunately, one of the other reasons I want to move to Nexenta is the more robust toolchain for troubleshooting just these kinds of problems. I started out by using 'truss' on thread 1 of the mysql daemon under the assumption that it was the thread responsible for managing incoming client connections - not a bad guess. Here is a trace of a mysql connection that works correctly vs one that breaks:

Works OK:

root@perftest-db01:~# truss  -w all  -p 6650/1
/1:     pollsys(0x080473C0, 2, 0x00000000, 0x00000000) (sleeping...)
/1:     pollsys(0x080473C0, 2, 0x00000000, 0x00000000)  = 1
/1:     fcntl(11, F_SETFL, FWRITE|FNONBLOCK)            = 0
/1:     accept(11, 0x08047948, 0x08047958, SOV_DEFAULT) = 57
/1:     fcntl(11, F_SETFL, FWRITE)                      = 0
/1:     sigaction(SIGCLD, 0x08047420, 0x080474A0)       = 0
/1:     getpid()                                        = 6650 [6589]
/1:     getpeername(57, 0xFEF67A90, 0x080474B8, SOV_DEFAULT) = 0
/1:     getsockname(57, 0xFEF67A80, 0x080474B8, SOV_DEFAULT) = 0
/1:     open("/etc/hosts.allow", O_RDONLY)              = 58
/1:     fstat64(58, 0x08046B20)                         = 0
/1:     fstat64(58, 0x08046A50)                         = 0
/1:     ioctl(58, TCGETA, 0x08046AEC)                   Err#25 ENOTTY
/1:     read(58, " #   / e t c / h o s t s".., 8192)    = 677
/1:     read(58, 0x504EA88C, 8192)                      = 0
/1:     llseek(58, 0, SEEK_CUR)                         = 677
/1:     close(58)                                       = 0
/1:     open("/etc/hosts.deny", O_RDONLY)               = 58
/1:     fstat64(58, 0x08046B20)                         = 0
/1:     fstat64(58, 0x08046A50)                         = 0
/1:     ioctl(58, TCGETA, 0x08046AEC)                   Err#25 ENOTTY
/1:     read(58, " #   / e t c / h o s t s".., 8192)    = 901
/1:     read(58, 0x504EA88C, 8192)                      = 0
/1:     llseek(58, 0, SEEK_CUR)                         = 901
/1:     close(58)                                       = 0
/1:     getsockname(57, 0x08047938, 0x08047958, SOV_DEFAULT) = 0
/1:     fcntl(57, F_SETFL, (no flags))                  = 0
/1:     fcntl(57, F_GETFL)                              = 2
/1:     fcntl(57, F_SETFL, FWRITE|FNONBLOCK)            = 0
/1:     setsockopt(57, ip, 3, 0x0804748C, 4, SOV_DEFAULT) = 0
/1:     setsockopt(57, tcp, TCP_NODELAY, 0x0804748C, 4, SOV_DEFAULT) = 0
/1:     time()                                          = 1169599669
/1:     lwp_kill(73, SIG#0)                             Err#3 ESRCH
/1:     lwp_create(0x08047240, LWP_DETACHED|LWP_SUSPENDED, 0x08047464) = 243
/1:     lwp_continue(243)                               = 0
/1:     pollsys(0x080473C0, 2, 0x00000000, 0x00000000) (sleeping...)

Immediately closes connection:

root@perftest-db01:~# truss  -w all  -p 6650/1
/1:     pollsys(0x080473C0, 2, 0x00000000, 0x00000000) (sleeping...)
/1:     pollsys(0x080473C0, 2, 0x00000000, 0x00000000)  = 1
/1:     fcntl(11, F_SETFL, FWRITE|FNONBLOCK)            = 0
/1:     accept(11, 0x08047948, 0x08047958, SOV_DEFAULT) = 255
/1:     fcntl(11, F_SETFL, FWRITE)                      = 0
/1:     sigaction(SIGCLD, 0x08047420, 0x080474A0)       = 0
/1:     getpid()                                        = 6650 [6589]
/1:     getpeername(255, 0xFEF67A90, 0x080474B8, SOV_DEFAULT) = 0
/1:     getsockname(255, 0xFEF67A80, 0x080474B8, SOV_DEFAULT) = 0
/1:     open("/etc/hosts.allow", O_RDONLY)              = 257
/1:     close(257)                                      = 0
/1:     fxstat(2, 256, 0x08045DF8)                      = 0
/1:     time()                                          = 1169599778
/1:     getpid()                                        = 6650 [6589]
/1:     putmsg(256, 0x080467B8, 0x080467C4, 0)          = 0
/1:     open("/var/run/syslog_door", O_RDONLY)          = 257
/1:     door_info(257, 0x08045C10)                      = 0
/1:     getpid()                                        = 6650 [6589]
/1:     door_call(257, 0x08045C48)                      = 0
/1:     close(257)                                      = 0
/1:     fxstat(2, 256, 0x080459B8)                      = 0
/1:     time()                                          = 1169599778
/1:     getpid()                                        = 6650 [6589]
/1:     putmsg(256, 0x08046378, 0x08046384, 0)          = 0
/1:     open("/var/run/syslog_door", O_RDONLY)          = 257
/1:     door_info(257, 0x080457D0)                      = 0
/1:     getpid()                                        = 6650 [6589]
/1:     door_call(257, 0x08045808)                      = 0
/1:     close(257)                                      = 0
/1:     fxstat(2, 256, 0x08046A88)                      = 0
/1:     time()                                          = 1169599778
/1:     getpid()                                        = 6650 [6589]
/1:     putmsg(256, 0x08047448, 0x08047454, 0)          = 0
/1:     open("/var/run/syslog_door", O_RDONLY)          = 257
/1:     door_info(257, 0x080468A0)                      = 0
/1:     getpid()                                        = 6650 [6589]
/1:     door_call(257, 0x080468D8)                      = 0
/1:     close(257)                                      = 0
/1:     shutdown(255, SHUT_RDWR, SOV_DEFAULT)           = 0
/1:     close(255)                                      = 0
/1:     pollsys(0x080473C0, 2, 0x00000000, 0x00000000) (sleeping...)

Looks like the main difference starts here:

/1:     open("/etc/hosts.allow", O_RDONLY)              = 257
/1:     close(257)                                      = 0

That explains a lot -- the "hosts.allow" file is part of the tcpwrappers system, which controls access to various daemons on the system based on access control rules set by the system administrator. No wonder I am getting a connection but then immediately getting booted. It is trying to open the hosts.allow file, but then is immediately closing it, vs actually reading and processing the file as seen in the working connection. Does the process not have enough filehandles?

root@perftest-db01:~# pfiles  6650 |head -2
6650:   /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql
Current rlimit: 8192 file descriptors

Nope, doesn't look that way -- it's configured to use 8192 filehandles. My next clue was the file descriptor number that was returned by the "open" system call, 257. That's awfully near one of those magic "power of 2" boundaries. I started snooping around in google.

It turns out that under Solaris, and maybe *BSD also, the tcpwrappers library (libwrap) uses the "stdio" library to manage IO. This library does not understand file handles above 255, therefore, as the mysql server continues to collect client processes and open tables for reading, eventually this file descriptor boundary is crossed and calls to open "hosts.allow"appear to fail because they return too high a file descriptor number. tcpwrappers appears to fail closed, so since it cannot read the "hosts.allow" file, it denies access to the service by immediately closing the communication channel.

Fortunately, there is a fix. Giri Mandalika has a blog entry that references the issue and is a good resource on the problem. The solution is to use the extendedFILE library that's provided in Solaris Express 06/06 or later (So this is included in Nexenta Alpha 6, and possibly earlier):

root@perftest-db01:~# export LD_PRELOAD_32=/usr/lib/extendedFILE.so.1
root@perftest-db01:~# /etc/init.d/mysql restart

(Obviously I will also need to modify the /etc/init.d/mysql startup script to include the LD_PRELOAD_32). Now, I start up a test program to artificially create a bunch of connections to the database, and see what a truss looks like now:

root@perftest-db01:~# truss  -w all  -p 6846/1
/1:     pollsys(0x08047390, 2, 0x00000000, 0x00000000) (sleeping...)
/1:     pollsys(0x08047390, 2, 0x00000000, 0x00000000)  = 1
/1:     fcntl(11, F_SETFL, FWRITE|FNONBLOCK)            = 0
/1:     accept(11, 0x08047918, 0x08047928, SOV_DEFAULT) = 294
/1:     fcntl(11, F_SETFL, FWRITE)                      = 0
/1:     sigaction(SIGCLD, 0x080473F0, 0x08047470)       = 0
/1:     getpid()                                        = 6846 [6785]
/1:     getpeername(294, 0xFEF47A90, 0x08047488, SOV_DEFAULT) = 0
/1:     getsockname(294, 0xFEF47A80, 0x08047488, SOV_DEFAULT) = 0
/1:     open("/etc/hosts.allow", O_RDONLY)              = 295
/1:     fstat64(295, 0x08046AF0)                        = 0
/1:     fstat64(295, 0x08046A20)                        = 0
/1:     ioctl(295, TCGETA, 0x08046ABC)                  Err#25 ENOTTY
/1:     read(295, " #   / e t c / h o s t s".., 8192)   = 677
/1:     read(295, 0x5122F9D4, 8192)                     = 0
/1:     llseek(295, 0, SEEK_CUR)                        = 677
/1:     close(295)                                      = 0
/1:     open("/etc/hosts.deny", O_RDONLY)               = 295
/1:     fstat64(295, 0x08046AF0)                        = 0
/1:     fstat64(295, 0x08046A20)                        = 0
/1:     ioctl(295, TCGETA, 0x08046ABC)                  Err#25 ENOTTY
/1:     read(295, " #   / e t c / h o s t s".., 8192)   = 901
/1:     read(295, 0x5122F9D4, 8192)                     = 0
/1:     llseek(295, 0, SEEK_CUR)                        = 901
/1:     close(295)                                      = 0
/1:     getsockname(294, 0x08047908, 0x08047928, SOV_DEFAULT) = 0
/1:     fcntl(294, F_SETFL, (no flags))                 = 0
/1:     fcntl(294, F_GETFL)                             = 2
/1:     fcntl(294, F_SETFL, FWRITE|FNONBLOCK)           = 0
/1:     setsockopt(294, ip, 3, 0x0804745C, 4, SOV_DEFAULT) = 0
/1:     setsockopt(294, tcp, TCP_NODELAY, 0x0804745C, 4, SOV_DEFAULT) = 0
/1:     time()                                          = 1169601081
/1:     lwp_kill(273, SIG#0)                            Err#3 ESRCH
/1:     lwp_create(0x08047210, LWP_DETACHED|LWP_SUSPENDED, 0x08047434) = 274
/1:     lwp_continue(274)                               = 0
/1:     pollsys(0x08047390, 2, 0x00000000, 0x00000000) (sleeping...)

As you can see above - the "open" command on the "hosts.allow" file is returning a filehandle greater than 255, but reading and processing the hosts.allow file proceeds normally, and the connection is accepted.

Yay for truss!

Technorati Tags: mysql5, nexenta, opensolaris, truss

ZFS features

Here's a post I just entered on the Nexenta/gnusolaris Beginners Forum that has some good info about ZFS. Apparently the formatting got eaten on the mailing list so I'm reposting it here:

Hi all,

Can I have it installed concurrently with linux and allocate linux partitions to the RAID Z? or RAID-Z takes the whole disks?

There are two "layers" of partitions in opensolaris; the first is managed with the "fdisk" utility, the second is managed with the "format" utility - these partitions are aka "slices". I am not an expert, but I believe that the "fdisk" managed partitions are the pieces that linux/windows/etc sees. You first would allocate one of these partitions to Solaris, and from there you can additionally split that fdisk partition into root/swap/data "slices". I believe that the linux partitions you'd see would be visible via the "fdisk" command.

According to some of the ZFS faq/wiki resources, ZFS is "better" if it manages the entire disk, however, it will work just fine managing either "partitions" or "slices". You can even make a ZFS pool with individual files.

Here is an example of one of my disks. There is one "fdisk" partition, and a few "slices":

root@medb01:~# fdisk -g /dev/rdsk/c0t0d0p0                                                                                       
* Label geometry for device /dev/rdsk/c0t0d0p0
* PCYL     NCYL     ACYL     BCYL     NHEAD NSECT SECSIZ
 48638    48638    2        0        255   63    512 

root@medb01:~# prtvtoc /dev/rdsk/c0t0d0p0                                                                                        
* /dev/rdsk/c0t0d0p0 partition map
*
* Dimensions:
*     512 bytes/sector
*      63 sectors/track
*     255 tracks/cylinder
*   16065 sectors/cylinder
*   48640 cylinders
*   48638 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
      0      0    00      16065   8401995   8418059
      1      0    00    8418060  16787925  25205984
      2      5    01          0 781369470 781369469
      6      0    00   25205985 756147420 781353404
      7      0    00  781353405     16065 781369469
      8      1    01          0     16065     16064

Note that in the following examples, I'll create ZFS pools with "c0tXd0s6", that is, the 6th "slice" listed in the solaris partition table.

Alternatively, Can I mount my Linux RAID partitions on Nexenta, at least for migration purposes? What about the LVM disks?

As far as I know, there is no LVM or linux-supported filesystem types built into Opensolaris/Nexenta. i.e. you could not just "mount -t ext3" a linux filesystem and be able to read it. Since you've mentioned that you're running a VMware server, I suppose it may be possible to have both guest operating systems running and copy the data over the 'network'. Also it's likely that Nexenta won't know about LVM managed partitions, it would have to be a real honest-to-goodness partition.

What about RAID-Z features:
Can I hot-swap a defective disk?

This should be possible, assuming that your hardware supports it. You may need to force a rescan of the devices if you replace a disk, check devfsadm. Reintegrating it into the pool would be accomplished with a "zpool replace pool device [new device]"

Can I add a disk to the server and tell it to enlarge the pool, to make more space available on the preexisting RAID?

Yes, with a caveat - ZFS doesn't do any magic stripe re-balancing. If you have a 4-disk pool, and add another disk, what you really have is a 4-disk raidz with a single disk tacked on at the end with no redundancy. Best practice would be to add space in 'chunks' of several disks. Fortunately I am in the middle of building a Nexenta-based box with 4 SATA drives so I can play around with some of the commands and show you the output:

Here is a 4-disk zpool using raidZ:

root@medb01:~# zpool create u01 raidz c0t0d0s6 c0t1d0s6 c0t2d0s6 c0t3d0s6                                                        
root@medb01:~# zpool status u01                                                                                                  
 pool: u01
state: ONLINE
scrub: none requested
config:

       NAME          STATE     READ WRITE CKSUM
       u01           ONLINE       0     0     0
         raidz1      ONLINE       0     0     0
           c0t0d0s6  ONLINE       0     0     0
           c0t1d0s6  ONLINE       0     0     0
           c0t2d0s6  ONLINE       0     0     0
           c0t3d0s6  ONLINE       0     0     0

Here is a 3-disk raidZ pool that I "grow" by adding a single additional disk. Note the subtle indentation difference on c0t3d0s6 in this example; it is not part of the original raidz1 and is just a standalone disk in the pool.

root@medb01:~# zpool destroy u01                                                                                                 
root@medb01:~# zpool create u01 raidz c0t0d0s6 c0t1d0s6 c0t2d0s6                                                                 
root@medb01:~# zpool add u01 c0t3d0s6                                                                                            
invalid vdev specification
use '-f' to override the following errors:
mismatched replication level: pool uses raidz and new vdev is disk
root@medb01:~# zpool add -f u01 c0t3d0s6                                                                                         
root@medb01:~# zpool status u01                                                                                                  
 pool: u01
state: ONLINE
scrub: none requested
config:

       NAME          STATE     READ WRITE CKSUM
       u01           ONLINE       0     0     0
         raidz1      ONLINE       0     0     0
           c0t0d0s6  ONLINE       0     0     0
           c0t1d0s6  ONLINE       0     0     0
           c0t2d0s6  ONLINE       0     0     0
         c0t3d0s6    ONLINE       0     0     0

Here is an example of adding space in "chunks", note the size of the volume is different in the "zpool list" before and after.

root@medb01:~# zpool destroy u01                                                                                                 
root@medb01:~# zpool create u01 mirror c0t0d0s6 c0t1d0s6                                                                         
root@medb01:~# zpool list u01                                                                                                    
NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
u01                     360G   53.5K    360G     0%  ONLINE     -
root@medb01:~# zpool add u01 mirror c0t2d0s6 c0t3d0s6                                                                            
root@medb01:~# zpool list u01                                                                                                    
NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
u01                     720G    190K    720G     0%  ONLINE     -
root@medb01:~# zpool status u01                                                                                                  
 pool: u01
state: ONLINE
scrub: none requested
config:

       NAME          STATE     READ WRITE CKSUM
       u01           ONLINE       0     0     0
         mirror      ONLINE       0     0     0
           c0t0d0s6  ONLINE       0     0     0
           c0t1d0s6  ONLINE       0     0     0
         mirror      ONLINE       0     0     0
           c0t2d0s6  ONLINE       0     0     0
           c0t3d0s6  ONLINE       0     0     0

PS, doing it this way appears to stripe writes across the two mirrored "subvolumes".

Does it have a facility similar to LVM, where I can create 'logical volumes' on top of the RAID and allocate/deallocate space as needed for flexible storage management (without putting the machine offline)?

Yes, there are two layers in ZFS, the pool management, managed through the "zpool" command, and the filesystem management, through the "zfs" command. Individual filesystems are created as subdirectories of the base pool, or can be relocated with the "zfs set mountpoint" option if you desire. Here I create a ZFS called /u01/opt with a 100MB quota, and then increase the quota to 250MB.

root@medb01:~# zfs create -oquota=100M u01/opt                                                                                   
root@medb01:~# df -k /u01 /u01/opt                                                                                               
Filesystem            kbytes    used   avail capacity  Mounted on
u01                  743178240      26 743178105     1%    /u01
u01/opt               102400      24  102375     1%    /u01/opt
root@medb01:~# zfs set quota=250m u01/opt                                                                                        
root@medb01:~# df -k /u01 /u01/opt                                                                                               
Filesystem            kbytes    used   avail capacity  Mounted on
u01                  743178240      26 743178105     1%    /u01
u01/opt               256000      24  255975     1%    /u01/opt

Also, things like atime update, compression, etc, can be set on a per filesystem basis.

Can I do fancy stuff like plug an e-sata disk to my machine and tell it to 'ghost' a 'logical volume' on-the-fly, online, without unmounting the volume?

Yes, this is possible. ZFS supports "snapshots" - moment in time copies of an entire ZFS filesystem. ZFS also supports a "send" and "receive" of a snapshot, so you can then take that moment in time copy of your filesystem and replicate it somewhere else. (Or just leave the snapshot laying around for recovery purpouses).

The procedure would be to create a ZFS volume on your external drive, and then "zpool import" that drive each time you plugged it in. Then create a snapshot on your filesystem and "send" it to the external drive, like so. (I don't have an external drive to import so I'll just create 2 pools). I test by creating a filesystem, creating a file in that filesystem, then snapshotting and sending that snapshot to a different pool. Note that the file I created exists in the destination when I'm done.

root@medb01:/# zpool destroy u01                                                                                                 
root@medb01:/# zpool destroy u02                                                                                                 
root@medb01:/# zpool create u01 mirror c0t0d0s6 c0t1d0s6                                                                         
root@medb01:/# zpool create u02 mirror c0t2d0s6 c0t3d0s6                                                                         
root@medb01:/# zfs create u01/data                                                                                               
root@medb01:/# echo "test test test" > /u01/data/testfile.txt                                                                    
root@medb01:/# zfs snapshot u01/data@send_test                                                                                   
root@medb01:/# zfs send u01/data@send_test | zfs receive u02/u01_copy                                                            
root@medb01:/# ls -l /u02/u01_copy                                                                                               
total 1
-rw-r--r-- 1 root root 15 Jan 23 04:49 testfile.txt
root@medb01:/# cat /u02/u01_copy/testfile.txt                                                                                    
test test test
root@medb01:/#    

Hope all this helps (and maybe makes it into the wiki too :-) )

Modifying a Nexenta Boot ISO

Today I needed to modify a nexenta Alpha-6 install ISO. There were several things I needed to fix.

1. We have two revisions of the same Supermicro H8DAR-T based beige-box systems. Solaris10/Nexenta works great on the 2.01 based motherboards. The older version - version 1.01 - does not have a SATA chipset that is supported by Nexenta out of the box. (The PCI ID is "pci11ab,6041.3" versus "pci11ab,6041.9" on the 2.01 version). I needed to add this PCI ID to the "/etc/driver_aliases" file. Aside: As far as I can tell, the only way to differentiate is to crack open the box and look for the silkscreened version on the back lefthand corner. :-( (Or, you could boot Nexenta/Solaris10 and see if it can see the disks :-) )

2. There is a bug in the nexenta-install.sh script that will hang the system while scanning for partitions after manual partitioning. I found a response to this post by LukeD in the Nexenta forums that is reported to fix this problem.

3. We wanted to add a couple of packages to the "minimal" set.

If we start a widespread Nexenta rollout there will be much more automation going into these CD images (Or even better, network images) to make for a hands-off install.

Here's how I made the changes:

1. Copied the .ISO image to another Nexenta system. (There is nothing Nexenta/Solaris specific to making the image, however, the commands used are somewhat different between different operating systems. On a linux system, you would use losetup instead of lofiadm. )

2. Create a loopback device for the .iso and mount it:

lofiadm -a /opt/temp/elatte_installcd_alpha6_i386.iso
( this creates /dev/lofi/1 )
mkdir /mnt
mount -F hsfs /dev/lofi/1 /mnt

3. Copy all of the files to a temporary location. This is necessary because the .ISO image is read only.

mkdir /opt/temp/cd
cp -av /mnt/. /opt/temp/cd/.

4. Find and mount the miniroot image as a second loopback device. The miniroot is a gzipped UFS filesystem created in the "boot" directory of the CD. Most of the filesystem is here, although it appears that nexenta remounts /usr from the CD-ROM later in the boot process. This step is required because I am changing the "driver_aliases" file, which lives inside the gzipped miniroot -- if you plan only to modify the nexenta_install.sh and/or add packages, these steps (4-6) are not necessary.

cd /opt/temp/cd/boot/
mv miniroot miniroot.gz
gunzip miniroot.gz
lofiadm -a /opt/temp/cd/boot/miniroot
( this creates /dev/lofi/2 )
mkdir /mnt2
mount /dev/lofi/2 /mnt2

5. Edit the driver_aliases file:

cd /mnt2/etc
(vi driver_aliases)

... If there are any other files that need to change within the miniroot, edit them now.

6. Unmount and re-gzip the miniroot, and clean up the other lofi mount too.

umount /mnt2
lofiadm -d /dev/lofi/2
umount /mnt
lofiadm -d /dev/lofi/1
cd /opt/temp/cd/boot
gzip miniroot
mv miniroot.gz miniroot

7. Fix the script mentioned in problem #2 and add some packages to the minimal set.

cd /opt/temp/cd/root/usr/gnusolaris
vi nexenta-install.sh
(I chose to do a "grep -i "Unknown_fstype" rather than run the sed script referenced in the forum post above)
vi base-minimal.lst
(add some packages here)

8. Create the new install CD. I found some good instructions on BigAdmin at the end of this article in the section "Using CD/DVD ISO Files":

mkisofs -o /opt/temp/nexenta_boot.iso -b boot/grub/stage2_eltorito \
   -c .catalog -no-emul-boot -boot-load-size 4 \
   -boot-info-table -relaxed-filenames -ldots -N -l -R \
   -d -D -V Elatte_InstallCD /opt/temp/cd/

This will create the new .ISO file. From there you'll need to burn it to disk.

Unfortunately, adding the new driver_alias line didn't work for me. It sees the controller now, but it pukes about not knowing how to talk to it. (Forgot the exact error message but it looked ominous). I will have to do some testing on a known-working box (one of the later revision ones) to see if the other changes I made to this CD were successful.

Edit: The mkisofs command I originally listed was incorrect. I have found the magic mkisofs command parameters and updated this post accordingly. Note that the volume name MUST be "Elatte_InstallCD".

Debian linux ethernet bonding

We're working on some fault tolerant delpoyments of debian linux systems. It turns out that this is available in the stock debian kernel (sarge) and luckily, in the custom kernel we'd built at a later time.

I dug up a few howto's out there, but none of them really had all of the pieces in one place, specifically for the packages that Debian uses out of the box.

Two software pieces are necessary - the "bonding" driver, and the "ifenslave" package. The bonding driver is in the default sarge kernel, and appears to be in the defaults during a kernel build. The "ifenslave" package is the userspace program used to control the binding of physical interfaces to the bonded driver. To install this, simply

apt-get install ifenslave-2.6

It's important to get the latest version; the version that installs with a plain "apt-get install ifenslave" doesn't seem to work properly.

Next, there are two files which need to change to tell the kernel to load the bonding driver. I appended these lines to the following files:

/etc/modules:
bonding

/etc/modprobe.d/aliases:
alias bond0 bonding
options bonding mode=active-backup miimon=100 max_bonds=1

If more than one bonding interface is needed, add additional aliases in this file, and increase the "max_bonds" option as necessary. We plan on using bonding on a few machines that act as routers, so they will need to have multiple bonded interface sets.

Finally, the bond0 interface must be set up in the /etc/network/interfaces file. More than likely there is an existing entry for your primary network interface, i.e. eth0. I just changed the interface name from eth0 to bond0, and added the following line:

up    ifenslave bond0 eth0 eth1

For reference, the entire /etc/network/interfaces file looks like this - notice that there are no individual entries for eth0 and eth1.

# generated by FAI
auto lo bond0
iface lo inet loopback
iface bond0 inet static
address 192.168.195.145
netmask 255.255.255.0
broadcast 192.168.195.255
gateway 192.168.195.1
up    ifenslave bond0 eth0 eth1
post-up  /opt/tools/bin/init-ipmi

The linux Bonding howto is very comprehenive and covers the different modes of operation of this driver, as well as installation instructions for different flavors of linux and some discussion of deployment scenarios. We'll be expirimenting with various bonding modes this week to see what we can get away with; currently we're planning on running in the "active-backup" mode which is a simple active/passive failover.

Here are some other resources about linux/debian ethernet bonding:
http://www.howtoforge.com/nic_bonding
http://glasnost.beeznest.org/articles/179
http://www.debianhelp.co.uk/bonding.htm

More about bonding later!