Updating split to support large files

Posted by Brian on January 20, 2007

The split utility is a highly useful tool for splitting a large file into pieces. This command-line utility may be used to split a large file so that it may fit on several CDs or DVDs. It also could be used to provide a large file in multiple segments for easier download over slow Internet connections.

Recently I discovered a small problem with the split utility that ships with Mac OS X 10.4. Attempting to split a really large file into 2GB pieces produced an error, as shown here.

$ split -b 2048m somelargefile
split: write: Invalid argument

An examination of the source code used for the split utility (244KB download) revealed that a signed long value was used to store the byte count, limiting it to a maximum of 2047MB.

Newer Source Code

Since the source code appeared to originate with NetBSD, I visited their web site to download a newer release of the split utility.

http://cvsweb.netbsd.org/bsdweb.cgi/src/usr.bin/split/split.c

This newer code uses the off_t type to store the byte count, which in Mac OS X 10.4 is a 64-bit integer value allowing a much greater maximum number of bytes.

The first attempt to compile the code failed because DEFFILEMODE was undeclared. This was resolved by adding the following line near the top of the split.c file, right in between the #include <sys/param.h> and #include <ctype.h> lines.

#include <sys/stat.h>

Compile and Install

I then used this command line to compile a universal binary suitable for both Intel and PowerPC Macintosh computers. This was important since our computers include at least one Intel Macintosh with more likely in the future.

cc -Os -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch i386 -arch ppc \
-o split split.c

If you are compiling for just one type of processor, then a simpler command could be used.

cc -Os -o split split.c

Installation of the newly compiled split utility was performed with the commands listed below. Notice how the Apple-provided split utility is renamed in case it is needed later on.

sudo mv /usr/bin/split /usr/bin/split.apple
sudo cp split /usr/bin/split
sudo chmod ugo-w /usr/bin/split
sudo chmod ugo+rx /usr/bin/split
sudo chown root:wheel /usr/bin/split

These instructions assumed some familiarity with the Terminal application.

This problem has been reported to Apple by means of a bug report (#4845285), so hopefully this work-around won’t be needed in a future release of Mac OS X. I also submitted this information as a hint to macosxhints.com.