hard drive format advice

[farmkid]

My lab uses some external hard drives to backup large datasets. I'm kind of in charge of getting and maintaining the drives. We currently have two 1TB hard drives. One is ext4 and the other is NTFS. I use the ext4 drive to backup data from a computer that generates about 30-40GB of data per experiment. That computer runs Red Hat Enterprise Linux. I also use it with my laptop, which runs Fedora most of the time. The NTFS drive is used for general backup of other computers.

The problem is that we have the Linux machines I mentioned, Windows machines, and Macs. Anyone who wants to backup a Mac has to transfer the data to a Windows computer first because they can't write to either NTFS or ext4. The Red Hat computer can't read the NTFS drive; that's not usually a problem, but occasionally I have had a reason to plug in the NTFS drive into it. There is another computer on an instrument running an older version of Red Hat that can't read NTFS or ext4.

I'm about to buy another hard drive, and I would like to format it in such a way that it can be used with all of the computers. I know I could use Fat32, but that doesn't work well with large drives (I'm probably going to buy a 2TB drive) and really don't want to partition the drive in to several smaller drives so that Fat32 will be okay on it. Does anyone know of a format I could use that will work with large drives and is universal with all three operating systems without too much fuss?

 

[Jadzia]

All computers can work with FAT32 and that does support 2TB drives.

If you format it with a 32k cluster size, it can go upto 8TB.

The problem you may have with it is file size. It has a 4GB file size limit. I don't know if your data dumps are one file or many files.

 

[farmkid]

Yeah, I've been tempted to use FAT32. Most (or perhaps all) of the data we generate will be smaller files, so the 4GB limit shouldn't be much of a problem. For example, the 30GB datasets I mentioned that I routinely generate is mostly composed of about 800 images that are about 32MB each. What I'm more concerned about, however, is the waste of space introduced by using 32K clusters in FAT32. For example, one run through a project I do regularly (from the 30GB raw data mentioned above) will generate about 1GB of data in about 20,000 files I need to backup. Of that 1GB, about 800MB is contained in 16 files. Probably about 1000-2000 of those files are about 100K-1MB in size, and the most of the rest of them are 0-2K in size. So, that 1GB of data will end up filling a lot more than 1GB on a FAT32 volume. Maybe that's not a big deal on a 2TB drive that costs $80, but that's something I need to decide before choosing FAT32.

FAT32 seems to be the best for universal access, but it comes with other problems. NTFS should work well if I could install NTFS-3g on the Macs and on the RHEL machine, but that's not up to me. What I really need is something as universal as FAT32 and as functional as NTFS. I haven't found any such thing.

 

[Robert Maxwell]

FAT32 is definitely the way to go if you want maximum compatibility, just as long as you keep the 4GB filesize limit in mind and don't mind how terribly inefficient its cluster sizes are.

I don't think you're going to find anything more universal. You could, however, install an ext2 driver on the Macs, which would be natively supported by the RHEL system. But then ext2 is not supported by Windows, either, unless you want to install a driver for it.

If you can't install any software on any of the systems involved, FAT32 is as good as you're gonna get.

 

[farmkid]

That's what I was thinking, but as stated, FAT32 isn't ideal. I guess I was just hoping that someone here would know of some file system I'm not familiar with that would work better, or that someone would have some creative solution I hadn't thought of.

 

[Jadzia]

For storing your data, could you tar/zip these 20000 files together? That way it wouldn't waste space from having 32k clusters.

I know that for windows, there is software that allows you to mount zip archives into the file system, so you can read/write files to an archive as though it is a drive. I don't know if there is similar software for osx/linux, but I expect there would be something like that for linux.

 

[Robert Maxwell]

That's what I was thinking, but as stated, FAT32 isn't ideal. I guess I was just hoping that someone here would know of some file system I'm not familiar with that would work better, or that someone would have some creative solution I hadn't thought of.

Nothing is going to be "ideal" in your situation because the situation isn't ideal in the first place, unfortunately.

If you could install whatever software you needed on all the systems involved, you'd have a lot more options. Since it sounds like you can't, FAT32 is the common denominator.