Query on the limitation of FAT32 File System of files per folder

[Remittag]

Dear Tim,

Thanks for your prompt response.

We wish to put forward a query about the issue of the FAT32 File System
limitation of files per folder in our application as follows:-

Background

We are developing a software application targeted at home users (using
Visual Basic 6.0 (for the Front-End GUI) and MS Access (as back-end
database)) that allows a user to build, store, and organize RTF (Rich Text
Format) documents. These RTF documents are stored under a folder named 'Data'
which is generated when our application is installed in the system drive of
the user’s computer.

Query – FAT32 File System limitation of files per folder in our application

While conducting a Stress Test on our application on a computer running on
Windows 98 Second Edition, and having the FAT32 File System, we found that
the 'Data' folder could not hold more than 32,767 files (RTF documents). On
doing some research on this issue, we found out that a directory/folder in
the FAT32 File System can contain upto 65,536 entries (i.e. files). The
reason that the 'Data' folder was limited to 32,767 files could be due to the
fact that each file is taking 2 entries on an average. This issue has put a
restriction on our application in that the user can create only 32,767 files.
For this, we have thought of the following solutions:-

i) In the Minimum System Requirements for using our application, we will
tell the user that he/she should have a computer running on Windows 2000 or
above which have the NTFS File System as it has no limit on the number of
file or subdirectories in a directory. However, we do not feel that this
solution would be feasible as the FAT32 File System is present on most
computers.

ii) We will programmatically create multiple folders under the 'Data'
folder. For example, within 'Data' folder we would have a folder named
something like "1-30000" in which the first 30000 files will be stored. Now
if the user creates the 30001th file, then a new folder named "30001-60000"
will be automatically created under the 'Data' folder, which will store the
next 30000 files and so on. In this manner, our application will have
multiple folders containing 30000 files each. Thus, this would remove the
restriction on our application in that the user can create MORE THAN 32,767
FILES.

We would request you to kindly verify and suggest which of the solutions
discussed above is feasible and should be implemented.

We would look forward to your reply at the earliest.

Regards,

Remittag Team

[Tim Slattery]

Post by Remittag
i) In the Minimum System Requirements for using our application, we will
tell the user that he/she should have a computer running on Windows 2000 or
above which have the NTFS File System as it has no limit on the number of
file or subdirectories in a directory. However, we do not feel that this
solution would be feasible as the FAT32 File System is present on most
computers.

FAT32 can be used on all Win95, Win98, WinME, WinNT, Win2000, and
WinXP computers. NTFS can be used only on the NT line: WinNT,
Win2000, WinXP. Since new computers for a few years now have come with
WinXP, I don't know whether it's still true that "FAT32 is present on
most computers". Maybe it is, I think that some new computers that
come with XP also come with the disks setup with FAT32. But I can only
guess about this.

Post by Remittag
ii) We will programmatically create multiple folders under the 'Data'
folder. For example, within 'Data' folder we would have a folder named
something like "1-30000" in which the first 30000 files will be stored. Now
if the user creates the 30001th file, then a new folder named "30001-60000"
will be automatically created under the 'Data' folder, which will store the
next 30000 files and so on. In this manner, our application will have
multiple folders containing 30000 files each. Thus, this would remove the
restriction on our application in that the user can create MORE THAN 32,767
FILES.

Either solution will work. Ultimately, dividing into multiple subdirs
may be better. Even though you can have lots and lots of files in an
NTFS directory, you're still dealing with a huge directory, and huge
directories take time to search and manipulate.

[Jeff Richards]

You don't indicate whether the filenames are automatically generated or
assigned by the user. If the stress tester has automatically assigned the
file name then you have accidentally hit on a scheme that requires exactly
two entries per file. If the user is going to be allowed to choose their
own names, then the stress testing has given you a distorted result for the
maximum number of files - the maximum might be much less than 32,000.

If you are controlling the file naming, then you should simply indicate that
the product has an absolute limit of 32,000 files if the user is using the
FAT file system. If it is really unacceptable to tell users that there is
this sort of limit because a significant number of users might want to
manage more than 32,000 files, then you have more problems than the file
system limits. The FAT file system performance starts to degrade
significantly at about 10,000 files in a folder, and at about 20,000 files
in a folder file operations can take too long to be acceptable in ordinary
usage. For many users, backing up a folder of 32,000 files would be
impractical.

If the user is controlling the filenames then you have to re-think the
design because it will be very difficult to tell when the folder is running
out of directory entries. It would be better if you were not relying on the
file system to do your filing for you. That way you determine your own
limits and you ensure adequate scalability.
--
Jeff Richards
MS MVP (Windows - Shell/User)

Post by Remittag
Dear Tim,
Thanks for your prompt response.
We wish to put forward a query about the issue of the FAT32 File System
limitation of files per folder in our application as follows:-
Background
We are developing a software application targeted at home users (using
Visual Basic 6.0 (for the Front-End GUI) and MS Access (as back-end
database)) that allows a user to build, store, and organize RTF (Rich Text
Format) documents. These RTF documents are stored under a folder named 'Data'
which is generated when our application is installed in the system drive of
the user's computer.
Query - FAT32 File System limitation of files per folder in our
application
While conducting a Stress Test on our application on a computer running on
Windows 98 Second Edition, and having the FAT32 File System, we found that
the 'Data' folder could not hold more than 32,767 files (RTF documents). On
doing some research on this issue, we found out that a directory/folder in
the FAT32 File System can contain upto 65,536 entries (i.e. files). The
reason that the 'Data' folder was limited to 32,767 files could be due to the
fact that each file is taking 2 entries on an average. This issue has put a
restriction on our application in that the user can create only 32,767 files.
For this, we have thought of the following solutions:-
i) In the Minimum System Requirements for using our application, we will
tell the user that he/she should have a computer running on Windows 2000 or
above which have the NTFS File System as it has no limit on the number of
file or subdirectories in a directory. However, we do not feel that this
solution would be feasible as the FAT32 File System is present on most
computers.
ii) We will programmatically create multiple folders under the 'Data'
folder. For example, within 'Data' folder we would have a folder named
something like "1-30000" in which the first 30000 files will be stored. Now
if the user creates the 30001th file, then a new folder named
"30001-60000"
will be automatically created under the 'Data' folder, which will store the
next 30000 files and so on. In this manner, our application will have
multiple folders containing 30000 files each. Thus, this would remove the
restriction on our application in that the user can create MORE THAN 32,767
FILES.
We would request you to kindly verify and suggest which of the solutions
discussed above is feasible and should be implemented.
We would look forward to your reply at the earliest.
Regards,
Remittag Team
--
Remittag Software Company Pvt. Ltd.