Still 'iffy' about 1 folder holding all photos

Not worried about storage so much but what if you have 5k members (theory, not reality) that have 50 pictures to display?

Calculate in the formats ranging from icon to the original that 250k number jumps to 1 million files in 1 folder. No? 

250,000 x 4 different file formats = 1,000,000 unique files in /photos/data/files

I've asked this scalability question before but no one seems too worried about it so I guess I'm just missing something.  Any comments to help make me feel better about this?

I will tell you that with an older version of Apache and Dolphin 6.1, I did hit a limit of like 32k files in one folder. You simply can not write the file to the folder once you hit the limit.

I've not load tested it with the latest server and dolphin software to see if that's still the case.

Ext2: Second extended file system
Introduced: January 1993
Directory contents: ?
File allocation: Inodes
Bad blocks: ?
Maximum volume size: 16TB
Maximum number of files: 10^18
Maximum file size: 2TB
Filenames: 255 characters, any byte except \0 and /.
Attributes: ?
Permission: POSIX
Transparent compression: Yes (optional)
Transparent encryption: No
Journaling: No
OS support: Native support under Linux and the BSDs. Read-write support in Windows 2000 and newer with a 3rd-party driver, MacOS X.

Here are the other major FSes and how they stack up:

Ext3: Third extended file system
Introduced: November 2001 (Linux 2.4.15)
Directory contents: Table, tree
File allocation: Bitmap, table
Bad blocks: Table
Maximum volume size: 2TB (1KB block) to 16TB (4KB block)
Maximum number of files: number of bytes in volume/2^13.
Maximum file size: 16GB (1KB block) to 2TB (4KB block)
Filenames: 255 bytes long, all bytes except \0 and /.
Attributes: no access time logging, append-only, synchronous write, no-dump, h-tree, immutable, journal, secure-delete, top, allow-undelete.
Permission: POSIX, ACLs and arbitrary permissions (Linux 2.6 and later.)
Transparent compression: no
Transparent encryption: no, provided by the OS instead.
Journal: metadata and file contents
OS support: Native support in Linux and the BSDs, read-write support in Windows 2000 and newer via a 3rd-party driver.

ReiserFS 3.x
Introduced: 2001 (Linux 2.4.1)
Directory contents: B+ tree
File allocation: Bitmap
Bad blocks: ?
Maximum volume size: 16TB
Maximum number of files: 4,294,967,296
Maximum file size: 8TB
Filenames: 255 bytes, all bytes except \0 and /.
Attributes: modification time, metadata change, access time. Tail packing.
Permissions: POSIX, ACLs, and arbitrary security attributes
Transparent compression: No
Transparent encryption: No
Journal: Metadata or block
OS support: Linux (2.4.1 and newer, 2.6.8 or newer for block journal.) Very limited read-only access via 3rd-party programs in Windows.

sweet. I feel better now. I meant I had problems at 65k folders within a folder using 6.1x, not 32k files in a folder. Not sure where I came up with that number. lol. But you answered what I wanted to know, how many files in a folder.

I'm making you my official Valentine's Day geek girl this year.

Thank you.

-g

You are welcomed.  At a command prompt on your system, type df -T to see the file type in use.

I'd be surprised to find someone running either Ext2 or ReiserFS (not counting political reasons). Ext4 is the default FS for current enterprise-level Linux distributions (from Ext3), and XFS will be the new default in RHEL 7 and its clones.

I would still be happier if the members media was in a folder under their ID.

Sometimes media is missed when the member leaves or is deleted.

Any ideas on this?

I was thinking the same thing, if a member made 500 video comments or uploaded 300 videos then these files are orphaned, taking up unnecessary space. 

Yes, I can see how this would make it hard to know which files to remove.

A solution that might be the easiest is to look at the hash that is created for the file and append the user ID to the hash;

xxxxxxxxxxxx-user01.ext.

Then one could use a simple script to check.  Or apply the user id to the beginning of the hash to make it easy to sort and delete using ftp client.

Dolphin uses the main files uploader for all files; so I am unsure of how to proceed.

However, it should also not be that difficult to modify the write routine for the files to check if the user directory exists, then write to it.  Currently Dolphin puts photos under the photo module, files under files, and videos under flash.

If there is not a problem with the file system having 10 thousand or a million directories, then one could have the directory structure:

Users

---UserID (where UserID is either just the ID or UserXX; however one might wish)

------Photos

------Files

------Videos

etc for each type of file.  I assume that modules work off of the default Dolphin files uploader so that it would be backwards compatible; if not, then that could be an issue.  That directory structure would make it very easy to monitor and delete orphan files.  However, removing the User should remove the User's directory and all subdirectories so that orphan files should not be an issue.

I'm still interested in how this would work.

I think there's 2 blocks of code that control where the files are stored.  IDK if simply adding the member ID to the string that creates the path will work or not.