putting members into their own folder

Is there any way to put member's uploads into their own folder within cpanel, for example, when someone uploads a video, their videos will be in a separate folder belonging to them instead of all videos just being dumped into the files folder. I can't imagine having tens of thousands of videos like this.

So what I need is for each person to have their own folder in cpanel where all of their media and information is stored. This is the best way I know how to explain it.

Not without a lot of code rewriting.

It's not just videos. Photos, files, avatars ect all do the same thing.

cPanel is a server control panel and has nothing to do with Dolphin.

 I'm just talking about how the media is stored by Dolphin. Not just for cpanel.

Only the developers can explain why they did it the way they did it.  Maybe it had to do with older file systems.  To change it at this point would mean a major change that would also affect third party modules.  It would have to wait for Dolphin 8.

What we have discussed to make the files easier to identify is to add on the user ID on front of the hash perhaps.  Then a simple sort of the files would show everything by that member.

Maybe the developers didn't want countless thousands of folders.

Or even more; what if you have a site with a million users?  File systems do have limits on the number of directories although that  limit may be very high.  Plus, the number of directories would be multiply by the media types; if one kept the current method of photos stored under the photo module, files under the files module, etc.  The question I have is it really important to have each member with their own files directory?  What are the benefits of that over the current way?  What do you expect to gain by the change?

I just believe in precise file systems. I can't complain though because I don't even know how to do it. But I know that when businesses want to analyse data, they like every little thing to be in a separate folder. What if I needed to download a person's profile or compare it to someone else's stuff and they had tons of videos scattered inside of the video folder, same with photos. I'm just giving an example, but either way I think it should be organized better. 

I would rather have millions of videos in their own separate folder containing the media, pic, and data rather than having millions of videos in one gigantic folder.

 RE:

 I can't imagine any business person in their right mind that would want to rifle though thousands of folders to get the data they wanted.  Any sensible business person would more likely want to query a database to get the information they wanted.

The way media is managed within Dolphin is fairly efficient.  Storing all those multimedia files in their own folder might seem more logical to the human brain, but if you're a database driven web app that needs to efficiently manage files, it's not really that logical.  Individual folders wouldn't speed things up or make file management more efficient... more likely it would have the reverse effect.

If exporting profiles and all the associated multimedia is the goal, there are better methods.  I don't believe a big bloated file system would help anyone's cause.  I can't explain why, but there aren't any mods on the market that will export ALL profile information and associated multimedia.  Maybe some day there will be.

You guys are probably right and I am happy to hear your point of views. Thank you.

This is highly depended on many factors like file system in use. What make you say this? I am certain that its actually reverse, storing bazillion files in one folder does NOT make it efficient (not talking about human brains here) and after you have a REALLY LARGE number of files in one folder you start suffering some performance problems as the file system will have trouble seeking for files. But an average dolphin site does not need to worry about this.

Yes this is how I feel. So how can we make Dolphin scalable to a global scale? I just discovered Hadoop and big data and I am confident that sooner or later Dolphin will need to work on a large scale. I am starting to take being a computer scientist seriously and I am beginning to prepare for the future.

What would happen if someone built a different version of Dolphin and incorporated the ideas that arise in the Boonex forums into it? Would Boonex allow us to use it or would they forbid it?

Like a fork on Github.

There are more things to consider than just seeking files... privacy settings/viewing permissions... site backups.  I just don't see how a million files in a million folders can be more efficient than a million files in one folder... especially when it comes to backups. But as you say, the average Dolphin site does not need to worry about this... nor do I. I don't even know why I waste my time on topics like this... must have been bored.

Hadoop is not for storing files or things alike.

You can't really scale Dolphin too much in its default state. You will mostly have to rewrite bunch most of the stuff to scale very large to like clusters of servers. Unless you're at that stage you just need to focus more on user satisfaction and if you really want then get a developer to rewrite the directory structure for you.

lol why that has anything to do with directory structure, that's program logic.

What with backups? I think you need to take a nap or some tea..

 RE:

 First of all, I can't stand tea. We obviously have different opinions.  Mine is that retrieving one million files from one million folders, will be a lot slower than retrieving one million files from ONE folder. If you don't see how that might relate to backups, or any other file operations, forget I mentioned it.

Your opinion is obviously the exact opposite of mine.  I can live with that.

I did some research and it appeared like a fact to me....but yeah you can make your own opinion about how a computer works.

Btw I thought you hated coffee so I "guessed" it could be tea...

I don't see any real benefits in each member having their own folder ...

On one of my sites alone I would be looking at 14,000 extra folders :/

I am happy with the way things are!

For a min stop thinking like a human brain...

File system in use for this example is NTFS:

Case 1: Bazillion files in single folder. 

Like a rational database. Each folder stores an index for each file/folder. Now suppose you have 500,001 files in a single folder. And you want to read a file named "file_number_799.jpg" but you have to locate it first. You start seeking through the index of all those files in that folder and its gets slower cuz your index will defrag after so many edits when you add/remove/rename files and grow in sizes of gigs.

Case 2: Distribute the files in sub folder.

Now having 500,001 folders inside a folder will *not* improve performance at all. Cuz it will be same thing as having that many files. But you can structure this in sub folders in sub folders. Something like /year/month/ even better is folders created with file hash breakdown. So you have a folder and inside it another 20 folder, you open it and as the index is way too small you instantly get the folder you're seeking for, then inside that folder you have 50 folders and thats instant too, then you open one of the folder from those 50 and you have like 1000 files in it which will be easily to seek in the very small index than seeing 1 from 500,001 files, You don't think that? Isn't that logical?

P.S: These days processors and new SSDs are way too powerful for tasks like these, so as i said average site does *not* have to worry about it.

Ok, i have wasted lot more time on this topic than i estimated and i guess you guys did too...so lets move on and keep your "opinion" if you like 

NTFS is an outdated file system.  Microsoft promised a better file system but dumped it because they could not complete it before the mandated release date of the OS; which may have been Vista at the time.

Linux, on the other hand, being an open source OS and not constrained by the problems of MicroCrap, has several file systems to choose from and you can installed a journalled file system in linux.

Both systems, one directory for all users' photos, one directory for all users' videos, etc, and each user with their own directory containing just their files have merit.  We would have to have each side by side with the same number of users and files and perform performance tests to see which one performs better.  There is more than just the OS involved, we also have the MySQL server involved in the retrieval of the files as well.

 RE:

It pains me to comment on this again, but by my math, 500,000 files in 1 folder = 500,001 file system objects.  500,000 files in 500,000 folders = 1,000,000 file system objects.

By all means, correct me if I am wrong on this.

Its not about the "object" itself, its about finding those object in lots of objects. Each folder builds an index of files/folders inside it aka its child. Finding something in smaller index is easier than finding something in a big one.

So in your NTFS example, doesn't case #2 double the entries in the MFT?

In linux. the normal file system used to be ext3 which had a limit of 32000 subfolders. Thus it could not be done. Once the member base exceded 32000 members the limitation of the ext3 file system would create a problem.

Now most linux systems now use ext4 which no longer has that limit. It also uses a btree type file indexing system, so the number of files in a folder does not really matter much anymore either.

If ext3 was used then Prashank is correct. Performace drops as the number of files increase and that is due to the size of the read buffer when reading the directory. Multiple reads have to be performed to get the entire directory. That is no longer a problem in ext4

So if the system was to be split into folders for each member and the file system is ext3 then you would run into the subfolder limit. If left as it is then performance problems would begin to occure, but at least would still function.

With ext4 neither of those problems is a issue. So it no longer really matters. But there are still some hosts out there that are still using ext3, so switching to a subfolder system while ext3 is still in wide spread use is a bad idea. 

We should ask radefantasy.  He has a site with half a billion members.

Yep...you are correct i am just started to read more about the new file systems and they are build to expand "quite large". Win 8 server edition OS got new file system too "ReFS" and its 100x better than NTFS. But there is a something about ext4 too, there is a file/folder limit on it too defined by "inode number". Oh and you will run out of your logical (not only physical) HDD space long before that, so don't worry . Btw XFS seems like a cool dude to me, might use it on some storage server some day.

P.S. I never said to give each member its own folder, thats really will not help.

 LMAO, where is he...i miss him