How to Handle 1/2 million files in XP

[tstolze]

I have a Windows XP 32 bit system that is dedicated to updating my weather website, archiving my webcam images, creating a daily time lapse from previous images and acting as our home document server. Core 2 Quad with 2 gigs of ram. CPU usage averages 25%....

I have 2 folders on a dedicated drive with over 500,000 files in them each. These are archives of my outdoor webcams, the reason I am keeping so many is to attempt a year long time lapse.

I have tried some "tweaks" found online to help speed things up, but feel like I am missing something, as I usually give up when trying to explore these 2 folders due to very long load times. This system is headless and typically the files are accessed by another system, although it is still a very drawn out process via remote desktop.

Thought I would find some help here at Satguys....I do have enough storage on an Unraid server to transfer and empty the drive if needed.

Thanks!

 

[StevenD]

I would use command line tools to further divide them into folders, like monthly or weekly.

 

[tstolze]

I would use command line tools to further divide them into folders, like monthly or weekly.

Unfortunately the software I use to automatically make the time lapse only looks into 1 folder for the files, which are named by year, month, day, hour , minute.

 

[mike123abc]

One solution I would suggest is an SSD.

 

[LostBoyinVA]

I understand the software only saves to one directory, but is there a reason you need to keep all the files in One Directory?

If you manually archive the files to sub folders by Week or Month and that would make the base directory much more usable.

 

[tstolze]

I understand the software only saves to one directory, but is there a reason you need to keep all the files in One Directory?

If you manually archive the files to sub folders by Week or Month and that would make the base directory much more usable.

The software which produces the time lapse only looks at the first layer in the directory. Thinking maybe archive them monthly, produce a time lapse from each month, then combine the video in Adobe.

 

[John Kotches]

The software which produces the time lapse only looks at the first layer in the directory. Thinking maybe archive them monthly, produce a time lapse from each month, then combine the video in Adobe.

can you change the frequency of file rolling from whatever it currently is to daily? best guess is that you are rolling at 5 minute intervals.



Sent from my MB855 using Tapatalk

 

[Pepper]

With Windows your only hope for relief is a smaller number of files per folder. NTFS has issues with such a large number of files and for sure don't try that on FAT.

It could be as simple as changing "YYMMDDHHMM" to "YY\MM\DD\HHMM" to get a workable folder structure. But you probably don't have access to any of the source code or the original developer to suggest/make such a change do you?

 

[tstolze]

can you change the frequency of file rolling from whatever it currently is to daily? best guess is that you are rolling at 5 minute intervals.
Sent from my MB855 using Tapatalk

Images are saved each minute, I guess if I changed it to every two minutes it would cut the number of files in half. Things worked decent until recently when the files exceeded 500,000.

With Windows your only hope for relief is a smaller number of files per folder. NTFS has issues with such a large number of files and for sure don't try that on FAT.

It could be as simple as changing "YYMMDDHHMM" to "YY\MM\DD\HHMM" to get a workable folder structure. But you probably don't have access to any of the source code or the original developer to suggest/make such a change do you?

No luck on source code, doesn't look like anything has been done on the programs used to archive images or produce the time lapse in several years.

I was hoping that maybe I could format the drive some other way to make things work like I would like....

Appreciate the help so far.

 

[FaT Air]

Disable the automatic display of thumbnails in certain folders Solution 4: Delete THUMBS.DB
Annoyances.org - Disable the automatic display of thumbnails in certain folders Did the job for me

 

[harshness]

I have 2 folders on a dedicated drive with over 500,000 files in them each. These are archives of my outdoor webcams, the reason I am keeping so many is to attempt a year long time lapse.

Did you stop to think how many hours the finished video was going to be??? At 30fps, that works out to 4.63 hours worth of video. I sincerely doubt anyone is going to sit for that.

 

[John Kotches]

Images are saved each minute, I guess if I changed it to every two minutes it would cut the number of files in half. Things worked decent until recently when the files exceeded 500,000.

You don't need to cut the number of files in half, you need to drop by ask order of magnitude (a factor of 10 or more)

this will make a significant performance difference at the file system level.



Sent from my MB855 using Tapatalk

 

[tstolze]

Did you stop to think how many hours the finished video was going to be??? At 30fps, that works out to 4.63 hours worth of video. I sincerely doubt anyone is going to sit for that.

I can specify how many frames each image is used, if I wanted to use all images from the year. I was thinking more about noon each day or maybe each sunset, as the software allows me to specify certain times or sunrise/sunset.

 

[tstolze]

You don't need to cut the number of files in half, you need to drop by ask order of magnitude (a factor of 10 or more)

this will make a significant performance difference at the file system level.

Sent from my MB855 using Tapatalk

It was bearable until recently when the numbers approached 500,000.

With only 50,000 images it would open very quick.

 

[tstolze]

Disable the automatic display of thumbnails in certain folders Solution 4: Delete THUMBS.DB
Annoyances.org - Disable the automatic display of thumbnails in certain folders Did the job for me

I have the whole drive open as a list to minimize the effects of thumbnails.... I will check on the link tonight to see if it has anything that would help.