Slow Import from Web

[gtg430i]

Hello - I have a question. I have a code that imports data from several websites. The code takes at least 6 hours to run and about 700,000 records are imported with each run. I then take the data and store it in an access database. All data and stored queries are then deleted before the next run. I usually run the program two to three times a week.

Now after using the program and collecting the data for three month, I noticed that the run is getting much slower and after couple of hours or so all icons, folders, and files on my desktop disappear.

Any suggestion why this is happening? Anything related to my computer processor?

Thanks

[p45cal]

I noticed that the run is getting much slower <snip> Any suggestion why this is happening? Anything related to my computer processor?

It is unlikely that the materials in your processor are degrading enough to cause this effect.
However, if you were to post some code that you're using to import that data, we could perhaps have a clue.

[snb]

I fear you not only import data form those websites but pictures/objects/formatting as well.
It all depends on the method you use to import from those websites.

[SamT]

Run Ccleaner (Ccleaner) after each data run.

Program with TCP/IP

[jwise]

You may have a problem with disk fragmentation. When you have large files, these files tend to be fragmented. Over time, fragmentation increases the average amount of time it takes to access the data (because of disk seeks). There are probably utilities that can tell you how fragmented your data is. The way Windows assigns disk storage accelerates fragmentation. This is remedied by regular defragmentation. If it is an SSD, you can scratch this idea, and moving to an SSD would undoubtedly cure it.

Another issue is Windows memory management, i.e. Windows is a notoriously poor manager of storage. This is why many people re-boot which clears the storage management issue. Program memory usage is dynamic and constantly changing. If memory requirements go beyond a certain level which is determined by the amount of real storage available at the time, then the app can begin to page. This can drastically alter execution on any virtual memory operating system. This problem is more difficult to diagnose, and I have no clue how to do it on windows. On other systems, utilities indicate paging rates in real time. Windows may have this, too.

[gtg430i]

Hi all - My apologies for the delayed response. I have downloaded Ccleaner as suggested from SamT. It actually solved the problem and the code is running faster.

Thank you all for your help and suggestions.