best way to cache RSS data

Hi,

juts to be sure...

after I parsed some RSS feed and want to cache this data, what is the best way to do that:

1. parse the xml into an array
2. serialize the data
3. write to a file

for reading a file:
read the file to get the string  and unserialize to get (again) the array?

It depends in the size of data you want to cache. That looks ok for some data, but for more feeds you probably need a database :)

sure its possible that there are 100 or more files but each file is not bigger the 30-50KB

I guess that's fine.

BTW magpie has caching embedded.

but magpie doesn't parse all youtube data ;)

I use simpleXML for parsing

does cached files overwrite itself or a new file is added everytime the old one expires thereby taking up storage space so there is a need to delete them from time to time?

the name of the cached file is still the same, script looks after the last modfied date, will say there is only one file for each url, yes right if you have 100.000 urls there are 100.000 cache files :)

BTW if this is going to be user triggered, be sure to use the flock function ;)

I read about flock before, but these file are created by the class (md5(url))

I was looking two days for the best method to cache contents. For plain data I'd use file_get_contents() and file_put_contents(). For caching loops I can recommend SQLite. It has the best performance to read out big data in a short period of time. I posted the full story at http://phpperformance.de/ausgaben-in-cache-speichern/ (last comment). Unfortunately it's in German.

You mean SQLite is faster than plain text files?

How many records you used for your test?

The MySQL table consists of 1.000.000 records but I did only read out 20.000 of them. I had a performance boost of ~500%.

Makes sense since sqllite is file based too.

At the moment that I started this thread the directory with cached file becomes bigger than 100.000 files.

this amount was not a problem, later I decided to create a cronjob to remove files older than a week. This job is holding the amount of files near the 100.000 files and I'm happy with that.

but anyway using sqllite for caching sound interesting.

Tim do you checked also the filsize for the database is this bigger smaller than for the file based cache?

how about the server load while reading some file/record?

Maybe it would be better if you check the creation time of the caching file before every usage. Then consider deleting it if it's older than one week, refresh the RSS newsfeed and rewrite the cache or use the data of the cache.

I haven't yet used my caching method in production use but I don't think the server load would be much higher than by using ordinary files. As I've already said, this method is high recommendable if you need to cache loops because you can't jump between records with fseek()/ftell()/fgets() as fast as you can with SQLite . I didn't find any better solution being based on files. Another solution would be to create a second MySQL server which runs at another port instead of using SQLite. Does MySQL have a better performance than SQLite?

Of course mysql has better performance. SQLite is good for small databases, and besides that I don't know how it will handle many concurrent connections (while the file system and mysql will be ok with this)