I wanted to know how you guys cached your stats you get from LFSWorld. Do you use a file, a database, or keep it in memory?

What do you guys use? How do you implament it? Do you mind giving code examples?

I would love to make a stats program for LFS in PHP. But I would want for it to be used by eveyone. I want for it to be open source. I want for it to be readily available and easly understood. Time frame for release? One Month.

Quote from Dygear :

I would love to make a stats program for LFS in PHP. But I would want for it to be used by eveyone. I want for it to be open source. I want for it to be readily available and easly understood. Time frame for release? One Month.

There are already quite a few. You can parse nearly all the LFSWorld data look here: http://www.lfsforum.net/showthread.php?t=183

Quote from Dygear :

I like the idea of a file, but I also hate it. I like the idea, as you can use this across all platforms and there are no 'special' requirements like there are with databases. It is also relatively simple for the end user to understand. But you littler you file system with unwanted files, and you require write access.

A specific cache directory resolves this. If you actually think about it, the key lies in the key generation for the cache. (pun not intended )
If you generate the key based on data sent to LFSWorld then requests for the same data overwrite the old data when they are updated, thus keeping files to a minimal.

Quote from Dygear :

I would perfer a database setup, end users must have the database type that you specify. But its easy to maintain.

No easier than files, and way more complicated than you might expect. This is the current sticking point of PPF2. If you just want a really simple SQL cache (one where you can't fetch out of it by individual fields) then you can just use the SQL container for PEAR::Cache. However, if you want to be able to select fields individually, how do you go about that? Do you split the data in to tables thus requiring multiple inserts per update but sticking to normalization rules, or do you just slap it all in one table, but flying in the face of common database design practice given that it is highly dynamic data?
In addition, you need SQL schemas and queries specific to each datasource if you go the 'individual field' route. The payoff to this method is the full leverage of the power of the RDBMS you implement it on to filter and sort data.

Quote from Dygear :

With memory as soon the file exits, you have lost your information but can be used by eveyone.

Shared memory does not work on windows, hence it's a non starter for me.
*edit* Actually, it does work, but not widely deployed, so it's still a non starter*edit*

Quote from Dygear :

What do you guys use? How do you implament it? Do you mind giving code examples?

A full cache implementation sample would be a pretty big chunk of PHP. Especially for PPF2 because it uses a generic interface allowing for swapping between cache implementations. (i.e. file, sql, any others you care to code)
PPF2 also utilizes a fallback mechanism. The whole process goes something like...

• Generate cache key based on requested data
  
• Check cache for valid entry
  
• If one exists, return data else continue
  
• Request data from LFSWorld
  
• Decompress & check for errors
  
• If data has error, check cache for an old entry (if enabled)
  
• If old entry does not exist, return false
  
• Otherwise return data

Because of this, PPF also implements a request meta-data system that will allow you to see stuff like last update time of cache, if the data is fresh, if the data is fallback data, the query url, error log (if any), and some other useful data.

As you can imagine, this is a not-so-simple affair, so it's all hidden from the user of the class with a nice friendly interface if they don't want to deal with it. However, if they do, almost everything can be overriden either in settings for the interface itself, overriding a specific component or by writing your own interface.

In addition to all of that, everything is pretty independant. If you want to add a datasource, it just requires coding an api compatible class, instantiating it and passing it in as the provider class. You could use the same class with your own cache & request mechanism if you wanted to, just by calling the parsing function.

This post is getting pretty long now, so I'll wrap up my thread hi-jack

A list of components in PPF2

Providers (parsers for all datasources (ch, highlight, hl, hllog, hosts, pb, pst, teams & wr))
Format lookup tools (cars, tracks, rules, flags)
Main programmer interface (LFSWorldSimpleInterface & LFSWorldInterface)
Caches (atm there are file & sql caches, but others are easily added)
String processing (colour code processing, codepage translation & Join2LFS compatible link creation)
LFSWorld error checking function
LFSWorld decompression function

In addition to those, there will be usage docs & api docs. Possibly in a wiki format since phpDocumentor is turning out to be... well a bit naff as far as extended docs go.

There may be more stuff like a PB/PST request chain mechanism to make batch grabs of PB / PST data easier.

PPF was in development for 2 months, and PPF2 (extending & refining PPF1 architecture) has taken an additional month.

If you still want to have a bash though, don't let me stop you
I hope this insanely long post has provided some insight in to what may lie ahead

Quote from Dygear :

I wanted to know how you guys cached your stats you get from LFSWorld. Do you use a file, a database, or keep it in memory?

When I originally started libLFSW, it used flat files - basically a dump from LFSW. Unfortunately this wasnt particularly practical for more complex queries (i.e. list all hosts, and for each player list their PB for this track, if cached). For the v0.1.6 version I've moved to using SQLite. Since libLFSW is a library written in C, all the database handling code is built in, and since its SQLite its not too heavy.

Implementing a duplicate detection system, and providing a good and simple way to interact with the data is certainly the most difficult thing to do. Keep things to a minimum where possible, but dont over simplify.

Quote from Dygear :

With memory as soon the file exits, you have lost your information but can be used by eveyone.

I'm not sure I'm following this. Once your program finishes execution it should free all the memory its used - otherwise thats a memory leak and probably wont get freed...

Quote from Dygear :

I would love to make a stats program for LFS in PHP. But I would want for it to be used by eveyone. I want for it to be open source. I want for it to be readily available and easly understood. Time frame for release? One Month.

Now I dont want to be mean, but theres a LOT of PHP source out there (in comparison to anything else). Its probably not under an OSI compliant license at the moment, but I'm sure if you talked to the original authors they wouldnt mind. I'd consider most stuff on the forums to be public domain, but of course its a bit of a legal grey area.

To the_angry_angel, I meant every computer has memory (RAM) and thank you for the insight.

To Anarchi-H, wow, that's an awsome post, thanks for all of that information. I'll consider all of the things you have said. I think I'll go with a file system, as you can then use a generic parser for the information from LFS world and the one from your own file system. I have down SQL querys, and they have become second nature to me, but I am drawn to files for this project.

To nikimere, I have seen that, but its not a parser, nor is it code, but insight it is. Thanks anyway .