madclarinet (madclarinet) wrote,


I'm trying to parse some information from months of web access logs. The only practical way of doing this is by stuffing all the logs into a database, deleting what I don't need etc etc etc.

Luckily I have a server which I use for testing etc which is ideal to hold the db. Mysql is already installed and quite quickly I set up the 'bones' of it. First, on my windows pc I removed all the '[' and ']' from the logs which took quite a while. Uploading the files took a while but the import went perfectly (thats Solaris for you, excellent memory/disk management). So, now I have a database with 3,910,791 records in which comes to about 1Gb in size.

After keeping a copy of this table I set about removing what I don't need - so far I've got it down to 50,751 records with more still to do. Once I bash it down more then I'll be able to get the task done I want to - then its plain sailing.

If this hasn't tired me out enough so I get decent nights sleep then I'm not sure what will.
  • Post a new comment


    default userpic

    Your IP address will be recorded