madclarinet (madclarinet) wrote,
madclarinet
madclarinet

Databases

I'm trying to parse some information from months of web access logs. The only practical way of doing this is by stuffing all the logs into a database, deleting what I don't need etc etc etc.

Luckily I have a server which I use for testing etc which is ideal to hold the db. Mysql is already installed and quite quickly I set up the 'bones' of it. First, on my windows pc I removed all the '[' and ']' from the logs which took quite a while. Uploading the files took a while but the import went perfectly (thats Solaris for you, excellent memory/disk management). So, now I have a database with 3,910,791 records in which comes to about 1Gb in size.

After keeping a copy of this table I set about removing what I don't need - so far I've got it down to 50,751 records with more still to do. Once I bash it down more then I'll be able to get the task done I want to - then its plain sailing.

If this hasn't tired me out enough so I get decent nights sleep then I'm not sure what will.
Subscribe
  • Post a new comment

    Error

    default userpic

    Your IP address will be recorded 

  • 0 comments