The majority of the time I need to analyze logs across multiple servers, I use logstash. Sometimes though I want to aggregate the actual files on the server and go through them myself with awk and grep. For doing that, I use two tools.
- In my bash config, I have a function called access_concat that reads out regular and gzipped access logs.
access_concat(){ find $1 -name "acc*" -not -name "*.gz" -exec cat '{}' \; find $1 -name "acc*" -name "*.gz" -exec zcat '{}' \; }I can pass a path that log files are stored in and it will search them to find the files I actually want.
- Dancer’s Shell (or DSH) makes it easy for me to run a command across multiple servers.
Combining these two, I can run: dsh -M -c -g prd-wp 'access_concat /logs >> ~/oct22.logs' to concatenate all of the log files that exist on the server today. I then just need to scp down oct22.logs and I can easily run my analysis locally.
Note that to do this, you need to configure dsh so that the servers you want to access are in the prd-wp group (or better yet, the logical name for whatever you are working on).