[Flow-tools] linkme script modification for nested date- based storage (Re: Trouble processing too many flows)

[jay alvarez]

----- Original Message ----
From: Ed Ravin <eravin at panix.com>
To: jay alvarez <netfl0wb0y at yahoo.com>
Cc: flow-tools at list.splintered.net
Sent: Saturday, December 23, 2006 2:33:26 AM
Subject: Re: [Flow-tools] Trouble processing too many flows (8640 5-minute flow in a month)

>   As the other poster suggested, you need to switch to a date-based
>   directory format.  See the flow-capture man page for the details -
>   it will do it automatically.  Then you can do things like
>   
>      flow-cat /var/netflow/ft/2006/2006-12/
>   
>   To get all of December 2006 - flow-cat is smart enough to follow
>   the directory tree recursively.


I got the script below from: http://www.dynamicnetworks.us/netflow/5-configure-flow-tools.html

#!/usr/bin/perl
$base = "/var/netflow/ft";
if ($ARGV[0] =~ /.*[\/]*(ft-v05[^\/]*$)/) {
 $fileName = $1;
 } else {
 print "Must specify file\n";
 exit 1;
 }
unless ( symlink("$base/all/$fileName","$base/$fileName") ) {
 print "Unable to create symbolic link: $base/$fileName\n";
 exit 1;
 }


As far as I can understand, this is used to avoid having flowscan to delete the each flows after processing it. It instead deletes the symbolic link.

Right now all my flows are in a single directory, and I want to implement nesting level as you have suggested. My problem is i'm guessing that the script above will no longer work. Are you using the a linkme script for nesting storage configuratoin? Can you show it to me? :-)


The man page for nesting flow-captured flows looks like this:

 -3    YYYY/YYYY-MM/YYYY-MM-DD/flow-file
 -2    YYYY-MM/YYYY-MM-DD/flow-file
  -1    YYYY-MM-DD/flow-file
   0    flow-file
   1    YYYY/flow-file
   2    YYYY/YYYY-MM/flow-file
   3    YYYY/YYYY-MM/YYYY-MM-DD/flow-file

I guess I need to use "2" or "3" perhaps, and I need to create 2 linkme scripts to support both, only that I'm not good in this. Btw, what's the difference between -3 and 3 ?


>   Blech.  You can do this in a much simpler fashion using find and xargs:

>        find /var/netflow/ft/all/ -type f -name "vt-v05.2006-12-*" -print |
>           xargs flow-cat > dec2006.flow

My problem is that after running my looping script or xargs which you have suggested and writing the output to a single flow file, I got an error when running flow-cat:

flow-cat: fstat(dec0): Value too large for defined data type

It seems like the file is too big (20G) for it to process at once.. 








__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.splintered.net/pipermail/flow-tools/attachments/20061222/f401360c/attachment.htm