*SPAM* [Flow-tools] Trouble processing too many flows (8640 5-minute flow in a month)

[Dawn Keenan]

jay alvarez wrote:
> Hi,
> 
> After running flow-capture beginning december 1, I looked into the flows 
> directory and it has tremendously grown into 16Gig in size.
> As of now it contains 6005 pieces of 5-minute flows. My goal is to run 
> "flow-cat all_December_flows  | flow-stat -f 8", however I'm having a 
> difficulty when executing this command.

Flow-cat's output is generally larger than its input because of the 
change in format.

Rather than keep all your flow files in the same directory, you may want 
to set up a date-based directory structure.  For example, you could 
store your files for December 21 in the directory 
/var/netflow/ft/all/2006/2006-12/2006-12-21/.  Since you can use a 
directory (all files under the directory will be read) or a file as 
input, that would avoid the shell expansion limit you've been hitting. 
You may run into a memory limit depending on your volume of data, or it 
may work just fine.

--
Dawn Keenan
Information Systems and Technology, University of Waterloo