[Flow-tools] RE: Flow-tools Digest [Store data]

Wed Sep 19 14:16:43 EDT 2007

On Wed, 19 Sep 2007, Michael Graziano wrote:

> Caio (& everyone else) -
>
> My company currently uses a nifty combination of flow-tools
> (flow-report, flow-filter) and a modified FlowScan/JKFlow codebase for
> traffic monitoring.  It's a pretty extensive system (it does per-client
> and per-purpose (CoLo, Leased Lines, Internal/Infrastructure) traffic
> monitoring, as well as implementing abnormal traffic detection (overall
> network & per-client)).
>
> The FlowScan code is an ugly hack (cooking flows to cflow format before
> handing them off to be processed rather than hacking FlowScan &
> Friends), but it all comes together quite nicely.  If there's any
> interest I'm sure I can convince the powers that be to let me package it
> for release :)

Please don't tease us. :)

Charles

>
> Numbers-wise our system takes a lot of disk.  For our network
> (medium-sized ISP) 5 days of stored flows (for reporting) is about 48GB,
> RRDs for all our graphs are about 1.5GB, and misc. bandwidth billing
> data (handled by the same system) is about 30GB.
> It's also a CPU-Intensive system (A shiny new Dell 1950 takes about 2.5
> minutes to process a 5 minute window of data with FlowScan - Anyone got
> a multithreaded version of that kicking around? :)
>
>
>> -----Original Message-----
>> From: flow-tools-bounces at list.splintered.net [mailto:flow-tools-
>> bounces at list.splintered.net] On Behalf Of flow-tools-
>> request at list.splintered.net
>> Sent: Wednesday, September 19, 2007 12:07 PM
>> To: flow-tools at list.splintered.net
>> Subject: Flow-tools Digest, Vol 46, Issue 6
>>
>> Send Flow-tools mailing list submissions to
>> 	flow-tools at list.splintered.net
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> 	http://mailman.splintered.net/mailman/listinfo/flow-tools
>> or, via email, send a message with subject or body 'help' to
>> 	flow-tools-request at list.splintered.net
>>
>> You can reach the person managing the list at
>> 	flow-tools-owner at list.splintered.net
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Flow-tools digest..."
>>
>>
>> Today's Topics:
>>
>>    1. report type in flow-report (Caio Brentano)
>>    2. Support Netflow v9 and IPFIX (Roque Gagliano)
>>    3. Store data (Caio Brentano)
>>    4. Re: Store data (Dave Plonka)
>>    5. RES: [Flow-tools] Store data (Caio Brentano)
>>    6. Re: RES: [Flow-tools] Store data (Dave Plonka)
>>    7. Re: RES: [Flow-tools] Store data (Dave Plonka)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Tue, 18 Sep 2007 15:43:12 -0300
>> From: "Caio Brentano" <caio.b at terra.com.br>
>> Subject: [Flow-tools] report type in flow-report
>> To: <flow-tools at list.splintered.net>
>> Message-ID: <004501c7fa23$c559bb20$5101b0c8 at terrabr.corp.terra.com.br>
>> Content-Type: text/plain; charset="us-ascii"
>>
>> Hi all
>>
>>
>>
>>
>>
>> I'm trying to create some reports with flow-report. Can I create my
> own
>> "Report Type" for flow-report?
>>
>> For example: I need a report about "ip-source-port" + "ip-protocol".
>> Can I
>> create my own "Report Type" for it?
>>
>>
>>
>> I know that there is a report type with these information, but it has
>> some
>> informations that don't care for me, such as "ip-tos".
>>
>>
>>
>> Regards.
>>
>>
>>
>> --
>>
>> Caio Brentano dos Passos
>>
>>
>>
>>
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: http://mailman.splintered.net/pipermail/flow-
>> tools/attachments/20070918/d72435e7/attachment-0001.htm
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Wed, 19 Sep 2007 08:48:34 -0300
>> From: Roque Gagliano <rgaglian at antel.net.uy>
>> Subject: [Flow-tools] Support Netflow v9 and IPFIX
>> To: flow-tools at list.splintered.net
>> Message-ID: <1190202514.11053.46.camel at jessy.antel.net.uy>
>> Content-Type: text/plain; charset="us-ascii"
>>
>> Skipped content of type multipart/alternative-------------- next part
> -
>> -------------
>> A non-text attachment was scrubbed...
>> Name: not available
>> Type: application/pgp-signature
>> Size: 189 bytes
>> Desc: This is a digitally signed message part
>> Url : http://mailman.splintered.net/pipermail/flow-
>> tools/attachments/20070919/c9f2aa33/attachment-0001.bin
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Wed, 19 Sep 2007 11:52:23 -0300
>> From: "Caio Brentano" <caio.b at terra.com.br>
>> Subject: [Flow-tools] Store data
>> To: <flow-tools at list.splintered.net>
>> Message-ID: <008401c7facc$b0a5b270$5101b0c8 at terrabr.corp.terra.com.br>
>> Content-Type: text/plain; charset="us-ascii"
>>
>> Hi all
>>
>>
>>
>> I'm developing a web-based system to show reports and graphs of data
>> collected from flows.
>>
>>
>>
>> What do you suggest me to store this data? I developed a netowork
>> monitoring
>> system based on SNMP that data is stored in RRD.
>>
>> Is it ok for flow? Is there a better way?
>>
>>
>>
>> Regards
>>
>> --
>>
>> Caio Brentano
>>
>>
>>
>>
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: http://mailman.splintered.net/pipermail/flow-
>> tools/attachments/20070919/dc7962ad/attachment-0001.htm
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Wed, 19 Sep 2007 10:02:17 -0500
>> From: Dave Plonka <plonka at doit.wisc.edu>
>> Subject: Re: [Flow-tools] Store data
>> To: flow-tools at list.splintered.net
>> Message-ID: <20070919150217.GA20075 at doit.wisc.edu>
>> Content-Type: text/plain; charset=us-ascii
>>
>>
>> Hi Caio,
>>
>> On Wed, Sep 19, 2007 at 11:52:23AM -0300, Caio Brentano wrote:
>> <snip>
>>> What do you suggest me to store this data? I developed a netowork
>> monitoring
>>> system based on SNMP that data is stored in RRD.
>>>
>>> Is it ok for flow? Is there a better way?
>>
>> There are a number of FlowScan reports that digest raw flow data, in
>> flow-tools format or others, and populate RRD files.  These include
>> the reports supplied with FlowScan and others such as CUFlow.
>>
>> There are mailing lists and online docs for both.
>> Here's one place to start: http://net.doit.wisc.edu/~plonka/FlowScan/
>>
>> Dave
>>
>> P.S. Most of my reports write to RRD files for time-series graphing,
>> but some flow data, such as top talkers works better of course as
>> tabular data.  Thus some reports produce HTML tables.
>>
>> --
>> plonka at doit.wisc.edu  http://net.doit.wisc.edu/~plonka/  Madison, WI
>>
>>
>> ------------------------------
>>
>> Message: 5
>> Date: Wed, 19 Sep 2007 12:15:10 -0300
>> From: "Caio Brentano" <caio.b at terra.com.br>
>> Subject: RES: [Flow-tools] Store data
>> To: <plonka at doit.wisc.edu>,	<flow-tools at list.splintered.net>
>> Message-ID: <008f01c7facf$defbb2c0$5101b0c8 at terrabr.corp.terra.com.br>
>> Content-Type: text/plain;	charset="us-ascii"
>>
>> These are RRD graphs http://wwwstats.net.wisc.edu/ ?
>>
>> --
>> Caio Brentano
>>
>>
>>
>> ------------------------------
>>
>> Message: 6
>> Date: Wed, 19 Sep 2007 10:23:43 -0500
>> From: Dave Plonka <plonka at doit.wisc.edu>
>> Subject: Re: RES: [Flow-tools] Store data
>> To: flow-tools at list.splintered.net
>> Message-ID: <20070919152343.GD20075 at doit.wisc.edu>
>> Content-Type: text/plain; charset=us-ascii
>>
>> On Wed, Sep 19, 2007 at 12:15:10PM -0300, Caio Brentano wrote:
>>> These are RRD graphs http://wwwstats.net.wisc.edu/ ?
>>
>> Yes, of course.
>>
>> If this is new to you, perhaps you'd like to read the original paper:
>>
>>    http://www.usenix.org/events/lisa2000/plonka.html
>>
>> Some of the most popular 3rd party documentation I've seen for using
>> FlowScan is from these onlamp articles. E.g.:
>>
>>    http://www.onlamp.com/pub/a/bsd/2005/10/27/Big_Scary_Daemons.html
>>
>> A number of people use FlowScan, but use the CUFlow or other reports
>> instead of the original ones I wrote (CampusIO SubNetIO)...
>>
>> Since it has been a long time since a FlowScan release, you need to
>> patch it up by hand a bit to get it all working.  This is documented
>> in the link "Tips on configuring FlowScan with flow-tools." at
>> http://www.splintered.net/sw/flow-tools/ :
>>
>>    http://net.doit.wisc.edu/~plonka/list/flowscan/archive/1117.html
>>
>> Dave
>>
>> P.S. beyond that the mailing list archives have a lot of FAQs covered.
>>    http://lists.wiscnet.net/mailman/listinfo/flowscan/
>>
>> --
>> plonka at doit.wisc.edu  http://net.doit.wisc.edu/~plonka/  Madison, WI
>>
>>
>> ------------------------------
>>
>> Message: 7
>> Date: Wed, 19 Sep 2007 10:49:39 -0500
>> From: Dave Plonka <plonka at doit.wisc.edu>
>> Subject: Re: RES: [Flow-tools] Store data
>> To: flow-tools at list.splintered.net
>> Message-ID: <20070919154939.GB9783 at doit.wisc.edu>
>> Content-Type: text/plain; charset=us-ascii
>>
>> On Wed, Sep 19, 2007 at 10:23:43AM -0500, Dave Plonka wrote:
>> <snip>
>>> Some of the most popular 3rd party documentation I've seen for using
>>> FlowScan is from these onlamp articles. E.g.:
>>>
>>>    http://www.onlamp.com/pub/a/bsd/2005/10/27/Big_Scary_Daemons.html
>>
>> Actually this is the link I meant:
>>
>>    "Visualizing Network Traffic with Netflow and FlowScan"
>>    http://www.onlamp.com/pub/a/bsd/2005/09/15/Big_Scary_Daemons.html
>>
>> --
>> plonka at doit.wisc.edu  http://net.doit.wisc.edu/~plonka/  Madison, WI
>>
>>
>> ------------------------------
>>
>> _______________________________________________
>> Flow-tools mailing list
>> Flow-tools at list.splintered.net
>> http://mailman.splintered.net/mailman/listinfo/flow-tools
>>
>>
>> End of Flow-tools Digest, Vol 46, Issue 6
>> *****************************************
> _______________________________________________
> Flow-tools mailing list
> flow-tools at splintered.net
> http://mailman.splintered.net/mailman/listinfo/flow-tools
>