[Flow-tools] RE: Flow-tools Digest [Store data]
Michael Graziano
mgraziano at invision.net
Wed Sep 19 12:25:36 EDT 2007
Caio (& everyone else) -
My company currently uses a nifty combination of flow-tools
(flow-report, flow-filter) and a modified FlowScan/JKFlow codebase for
traffic monitoring. It's a pretty extensive system (it does per-client
and per-purpose (CoLo, Leased Lines, Internal/Infrastructure) traffic
monitoring, as well as implementing abnormal traffic detection (overall
network & per-client)).
The FlowScan code is an ugly hack (cooking flows to cflow format before
handing them off to be processed rather than hacking FlowScan &
Friends), but it all comes together quite nicely. If there's any
interest I'm sure I can convince the powers that be to let me package it
for release :)
Numbers-wise our system takes a lot of disk. For our network
(medium-sized ISP) 5 days of stored flows (for reporting) is about 48GB,
RRDs for all our graphs are about 1.5GB, and misc. bandwidth billing
data (handled by the same system) is about 30GB.
It's also a CPU-Intensive system (A shiny new Dell 1950 takes about 2.5
minutes to process a 5 minute window of data with FlowScan - Anyone got
a multithreaded version of that kicking around? :)
> -----Original Message-----
> From: flow-tools-bounces at list.splintered.net [mailto:flow-tools-
> bounces at list.splintered.net] On Behalf Of flow-tools-
> request at list.splintered.net
> Sent: Wednesday, September 19, 2007 12:07 PM
> To: flow-tools at list.splintered.net
> Subject: Flow-tools Digest, Vol 46, Issue 6
>
> Send Flow-tools mailing list submissions to
> flow-tools at list.splintered.net
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.splintered.net/mailman/listinfo/flow-tools
> or, via email, send a message with subject or body 'help' to
> flow-tools-request at list.splintered.net
>
> You can reach the person managing the list at
> flow-tools-owner at list.splintered.net
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Flow-tools digest..."
>
>
> Today's Topics:
>
> 1. report type in flow-report (Caio Brentano)
> 2. Support Netflow v9 and IPFIX (Roque Gagliano)
> 3. Store data (Caio Brentano)
> 4. Re: Store data (Dave Plonka)
> 5. RES: [Flow-tools] Store data (Caio Brentano)
> 6. Re: RES: [Flow-tools] Store data (Dave Plonka)
> 7. Re: RES: [Flow-tools] Store data (Dave Plonka)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 18 Sep 2007 15:43:12 -0300
> From: "Caio Brentano" <caio.b at terra.com.br>
> Subject: [Flow-tools] report type in flow-report
> To: <flow-tools at list.splintered.net>
> Message-ID: <004501c7fa23$c559bb20$5101b0c8 at terrabr.corp.terra.com.br>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi all
>
>
>
>
>
> I'm trying to create some reports with flow-report. Can I create my
own
> "Report Type" for flow-report?
>
> For example: I need a report about "ip-source-port" + "ip-protocol".
> Can I
> create my own "Report Type" for it?
>
>
>
> I know that there is a report type with these information, but it has
> some
> informations that don't care for me, such as "ip-tos".
>
>
>
> Regards.
>
>
>
> --
>
> Caio Brentano dos Passos
>
>
>
>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://mailman.splintered.net/pipermail/flow-
> tools/attachments/20070918/d72435e7/attachment-0001.htm
>
> ------------------------------
>
> Message: 2
> Date: Wed, 19 Sep 2007 08:48:34 -0300
> From: Roque Gagliano <rgaglian at antel.net.uy>
> Subject: [Flow-tools] Support Netflow v9 and IPFIX
> To: flow-tools at list.splintered.net
> Message-ID: <1190202514.11053.46.camel at jessy.antel.net.uy>
> Content-Type: text/plain; charset="us-ascii"
>
> Skipped content of type multipart/alternative-------------- next part
-
> -------------
> A non-text attachment was scrubbed...
> Name: not available
> Type: application/pgp-signature
> Size: 189 bytes
> Desc: This is a digitally signed message part
> Url : http://mailman.splintered.net/pipermail/flow-
> tools/attachments/20070919/c9f2aa33/attachment-0001.bin
>
> ------------------------------
>
> Message: 3
> Date: Wed, 19 Sep 2007 11:52:23 -0300
> From: "Caio Brentano" <caio.b at terra.com.br>
> Subject: [Flow-tools] Store data
> To: <flow-tools at list.splintered.net>
> Message-ID: <008401c7facc$b0a5b270$5101b0c8 at terrabr.corp.terra.com.br>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi all
>
>
>
> I'm developing a web-based system to show reports and graphs of data
> collected from flows.
>
>
>
> What do you suggest me to store this data? I developed a netowork
> monitoring
> system based on SNMP that data is stored in RRD.
>
> Is it ok for flow? Is there a better way?
>
>
>
> Regards
>
> --
>
> Caio Brentano
>
>
>
>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://mailman.splintered.net/pipermail/flow-
> tools/attachments/20070919/dc7962ad/attachment-0001.htm
>
> ------------------------------
>
> Message: 4
> Date: Wed, 19 Sep 2007 10:02:17 -0500
> From: Dave Plonka <plonka at doit.wisc.edu>
> Subject: Re: [Flow-tools] Store data
> To: flow-tools at list.splintered.net
> Message-ID: <20070919150217.GA20075 at doit.wisc.edu>
> Content-Type: text/plain; charset=us-ascii
>
>
> Hi Caio,
>
> On Wed, Sep 19, 2007 at 11:52:23AM -0300, Caio Brentano wrote:
> <snip>
> > What do you suggest me to store this data? I developed a netowork
> monitoring
> > system based on SNMP that data is stored in RRD.
> >
> > Is it ok for flow? Is there a better way?
>
> There are a number of FlowScan reports that digest raw flow data, in
> flow-tools format or others, and populate RRD files. These include
> the reports supplied with FlowScan and others such as CUFlow.
>
> There are mailing lists and online docs for both.
> Here's one place to start: http://net.doit.wisc.edu/~plonka/FlowScan/
>
> Dave
>
> P.S. Most of my reports write to RRD files for time-series graphing,
> but some flow data, such as top talkers works better of course as
> tabular data. Thus some reports produce HTML tables.
>
> --
> plonka at doit.wisc.edu http://net.doit.wisc.edu/~plonka/ Madison, WI
>
>
> ------------------------------
>
> Message: 5
> Date: Wed, 19 Sep 2007 12:15:10 -0300
> From: "Caio Brentano" <caio.b at terra.com.br>
> Subject: RES: [Flow-tools] Store data
> To: <plonka at doit.wisc.edu>, <flow-tools at list.splintered.net>
> Message-ID: <008f01c7facf$defbb2c0$5101b0c8 at terrabr.corp.terra.com.br>
> Content-Type: text/plain; charset="us-ascii"
>
> These are RRD graphs http://wwwstats.net.wisc.edu/ ?
>
> --
> Caio Brentano
>
>
>
> ------------------------------
>
> Message: 6
> Date: Wed, 19 Sep 2007 10:23:43 -0500
> From: Dave Plonka <plonka at doit.wisc.edu>
> Subject: Re: RES: [Flow-tools] Store data
> To: flow-tools at list.splintered.net
> Message-ID: <20070919152343.GD20075 at doit.wisc.edu>
> Content-Type: text/plain; charset=us-ascii
>
> On Wed, Sep 19, 2007 at 12:15:10PM -0300, Caio Brentano wrote:
> > These are RRD graphs http://wwwstats.net.wisc.edu/ ?
>
> Yes, of course.
>
> If this is new to you, perhaps you'd like to read the original paper:
>
> http://www.usenix.org/events/lisa2000/plonka.html
>
> Some of the most popular 3rd party documentation I've seen for using
> FlowScan is from these onlamp articles. E.g.:
>
> http://www.onlamp.com/pub/a/bsd/2005/10/27/Big_Scary_Daemons.html
>
> A number of people use FlowScan, but use the CUFlow or other reports
> instead of the original ones I wrote (CampusIO SubNetIO)...
>
> Since it has been a long time since a FlowScan release, you need to
> patch it up by hand a bit to get it all working. This is documented
> in the link "Tips on configuring FlowScan with flow-tools." at
> http://www.splintered.net/sw/flow-tools/ :
>
> http://net.doit.wisc.edu/~plonka/list/flowscan/archive/1117.html
>
> Dave
>
> P.S. beyond that the mailing list archives have a lot of FAQs covered.
> http://lists.wiscnet.net/mailman/listinfo/flowscan/
>
> --
> plonka at doit.wisc.edu http://net.doit.wisc.edu/~plonka/ Madison, WI
>
>
> ------------------------------
>
> Message: 7
> Date: Wed, 19 Sep 2007 10:49:39 -0500
> From: Dave Plonka <plonka at doit.wisc.edu>
> Subject: Re: RES: [Flow-tools] Store data
> To: flow-tools at list.splintered.net
> Message-ID: <20070919154939.GB9783 at doit.wisc.edu>
> Content-Type: text/plain; charset=us-ascii
>
> On Wed, Sep 19, 2007 at 10:23:43AM -0500, Dave Plonka wrote:
> <snip>
> > Some of the most popular 3rd party documentation I've seen for using
> > FlowScan is from these onlamp articles. E.g.:
> >
> > http://www.onlamp.com/pub/a/bsd/2005/10/27/Big_Scary_Daemons.html
>
> Actually this is the link I meant:
>
> "Visualizing Network Traffic with Netflow and FlowScan"
> http://www.onlamp.com/pub/a/bsd/2005/09/15/Big_Scary_Daemons.html
>
> --
> plonka at doit.wisc.edu http://net.doit.wisc.edu/~plonka/ Madison, WI
>
>
> ------------------------------
>
> _______________________________________________
> Flow-tools mailing list
> Flow-tools at list.splintered.net
> http://mailman.splintered.net/mailman/listinfo/flow-tools
>
>
> End of Flow-tools Digest, Vol 46, Issue 6
> *****************************************
More information about the Flow-tools
mailing list