From bigserpent at gmail.com Tue May 4 01:51:10 2010 From: bigserpent at gmail.com (Jacky Chan) Date: Tue May 4 01:51:14 2010 Subject: [Flow-tools] Flow-stat vs MRTG Message-ID: Dear Sir, I have flow-tools 0.68 running on a Fedora Core 11 workstation and the system collecting flow-data from Juniper and Cisco routers. I tried obtain the link utilization from the flow-stat output but there is big different when compared to MRTG reading. Example-1 I have a GE link from my Juniper router to INTERNET upstream-1 >From MRTG, the average input / output speed @ 21:00 are 110Mbps/ 136Mbps. >From Flow-stat, the input/ output speed @21:00 are 292.6613Kbps/ 323.1716Kbps. Example-2 I have a GE link from my Cisco router to INTERNET upstream-2 >From MRTG, the average input / output speed @ 21:00 are 24Mbps/ 96Mbps. >From Flow-stat, the input/ output speed @21:00 are 225.0744Kbps/ 1462.7255Kbps. Here are the commands I used to obtain the link utilization from the flow-data. I did something wrong or misused the flow-stat application? flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e JUNIPER-IP -i116 | flow-stat flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e JUNIPER-IP -I116 | flow-stat flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e CISCO-IP -i13 | flow-stat flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e CISCO-IP -I13 | flow-stat -- Jacky :) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.splintered.net/pipermail/flow-tools/attachments/20100504/528e9cfb/attachment.htm From jloiacon at csc.com Tue May 4 09:08:31 2010 From: jloiacon at csc.com (Joe Loiacono) Date: Tue May 4 09:08:38 2010 Subject: [Flow-tools] Flow-stat vs MRTG In-Reply-To: References: Message-ID: When you apply flow-stat to a single ft file, you're getting the average rate over the length of time associated with the file. My ft files are typically 15 minutes long. If we assume yours are 15 minutes also, you are comparing a 15 minute average with an MRTG 5-minute SNMP sample of all bytes (including IP and TCP headers.) Looking at sampling as a possibility, here are the 'multipliers': 110/0.293 = 375 136/0.323 = 421 24/0.225 = 107 96/1.462 = 66 They're not consistent. But if you create an MRTG number from the average of the three readings that make up the 15 minute netflow period, you might find a consistent multiplier. But - you could first just check to see if you're sampling on the Juniper :-) By the way - have you checked out FlowViewer as a web interface to flow-tools? http://ensight.eos.nasa.gov/FlowViewer/ Joe From: Jacky Chan To: flow-tools@list.splintered.net Date: 05/04/2010 01:52 AM Subject: [Flow-tools] Flow-stat vs MRTG Dear Sir, I have flow-tools 0.68 running on a Fedora Core 11 workstation and the system collecting flow-data from Juniper and Cisco routers. I tried obtain the link utilization from the flow-stat output but there is big different when compared to MRTG reading. Example-1 I have a GE link from my Juniper router to INTERNET upstream-1 >From MRTG, the average input / output speed @ 21:00 are 110Mbps/ 136Mbps. >From Flow-stat, the input/ output speed @21:00 are 292.6613Kbps/ 323.1716Kbps. Example-2 I have a GE link from my Cisco router to INTERNET upstream-2 >From MRTG, the average input / output speed @ 21:00 are 24Mbps/ 96Mbps. >From Flow-stat, the input/ output speed @21:00 are 225.0744Kbps/ 1462.7255Kbps. Here are the commands I used to obtain the link utilization from the flow-data. I did something wrong or misused the flow-stat application? flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e JUNIPER-IP -i116 | flow-stat flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e JUNIPER-IP -I116 | flow-stat flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e CISCO-IP -i13 | flow-stat flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e CISCO-IP -I13 | flow-stat -- Jacky :)_______________________________________________ Flow-tools mailing list flow-tools@splintered.net http://mailman.splintered.net/mailman/listinfo/flow-tools -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.splintered.net/pipermail/flow-tools/attachments/20100504/d5f99c22/attachment.htm From bigserpent at gmail.com Tue May 4 11:30:41 2010 From: bigserpent at gmail.com (Jacky Chan) Date: Tue May 4 11:30:44 2010 Subject: [Flow-tools] Flow-stat vs MRTG In-Reply-To: References: Message-ID: HI Joe, My ft file is 5 minutes long (the stat interval is 5). Both my Juniper and Cisco do sampling and packet-interval is 100. For this case, I need to change the ft file to 15 minutes long? on the other hand, I am not sure how to calculate the "multipliers". Regards, Jacky On Tue, May 4, 2010 at 9:08 PM, Joe Loiacono wrote: > > When you apply flow-stat to a single ft file, you're getting the average > rate over the length of time associated with the file. My ft files are > typically 15 minutes long. If we assume yours are 15 minutes also, you are > comparing a 15 minute average with an MRTG 5-minute SNMP sample of all bytes > (including IP and TCP headers.) > > Looking at sampling as a possibility, here are the 'multipliers': > > 110/0.293 = 375 > 136/0.323 = 421 > 24/0.225 = 107 > 96/1.462 = 66 > > They're not consistent. But if you create an MRTG number from the average > of the three readings that make up the 15 minute netflow period, you might > find a consistent multiplier. > > But - you could first just check to see if you're sampling on the Juniper > :-) > > By the way - have you checked out FlowViewer as a web interface to > flow-tools? > > http://ensight.eos.nasa.gov/FlowViewer/ > > Joe > > > From: Jacky Chan To: > flow-tools@list.splintered.net Date: 05/04/2010 01:52 AM Subject: [Flow-tools] > Flow-stat vs MRTG > ------------------------------ > > > > Dear Sir, > > I have flow-tools 0.68 running on a Fedora Core 11 workstation and the > system collecting flow-data from Juniper and Cisco routers. > > I tried obtain the link utilization from the flow-stat output but there is > big different when compared to MRTG reading. > > Example-1 > I have a GE link from my Juniper router to INTERNET upstream-1 > From MRTG, the average input / output speed @ 21:00 are 110Mbps/ 136Mbps. > From Flow-stat, the input/ output speed @21:00 are 292.6613Kbps/ > 323.1716Kbps. > > Example-2 > I have a GE link from my Cisco router to INTERNET upstream-2 > From MRTG, the average input / output speed @ 21:00 are 24Mbps/ 96Mbps. > From Flow-stat, the input/ output speed @21:00 are 225.0744Kbps/ > 1462.7255Kbps. > > Here are the commands I used to obtain the link utilization from the > flow-data. I did something wrong or misused the flow-stat application? > > flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e JUNIPER-IP -i116 | > flow-stat > flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e JUNIPER-IP -I116 | > flow-stat > flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e CISCO-IP -i13 | > flow-stat > flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e CISCO-IP -I13 | > flow-stat > > > -- > Jacky :)_______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools > > -- Jacky Chan :) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.splintered.net/pipermail/flow-tools/attachments/20100504/9f135011/attachment.htm From jloiacon at csc.com Tue May 4 12:59:26 2010 From: jloiacon at csc.com (Joe Loiacono) Date: Tue May 4 12:59:33 2010 Subject: [Flow-tools] Flow-stat vs MRTG In-Reply-To: References: Message-ID: Jacky - OK so it looks like sampling is responsible for the difference between MRTG and flow-tools, but I'm surprised the 'multiplier' varies so much. My experience has had them closer. Maybe you can play with the sampling parameters on your routers. Though at the traffic levels you're talking about, I wouldn't sample at all. It's not a very heavy load, and you'll be happier with the results. You don't have to change the ft file times. Joe From: Jacky Chan To: Joe Loiacono/USA/CSC@CSC Cc: flow-tools@list.splintered.net Date: 05/04/2010 11:30 AM Subject: Re: [Flow-tools] Flow-stat vs MRTG HI Joe, My ft file is 5 minutes long (the stat interval is 5). Both my Juniper and Cisco do sampling and packet-interval is 100. For this case, I need to change the ft file to 15 minutes long? on the other hand, I am not sure how to calculate the "multipliers". Regards, Jacky On Tue, May 4, 2010 at 9:08 PM, Joe Loiacono wrote: When you apply flow-stat to a single ft file, you're getting the average rate over the length of time associated with the file. My ft files are typically 15 minutes long. If we assume yours are 15 minutes also, you are comparing a 15 minute average with an MRTG 5-minute SNMP sample of all bytes (including IP and TCP headers.) Looking at sampling as a possibility, here are the 'multipliers': 110/0.293 = 375 136/0.323 = 421 24/0.225 = 107 96/1.462 = 66 They're not consistent. But if you create an MRTG number from the average of the three readings that make up the 15 minute netflow period, you might find a consistent multiplier. But - you could first just check to see if you're sampling on the Juniper :-) By the way - have you checked out FlowViewer as a web interface to flow-tools? http://ensight.eos.nasa.gov/FlowViewer/ Joe From: Jacky Chan To: flow-tools@list.splintered.net Date: 05/04/2010 01:52 AM Subject: [Flow-tools] Flow-stat vs MRTG Dear Sir, I have flow-tools 0.68 running on a Fedora Core 11 workstation and the system collecting flow-data from Juniper and Cisco routers. I tried obtain the link utilization from the flow-stat output but there is big different when compared to MRTG reading. Example-1 I have a GE link from my Juniper router to INTERNET upstream-1 >From MRTG, the average input / output speed @ 21:00 are 110Mbps/ 136Mbps. >From Flow-stat, the input/ output speed @21:00 are 292.6613Kbps/ 323.1716Kbps. Example-2 I have a GE link from my Cisco router to INTERNET upstream-2 >From MRTG, the average input / output speed @ 21:00 are 24Mbps/ 96Mbps. >From Flow-stat, the input/ output speed @21:00 are 225.0744Kbps/ 1462.7255Kbps. Here are the commands I used to obtain the link utilization from the flow-data. I did something wrong or misused the flow-stat application? flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e JUNIPER-IP -i116 | flow-stat flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e JUNIPER-IP -I116 | flow-stat flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e CISCO-IP -i13 | flow-stat flow-cat ft-v05-2010-05-03.210000+0800 | flow-filter -e CISCO-IP -I13 | flow-stat -- Jacky :)_______________________________________________ Flow-tools mailing list flow-tools@splintered.net http://mailman.splintered.net/mailman/listinfo/flow-tools -- Jacky Chan :) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.splintered.net/pipermail/flow-tools/attachments/20100504/39604566/attachment.htm From jloiacon at csc.com Wed May 19 15:31:43 2010 From: jloiacon at csc.com (Joe Loiacono) Date: Wed May 19 15:31:50 2010 Subject: [Flow-tools] NetFlow v9 support in flow-tools? In-Reply-To: References: Message-ID: Craig, Thanks for the 'flowd2ft' script to enable conversion between flowd V9 captures and flow-tools ft files ( http://mailman.splintered.net/pipermail/flow-tools/2009-March/003765.html ) From your accompanying email message of Mar 13, 2009: > Naturally it only supports for V5 fields (use 'record netflow-original' when setting up Cisco flexible netflow). Does this require that the exporter be set this way only? In other words, will 'flowd-reader -c' be able to take any v9 file and convert it for flow-tools (V5 only - which is OK), or just those exported with 'record netflow-original'? Second question: Do you have to be careful syncing up the cron times with the flowd file times? Many thanks! Joe From: Craig Weinhold To: Adam Powers Cc: "" Date: 04/22/2010 01:04 PM Subject: Re: [Flow-tools] NetFlow v9 support in flow-tools? There's an active flow-tools code fork, but it also lacks netflow v9 support: http://code.google.com/p/flow-tools/updates/list Adding v9 to flow-tools is not that easy; the fixed-length file structure currently used doesn't lend itself to the arbitarry field/protocol capabilities of netflow v9. It's a substantial effort. See this post for a workaround (for IPv4 only): http://mailman.splintered.net/pipermail/flow-tools/2009-March/003765.html -Craig On Thu, 22 Apr 2010, Adam Powers wrote: > A colleague of mine mentioned the other day that he heard someone had updated flow-tools to support NetFlow v9. Truth? I can?t find anything about such support. > > -- > > Adam Powers > NetFlow Ninja & CTO > Lancope, Inc. > c. 678.725.1028 > e. apowers@lancope.com > _______________________________________________ Flow-tools mailing list flow-tools@splintered.net http://mailman.splintered.net/mailman/listinfo/flow-tools -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.splintered.net/pipermail/flow-tools/attachments/20100519/7224e6ca/attachment.htm From craig.weinhold at cdw.com Wed May 19 17:40:53 2010 From: craig.weinhold at cdw.com (Craig Weinhold) Date: Wed May 19 17:40:57 2010 Subject: [Flow-tools] NetFlow v9 support in flow-tools? In-Reply-To: References: Message-ID: Joe, 'flowd-reader -c' takes any flowd capture as input. If the capture contains IPv6, the output will include IPv6 addresses which'll break flow-import. You can certainly use a subset of netflow-original (e.g., leave out L4 info if you like). Using more than netflow-original will just waste router/server CPU and export bandwidth since flowd silently drops other fields anyway. Think of flowd as a netflow-original + IPv6 collector... Re sync issues... While the import is taking place, other scripts might mistakenly think the "ft-" file is complete when it's not. Here is a revised script that uses FT's "tmp-" file name to avoid that case: #!/usr/bin/perl # "flowd2ft" crontab script to move flowd capture files into flow-tools # -- flow-tools variables our $ftImport = "/usr/local/netflow/bin/flow-import"; # where ft's flow-import is our $ftDir = "/var/log/flow-tools-capture"; # where ft's capture files go our $ftTZ = "-0500"; # timezone for ft capture files our $ftPeriod = 300; # seconds per ft capture file and cron interval # -- flowd variables our $flowdReader = "/usr/local/bin/flowd-reader"; # where flowd-reader is our $flowdConf = "/usr/local/etc/flowd.conf"; # where flowd.conf is our $flowdHup = "10"; # SIGUSR1 our ($flowdPid, $flowdLog); ### START OF NEW STUFF our $ftStem = sprintf("v05.%04d-%02d-%02d.%02d%02d%02d$ftTZ", $year + 1900, $mon + 1, $mday, $hour, $min, $sec); our $ftFile = "tmp-$ftStem"; our $ftFileFinal = "ft-$ftStem"; ### END OF NEW STUFF open(IN, $flowdConf) || die "Could not read $flowdConf"; while ( ) { if (/^\s*logfile ["]?([^"\s]+)/) { $flowdLog = $1; } if (/^\s*pidfile ["]?([^"\s]+)/) { $flowdPid = $1; } } close(IN); exit if (! -f $flowdLog); # exit silently on empty file die "$flowdPid does not exist: $!" if (! -f $flowdPid); my $pid = `cat $flowdPid`; `mv $flowdLog $flowdLog.tmp`; die "$flowdPid ($pid) invalid: $!" if (! kill $flowdHup, $pid); `$flowdReader -c $flowdLog.tmp | $flowImport -f 2 -V 5 -z 1 > $ftDir/$ftFile`; unlink("$flowdLog.tmp"); `mv $ftDir/$ftFile $ftDir/$ftFileFinal`; ##### ALSO NEW -Craig On Wed, 19 May 2010, Joe Loiacono wrote: > > Craig, > > Thanks for the 'flowd2ft' script to enable conversion between flowd V9 captures and flow-tools ft files ( http://mailman.splintered.net/pipermail/flow-tools/2009-March/003765.html ) > > From your accompanying email message of Mar 13, 2009: > > > Naturally it only supports for V5 fields (use 'record netflow-original' when setting up Cisco flexible netflow). > > Does this require that the exporter be set this way only? In other words, will 'flowd-reader -c' be able to take any v9 file and convert it for flow-tools (V5 only - which is OK), or just those exported with 'record netflow-original'? > > Second question: > > Do you have to be careful syncing up the cron times with the flowd file times? > > Many thanks! > > Joe > > > From: Craig Weinhold > To: Adam Powers > Cc: "" > Date: 04/22/2010 01:04 PM > Subject: Re: [Flow-tools] NetFlow v9 support in flow-tools? > > ________________________________ > > > > There's an active flow-tools code fork, but it also lacks netflow v9 support: > http://code.google.com/p/flow-tools/updates/list > > Adding v9 to flow-tools is not that easy; the fixed-length file structure currently used doesn't lend itself to the arbitarry field/protocol capabilities of netflow v9. It's a substantial effort. > > See this post for a workaround (for IPv4 only): > http://mailman.splintered.net/pipermail/flow-tools/2009-March/003765.html > > -Craig > > > > On Thu, 22 Apr 2010, Adam Powers wrote: > > > A colleague of mine mentioned the other day that he heard someone had updated flow-tools to support NetFlow v9. Truth? I can?t find anything about such support. > > > > -- > > > > Adam Powers > > NetFlow Ninja & CTO > > Lancope, Inc. > > c. 678.725.1028 > > e. apowers@lancope.com > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools > > From Ken.Hagen at seattle.gov Thu May 20 12:16:36 2010 From: Ken.Hagen at seattle.gov (Hagen, Ken) Date: Thu May 20 12:16:43 2010 Subject: [Flow-tools] unsubscribe Message-ID: Ken Hagen CCNP Department of Information Technology City of Seattle W: (206) 386-1503 C: (206) 255-8391 E: ken.hagen@seattle.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.splintered.net/pipermail/flow-tools/attachments/20100520/5573388f/attachment.htm