From eravin at panix.com Mon Feb 1 12:19:29 2010 From: eravin at panix.com (Ed Ravin) Date: Mon Feb 1 12:19:37 2010 Subject: [Flow-tools] ipflowsearch, a quick way to grab flow data by IP address Message-ID: <20100201171929.GA16016@panix.com> Attached is the shell script "ipflowsearch", which lets you pull out flow data from your flow-tools tree by IP address or subnet with a minimum of intellectual effort. Feel free to toss this into the contrib section of the current flow-tools distribution, or anywhere else it might be useful. -- Ed -------------- next part -------------- #!/bin/sh set -u set -e USAGE="Usage: $0 flowdir-top ip-address[/mask] [...]" # EDIT THIS LINE - use a default dir with lots of free space export flowtemp=${TMPDIR:-/logs/tmp} # ipflowsearch - run flow-cat and flow-filter on a flow-tools data # tree, filtering out traffic for the requested IP address(es). # ipflowsearch was written by Ed Ravin , and is # made available to the public by courtesy of PANIX Public Access Networks # ( http://www.panix.com ). License is GPL. export TMPDIR=$flowtemp # so mktemp will work properly if [ ! -d $flowtemp ] then echo "$0: no such directory: $flowtemp" exit 23 fi if [ "${DEBUG:-no}" = YES ] then set -x fi convert_ip_slash() { slashpart=${1##*/} if [ "$slashpart" != $1 ] then ippart=${1%%/*} case $slashpart in 32) wildcard=;; 31) wildcard=0.0.0.1;; 30) wildcard=0.0.0.3;; 29) wildcard=0.0.0.7;; 28) wildcard=0.0.0.15;; 27) wildcard=0.0.0.31;; 26) wildcard=0.0.0.63;; 25) wildcard=0.0.0.127;; 24) wildcard=0.0.0.255;; 23) wildcard=0.0.1.255;; 22) wildcard=0.0.3.255;; 21) wildcard=0.0.7.255;; 20) wildcard=0.0.15.255;; 19) wildcard=0.0.31.255;; 18) wildcard=0.0.63.255;; 17) wildcard=0.0.127.255;; 16) wildcard=0.0.255.255;; 15) wildcard=0.1.255.255;; 14) wildcard=0.3.255.255;; 13) wildcard=0.7.255.255;; 12) wildcard=0.15.255.255;; 11) wildcard=0.31.255.255;; 10) wildcard=0.63.255.255;; 9) wildcard=0.127.255.255;; 8) wildcard=0.255.255.255;; 7) wildcard=1.255.255.255;; 6) wildcard=3.255.255.255;; 5) wildcard=7.255.255.255;; 4) wildcard=15.255.255.255;; 3) wildcard=31.255.255.255;; 2) wildcard=63.255.255.255;; 1) wildcard=127.255.255.255;; *) echo "Unrecognized/unsupported mask: $slashpart" 1>&2; exit 1;; esac echo $ippart $wildcard else echo $1 fi } # search flow logs for all traffic from specified IP address(es) flowdirtop=${1:?$USAGE} shift ipaddresses=${*:?$USAGE} # create the flow-filter tempacl=$(mktemp -t ipsearch.acl) if [ ! -f $tempacl ] then echo "$0: cannot create temp ACL file $tempacl" exit 24 fi trap 'rm -f $tempacl' 0 # datestamp for output file now=$(date +%Y%m%d-%H%M%S) filestr=$(echo "$*" | sed -e 's,[ /],_,g') for ip in $ipaddresses do ipacl=$(convert_ip_slash $ip) echo "ip access-list standard ipwanted permit $ipacl" done > $tempacl outputfile=$flowtemp/ipsearch-$filestr-$now.flow flow-cat $flowdirtop | flow-filter -f $tempacl -S ipwanted -o -D ipwanted \ > $outputfile echo "$0: output file is $outputfile" From drew.weaver at thenap.com Tue Feb 23 08:15:29 2010 From: drew.weaver at thenap.com (Drew Weaver) Date: Tue Feb 23 08:15:32 2010 Subject: [Flow-tools] Finding useful information from collected data Message-ID: Does anyone have any scripts or advice for getting useful data out of flow-cat/flow-print, etc? I am just looking for examples for simple things like finding the top 10 IPs doing SMTP, or SSH, or FTP, etc nothing too fancy.. also it would be cool to find the top 10 talkers in BPS/PPS, etc. Any resources available? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.splintered.net/pipermail/flow-tools/attachments/20100223/eeb951c3/attachment.htm From mansoor.ali at hbl.com Tue Feb 23 09:01:58 2010 From: mansoor.ali at hbl.com (Mansoor Ali Khan) Date: Tue Feb 23 09:04:50 2010 Subject: [Flow-tools] flow-fanout socket problem on Cygwin/WinXP Message-ID: <536F9F4465FD42499ECEE2E81FF05A2CD043843E87@PKKHICS01E07.domestic.hbl.com> I get 'Address family not supported by protocol' as indicated below. $ ./flow-fanout -D 172.15.15.12/172.11.1.45/9996 172.15.15.12/172.15.16.12/9997 flow-fanout: setsockopt(size=4194304) flow-fanout: bind(): Address family not supported by protocol $ netstat -an -p udp | grep 9996 However, flow-capture runs just fine and captures the data from the router Following are my configurations: Local IP: 172.15.15.12 Router IP/port: 172.11.1.45/9996 Remote IP (to which I want to forward NetFlow UDP packets): 172.15.16.12 CYGWIN_NT-5.1 1.7.1(0.218/5/3) 2009-12-07 11:48 on Win XP SP3 Flow-tools v0.66 Any help will be highly appreciated Regards, MAK. ________________________________ This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the author and delete the email. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Finally, the recipient should check this email and any attachments for the presence of viruses. HBL accepts no liability for any damage caused by any virus transmitted by this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.splintered.net/pipermail/flow-tools/attachments/20100223/4bc67d11/attachment.htm From jloiacon at csc.com Tue Feb 23 13:17:32 2010 From: jloiacon at csc.com (Joe Loiacono) Date: Tue Feb 23 13:17:35 2010 Subject: [Flow-tools] Finding useful information from collected data In-Reply-To: References: Message-ID: Why don't you use FlowViewer? It is a web companion tool for flow-tools, and meets your requirements. Check out: http://ensight.eos.nasa.gov/FlowViewer Look through the screenshots, and read through the User's Guide. Joe |------------> | From: | |------------> >------------------------------------------------------------------------------------------------------------------------------------------| |Drew Weaver | >------------------------------------------------------------------------------------------------------------------------------------------| |------------> | To: | |------------> >------------------------------------------------------------------------------------------------------------------------------------------| |"'flow-tools@list.splintered.net'" | >------------------------------------------------------------------------------------------------------------------------------------------| |------------> | Date: | |------------> >------------------------------------------------------------------------------------------------------------------------------------------| |02/23/2010 08:16 AM | >------------------------------------------------------------------------------------------------------------------------------------------| |------------> | Subject: | |------------> >------------------------------------------------------------------------------------------------------------------------------------------| |[Flow-tools] Finding useful information from collected data | >------------------------------------------------------------------------------------------------------------------------------------------| Does anyone have any scripts or advice for getting useful data out of flow-cat/flow-print, etc? I am just looking for examples for simple things like finding the top 10 IPs doing SMTP, or SSH, or FTP, etc nothing too fancy.. also it would be cool to find the top 10 talkers in BPS/PPS, etc. Any resources available? _______________________________________________ Flow-tools mailing list flow-tools@splintered.net http://mailman.splintered.net/mailman/listinfo/flow-tools From eravin at panix.com Tue Feb 23 13:25:56 2010 From: eravin at panix.com (Ed Ravin) Date: Tue Feb 23 13:25:58 2010 Subject: [Flow-tools] Finding useful information from collected data In-Reply-To: References: Message-ID: <20100223182556.GA19051@panix.com> On Tue, Feb 23, 2010 at 08:15:29AM -0500, Drew Weaver wrote: > Does anyone have any scripts or advice for getting useful data out of > flow-cat/flow-print, etc? > > > I am just looking for examples for simple things like finding the top > 10 IPs doing SMTP, or SSH, or FTP, etc nothing too fancy.. Here's a template you can use to sort by top 10 source IP: flow-cat $NETFLOW_FILES | flow-filter -I $OUTGOING_INTERFACE_OID | flow-stat -f9 -S2 | head -20 You can skip the line for filtering on interface if it's not needed in your environment. The "old style" tools, flow-filter and flow-stat, can be used for quick and dirty reports like the one above. The "new style" tools, flow-nfilter and flow-report, are much harder to use but have more functionality. For immediate views with graphing, you should use flow-viewer as Joe suggests. From craig.weinhold at cdw.com Tue Feb 23 14:28:00 2010 From: craig.weinhold at cdw.com (Craig Weinhold) Date: Tue Feb 23 14:28:03 2010 Subject: [Flow-tools] Finding useful information from collected data In-Reply-To: <20100223182556.GA19051@panix.com> References: <20100223182556.GA19051@panix.com> Message-ID: My favorite, rarely-used flow-report is a "peer" report. The report shows each IP and a count of how many other IP's it has communicated with. ip-destination-address-source-count finds hosts that are very popular. E.g., DNS, SNMP, web proxy, etc ip-source-address-destination-count finds the same hosts, but also includes those that are unsuccesfully attempting to talk to lots of other hosts. E.g., worms attempting to spread, people doing IP scans, torrent users, etc. The top IPs from these reports should never be surprising to a network admin (but they often are!) -Craig On Tue, 23 Feb 2010, Ed Ravin wrote: > On Tue, Feb 23, 2010 at 08:15:29AM -0500, Drew Weaver wrote: > > Does anyone have any scripts or advice for getting useful data out of > > flow-cat/flow-print, etc? > > > > > > I am just looking for examples for simple things like finding the top > > 10 IPs doing SMTP, or SSH, or FTP, etc nothing too fancy.. > > Here's a template you can use to sort by top 10 source IP: > > flow-cat $NETFLOW_FILES | > flow-filter -I $OUTGOING_INTERFACE_OID | > flow-stat -f9 -S2 | > head -20 > > You can skip the line for filtering on interface if it's not needed in > your environment. > > The "old style" tools, flow-filter and flow-stat, can be used for > quick and dirty reports like the one above. The "new style" tools, > flow-nfilter and flow-report, are much harder to use but have more > functionality. > > For immediate views with graphing, you should use flow-viewer as Joe > suggests. > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools > From i at stingr.net Wed Feb 24 07:21:13 2010 From: i at stingr.net (Paul Komkoff) Date: Wed Feb 24 07:21:17 2010 Subject: [Flow-tools] flow-fanout socket problem on Cygwin/WinXP In-Reply-To: <536F9F4465FD42499ECEE2E81FF05A2CD043843E87@PKKHICS01E07.domestic.hbl.com> References: <536F9F4465FD42499ECEE2E81FF05A2CD043843E87@PKKHICS01E07.domestic.hbl.com> Message-ID: <715ea5c11002240421o499107b7h768aa8bbe7a4a4d6@mail.gmail.com> On Tue, Feb 23, 2010 at 2:01 PM, Mansoor Ali Khan wrote: > Flow-tools v0.66 Can you try newer flow-tools? Like, this one: http://flow-tools.googlecode.com/files/flow-tools-0.68.4.3.tar.bz2 -- This message represents the official view of the voices in my head