From splintered-flow-tools-owner@splintered.net Mon Dec 01 18:01:53 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 76853 invoked by uid 4001); 1 Dec 2003 18:01:53 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 76851 invoked by alias); 1 Dec 2003 18:01:53 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 1 Dec 2003 18:01:53 -0000 Mime-Version: 1.0 (Apple Message framework v606) Content-Transfer-Encoding: 7bit Message-Id: <7174894C-2428-11D8-BC39-000A95DA1C38@eng.oar.net> Content-Type: text/plain; charset=US-ASCII; format=flowed To: flow-tools@splintered.net From: Mark Fullmer Date: Mon, 1 Dec 2003 13:01:52 -0500 X-Mailer: Apple Mail (2.606) Cc: Subject: [Flow-tools] New list host X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Dec 2003 18:01:53 -0000 I've moved the flow-tools mailing list to a new host, hopefully this will fix the dropped mail problems I've had with pair.com's mailing list service. The new list home is http://mailman.splintered.net/mailman/listinfo/flow-tools In the process any accounts which were disabled due to excessive bounces have been re-enabled. You may need to set your digest flag again... mark From splintered-flow-tools-owner@splintered.net Tue Dec 02 00:35:43 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 79445 invoked by uid 4001); 2 Dec 2003 00:35:43 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 79442 invoked by alias); 2 Dec 2003 00:35:43 -0000 Received: from dhcp9578217.columbus.rr.com (HELO ?10.0.0.25?) (24.95.78.217) by 66.250.216.131 with SMTP; 2 Dec 2003 00:35:43 -0000 In-Reply-To: <00e001c3b817$23798170$61af0a0a@Accenture.com> References: <00e701c3b4f4$319787a0$61af0a0a@Accenture.com> <20031127202819.GE91301@elvis.mu.org> <002001c3b5b2$2e7f80e0$61af0a0a@Accenture.com> <43B31182-220A-11D8-9AA7-000A95DA1C38@splintered.net> <002801c3b7f3$7d645850$61af0a0a@Accenture.com> <1D50FC29-2400-11D8-BC39-000A95DA1C38@splintered.net> <00e001c3b817$23798170$61af0a0a@Accenture.com> Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <753B3D6F-245F-11D8-BC39-000A95DA1C38@splintered.net> Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [flow-tools] format flow description Date: Mon, 1 Dec 2003 19:35:41 -0500 To: "Orlando Onorato" X-Mailer: Apple Mail (2.606) Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Dec 2003 00:35:43 -0000 Flow-tools does not store these fields, they're only necessary to get the data from the exporter to the collector. The 'pad' field is there to provide alignment. You could look in lib/ftdecode.c or src/fdg.c for some examples on how to find these fields mark On Dec 1, 2003, at 9:17 AM, Orlando Onorato wrote: > In particular I need of these header fields: > ushort version, ushort count, ulong flow_sequence. > > Why the field "uchar pad" isn't there in the flow record? > > Thanks for your patience! > > bye > > > ----- Original Message ----- > From: "Mark Fullmer" > To: "Orlando Onorato" > Cc: > Sent: Monday, December 01, 2003 2:13 PM > Subject: Re: [flow-tools] format flow description > > >> Which header are you trying to look at? >> >> mark >> >> On Dec 1, 2003, at 5:09 AM, Orlando Onorato wrote: >> >>> Thank you very much. I've solved my 2nd problem. >>> >>> By mean the flow-export I'm not able to view the header yet, >>> and I don't want to use the tcpdump. >>> >>> Can you help me? >>> >>> >>> ----- Original Message ----- >>> From: "Mark Fullmer" >>> To: "Orlando Onorato" >>> Cc: >>> Sent: Saturday, November 29, 2003 2:20 AM >>> Subject: Re: [flow-tools] format flow description >>> >>> >>>> It's there. Use flow-export -f2 (ASCII). >>>> >>>> #: >>>> unix_secs,unix_nsecs,sysuptime,exaddr,dpkts,doctets,first,last,engin >>>> e_ >>>> ty >>>> pe,eng >>>> ine_id,srcaddr,dstaddr,nexthop,input,output,srcport,dstport,prot,tos >>>> ,t >>>> cp >>>> _flags,s >>>> rc_mask,dst_mask,src_as,dst_as >>>> 1018404899,502659552,3685185404,192.88.192.245,1,40,3685169372,36851 >>>> 69 >>>> 37 >>>> 2,4,0,12 >>>> 9.137.254.5,129.21.7.49,192.88.192.33,40,25,56279,33508,17,0,16,16,1 >>>> 6, >>>> 20 >>>> 126,4385 >>>> ^^^^^^^^^^^^^ >>>> >>>> mark >>>> >>>> >>>> On Nov 28, 2003, at 8:18 AM, Orlando Onorato wrote: >>>> >>>>> By means of flow-export I'm not able to view the Nexhop field, >>>>> although this filed there is in Netflow ver.5. >>>>> >>>>> >>>>> ----- Original Message ----- >>>>> From: "Bill Fumerola" >>>>> To: "Orlando Onorato" >>>>> Cc: >>>>> Sent: Thursday, November 27, 2003 9:28 PM >>>>> Subject: Re: [flow-tools] format flow description >>>>> >>>>> >>>>>> On Thu, Nov 27, 2003 at 03:38:57PM +0100, Orlando Onorato wrote: >>>>>> >>>>>>> 1) View header description of datagramm generated. >>>>>> >>>>>> dunno what you mean here, but increasing the debug level will show >>>>>> the >>>>>> header of the flow file. if you want to see ip or udp header >>>>>> information >>>>>> from the actual netflow packet, i'd suggest tcpdump. >>>>>> >>>>>>> 2) View all fields of flow description (e.g. nexthop field. >>>>>> >>>>>> flow-export >>>>>> >>>>>> -- >>>>>> - bill fumerola / fumerola@yahoo-inc.com / billf@FreeBSD.org >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> flow-tools@splintered.net >>>>> http://www.splintered.net/sw/flow-tools >>>>> >>>> >>>> >>>> _______________________________________________ >>>> flow-tools@splintered.net >>>> http://www.splintered.net/sw/flow-tools >> >> >> _______________________________________________ >> flow-tools@splintered.net >> http://www.splintered.net/sw/flow-tools > From splintered-flow-tools-owner@splintered.net Tue Dec 02 02:43:40 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 80607 invoked by uid 4001); 2 Dec 2003 02:43:40 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 80603 invoked by alias); 2 Dec 2003 02:43:39 -0000 Received: from mail.sunet.com.au (HELO jupiter.sunet.com.au) (203.166.102.39) by 66.250.216.131 with SMTP; 2 Dec 2003 02:43:39 -0000 Received: from ganymede.internal.sunet.com.au (canopus.sunet.com.au [::ffff:203.166.102.49]) by jupiter.sunet.com.au with esmtp; Tue, 02 Dec 2003 13:38:15 +1100 Date: Tue, 2 Dec 2003 13:43:59 +1100 (EST) From: Systems Administrator X-X-Sender: sysadmin@ganymede.bcc.local To: Mark Fullmer Subject: Re: [flow-tools] flow-nfilter end-time comparisons In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Dec 2003 02:43:40 -0000 On Mon, 1 Dec 2003, Mark Fullmer wrote: > Yep. It's a one line fix. Great! I'd been wanting that for a year or so :). But if I needed it *that* badly, I would've done it myself :). I presume this is making it into the next version of flow-tools? (is it 0.67 next?) Thanks, -- Tim Nelson Systems Administrator Sunet Internet Tel: +61 3 5241 1155 Fax: +61 3 5241 6187 Web: http://www.sunet.com.au/ Email: sysadmin@sunet.com.au From foulonneau@net-outremer.nc Tue Dec 02 06:35:23 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 82308 invoked by alias); 2 Dec 2003 06:35:22 -0000 Received: from gw-smtp.net-outremer.nc (HELO gw-smtp.canl.nc) (202.171.64.2) by 66.250.216.131 with SMTP; 2 Dec 2003 06:35:22 -0000 Received: from mail1.canl.nc ([202.87.159.22]) by gw-smtp.canl.nc with esmtp (Exim 4.14) id 1AR480-0003XJ-Fp for flow-tools@list.splintered.net; Tue, 02 Dec 2003 17:35:20 +1100 Received: from [192.168.0.66] (helo=callispe.net-outremer.nc) by mail1.canl.nc with asmtp (Exim 4.24) id 1AR47z-0002DG-Ul for flow-tools@list.splintered.net; Tue, 02 Dec 2003 17:35:19 +1100 Message-Id: <5.1.0.14.2.20031202173526.02b82ac0@mail.canl.nc> X-Sender: Foulonneau@net-outremer.nc@mail.canl.nc X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Tue, 02 Dec 2003 17:35:33 +1100 To: flow-tools@list.splintered.net From: Laurent Foulonneau Subject: [flow-tools] some php scripts Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Dec 2003 06:35:23 -0000 Great ! it works fine for me... Thank you From splintered-flow-tools-owner@splintered.net Tue Dec 02 08:29:23 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 83586 invoked by uid 4001); 2 Dec 2003 08:29:23 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 83583 invoked by alias); 2 Dec 2003 08:29:23 -0000 Received: from hal-5.inet.it (213.92.5.24) by 66.250.216.131 with SMTP; 2 Dec 2003 08:29:23 -0000 Received: from 194-177-126-137.f4.ngi.it [::ffff:194.177.126.137] by hal-5.inet.it via I-SMTP-4.7.0-470 id ::ffff:194.177.126.137+wrYt1WvYG; Tue, 02 Dec 2003 09:29:21 +0100 Message-ID: <3FCC4AAD.7070700@tekNico.net> Date: Tue, 02 Dec 2003 09:17:49 +0100 From: Nicola Larosa User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031105 Thunderbird/0.3 X-Accept-Language: en-us, en MIME-Version: 1.0 To: flow-tools@splintered.net Subject: Re: [flow-tools] flow-nfilter end-time comparisons References: In-Reply-To: X-Enigmail-Version: 0.81.6.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Dec 2003 08:29:23 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > I presume this is making it into the next version of flow-tools? (is it > 0.67 next?) Speaking of which, what's the timeline for the release? I have to find the time to report patches and suggestions for three items related to flow-capture. They're admittedly minor, but it would be nice if they had a chance to be at least considered for the next release, since they seem to appear not that often. :^) - -- "Workaholics and others who can't tear themselves away from the mouse and keyboard need to keep their legs active and get away from the computer for some exercise every now and then. So, please, get up and stretch every so often. You certainly don't want to wind up eDead from eThrombosis." -- Jenny Thompson, HSI Nicola Larosa - nico@tekNico.net -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux) iD8DBQE/zEqsXv0hgDImBm4RAgW8AJ4+8KXQJjObE2kTZfxHpXeli/5EUwCguVss 9oYZR80KQRQkwFAcPXYqiNA= =7MqC -----END PGP SIGNATURE----- From splintered-flow-tools-owner@splintered.net Tue Dec 02 14:38:57 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 85355 invoked by uid 4001); 2 Dec 2003 14:38:57 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 85351 invoked by alias); 2 Dec 2003 14:38:57 -0000 Received: from smtp0.libero.it (193.70.192.33) by 66.250.216.131 with SMTP; 2 Dec 2003 14:38:57 -0000 Received: from H7J31KT8Z604D (151.24.171.246) by smtp0.libero.it (7.0.020-DD01) id 3F6F1CE701678CC9; Tue, 2 Dec 2003 15:38:54 +0100 Message-ID: <00eb01c3b8e1$a503d4f0$61af0a0a@Accenture.com> From: "Orlando Onorato" To: "Mark Fullmer" , Subject: Re: [flow-tools] format flow description Date: Tue, 2 Dec 2003 15:36:10 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Dec 2003 14:38:57 -0000 Ok! I want understand if flow_gen produce this fields (ushort version, ushort count, ulong flow_sequence). I believe yes! Then, for obtain this fields in ASCII CSV format I must: 1) Add this fields in struct fts3rec_v5_gen or struct fts3rec_v5 ? 2) Modify the function ftio_read(). Is it ok? If you can to advice some specific code solution, I will be very happy! Thanks a lot. ----- Original Message ----- From: "Orlando Onorato" Sent: Tuesday, December 02, 2003 2:14 PM Subject: Fw: [flow-tools] format flow description > > ----- Original Message ----- > From: "Mark Fullmer" > To: "Orlando Onorato" > Cc: > Sent: Tuesday, December 02, 2003 1:35 AM > Subject: Re: [flow-tools] format flow description > > > > > > Flow-tools does not store these fields, they're only necessary > > to get the data from the exporter to the collector. > > > > The 'pad' field is there to provide alignment. > > > > You could look in lib/ftdecode.c or src/fdg.c for some examples on > > how to find these fields > > > > mark > > > > On Dec 1, 2003, at 9:17 AM, Orlando Onorato wrote: > > > > > In particular I need of these header fields: > > > ushort version, ushort count, ulong flow_sequence. > > > > > > Why the field "uchar pad" isn't there in the flow record? > > > > > > Thanks for your patience! > > > > > > bye > > > > > > > > > ----- Original Message ----- > > > From: "Mark Fullmer" > > > To: "Orlando Onorato" > > > Cc: > > > Sent: Monday, December 01, 2003 2:13 PM > > > Subject: Re: [flow-tools] format flow description > > > > > > > > >> Which header are you trying to look at? > > >> > > >> mark > > >> > > >> On Dec 1, 2003, at 5:09 AM, Orlando Onorato wrote: > > >> > > >>> Thank you very much. I've solved my 2nd problem. > > >>> > > >>> By mean the flow-export I'm not able to view the header yet, > > >>> and I don't want to use the tcpdump. > > >>> > > >>> Can you help me? > > >>> > > >>> > > >>> ----- Original Message ----- > > >>> From: "Mark Fullmer" > > >>> To: "Orlando Onorato" > > >>> Cc: > > >>> Sent: Saturday, November 29, 2003 2:20 AM > > >>> Subject: Re: [flow-tools] format flow description > > >>> > > >>> > > >>>> It's there. Use flow-export -f2 (ASCII). > > >>>> > > >>>> #: > > >>>> unix_secs,unix_nsecs,sysuptime,exaddr,dpkts,doctets,first,last,engin > > >>>> e_ > > >>>> ty > > >>>> pe,eng > > >>>> ine_id,srcaddr,dstaddr,nexthop,input,output,srcport,dstport,prot,tos > > >>>> ,t > > >>>> cp > > >>>> _flags,s > > >>>> rc_mask,dst_mask,src_as,dst_as > > >>>> 1018404899,502659552,3685185404,192.88.192.245,1,40,3685169372,36851 > > >>>> 69 > > >>>> 37 > > >>>> 2,4,0,12 > > >>>> 9.137.254.5,129.21.7.49,192.88.192.33,40,25,56279,33508,17,0,16,16,1 > > >>>> 6, > > >>>> 20 > > >>>> 126,4385 > > >>>> ^^^^^^^^^^^^^ > > >>>> > > >>>> mark > > >>>> > > >>>> > > >>>> On Nov 28, 2003, at 8:18 AM, Orlando Onorato wrote: > > >>>> > > >>>>> By means of flow-export I'm not able to view the Nexhop field, > > >>>>> although this filed there is in Netflow ver.5. > > >>>>> > > >>>>> > > >>>>> ----- Original Message ----- > > >>>>> From: "Bill Fumerola" > > >>>>> To: "Orlando Onorato" > > >>>>> Cc: > > >>>>> Sent: Thursday, November 27, 2003 9:28 PM > > >>>>> Subject: Re: [flow-tools] format flow description > > >>>>> > > >>>>> > > >>>>>> On Thu, Nov 27, 2003 at 03:38:57PM +0100, Orlando Onorato wrote: > > >>>>>> > > >>>>>>> 1) View header description of datagramm generated. > > >>>>>> > > >>>>>> dunno what you mean here, but increasing the debug level will show > > >>>>>> the > > >>>>>> header of the flow file. if you want to see ip or udp header > > >>>>>> information > > >>>>>> from the actual netflow packet, i'd suggest tcpdump. > > >>>>>> > > >>>>>>> 2) View all fields of flow description (e.g. nexthop field. > > >>>>>> > > >>>>>> flow-export > > >>>>>> > > >>>>>> -- > > >>>>>> - bill fumerola / fumerola@yahoo-inc.com / billf@FreeBSD.org > > >>>>>> > > >>>>>> > > >>>>> > > >>>>> _______________________________________________ > > >>>>> flow-tools@splintered.net > > >>>>> http://www.splintered.net/sw/flow-tools > > >>>>> > > >>>> > > >>>> > > >>>> _______________________________________________ > > >>>> flow-tools@splintered.net > > >>>> http://www.splintered.net/sw/flow-tools > > >> > > >> > > >> _______________________________________________ > > >> flow-tools@splintered.net > > >> http://www.splintered.net/sw/flow-tools > > > > > > From splintered-flow-tools-owner@splintered.net Tue Dec 02 15:33:33 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 86317 invoked by uid 4001); 2 Dec 2003 15:33:33 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 86314 invoked by alias); 2 Dec 2003 15:33:32 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 2 Dec 2003 15:33:32 -0000 In-Reply-To: <3FCC4AAD.7070700@tekNico.net> References: <3FCC4AAD.7070700@tekNico.net> Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [flow-tools] flow-nfilter end-time comparisons Date: Tue, 2 Dec 2003 10:33:32 -0500 To: Nicola Larosa X-Mailer: Apple Mail (2.606) Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Dec 2003 15:33:33 -0000 0.67 will be released in the next few days. Yes, the last snapshot was over 6 months ago. I had to take a break from ft development to work on other projects. mark On Dec 2, 2003, at 3:17 AM, Nicola Larosa wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > >> I presume this is making it into the next version of flow-tools? (is >> it >> 0.67 next?) > > Speaking of which, what's the timeline for the release? I have to find > the > time to report patches and suggestions for three items related to > flow-capture. > > They're admittedly minor, but it would be nice if they had a chance to > be at > least considered for the next release, since they seem to appear not > that > often. :^) > > > - -- > "Workaholics and others who can't tear themselves away from the mouse > and > keyboard need to keep their legs active and get away from the computer > for > some exercise every now and then. So, please, get up and stretch every > so > often. You certainly don't want to wind up eDead from eThrombosis." > -- Jenny Thompson, HSI > > Nicola Larosa - nico@tekNico.net > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.2.3 (GNU/Linux) > > iD8DBQE/zEqsXv0hgDImBm4RAgW8AJ4+8KXQJjObE2kTZfxHpXeli/5EUwCguVss > 9oYZR80KQRQkwFAcPXYqiNA= > =7MqC > -----END PGP SIGNATURE----- > > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From splintered-flow-tools-owner@splintered.net Tue Dec 02 15:37:50 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 87056 invoked by uid 4001); 2 Dec 2003 15:37:50 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 87053 invoked by alias); 2 Dec 2003 15:37:50 -0000 Received: from dhcp9578217.columbus.rr.com (HELO ?10.0.0.25?) (24.95.78.217) by 66.250.216.131 with SMTP; 2 Dec 2003 15:37:50 -0000 In-Reply-To: <00eb01c3b8e1$a503d4f0$61af0a0a@Accenture.com> References: <00eb01c3b8e1$a503d4f0$61af0a0a@Accenture.com> Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <7C0EBFEA-24DD-11D8-87CF-000A95DA1C38@splintered.net> Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [flow-tools] format flow description Date: Tue, 2 Dec 2003 10:37:49 -0500 To: "Orlando Onorato" X-Mailer: Apple Mail (2.606) Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Dec 2003 15:37:50 -0000 These fields are only necessary when sending data in Cisco NetFlow format over a network. Modifying flow-tools to store them is probably not the correct approach. What exactly are you trying to do? The "wire" option to flow-export will produce output that looks like what would be sent over the wire. This can then be stored to a file. flow-send will transmit flow-tools data in Cisco NetFlow format over a network. Both the above two options will create the fields you're looking for. mark On Dec 2, 2003, at 9:36 AM, Orlando Onorato wrote: > Ok! > I want understand if flow_gen produce this fields (ushort version, > ushort > count, ulong flow_sequence). I believe yes! > > Then, for obtain this fields in ASCII CSV format I must: > 1) Add this fields in struct fts3rec_v5_gen or struct fts3rec_v5 ? > 2) Modify the function ftio_read(). > > Is it ok? If you can to advice some specific code solution, I will be > very > happy! > > Thanks a lot. > > ----- Original Message ----- > From: "Orlando Onorato" > Sent: Tuesday, December 02, 2003 2:14 PM > Subject: Fw: [flow-tools] format flow description > > >> >> ----- Original Message ----- >> From: "Mark Fullmer" >> To: "Orlando Onorato" >> Cc: >> Sent: Tuesday, December 02, 2003 1:35 AM >> Subject: Re: [flow-tools] format flow description >> >> >>> >>> Flow-tools does not store these fields, they're only necessary >>> to get the data from the exporter to the collector. >>> >>> The 'pad' field is there to provide alignment. >>> >>> You could look in lib/ftdecode.c or src/fdg.c for some examples on >>> how to find these fields >>> >>> mark >>> >>> On Dec 1, 2003, at 9:17 AM, Orlando Onorato wrote: >>> >>>> In particular I need of these header fields: >>>> ushort version, ushort count, ulong flow_sequence. >>>> >>>> Why the field "uchar pad" isn't there in the flow record? >>>> >>>> Thanks for your patience! >>>> >>>> bye >>>> >>>> >>>> ----- Original Message ----- >>>> From: "Mark Fullmer" >>>> To: "Orlando Onorato" >>>> Cc: >>>> Sent: Monday, December 01, 2003 2:13 PM >>>> Subject: Re: [flow-tools] format flow description >>>> >>>> >>>>> Which header are you trying to look at? >>>>> >>>>> mark >>>>> >>>>> On Dec 1, 2003, at 5:09 AM, Orlando Onorato wrote: >>>>> >>>>>> Thank you very much. I've solved my 2nd problem. >>>>>> >>>>>> By mean the flow-export I'm not able to view the header yet, >>>>>> and I don't want to use the tcpdump. >>>>>> >>>>>> Can you help me? >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>> From: "Mark Fullmer" >>>>>> To: "Orlando Onorato" >>>>>> Cc: >>>>>> Sent: Saturday, November 29, 2003 2:20 AM >>>>>> Subject: Re: [flow-tools] format flow description >>>>>> >>>>>> >>>>>>> It's there. Use flow-export -f2 (ASCII). >>>>>>> >>>>>>> #: >>>>>>> > unix_secs,unix_nsecs,sysuptime,exaddr,dpkts,doctets,first,last,engin >>>>>>> e_ >>>>>>> ty >>>>>>> pe,eng >>>>>>> > ine_id,srcaddr,dstaddr,nexthop,input,output,srcport,dstport,prot,tos >>>>>>> ,t >>>>>>> cp >>>>>>> _flags,s >>>>>>> rc_mask,dst_mask,src_as,dst_as >>>>>>> > 1018404899,502659552,3685185404,192.88.192.245,1,40,3685169372,36851 >>>>>>> 69 >>>>>>> 37 >>>>>>> 2,4,0,12 >>>>>>> > 9.137.254.5,129.21.7.49,192.88.192.33,40,25,56279,33508,17,0,16,16,1 >>>>>>> 6, >>>>>>> 20 >>>>>>> 126,4385 >>>>>>> ^^^^^^^^^^^^^ >>>>>>> >>>>>>> mark >>>>>>> >>>>>>> >>>>>>> On Nov 28, 2003, at 8:18 AM, Orlando Onorato wrote: >>>>>>> >>>>>>>> By means of flow-export I'm not able to view the Nexhop field, >>>>>>>> although this filed there is in Netflow ver.5. >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>> From: "Bill Fumerola" >>>>>>>> To: "Orlando Onorato" >>>>>>>> Cc: >>>>>>>> Sent: Thursday, November 27, 2003 9:28 PM >>>>>>>> Subject: Re: [flow-tools] format flow description >>>>>>>> >>>>>>>> >>>>>>>>> On Thu, Nov 27, 2003 at 03:38:57PM +0100, Orlando Onorato >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> 1) View header description of datagramm generated. >>>>>>>>> >>>>>>>>> dunno what you mean here, but increasing the debug level will > show >>>>>>>>> the >>>>>>>>> header of the flow file. if you want to see ip or udp header >>>>>>>>> information >>>>>>>>> from the actual netflow packet, i'd suggest tcpdump. >>>>>>>>> >>>>>>>>>> 2) View all fields of flow description (e.g. nexthop field. >>>>>>>>> >>>>>>>>> flow-export >>>>>>>>> >>>>>>>>> -- >>>>>>>>> - bill fumerola / fumerola@yahoo-inc.com / billf@FreeBSD.org >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> flow-tools@splintered.net >>>>>>>> http://www.splintered.net/sw/flow-tools >>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> flow-tools@splintered.net >>>>>>> http://www.splintered.net/sw/flow-tools >>>>> >>>>> >>>>> _______________________________________________ >>>>> flow-tools@splintered.net >>>>> http://www.splintered.net/sw/flow-tools >>>> >>> >> > From splintered-flow-tools-owner@splintered.net Tue Dec 02 15:58:24 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 87913 invoked by uid 4001); 2 Dec 2003 15:58:24 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 87909 invoked by alias); 2 Dec 2003 15:58:24 -0000 Received: from smtp1.libero.it (193.70.192.51) by 66.250.216.131 with SMTP; 2 Dec 2003 15:58:24 -0000 Received: from H7J31KT8Z604D (151.24.221.152) by smtp1.libero.it (7.0.020-DD01) id 3F6F0E4801683E3F; Tue, 2 Dec 2003 16:58:56 +0100 Message-ID: <00f801c3b8ec$bf0a1ed0$61af0a0a@Accenture.com> From: "Orlando Onorato" To: "Mark Fullmer" , Subject: Fw: [flow-tools] format flow description Date: Tue, 2 Dec 2003 16:55:43 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Dec 2003 15:58:24 -0000 Ok! Perhaps is arrive to speak one's mind! #;-) This is my goal: I receive real-time traffic from Router CiscoNetFlow then I store it into files for future, not real-time, sent to CiscoNetFlow Reader, over network. Since CiscoNetFlow Reader aims some size-format control I must be able to verified the correctness of results. An example CiscoNetFlow Reader aims the control of flow_sequence for reject flow not corret, then I must view this field. If you need about other details, please write me again. You are very kind #;-) ----- Original Message ----- From: "Mark Fullmer" To: "Orlando Onorato" Cc: Sent: Tuesday, December 02, 2003 4:37 PM Subject: Re: [flow-tools] format flow description > > These fields are only necessary when sending data in Cisco NetFlow > format > over a network. Modifying flow-tools to store them is probably not the > correct approach. > > What exactly are you trying to do? > > The "wire" option to flow-export will produce output that looks like > what > would be sent over the wire. This can then be stored to a file. > > flow-send will transmit flow-tools data in Cisco NetFlow format over a > network. > > Both the above two options will create the fields you're looking for. > > mark > > > On Dec 2, 2003, at 9:36 AM, Orlando Onorato wrote: > > > Ok! > > I want understand if flow_gen produce this fields (ushort version, > > ushort > > count, ulong flow_sequence). I believe yes! > > > > Then, for obtain this fields in ASCII CSV format I must: > > 1) Add this fields in struct fts3rec_v5_gen or struct fts3rec_v5 ? > > 2) Modify the function ftio_read(). > > > > Is it ok? If you can to advice some specific code solution, I will be > > very > > happy! > > > > Thanks a lot. > > > > ----- Original Message ----- > > From: "Orlando Onorato" > > Sent: Tuesday, December 02, 2003 2:14 PM > > Subject: Fw: [flow-tools] format flow description > > > > > >> > >> ----- Original Message ----- > >> From: "Mark Fullmer" > >> To: "Orlando Onorato" > >> Cc: > >> Sent: Tuesday, December 02, 2003 1:35 AM > >> Subject: Re: [flow-tools] format flow description > >> > >> > >>> > >>> Flow-tools does not store these fields, they're only necessary > >>> to get the data from the exporter to the collector. > >>> > >>> The 'pad' field is there to provide alignment. > >>> > >>> You could look in lib/ftdecode.c or src/fdg.c for some examples on > >>> how to find these fields > >>> > >>> mark > >>> > >>> On Dec 1, 2003, at 9:17 AM, Orlando Onorato wrote: > >>> > >>>> In particular I need of these header fields: > >>>> ushort version, ushort count, ulong flow_sequence. > >>>> > >>>> Why the field "uchar pad" isn't there in the flow record? > >>>> > >>>> Thanks for your patience! > >>>> > >>>> bye > >>>> > >>>> > >>>> ----- Original Message ----- > >>>> From: "Mark Fullmer" > >>>> To: "Orlando Onorato" > >>>> Cc: > >>>> Sent: Monday, December 01, 2003 2:13 PM > >>>> Subject: Re: [flow-tools] format flow description > >>>> > >>>> > >>>>> Which header are you trying to look at? > >>>>> > >>>>> mark > >>>>> > >>>>> On Dec 1, 2003, at 5:09 AM, Orlando Onorato wrote: > >>>>> > >>>>>> Thank you very much. I've solved my 2nd problem. > >>>>>> > >>>>>> By mean the flow-export I'm not able to view the header yet, > >>>>>> and I don't want to use the tcpdump. > >>>>>> > >>>>>> Can you help me? > >>>>>> > >>>>>> > >>>>>> ----- Original Message ----- > >>>>>> From: "Mark Fullmer" > >>>>>> To: "Orlando Onorato" > >>>>>> Cc: > >>>>>> Sent: Saturday, November 29, 2003 2:20 AM > >>>>>> Subject: Re: [flow-tools] format flow description > >>>>>> > >>>>>> > >>>>>>> It's there. Use flow-export -f2 (ASCII). > >>>>>>> > >>>>>>> #: > >>>>>>> > > unix_secs,unix_nsecs,sysuptime,exaddr,dpkts,doctets,first,last,engin > >>>>>>> e_ > >>>>>>> ty > >>>>>>> pe,eng > >>>>>>> > > ine_id,srcaddr,dstaddr,nexthop,input,output,srcport,dstport,prot,tos > >>>>>>> ,t > >>>>>>> cp > >>>>>>> _flags,s > >>>>>>> rc_mask,dst_mask,src_as,dst_as > >>>>>>> > > 1018404899,502659552,3685185404,192.88.192.245,1,40,3685169372,36851 > >>>>>>> 69 > >>>>>>> 37 > >>>>>>> 2,4,0,12 > >>>>>>> > > 9.137.254.5,129.21.7.49,192.88.192.33,40,25,56279,33508,17,0,16,16,1 > >>>>>>> 6, > >>>>>>> 20 > >>>>>>> 126,4385 > >>>>>>> ^^^^^^^^^^^^^ > >>>>>>> > >>>>>>> mark > >>>>>>> > >>>>>>> > >>>>>>> On Nov 28, 2003, at 8:18 AM, Orlando Onorato wrote: > >>>>>>> > >>>>>>>> By means of flow-export I'm not able to view the Nexhop field, > >>>>>>>> although this filed there is in Netflow ver.5. > >>>>>>>> > >>>>>>>> > >>>>>>>> ----- Original Message ----- > >>>>>>>> From: "Bill Fumerola" > >>>>>>>> To: "Orlando Onorato" > >>>>>>>> Cc: > >>>>>>>> Sent: Thursday, November 27, 2003 9:28 PM > >>>>>>>> Subject: Re: [flow-tools] format flow description > >>>>>>>> > >>>>>>>> > >>>>>>>>> On Thu, Nov 27, 2003 at 03:38:57PM +0100, Orlando Onorato > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> 1) View header description of datagramm generated. > >>>>>>>>> > >>>>>>>>> dunno what you mean here, but increasing the debug level will > > show > >>>>>>>>> the > >>>>>>>>> header of the flow file. if you want to see ip or udp header > >>>>>>>>> information > >>>>>>>>> from the actual netflow packet, i'd suggest tcpdump. > >>>>>>>>> > >>>>>>>>>> 2) View all fields of flow description (e.g. nexthop field. > >>>>>>>>> > >>>>>>>>> flow-export > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> - bill fumerola / fumerola@yahoo-inc.com / billf@FreeBSD.org > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> flow-tools@splintered.net > >>>>>>>> http://www.splintered.net/sw/flow-tools > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> flow-tools@splintered.net > >>>>>>> http://www.splintered.net/sw/flow-tools > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> flow-tools@splintered.net > >>>>> http://www.splintered.net/sw/flow-tools > >>>> > >>> > >> > > > From splintered-Flow-tools-owner@splintered.net Tue Dec 02 19:42:53 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 90080 invoked by uid 4001); 2 Dec 2003 19:42:53 -0000 Delivered-To: splintered-Flow-tools@splintered.net Received: (qmail 90077 invoked by alias); 2 Dec 2003 19:42:53 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 2 Dec 2003 19:42:53 -0000 In-Reply-To: <71E9FA687B04C94EBD1CCEE7DCEDA8EE5EA74B@rodan.motive.com> References: <71E9FA687B04C94EBD1CCEE7DCEDA8EE5EA74B@rodan.motive.com> Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [flow-tools] Networks Date: Tue, 2 Dec 2003 14:42:51 -0500 To: "Samson Martinez" X-Mailer: Apple Mail (2.606) Cc: flowscan@net.doit.wisc.edu, Flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Dec 2003 19:42:53 -0000 Redirected to the FlowScan mailing list.... mark On Dec 1, 2003, at 9:16 AM, Samson Martinez wrote: > Hello, > > > I've configured flow-tools with Flowscan & CUFlow and, so far, > everything appears to be working well. However, I'm trying to identify > traffic that is showing up as a substantial part of the reported flows. > Our network, in a nutshell, is as follows: > > 2 Cisco 7204VXRs that sit on our network boundary, both exporting > version 5 flows to a Sun Solaris server. > > I have configured 7 subnets as local as these are networks that sit > behind the 7204s. I then added these same subnets in their subnetted > form to the "Networks Interested In" portion of the configuration. > > All the traffic appears to be properly accounted for but when I show > the > graphs I see 20% In & 41% Out traffic identified as "Other networks". > > What is the best way to isolate and identify those "Other Networks"? > > Many thanks for all your assistance. > > By the way, the same is occurring with the "Services" and "Protocols". > > Regards, > > -Samson Martinez > > > _______________________________________________ > flow-tools@splintered.net > http://www.splintered.net/sw/flow-tools > From splintered-flow-tools-owner@splintered.net Wed Dec 03 16:56:36 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 95043 invoked by uid 4001); 3 Dec 2003 16:56:36 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 95039 invoked by alias); 3 Dec 2003 16:56:35 -0000 Received: from smtp2.libero.it (193.70.192.52) by 66.250.216.131 with SMTP; 3 Dec 2003 16:56:35 -0000 Received: from H7J31KT8Z604D (151.24.216.135) by smtp2.libero.it (7.0.020-DD01) id 3F6F0DA9016D6964; Wed, 3 Dec 2003 17:57:19 +0100 Message-ID: <002401c3b9be$09cc88f0$61af0a0a@Accenture.com> From: "Orlando Onorato" To: "Mark Fullmer" References: <00eb01c3b8e1$a503d4f0$61af0a0a@Accenture.com> <7C0EBFEA-24DD-11D8-87CF-000A95DA1C38@splintered.net> Subject: Re: [flow-tools] format flow description Date: Wed, 3 Dec 2003 17:53:54 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Dec 2003 16:56:36 -0000 Hi Mark, probably I've success about my goal - thanks to you #;-) Now I've this problem: whit flow-gen the flow_sequence is ever equals to 0. Since for now I haven't got a real netflow traffic I wold be test my changes... #;-( have you got some real netflow traffic stored into files? Thanks a lot. Orlando ----- Original Message ----- From: "Mark Fullmer" To: "Orlando Onorato" Cc: Sent: Tuesday, December 02, 2003 4:37 PM Subject: Re: [flow-tools] format flow description > > These fields are only necessary when sending data in Cisco NetFlow > format > over a network. Modifying flow-tools to store them is probably not the > correct approach. > > What exactly are you trying to do? > > The "wire" option to flow-export will produce output that looks like > what > would be sent over the wire. This can then be stored to a file. > > flow-send will transmit flow-tools data in Cisco NetFlow format over a > network. > > Both the above two options will create the fields you're looking for. > > mark > > > On Dec 2, 2003, at 9:36 AM, Orlando Onorato wrote: > > > Ok! > > I want understand if flow_gen produce this fields (ushort version, > > ushort > > count, ulong flow_sequence). I believe yes! > > > > Then, for obtain this fields in ASCII CSV format I must: > > 1) Add this fields in struct fts3rec_v5_gen or struct fts3rec_v5 ? > > 2) Modify the function ftio_read(). > > > > Is it ok? If you can to advice some specific code solution, I will be > > very > > happy! > > > > Thanks a lot. > > > > ----- Original Message ----- > > From: "Orlando Onorato" > > Sent: Tuesday, December 02, 2003 2:14 PM > > Subject: Fw: [flow-tools] format flow description > > > > > >> > >> ----- Original Message ----- > >> From: "Mark Fullmer" > >> To: "Orlando Onorato" > >> Cc: > >> Sent: Tuesday, December 02, 2003 1:35 AM > >> Subject: Re: [flow-tools] format flow description > >> > >> > >>> > >>> Flow-tools does not store these fields, they're only necessary > >>> to get the data from the exporter to the collector. > >>> > >>> The 'pad' field is there to provide alignment. > >>> > >>> You could look in lib/ftdecode.c or src/fdg.c for some examples on > >>> how to find these fields > >>> > >>> mark > >>> > >>> On Dec 1, 2003, at 9:17 AM, Orlando Onorato wrote: > >>> > >>>> In particular I need of these header fields: > >>>> ushort version, ushort count, ulong flow_sequence. > >>>> > >>>> Why the field "uchar pad" isn't there in the flow record? > >>>> > >>>> Thanks for your patience! > >>>> > >>>> bye > >>>> > >>>> > >>>> ----- Original Message ----- > >>>> From: "Mark Fullmer" > >>>> To: "Orlando Onorato" > >>>> Cc: > >>>> Sent: Monday, December 01, 2003 2:13 PM > >>>> Subject: Re: [flow-tools] format flow description > >>>> > >>>> > >>>>> Which header are you trying to look at? > >>>>> > >>>>> mark > >>>>> > >>>>> On Dec 1, 2003, at 5:09 AM, Orlando Onorato wrote: > >>>>> > >>>>>> Thank you very much. I've solved my 2nd problem. > >>>>>> > >>>>>> By mean the flow-export I'm not able to view the header yet, > >>>>>> and I don't want to use the tcpdump. > >>>>>> > >>>>>> Can you help me? > >>>>>> > >>>>>> > >>>>>> ----- Original Message ----- > >>>>>> From: "Mark Fullmer" > >>>>>> To: "Orlando Onorato" > >>>>>> Cc: > >>>>>> Sent: Saturday, November 29, 2003 2:20 AM > >>>>>> Subject: Re: [flow-tools] format flow description > >>>>>> > >>>>>> > >>>>>>> It's there. Use flow-export -f2 (ASCII). > >>>>>>> > >>>>>>> #: > >>>>>>> > > unix_secs,unix_nsecs,sysuptime,exaddr,dpkts,doctets,first,last,engin > >>>>>>> e_ > >>>>>>> ty > >>>>>>> pe,eng > >>>>>>> > > ine_id,srcaddr,dstaddr,nexthop,input,output,srcport,dstport,prot,tos > >>>>>>> ,t > >>>>>>> cp > >>>>>>> _flags,s > >>>>>>> rc_mask,dst_mask,src_as,dst_as > >>>>>>> > > 1018404899,502659552,3685185404,192.88.192.245,1,40,3685169372,36851 > >>>>>>> 69 > >>>>>>> 37 > >>>>>>> 2,4,0,12 > >>>>>>> > > 9.137.254.5,129.21.7.49,192.88.192.33,40,25,56279,33508,17,0,16,16,1 > >>>>>>> 6, > >>>>>>> 20 > >>>>>>> 126,4385 > >>>>>>> ^^^^^^^^^^^^^ > >>>>>>> > >>>>>>> mark > >>>>>>> > >>>>>>> > >>>>>>> On Nov 28, 2003, at 8:18 AM, Orlando Onorato wrote: > >>>>>>> > >>>>>>>> By means of flow-export I'm not able to view the Nexhop field, > >>>>>>>> although this filed there is in Netflow ver.5. > >>>>>>>> > >>>>>>>> > >>>>>>>> ----- Original Message ----- > >>>>>>>> From: "Bill Fumerola" > >>>>>>>> To: "Orlando Onorato" > >>>>>>>> Cc: > >>>>>>>> Sent: Thursday, November 27, 2003 9:28 PM > >>>>>>>> Subject: Re: [flow-tools] format flow description > >>>>>>>> > >>>>>>>> > >>>>>>>>> On Thu, Nov 27, 2003 at 03:38:57PM +0100, Orlando Onorato > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> 1) View header description of datagramm generated. > >>>>>>>>> > >>>>>>>>> dunno what you mean here, but increasing the debug level will > > show > >>>>>>>>> the > >>>>>>>>> header of the flow file. if you want to see ip or udp header > >>>>>>>>> information > >>>>>>>>> from the actual netflow packet, i'd suggest tcpdump. > >>>>>>>>> > >>>>>>>>>> 2) View all fields of flow description (e.g. nexthop field. > >>>>>>>>> > >>>>>>>>> flow-export > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> - bill fumerola / fumerola@yahoo-inc.com / billf@FreeBSD.org > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> flow-tools@splintered.net > >>>>>>>> http://www.splintered.net/sw/flow-tools > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> flow-tools@splintered.net > >>>>>>> http://www.splintered.net/sw/flow-tools > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> flow-tools@splintered.net > >>>>> http://www.splintered.net/sw/flow-tools > >>>> > >>> > >> > > > From splintered-flow-tools-owner@splintered.net Wed Dec 03 21:55:09 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 96790 invoked by uid 4001); 3 Dec 2003 21:55:09 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 96788 invoked by alias); 3 Dec 2003 21:55:08 -0000 Received: from web21501.mail.yahoo.com (66.163.169.12) by 66.250.216.131 with SMTP; 3 Dec 2003 21:55:08 -0000 Message-ID: <20031203215507.51408.qmail@web21501.mail.yahoo.com> Received: from [138.222.250.65] by web21501.mail.yahoo.com via HTTP; Wed, 03 Dec 2003 13:55:07 PST Date: Wed, 3 Dec 2003 13:55:07 -0800 (PST) From: johann lafer To: flow-tools@splintered.net MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-985996298-1070488507=:50736" Cc: Subject: [Flow-tools] Performance / compile question X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Dec 2003 21:55:09 -0000 --0-985996298-1070488507=:50736 Content-Type: text/plain; charset=us-ascii Hello Mark, in the mailing list i found http://www.pairlist.net/pipermail/flow-tools/2003-January/001007.html where you wrote, that compiling flow-tools with -O or -O2 increases the performance. I tried to use the cflags, but running "make" still shows -g -Wall. Which modifications do i have to do where? Have you ever tried to compile flow-tools with the processor option ?(i know this is more a linux question). Thanks Janno --------------------------------- Do you Yahoo!? Free Pop-Up Blocker - Get it now --0-985996298-1070488507=:50736 Content-Type: text/html; charset=us-ascii
Hello Mark,
 
in the mailing list i found
 
 
where you wrote, that compiling flow-tools with -O or -O2 increases the performance.
 
I tried to use the cflags, but running "make" still shows -g -Wall. Which modifications do i have to do where? Have you ever tried to compile flow-tools with the processor option ?(i know this is more a linux question). 
 
Thanks
 
Janno

Do you Yahoo!?
Free Pop-Up Blocker - Get it now --0-985996298-1070488507=:50736-- From splintered-flow-tools-owner@splintered.net Wed Dec 03 23:42:03 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 97945 invoked by uid 4001); 3 Dec 2003 23:42:03 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 97942 invoked by alias); 3 Dec 2003 23:42:03 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 3 Dec 2003 23:42:03 -0000 In-Reply-To: <20031203215507.51408.qmail@web21501.mail.yahoo.com> References: <20031203215507.51408.qmail@web21501.mail.yahoo.com> Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=ISO-8859-1; format=flowed Message-Id: <4B20BDF4-25EA-11D8-A8C6-000A95DA1C38@splintered.net> Content-Transfer-Encoding: quoted-printable From: Mark Fullmer Subject: Re: [Flow-tools] Performance / compile question Date: Wed, 3 Dec 2003 18:42:02 -0500 To: johann lafer X-Mailer: Apple Mail (2.606) Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Dec 2003 23:42:03 -0000 src/Makefile.am and lib/Makefile.am. You'll need to have automake installed. Haven't tried any of the processor options to gcc. Are you having performance problems? mark On Dec 3, 2003, at 4:55 PM, johann lafer wrote: > Hello Mark, > =A0 > in the mailing list i found > =A0 > http://www.pairlist.net/pipermail/flow-tools/2003-January/001007.html > =A0 > where you wrote, that compiling flow-tools with -O or -O2 increases=20 > the performance. > =A0 > I tried to use the cflags, but running "make" still shows -g -Wall.=20 > Which modifications do i have to do where? Have you ever tried to=20 > compile flow-tools with the processor option ?(i know this is more a=20= > linux question).=A0 > =A0 > Thanks > =A0 > Janno > > Do you Yahoo!? > Free Pop-Up Blocker - Get it=20 > now_______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools= From splintered-flow-tools-owner@splintered.net Thu Dec 04 05:08:23 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 99956 invoked by uid 4001); 4 Dec 2003 05:08:23 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 99953 invoked by alias); 4 Dec 2003 05:08:23 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 4 Dec 2003 05:08:23 -0000 In-Reply-To: <002401c3b9be$09cc88f0$61af0a0a@Accenture.com> References: <00eb01c3b8e1$a503d4f0$61af0a0a@Accenture.com> <7C0EBFEA-24DD-11D8-87CF-000A95DA1C38@splintered.net> <002401c3b9be$09cc88f0$61af0a0a@Accenture.com> Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [flow-tools] format flow description Date: Thu, 4 Dec 2003 00:08:22 -0500 To: "Orlando Onorato" X-Mailer: Apple Mail (2.606) Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Dec 2003 05:08:23 -0000 It sounds like you want to be using flow-fanout to replicate the data from a router to multiple destinations... The data does not have sequence numbers in it. It's possible flow-tools will create the first flow with a sequence number with the number of flows encoded, this is just how its implemented. There's nothing special about when the sequence numbers start. mark On Dec 3, 2003, at 11:53 AM, Orlando Onorato wrote: > Hi Mark, > probably I've success about my goal - thanks to you #;-) > > Now I've this problem: whit flow-gen the flow_sequence is ever equals > to 0. > Since for now I haven't got a real netflow traffic I wold be test my > changes... #;-( > have you got some real netflow traffic stored into files? > > Thanks a lot. > > Orlando > > > ----- Original Message ----- > From: "Mark Fullmer" > To: "Orlando Onorato" > Cc: > Sent: Tuesday, December 02, 2003 4:37 PM > Subject: Re: [flow-tools] format flow description > > >> >> These fields are only necessary when sending data in Cisco NetFlow >> format >> over a network. Modifying flow-tools to store them is probably not >> the >> correct approach. >> >> What exactly are you trying to do? >> >> The "wire" option to flow-export will produce output that looks like >> what >> would be sent over the wire. This can then be stored to a file. >> >> flow-send will transmit flow-tools data in Cisco NetFlow format over a >> network. >> >> Both the above two options will create the fields you're looking for. >> >> mark >> >> >> On Dec 2, 2003, at 9:36 AM, Orlando Onorato wrote: >> >>> Ok! >>> I want understand if flow_gen produce this fields (ushort version, >>> ushort >>> count, ulong flow_sequence). I believe yes! >>> >>> Then, for obtain this fields in ASCII CSV format I must: >>> 1) Add this fields in struct fts3rec_v5_gen or struct fts3rec_v5 ? >>> 2) Modify the function ftio_read(). >>> >>> Is it ok? If you can to advice some specific code solution, I will be >>> very >>> happy! >>> >>> Thanks a lot. >>> >>> ----- Original Message ----- >>> From: "Orlando Onorato" >>> Sent: Tuesday, December 02, 2003 2:14 PM >>> Subject: Fw: [flow-tools] format flow description >>> >>> >>>> >>>> ----- Original Message ----- >>>> From: "Mark Fullmer" >>>> To: "Orlando Onorato" >>>> Cc: >>>> Sent: Tuesday, December 02, 2003 1:35 AM >>>> Subject: Re: [flow-tools] format flow description >>>> >>>> >>>>> >>>>> Flow-tools does not store these fields, they're only necessary >>>>> to get the data from the exporter to the collector. >>>>> >>>>> The 'pad' field is there to provide alignment. >>>>> >>>>> You could look in lib/ftdecode.c or src/fdg.c for some examples on >>>>> how to find these fields >>>>> >>>>> mark >>>>> >>>>> On Dec 1, 2003, at 9:17 AM, Orlando Onorato wrote: >>>>> >>>>>> In particular I need of these header fields: >>>>>> ushort version, ushort count, ulong flow_sequence. >>>>>> >>>>>> Why the field "uchar pad" isn't there in the flow record? >>>>>> >>>>>> Thanks for your patience! >>>>>> >>>>>> bye >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>> From: "Mark Fullmer" >>>>>> To: "Orlando Onorato" >>>>>> Cc: >>>>>> Sent: Monday, December 01, 2003 2:13 PM >>>>>> Subject: Re: [flow-tools] format flow description >>>>>> >>>>>> >>>>>>> Which header are you trying to look at? >>>>>>> >>>>>>> mark >>>>>>> >>>>>>> On Dec 1, 2003, at 5:09 AM, Orlando Onorato wrote: >>>>>>> >>>>>>>> Thank you very much. I've solved my 2nd problem. >>>>>>>> >>>>>>>> By mean the flow-export I'm not able to view the header yet, >>>>>>>> and I don't want to use the tcpdump. >>>>>>>> >>>>>>>> Can you help me? >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>> From: "Mark Fullmer" >>>>>>>> To: "Orlando Onorato" >>>>>>>> Cc: >>>>>>>> Sent: Saturday, November 29, 2003 2:20 AM >>>>>>>> Subject: Re: [flow-tools] format flow description >>>>>>>> >>>>>>>> >>>>>>>>> It's there. Use flow-export -f2 (ASCII). >>>>>>>>> >>>>>>>>> #: >>>>>>>>> >>> unix_secs,unix_nsecs,sysuptime,exaddr,dpkts,doctets,first,last,engin >>>>>>>>> e_ >>>>>>>>> ty >>>>>>>>> pe,eng >>>>>>>>> >>> ine_id,srcaddr,dstaddr,nexthop,input,output,srcport,dstport,prot,tos >>>>>>>>> ,t >>>>>>>>> cp >>>>>>>>> _flags,s >>>>>>>>> rc_mask,dst_mask,src_as,dst_as >>>>>>>>> >>> 1018404899,502659552,3685185404,192.88.192.245,1,40,3685169372,36851 >>>>>>>>> 69 >>>>>>>>> 37 >>>>>>>>> 2,4,0,12 >>>>>>>>> >>> 9.137.254.5,129.21.7.49,192.88.192.33,40,25,56279,33508,17,0,16,16,1 >>>>>>>>> 6, >>>>>>>>> 20 >>>>>>>>> 126,4385 >>>>>>>>> ^^^^^^^^^^^^^ >>>>>>>>> >>>>>>>>> mark >>>>>>>>> >>>>>>>>> >>>>>>>>> On Nov 28, 2003, at 8:18 AM, Orlando Onorato wrote: >>>>>>>>> >>>>>>>>>> By means of flow-export I'm not able to view the Nexhop field, >>>>>>>>>> although this filed there is in Netflow ver.5. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Bill Fumerola" >>>>>>>>>> To: "Orlando Onorato" >>>>>>>>>> Cc: >>>>>>>>>> Sent: Thursday, November 27, 2003 9:28 PM >>>>>>>>>> Subject: Re: [flow-tools] format flow description >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Thu, Nov 27, 2003 at 03:38:57PM +0100, Orlando Onorato >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> 1) View header description of datagramm generated. >>>>>>>>>>> >>>>>>>>>>> dunno what you mean here, but increasing the debug level will >>> show >>>>>>>>>>> the >>>>>>>>>>> header of the flow file. if you want to see ip or udp header >>>>>>>>>>> information >>>>>>>>>>> from the actual netflow packet, i'd suggest tcpdump. >>>>>>>>>>> >>>>>>>>>>>> 2) View all fields of flow description (e.g. nexthop field. >>>>>>>>>>> >>>>>>>>>>> flow-export >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> - bill fumerola / fumerola@yahoo-inc.com / billf@FreeBSD.org >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> flow-tools@splintered.net >>>>>>>>>> http://www.splintered.net/sw/flow-tools >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> flow-tools@splintered.net >>>>>>>>> http://www.splintered.net/sw/flow-tools >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> flow-tools@splintered.net >>>>>>> http://www.splintered.net/sw/flow-tools >>>>>> >>>>> >>>> >>> >> > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From splintered-flow-tools-owner@splintered.net Thu Dec 04 06:08:28 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 1176 invoked by uid 4001); 4 Dec 2003 06:08:28 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 1172 invoked by alias); 4 Dec 2003 06:08:28 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 4 Dec 2003 06:08:28 -0000 In-Reply-To: <3EE9EC1A0F3FA640B5F99FF3B7C915FD02EE61A5@snkxs001.scrippsnetworks.com> References: <3EE9EC1A0F3FA640B5F99FF3B7C915FD02EE61A5@snkxs001.scrippsnetworks.com> Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <46977FA2-2620-11D8-A8C6-000A95DA1C38@splintered.net> Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [flow-tools] Tools to watch for viruses or worms? Date: Thu, 4 Dec 2003 01:08:27 -0500 To: "Cowell, Andrew" X-Mailer: Apple Mail (2.606) Cc: "'flow-tools@splintered.net'" X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Dec 2003 06:08:28 -0000 You can IP catch scanners with the ip-source-address-destination-count and/or ip-destination-address-source-count. The ip-destination-address-source count was broken in 0.66. It's fixed in 0.67. mark On Oct 22, 2003, at 1:00 PM, Cowell, Andrew wrote: > Hey, I'm trying to use flow exports to watch for viruses and worms on > our > network. I haven't found any tools to do so yet. Anybody know of one? > Does anybody already have flow-tools filters for various worm > signatures? > The main suspicious activity I've been watching for has been sequential > network mapping, but I don't see how to catch that with flow-tools. > Any > ideas? > > -- > Andy Cowell > acowell@scrippsops.com > Senior Network Administrator > E.W. Scripps Corp. IT Operations and Engineering > ph: (865) 560-4652 > > > _______________________________________________ > flow-tools@splintered.net > http://www.splintered.net/sw/flow-tools > > From splintered-flow-tools-owner@splintered.net Thu Dec 04 06:41:55 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 2157 invoked by uid 4001); 4 Dec 2003 06:41:55 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 2155 invoked by alias); 4 Dec 2003 06:41:55 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 4 Dec 2003 06:41:55 -0000 Mime-Version: 1.0 (Apple Message framework v606) Content-Transfer-Encoding: 7bit Message-Id: Content-Type: text/plain; charset=US-ASCII; format=flowed To: 'flow-tools@splintered.net' From: Mark Fullmer Date: Thu, 4 Dec 2003 01:41:54 -0500 X-Mailer: Apple Mail (2.606) Cc: Subject: [Flow-tools] flow-tools 0.67 X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Dec 2003 06:41:55 -0000 A new snapshot of flow-tools is available at http://www.splintered.net/sw/flow-tools. I've been away from ft development for about 6 months, so there are a lot of accumulated contributions, fixes for reported bugs, and some new code. I'm expecting to have another snapshot out around the first of the year which will include initial support for NetFlow v9. Full support for v9 requires a file format change which probably won't be done until a little later. If you've sent a bug fix, request, or just a bug report that didn't make it in to 0.67 please send it on again, it probably got lost in my junk mailbox. mark * 12-4-2003 flow-tools 0.67 released. * flow-export: pgsql support from wyu@ateneo.edu * docs: flow-report: Added description of reports. * ftlib: ftfil.c - match_end_time() broken - noted by "Joe Loiacono" * ftlib: fttag.c - better syntax checking for or-src/dst and set-src/dst * ftlib: ftlib.h FT_TAG_TYPE_MATCH_NEXTHOP duplicated - flow-tag crash with next-hop type noted by Maxim Grigoriev * ftlib: ftstat.c - broken ip-destination-address-source-count. patch from "Shigeki Taniguchi" * flow-fanout: filters not loaded - noted by RAR@syssrc.com * ftlib: missing function prototypes for ftstat_*, rename bind to binding to prevent shadowing bind(). patch from Bill Fumerola * flow-fanout, flow-capture. Process SIGTERM like SIGQUIT so flow-tools will work better under daemontools - req by Bernhard Weisshuhn * docs: flow-nfilter and flow-cat TIME/DATE parsing section. * flow-dscan: drp->flags not updated when loading saved state - patch from Jon Snyder * flow-dscan: allow concurrent -w and -W, patch from Dan Thorson * docs: flow-print -f24 - noted by noted by Christian Bauer * dist: tag.sym and tag.cfg example files reversed - noted by * ftlib: ftlib.h - FT_TAG_SET and FT_TAG_OR are broken - patch from Valtteri Vuorikoski * ftlib: ftrec.c - add 1005to5 translation - patch from Valtteri Vuorikoski * flow-stat -f0 will try to divide by 0 with an empty flow file - noted by Mike Hunter * flow-capture: -u preserve unherited umask - patch from Everton da Silva Marques * flow-receive: remove -m and -A. * flow-capture: remove -m and -A, functionality is now in xlate -x -X. * flow-xlate: - config file based now. * docs: flow-report: note which fields are sortable and what the key field is. * flow-capture: accept()'s 3rd arg should be casted to socklen_t*, noted by Alistair.McGlinchy@marks-and-spencer.com * docs: flow-nfilter, port is 0..65535 not 0..255 - noted by Mike Hunter * ftlib: ftlib.h - set-{dst,src} and or-{dst,src} constants not correct - patch from Valtteri Vuorikoski * ftlib: ftchash.c - ftchash_sort() should not try to sort 0 entry table - noted by "Shane D." * flow-import: missing !HAVE_STRSEP compatability - patch from Alistair.McGlinchy@marks-and-spencer.com * ftlib: ftstat.c - output path not parsed correctly with leading whitespace -- noted by Maxim Grigoriev * ftlib: fttag.c - src->source dst->destination * ftlib: fttag.c - ip-address, exporter, interface tag actions, requested by Tim Irwin * ftlib: ftsym.c - ftsym_new() should handle null filename - noted by Celso Alves Vieira * flow-dscan: buf len 64, not 54 - Anil Madhavapeddy From splintered-flow-tools-owner@splintered.net Thu Dec 04 10:30:16 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 4198 invoked by uid 4001); 4 Dec 2003 10:30:16 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 4195 invoked by alias); 4 Dec 2003 10:30:15 -0000 Received: from webmail7.rediffmail.com (HELO rediffmail.com) (202.54.124.152) by 66.250.216.131 with SMTP; 4 Dec 2003 10:30:15 -0000 Received: (qmail 16476 invoked by uid 510); 4 Dec 2003 10:29:50 -0000 Date: 4 Dec 2003 10:29:50 -0000 Message-ID: <20031204102950.16474.qmail@webmail7.rediffmail.com> Received: from unknown (203.200.25.5) by rediffmail.com via HTTP; 04 dec 2003 10:29:50 -0000 MIME-Version: 1.0 From: "Chitman Kaur " To: flow-tools@splintered.net Content-type: multipart/alternative; boundary="Next_1070533790---0-202.54.124.152-16465" Cc: Subject: [Flow-tools] (no subject) X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list Reply-To: Chitman Kaur List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Dec 2003 10:30:16 -0000 This is a multipart mime message --Next_1070533790---0-202.54.124.152-16465 Content-type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline

=0AHi All....
=0AEven after putting the option -z0 I am getting lost = flows.....
=0AAny reason why....
=0AI am sure that there is no conges= tion between my router and collector.....
=0AAm I giving the option corr= ectly....
=0A-----------------------------------------------------------= ------------
=0A/usr/local/netflow/bin/flow-capture -w /var/netflow/ft 0= /0/2055 -z0 -S5 -V5 -E1G -n 287 -N 0 -R /usr/local/netflow/bin/linkme
= =0Atouch /var/lock/subsys/startflows
=0A--------------------------------= ---------------------------------------
=0ARegards
=0AChitman=0A

= =0A=0A=0A

=0A=0A --Next_1070533790---0-202.54.124.152-16465 Content-type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi All....=0AEven after putting the option -z0 I am getting lost flows.....= =0AAny reason why....=0AI am sure that there is no congestion between my ro= uter and collector.....=0AAm I giving the option correctly....=0A----------= -------------------------------------------------------------=0A/usr/local/= netflow/bin/flow-capture -w /var/netflow/ft 0/0/2055 -z0 -S5 -V5 -E1G -n 28= 7 -N 0 -R /usr/local/netflow/bin/linkme=0Atouch /var/lock/subsys/startflows= =0A-----------------------------------------------------------------------= =0ARegards=0AChitman=0A=0A --Next_1070533790---0-202.54.124.152-16465-- From splintered-flow-tools-owner@splintered.net Thu Dec 04 15:55:03 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 5981 invoked by uid 4001); 4 Dec 2003 15:55:03 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 5979 invoked by alias); 4 Dec 2003 15:55:02 -0000 Received: from web21504.mail.yahoo.com (66.163.169.15) by 66.250.216.131 with SMTP; 4 Dec 2003 15:55:02 -0000 Message-ID: <20031204155458.86623.qmail@web21504.mail.yahoo.com> Received: from [138.222.250.65] by web21504.mail.yahoo.com via HTTP; Thu, 04 Dec 2003 07:54:58 PST Date: Thu, 4 Dec 2003 07:54:58 -0800 (PST) From: johann lafer Subject: Re: [Flow-tools] Performance / compile question To: flow-tools@splintered.net In-Reply-To: <4B20BDF4-25EA-11D8-A8C6-000A95DA1C38@splintered.net> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-875184259-1070553298=:86102" Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Dec 2003 15:55:03 -0000 --0-875184259-1070553298=:86102 Content-Type: text/plain; charset=us-ascii Hm, that wasn't so easy. First I had to install autoconf and automake. Then I changed Makefile.am in both dirs. Running make, there is still a "-g -Wall" visible. So I am not really shure, if the optimization happend. I also appended -mpentium. But i do not notice any performance optimization. I don't know if it is really a performance problem, but the whole story is, that i have written a php frontend with flow-tools (flow-cat,flow-nfilter,flow-stat) in the background. If a web-user wants to generate a 1,3,12 or 24 hour report, it takes up to 5 minutes. But creating a "week-report" or "month-report" takes more than 25 minutes, depending on the generated data. The size of 5 minute flow-file differs between 1 and 4 Mbytes. Maybe i am impatient?! It looks like a hanging application if you have to wait to long. Yesterday i tried flow-export -z to compress all files generated the day before to single compressed file. The volume seems to be compress 10% and a report was generated 30% faster, but i do not know if this is an accident. Another idea is to run more than 1 process for a report (up to 31 process for a month report, bottleneck competing CPU and/or HD access?!) Is there a possibility to make the data volume smaller without losing important information? Thanks Janno Mark Fullmer wrote: src/Makefile.am and lib/Makefile.am. You'll need to have automake installed. Haven't tried any of the processor options to gcc. Are you having performance problems? mark On Dec 3, 2003, at 4:55 PM, johann lafer wrote: > Hello Mark, > > in the mailing list i found > > http://www.pairlist.net/pipermail/flow-tools/2003-January/001007.html > > where you wrote, that compiling flow-tools with -O or -O2 increases > the performance. > > I tried to use the cflags, but running "make" still shows -g -Wall. > Which modifications do i have to do where? Have you ever tried to > compile flow-tools with the processor option ?(i know this is more a > linux question). > > Thanks > > Janno > > Do you Yahoo!? > Free Pop-Up Blocker - Get it > now_______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools --------------------------------- Do you Yahoo!? Free Pop-Up Blocker - Get it now --0-875184259-1070553298=:86102 Content-Type: text/html; charset=us-ascii
Hm,
 
that wasn't so easy. First I had to install autoconf and automake. Then I changed Makefile.am in both dirs. Running make, there is still a "-g -Wall" visible. So I am not really shure, if the optimization happend. I also appended -mpentium. But i do not notice any performance optimization.
 
I don't know if it is really a performance problem, but the whole story is, that i have written a php frontend with flow-tools (flow-cat,flow-nfilter,flow-stat) in the background. If a web-user wants to generate a 1,3,12 or 24 hour report, it takes up to 5 minutes. But creating a "week-report" or "month-report"  takes more than 25 minutes, depending on the generated data. The size of 5 minute flow-file differs between 1 and 4 Mbytes. Maybe i am impatient?! It looks like a hanging application if you have to wait to long.
 
Yesterday i tried flow-export -z to compress all files generated the day before to single compressed file. The volume seems to be compress 10% and a report was generated 30% faster, but i do not know if this is an accident.
 
Another idea is to run more than 1 process for a report (up to 31 process for a month report, bottleneck competing CPU and/or HD access?!)
 
Is there a possibility to make the data volume smaller without losing important information?
 
Thanks
Janno

Mark Fullmer <maf@splintered.net> wrote:
src/Makefile.am and lib/Makefile.am. You'll need to have automake
installed. Haven't tried any of the processor options to gcc.

Are you having performance problems?

mark

On Dec 3, 2003, at 4:55 PM, johann lafer wrote:

> Hello Mark,
>  
> in the mailing list i found
>  
> http://www.pairlist.net/pipermail/flow-tools/2003-January/001007.html
>  
> where you wrote, that compiling flow-tools with -O or -O2 increases
> the performance.
>  
> I tried to use the cflags, but running "make" still shows -g -Wall.
> Which modifications do i have to do where? Have you ever tried to
> compile flow-tools with the processor option ?(i know this is more a
> linux question). 
>  
> Thanks
>  
> Janno
>
> Do you Yahoo!?
> Free Pop-Up Blocker - Get it
> now_______________________________________________
> Flow-tools mailing list
> flow-tools@splintered.net
> http://mailman.splintered.net/mailman/listinfo/flow-tools

Do you Yahoo!?
Free Pop-Up Blocker - Get it now --0-875184259-1070553298=:86102-- From splintered-flow-tools-owner@splintered.net Thu Dec 04 18:53:13 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 7254 invoked by uid 4001); 4 Dec 2003 18:53:13 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 7252 invoked by alias); 4 Dec 2003 18:53:12 -0000 Received: from ip166.usw253.dsl-acs2.sea.iinet.com (HELO ran.psg.com) (209.20.253.166) by 66.250.216.131 with SMTP; 4 Dec 2003 18:53:12 -0000 Received: from localhost ([127.0.0.1] helo=ran.psg.com) by ran.psg.com with esmtp (Exim 4.24; FreeBSD) id 1ARyb9-000FBz-LZ for flow-tools@splintered.net; Thu, 04 Dec 2003 10:53:11 -0800 From: Randy Bush MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Date: Thu, 4 Dec 2003 10:53:11 -0800 To: flow list Message-Id: Cc: Subject: [Flow-tools] diagnosing source of ping flood X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Dec 2003 18:53:13 -0000 so, i am experiencing a mild ping attack, only a few hundred k. but i would like to use the experience to sort out how to diagnose this. it seems to be targeting a single ip address inside the lan. how do i ask flow-tools to tell me the source ip addresses of all echo requests toward a specific ip address? also, how do i know if manufacture c's box is giving me the snmp and flow stats before or after rate limiting? thanks. randy From splintered-flow-tools-owner@splintered.net Thu Dec 04 19:37:24 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 8188 invoked by uid 4001); 4 Dec 2003 19:37:24 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 8186 invoked by alias); 4 Dec 2003 19:37:24 -0000 Received: from seabridg01.gettyimages.com (HELO seabridg01.seattle.gettyimages.com) (207.1.176.170) by 66.250.216.131 with SMTP; 4 Dec 2003 19:37:24 -0000 Received: by seabridg01.seattle.gettyimages.com with Internet Mail Service (5.5.2657.72) id ; Thu, 4 Dec 2003 11:37:23 -0800 Message-ID: From: Alex Shepard To: flow list Subject: RE: [Flow-tools] diagnosing source of ping flood Date: Thu, 4 Dec 2003 11:37:19 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2657.72) Content-Type: text/plain Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Dec 2003 19:37:24 -0000 Randy, One way to do this is to build yourself a filter and run your flow files through it. [alexs@mybox alexs]$ cat filter.txt filter-primitive icmp type ip-protocol permit 1 filter-primitive host_being_attacked type ip-address permit 10.10.10.10 filter-definition ping_attack match ip-destination-address host_being_attacked match ip-protocol icmp [alexs@mybox alexs]$ flow-nfilter lets you build primitives out of protocols, TCP flags, ip addresses, ports, AS, ifindex, mask, and a bunch of other stuff. then you can use build filters out of those primitives, which contain once you've got your filter built, run it with: flow-cat | flow-nfilter -f -F | flow-print you should see something like: [alexs@mybox alexs]$ flow-cat /var/local/flows/processed/ft-v05.2003-12-04.115231-0800 | flow-nfilter -f filter.txt -F ping_attack | flow-print srcIP dstIP prot srcPort dstPort octets packets 10.8.8.10 10.10.10.10 1 0 0 92 1 192.168.13.1 10.10.10.10 1 0 0 92 1 192.168.14.1 10.10.10.10 1 0 0 92 1 10.9.9.10 10.10.10.10 1 0 0 92 1 192.168.15.1 10.10.10.10 1 0 0 92 1 192.168.14.1 10.10.10.10 1 0 0 78 1 [... etc ...] [alexs@mybox alexs] I'm not sure if flow-tools looks deep enough into icmp packets to differentiate between echo request and other icmp packet types. HTH, alex -----Original Message----- From: flow-tools-bounces@list.splintered.net [mailto:flow-tools-bounces@list.splintered.net] On Behalf Of Randy Bush Sent: Thursday, December 04, 2003 10:53 AM To: flow list Subject: [Flow-tools] diagnosing source of ping flood so, i am experiencing a mild ping attack, only a few hundred k. but i would like to use the experience to sort out how to diagnose this. it seems to be targeting a single ip address inside the lan. how do i ask flow-tools to tell me the source ip addresses of all echo requests toward a specific ip address? also, how do i know if manufacture c's box is giving me the snmp and flow stats before or after rate limiting? thanks. randy _______________________________________________ Flow-tools mailing list flow-tools@splintered.net http://mailman.splintered.net/mailman/listinfo/flow-tools ======================================================= This email and its contents are confidential. If you are not the intended recipient, please do not disclose or use the information within this email or its attachments. If you have received this email in error, please delete it immediately. Thank you. ======================================================= From splintered-flow-tools-owner@splintered.net Thu Dec 04 19:37:32 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 8269 invoked by uid 4001); 4 Dec 2003 19:37:32 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 8266 invoked by alias); 4 Dec 2003 19:37:32 -0000 Received: from mx3.versus.com (HELO mx1.versus.com) (205.205.118.15) by 66.250.216.131 with SMTP; 4 Dec 2003 19:37:32 -0000 Received: from versus.com (boyle-new.versus.com [206.83.39.36]) by mx1.versus.com (Postfix) with ESMTP id B6F687379 for ; Thu, 4 Dec 2003 14:37:31 -0500 (EST) X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Date: Thu, 4 Dec 2003 14:37:31 -0500 Message-ID: <5B38BEB87ED6C449BD0CC4770F69857F01657EE9@boyle.versus.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Working the right way... ;) Thread-Index: AcO6ng5f/pJNIj90TN+BUMup6mTDIw== From: "Eric Rousse" To: Cc: Subject: [Flow-tools] Working the right way... ;) X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Dec 2003 19:37:33 -0000 Hi, I got 2 questions. First, each day I run a command on the flows of the previous day.=20 /usr/local/netflow/bin/flow-merge * | = /usr/local/netflow/bin/flow-nfilter -f /usr/local/scripts/nfilter.cfg -F = backup-match | /usr/local/netflow/bin/flow-stat -f11 > = /export2/netflows/flow-stat/20031204.log Is that the best way of doing my merge and nfilter and flowstat to a = file ? First, I can't do a flow-merge to a file, the file gets too big (more = than 2 gigs, and I get an error about the filesize most probably because of my glib version). Also doing all that at the same time use a lot of memory, I have 512MB = installed on that machine. And sometimes, the hole process gets stuck and it stays there for a = while... So is there a way to improve this ? Also here's the content of my file nfilter.cfg and my second question: filter-primitive backup-hosts type ip-address deny x.x.x.x deny x.x.x.x deny x.x.x.x default permit filter-definition backup-match match ip-source-address backup-hosts or match ip-destination-address backup-hosts Is that the right way of doing this kind of things ? I wanna filter 3 IP (backup servers) out from my flow files. Incoming or outgoing traffic. Is that the right way of doing this, because it doesn't seems to filter all the time, some days I have nothing from these IP, other days I have a huge load of traffic. mmm after all I have another question, I posted that = (http://www.pairlist.net/pipermail/flow-tools/2003-October/001674.html) a few weeks earlier, I changed few things since then, but things are not fixed, I still see that error in a strace, is that normal ? Anyone seeing this also ? Thanks! _________________________________________________ Eric Rousse Versus=AE=20 2050, rue De Bleury, bureau 520 Montr=E9al (Qu=E9bec) H3A 2J5=20 Canada T=E9l.: 514.284.9001 ext. 221 Fax: 514.284.9002 From splintered-flow-tools-owner@splintered.net Thu Dec 04 21:40:10 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 10245 invoked by uid 4001); 4 Dec 2003 21:40:10 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 10243 invoked by alias); 4 Dec 2003 21:40:10 -0000 Received: from ip166.usw253.dsl-acs2.sea.iinet.com (HELO ran.psg.com) (209.20.253.166) by 66.250.216.131 with SMTP; 4 Dec 2003 21:40:10 -0000 Received: from localhost ([127.0.0.1] helo=ran.psg.com) by ran.psg.com with esmtp (Exim 4.24; FreeBSD) id 1AS1Ci-000JhX-Pg; Thu, 04 Dec 2003 13:40:08 -0800 From: Randy Bush MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Date: Thu, 4 Dec 2003 13:40:08 -0800 To: Alex Shepard Subject: RE: [Flow-tools] diagnosing source of ping flood References: Message-Id: Cc: flow list X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Dec 2003 21:40:10 -0000 hmmm. % cat flow.filter filter-primitive icmp type ip-protocol permit 1 filter-primitive host_being_attacked type ip-address permit 666.42.7.11 filter-definition ping_attack match ip-destination-address host_being_attacked match ip-protocol icmp % flow-cat $FS | flow-nfilter -f flow.filter -F ping_attack | flow-print srcIP dstIP prot srcPort dstPort octets packets 38.113.11.4 666.42.7.11 1 0 771 405 4 38.113.11.4 666.42.7.11 1 0 771 591 6 81.28.0.133 666.42.7.11 1 0 2816 56 1 140.251.0.25 666.42.7.11 1 0 2048 1500 1 38.113.11.4 666.42.7.11 1 0 771 279 3 but look at those first two lines. so i tried % flow-cat $FS | flow-nfilter -f flow.filter -F ping_attack \ | flow-print -f 12 flow-print: Flow record missing required field for format. clue missing From splintered-flow-tools-owner@splintered.net Thu Dec 04 22:12:17 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 11118 invoked by uid 4001); 4 Dec 2003 22:12:17 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 11116 invoked by alias); 4 Dec 2003 22:12:17 -0000 Received: from seabridg01.gettyimages.com (HELO seabridg01.seattle.gettyimages.com) (207.1.176.170) by 66.250.216.131 with SMTP; 4 Dec 2003 22:12:17 -0000 Received: by seabridg01.seattle.gettyimages.com with Internet Mail Service (5.5.2657.72) id ; Thu, 4 Dec 2003 14:12:16 -0800 Message-ID: From: Alex Shepard To: 'Randy Bush' Subject: RE: [Flow-tools] diagnosing source of ping flood Date: Thu, 4 Dec 2003 14:12:14 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2657.72) Content-Type: text/plain Cc: flow list X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Dec 2003 22:12:17 -0000 "flow-print -f12" tries to print flows reported in Netflow v8.3 format (source aggregation). flow-print options only really add or remove or change which fields are printed. if you want to aggregate data into reports, you'll want to use flow-stat or flow-report. try: % flow-cat $FS | flow-nfilter -f flow.filter -F ping_attack | flow-stat -f10 flow-stat has about 30 canned reports that are very fast. flow-report is much more flexible but is also more complicated. it works more like flow-nfilter. HTH, alex -----Original Message----- From: Randy Bush [mailto:randy@psg.com] Sent: Thursday, December 04, 2003 1:40 PM To: Alex Shepard Cc: flow list Subject: RE: [Flow-tools] diagnosing source of ping flood hmmm. % cat flow.filter filter-primitive icmp type ip-protocol permit 1 filter-primitive host_being_attacked type ip-address permit 666.42.7.11 filter-definition ping_attack match ip-destination-address host_being_attacked match ip-protocol icmp % flow-cat $FS | flow-nfilter -f flow.filter -F ping_attack | flow-print srcIP dstIP prot srcPort dstPort octets packets 38.113.11.4 666.42.7.11 1 0 771 405 4 38.113.11.4 666.42.7.11 1 0 771 591 6 81.28.0.133 666.42.7.11 1 0 2816 56 1 140.251.0.25 666.42.7.11 1 0 2048 1500 1 38.113.11.4 666.42.7.11 1 0 771 279 3 but look at those first two lines. so i tried % flow-cat $FS | flow-nfilter -f flow.filter -F ping_attack \ | flow-print -f 12 flow-print: Flow record missing required field for format. clue missing ======================================================= This email and its contents are confidential. If you are not the intended recipient, please do not disclose or use the information within this email or its attachments. If you have received this email in error, please delete it immediately. Thank you. ======================================================= From splintered-flow-tools-owner@splintered.net Fri Dec 05 01:21:15 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 12389 invoked by uid 4001); 5 Dec 2003 01:21:15 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 12387 invoked by alias); 5 Dec 2003 01:21:14 -0000 Received: from mail.sunet.com.au (HELO jupiter.sunet.com.au) (203.166.102.39) by 66.250.216.131 with SMTP; 5 Dec 2003 01:21:14 -0000 Received: from ganymede.internal.sunet.com.au (canopus.sunet.com.au [::ffff:203.166.102.49]) by jupiter.sunet.com.au with esmtp; Fri, 05 Dec 2003 12:15:45 +1100 Date: Fri, 5 Dec 2003 12:21:33 +1100 (EST) From: Systems Administrator X-X-Sender: sysadmin@ganymede.bcc.local To: johann lafer Subject: Re: [Flow-tools] Performance / compile question In-Reply-To: <20031204155458.86623.qmail@web21504.mail.yahoo.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Dec 2003 01:21:15 -0000 On Thu, 4 Dec 2003, johann lafer wrote: > I don't know if it is really a performance problem, but the whole story > is, that i have written a php frontend with flow-tools > (flow-cat,flow-nfilter,flow-stat) in the background. If a web-user wants > to generate a 1,3,12 or 24 hour report, it takes up to 5 minutes. But > creating a "week-report" or "month-report" takes more than 25 minutes, > depending on the generated data. The size of 5 minute flow-file differs > between 1 and 4 Mbytes. Maybe i am impatient?! It looks like a hanging > application if you have to wait to long. ... > Another idea is to run more than 1 process for a report (up to 31 > process for a month report, bottleneck competing CPU and/or HD access?!) Another option would be, if you've got a limited number of reports (ie. if people are going to be saying "data from my subnet", and not "port x on machine y"), you could generate summaries on a daily basis and log them to a database, and then pull them out when you need them. I'm going to be providing something to our customers where they can view their usage on a monthly basis, divided up by day, and so I can just log all the daily summaries to a database, and query that when I need the info. :) -- Tim Nelson Systems Administrator Sunet Internet Tel: +61 3 5241 1155 Fax: +61 3 5241 6187 Web: http://www.sunet.com.au/ Email: sysadmin@sunet.com.au From splintered-flow-tools-owner@splintered.net Fri Dec 05 02:52:04 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 13487 invoked by uid 4001); 5 Dec 2003 02:52:04 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 13485 invoked by alias); 5 Dec 2003 02:52:04 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 5 Dec 2003 02:52:04 -0000 Mime-Version: 1.0 (Apple Message framework v606) Content-Transfer-Encoding: 7bit Message-Id: <018B7A34-26CE-11D8-A8C6-000A95DA1C38@splintered.net> Content-Type: text/plain; charset=US-ASCII; format=flowed To: 'flow-tools@splintered.net' From: Mark Fullmer Date: Thu, 4 Dec 2003 21:52:03 -0500 X-Mailer: Apple Mail (2.606) Cc: Subject: [Flow-tools] rsync exploit X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Dec 2003 02:52:04 -0000 I reference rsync a lot for use in distributed flow processing, so this latest vulnerability may be of interest. Details at http://rsync.samba.org/ mark From splintered-flow-tools-owner@splintered.net Fri Dec 05 04:04:22 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 14464 invoked by uid 4001); 5 Dec 2003 04:04:22 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 14462 invoked by alias); 5 Dec 2003 04:04:22 -0000 Received: from ip166.usw253.dsl-acs2.sea.iinet.com (HELO ran.psg.com) (209.20.253.166) by 66.250.216.131 with SMTP; 5 Dec 2003 04:04:22 -0000 Received: from localhost ([127.0.0.1] helo=ran.psg.com) by ran.psg.com with esmtp (Exim 4.24; FreeBSD) id 1AS7CX-0003lh-BS; Thu, 04 Dec 2003 20:04:21 -0800 From: Randy Bush MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Date: Thu, 4 Dec 2003 20:04:20 -0800 To: Alex Shepard Subject: RE: [Flow-tools] diagnosing source of ping flood References: Message-Id: Cc: flow list X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Dec 2003 04:04:22 -0000 > % flow-cat $FS | flow-nfilter -f flow.filter -F ping_attack | flow-stat -f10 worked. any way i can tell which icmp type i am seeing? randy From splintered-flow-tools-owner@splintered.net Fri Dec 05 06:29:03 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 15707 invoked by uid 4001); 5 Dec 2003 06:29:03 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 15705 invoked by alias); 5 Dec 2003 06:29:03 -0000 Received: from mail3.panix.com (166.84.1.74) by 66.250.216.131 with SMTP; 5 Dec 2003 06:29:03 -0000 Received: from panix5.panix.com (panix5.panix.com [166.84.1.5]) by mail3.panix.com (Postfix) with ESMTP id 3444C98359; Fri, 5 Dec 2003 01:29:02 -0500 (EST) Received: (from eravin@localhost) by panix5.panix.com (8.11.6p2-a/8.8.8/PanixN1.1) id hB56T2N11597; Fri, 5 Dec 2003 01:29:02 -0500 (EST) Date: Fri, 5 Dec 2003 01:29:02 -0500 From: Ed Ravin To: Randy Bush Message-ID: <20031205062901.GA11475@panix.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.1i X-Y-Z: 1, 2, 3? Cc: flow list Subject: [Flow-tools] flow stats and rate limiting? X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Dec 2003 06:29:03 -0000 On Thu, Dec 04, 2003 at 10:53:11AM -0800, Randy Bush wrote: > also, how do i know if manufacture c's box is giving me the snmp > and flow stats before or after rate limiting? Empirical testing, methinks, might be the only way to be sure. I seem to recall that packets screened by ACLs are still reported in Netflow from my Cisco 7200 - I suspect it would do the same thing with rate-limited packets. From splintered-flow-tools-owner@splintered.net Fri Dec 05 11:41:14 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 17889 invoked by uid 4001); 5 Dec 2003 11:41:14 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 17887 invoked by alias); 5 Dec 2003 11:41:14 -0000 Received: from dymwsm08.mailwatch.com (204.253.83.44) by 66.250.216.131 with SMTP; 5 Dec 2003 11:41:14 -0000 Received: from MWSC0225.mw4.mailwatch.com (mwsc0225.mw4.mailwatch.com [204.253.83.172]) by dymwsm08.mailwatch.com (8.12.9/8.12.9) with ESMTP id hB5BfEtH016080 for ; Fri, 5 Dec 2003 06:41:14 -0500 Received: from mail pickup service by MWSC0225.mw4.mailwatch.com with Microsoft SMTPSVC; Fri, 5 Dec 2003 06:41:14 -0500 Received: from 204.253.83.77 ([204.253.83.77]) by MWSC0225 with SMTP id 000200197b2a9324-9343-472a-8ed0-b9203b4bea3d; Fri, 05 Dec 2003 06:41:13 -0500 Received: from fmpo1.azell.com (fmpo1.azell.com [136.1.7.9]) by dymwsm03.mailwatch.com (8.12.9/8.12.9) with ESMTP id hB5BecJ1028818 for ; Fri, 5 Dec 2003 06:40:38 -0500 Received: from na1ecs06.dearborn.ford.com ([19.5.116.123]) by fmpo1.azell.com (Mirapoint Messaging Server MOS 3.3.5-GR) with ESMTP id BIM11911; Fri, 5 Dec 2003 06:41:13 -0500 (EST) Received: by na1ecs06.dearborn.ford.com with Internet Mail Service (5.5.2657.72) id ; Fri, 5 Dec 2003 06:41:12 -0500 Message-ID: From: "Nunn, Mike (M.)" To: "'flow-tools@splintered.net'" Date: Fri, 5 Dec 2003 06:40:09 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2657.72) Content-Type: text/plain; charset="iso-8859-1" X-MW-BTID: 090625000020033394203800003 X-MW-CTIME: 1070624438 X-MW-SENDING-MTA: 136.1.7.9 HOP-COUNT: 1 X-MAILWATCH-INSTANCEID: 010200197b2a9324-9343-472a-8ed0-b9203b4bea3d X-OriginalArrivalTime: 05 Dec 2003 11:41:14.0061 (UTC) FILETIME=[AFE2F7D0:01C3BB24] Cc: Subject: [Flow-tools] flows dropped by acl X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Dec 2003 11:41:15 -0000 There seems to be conflicting evidence about Netflow reporting traffic dropped by an acl on the exporting router, is there a definitive answer ? Thanks, Mike From splintered-flow-tools-owner@splintered.net Sat Dec 06 01:41:00 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 22402 invoked by uid 4001); 6 Dec 2003 01:41:00 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 22400 invoked by alias); 6 Dec 2003 01:41:00 -0000 Received: from elvis.mu.org (192.203.228.196) by 66.250.216.131 with SMTP; 6 Dec 2003 01:41:00 -0000 Received: by elvis.mu.org (Postfix, from userid 1098) id 01DC72ED472; Fri, 5 Dec 2003 17:40:50 -0800 (PST) Date: Fri, 5 Dec 2003 17:40:49 -0800 From: Bill Fumerola To: Ed Ravin Subject: Re: [Flow-tools] flow stats and rate limiting? Message-ID: <20031206014049.GT91301@elvis.mu.org> References: <20031205062901.GA11475@panix.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20031205062901.GA11475@panix.com> User-Agent: Mutt/1.4.1i X-Operating-System: FreeBSD 4.8-MUORG-20030806 i386 Cc: flow list X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Dec 2003 01:41:01 -0000 On Fri, Dec 05, 2003 at 01:29:02AM -0500, Ed Ravin wrote: > On Thu, Dec 04, 2003 at 10:53:11AM -0800, Randy Bush wrote: > > also, how do i know if manufacture c's box is giving me the snmp > > and flow stats before or after rate limiting? > > Empirical testing, methinks, might be the only way to be sure. I seem > to recall that packets screened by ACLs are still reported in Netflow > from my Cisco 7200 - I suspect it would do the same thing with rate-limited > packets. the destination interface may be zero for flows that summarize traffic that was dropped as part of policy. or it may not. i think it's architecture specific for vendor C. -- - bill fumerola / fumerola@yahoo-inc.com / billf@FreeBSD.org From splintered-flow-tools-owner@splintered.net Mon Dec 08 17:19:46 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 36427 invoked by uid 4001); 8 Dec 2003 17:19:46 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 36424 invoked by alias); 8 Dec 2003 17:19:46 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 8 Dec 2003 17:19:46 -0000 In-Reply-To: <20031204102950.16474.qmail@webmail7.rediffmail.com> References: <20031204102950.16474.qmail@webmail7.rediffmail.com> Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [Flow-tools] (no subject) Date: Mon, 8 Dec 2003 12:19:46 -0500 To: Chitman Kaur X-Mailer: Apple Mail (2.606) Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Dec 2003 17:19:46 -0000 1) look for packet loss between the exporter and collector. 2) Make sure the collector isn't overloaded and dropping flows on input. on *BSD you can do this with netstat -s | grep buf 0 dropped due to full socket buffers 0 output packets dropped due to no bufs, etc. 0 output packets dropped due to no bufs, etc. Mbuf statistics: 519 one mbuf two or more mbuf: 0 one ext mbuf 0 two or more ext mbuf Note the "dropped due to full socket buffers" 3) If the collector is dropping flows raise the priority of flow-capture. With FreeBSD I use 'rtprio 5 flow-capture ...'. Rtprio makes a big difference if you're running other software on the same server. 4) If it's an IOS device look at the output of 'sh ip flow export' krc5>sh ip flow export Flow export v5 is enabled for main cache Exporting flows to X.X.X.X (7998) Exporting using source interface Loopback0 Version 5 flow records, origin-as 4044972686 flows exported in 134907285 udp datagrams 1827 flows failed due to lack of export packet 186 export packets were sent up to process level 0 export packets were dropped due to no fib 642587 export packets were dropped due to adjacency issues 0 export packets were dropped due to fragmentation failures 0 export packets were dropped due to encapsulation fixup failures 112151 export packets were dropped enqueuing for the RP 834609 export packets were dropped due to IPC rate limiting 0 export packets were dropped due to output drops Note lines like the "834609 export packets dropped to IPC rate limiting". In this case the router can't handle the flow export rate and is dropping the flows internally. mark On Dec 4, 2003, at 5:29 AM, Chitman Kaur wrote: > Hi All.... > Even after putting the option -z0 I am getting lost flows..... > Any reason why.... > I am sure that there is no congestion between my router and > collector..... > Am I giving the option correctly.... > ----------------------------------------------------------------------- > /usr/local/netflow/bin/flow-capture -w /var/netflow/ft 0/0/2055 -z0 > -S5 -V5 -E1G -n 287 -N 0 -R /usr/local/netflow/bin/linkme > touch /var/lock/subsys/startflows > ----------------------------------------------------------------------- > Regards > Chitman > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From splintered-flow-tools-owner@splintered.net Mon Dec 08 17:23:44 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 36999 invoked by uid 4001); 8 Dec 2003 17:23:44 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 36997 invoked by alias); 8 Dec 2003 17:23:44 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 8 Dec 2003 17:23:44 -0000 In-Reply-To: <5B38BEB87ED6C449BD0CC4770F69857F01657EE9@boyle.versus.com> References: <5B38BEB87ED6C449BD0CC4770F69857F01657EE9@boyle.versus.com> Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=ISO-8859-1; delsp=yes; format=flowed Message-Id: <46966BF4-29A3-11D8-81D6-00039304FAA6@eng.oar.net> Content-Transfer-Encoding: quoted-printable From: Mark Fullmer Subject: Re: [Flow-tools] Working the right way... ;) Date: Mon, 8 Dec 2003 12:23:44 -0500 To: "Eric Rousse" X-Mailer: Apple Mail (2.606) Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Dec 2003 17:23:44 -0000 flow-merge by its nature is going to be slow, it sounds like you want to be using flow-cat. For files larger than 2 gig you'll probably need to use the -m (disable =20= mmap) option to flow-cat. Yes flow-tools should do this automatically... Your filters look correct...Can you send me a more specific example when it's not filtering correctly? The strace you sent looks like you have tags configured, are using the =20= "OSU" tag name and it doesn't exist in your config fle. mark On Dec 4, 2003, at 2:37 PM, Eric Rousse wrote: > > Hi, > > I got 2 questions. > > First, each day I run a command on the flows of the previous > day. > > /usr/local/netflow/bin/flow-merge * | =20 > /usr/local/netflow/bin/flow-nfilter -f /usr/local/scripts/nfilter.cfg =20= > -F backup-match | /usr/local/netflow/bin/flow-stat -f11 > =20 > /export2/netflows/flow-stat/20031204.log > > Is that the best way of doing my merge and nfilter and flowstat to a =20= > file ? > First, I can't do a flow-merge to a file, the file gets too big (more =20= > than 2 gigs, and I get > an error about the filesize most probably because of my glib version). > Also doing all that at the same time use a lot of memory, I have 512MB = =20 > installed on that machine. > And sometimes, the hole process gets stuck and it stays there for a =20= > while... > So is there a way to improve this ? > > > Also here's the content of my file nfilter.cfg and my second question: > > filter-primitive backup-hosts > type ip-address > deny x.x.x.x > deny x.x.x.x > deny x.x.x.x > default permit > > filter-definition backup-match > match ip-source-address backup-hosts > or > match ip-destination-address backup-hosts > > > Is that the right way of doing this kind of things ? > I wanna filter 3 IP (backup servers) out from my flow files. > Incoming or outgoing traffic. > > Is that the right way of doing this, because it doesn't seems to = filter > all the time, some days I have nothing from these IP, other days > I have a huge load of traffic. > > > mmm after all I have another question, > I posted that =20 > (http://www.pairlist.net/pipermail/flow-tools/2003-October/=20 > 001674.html) > a few weeks earlier, I changed few things since then, > but things are not fixed, I still see that error in a > strace, is that normal ? Anyone seeing this also ? > > Thanks! > > _________________________________________________ > Eric Rousse > Versus=AE > 2050, rue De Bleury, bureau 520 > Montr=E9al (Qu=E9bec) H3A 2J5 > Canada > T=E9l.: 514.284.9001 ext. 221 Fax: 514.284.9002 > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From splintered-flow-tools-owner@splintered.net Mon Dec 08 17:25:34 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 37197 invoked by uid 4001); 8 Dec 2003 17:25:34 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 37195 invoked by alias); 8 Dec 2003 17:25:34 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 8 Dec 2003 17:25:34 -0000 In-Reply-To: References: Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <88160A15-29A3-11D8-81D6-00039304FAA6@eng.oar.net> Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [Flow-tools] flows dropped by acl Date: Mon, 8 Dec 2003 12:25:34 -0500 To: "Nunn, Mike (M.)" X-Mailer: Apple Mail (2.606) Cc: "'flow-tools@splintered.net'" X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Dec 2003 17:25:34 -0000 On IOS based devices access lists and RPF checks will set the output interface to 0 of the packets in the flow are being dropped. Last time I asked CAR related drops are not reflected in the flow exports. In general if you want to look for dropped packets just filter on output interface 0. mark On Dec 5, 2003, at 6:40 AM, Nunn, Mike (M.) wrote: > There seems to be conflicting evidence about Netflow reporting traffic > dropped by an acl on the exporting router, is there a definitive > answer ? > > Thanks, Mike > > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From splintered-flow-tools-owner@splintered.net Mon Dec 08 17:35:23 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 38639 invoked by uid 4001); 8 Dec 2003 17:35:22 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 38637 invoked by alias); 8 Dec 2003 17:35:22 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 8 Dec 2003 17:35:22 -0000 In-Reply-To: References: Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [Flow-tools] diagnosing source of ping flood Date: Mon, 8 Dec 2003 12:35:23 -0500 To: Randy Bush X-Mailer: Apple Mail (2.606) Cc: flow list X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Dec 2003 17:35:23 -0000 There's a separate CAR mib to count dropped traffic. The port fields are used with ICMP to indicate the type and code. For example 199.18.139.136% ping 192.148.251.71 SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts AT1/0.14 199.18.139.136 Fa0/0 192.148.251.71 01 0000 0800 6 Fa0/0 192.148.251.71 AT1/0.14 199.18.139.136 01 0000 0000 6 From RFC792: Summary of Message Types 0 Echo Reply 3 Destination Unreachable 4 Source Quench 5 Redirect 8 Echo 11 Time Exceeded 12 Parameter Problem 13 Timestamp 14 Timestamp Reply 15 Information Request 16 Information Reply mark On Dec 4, 2003, at 1:53 PM, Randy Bush wrote: > so, i am experiencing a mild ping attack, only a few hundred k. but > i would like to use the experience to sort out how to diagnose this. > > it seems to be targeting a single ip address inside the lan. how > do i ask flow-tools to tell me the source ip addresses of all echo > requests toward a specific ip address? > > also, how do i know if manufacture c's box is giving me the snmp > and flow stats before or after rate limiting? > > thanks. > > randy > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From splintered-flow-tools-owner@splintered.net Mon Dec 08 18:16:12 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 39822 invoked by uid 4001); 8 Dec 2003 18:16:12 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 39820 invoked by alias); 8 Dec 2003 18:16:12 -0000 Received: from sj-iport-1-in.cisco.com (HELO sj-iport-1.cisco.com) (171.71.176.70) by 66.250.216.131 with SMTP; 8 Dec 2003 18:16:12 -0000 Received: from mira-sjc5-f.cisco.com (IDENT:mirapoint@mira-sjc5-f.cisco.com [171.71.163.13]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hB8IG9At014073; Mon, 8 Dec 2003 10:16:09 -0800 (PST) Received: from icox-w2k01.cisco.com (dhcp-128-107-159-10.cisco.com [128.107.159.10]) by mira-sjc5-f.cisco.com (Mirapoint Messaging Server MOS 3.3.6-GR) with ESMTP id ANE36797; Mon, 8 Dec 2003 10:16:09 -0800 (PST) Message-Id: <6.0.1.1.2.20031208101342.0484d6d8@mira-sjcd-1.cisco.com> X-Sender: icox@mira-sjcd-1.cisco.com (Unverified) X-Mailer: QUALCOMM Windows Eudora Version 6.0.1.1 Date: Mon, 08 Dec 2003 10:16:02 -0800 To: Mark Fullmer , "Nunn, Mike (M.)" From: Ian Cox Subject: Re: [Flow-tools] flows dropped by acl In-Reply-To: <88160A15-29A3-11D8-81D6-00039304FAA6@eng.oar.net> References: <88160A15-29A3-11D8-81D6-00039304FAA6@eng.oar.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Cc: "'flow-tools@splintered.net'" X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Dec 2003 18:16:12 -0000 If the platform is a Cat6k with Supervisor 2 or Supervisor 720 then the packets dropped by ACL are not sent to the collector. Ian At 12:25 PM 12/8/2003 -0500, Mark Fullmer wrote: >On IOS based devices access lists and RPF checks will set the output >interface to 0 of the packets in the flow are being dropped. > >Last time I asked CAR related drops are not reflected in the flow exports. > >In general if you want to look for dropped packets just filter on output >interface 0. > >mark > >On Dec 5, 2003, at 6:40 AM, Nunn, Mike (M.) wrote: > >>There seems to be conflicting evidence about Netflow reporting traffic >>dropped by an acl on the exporting router, is there a definitive answer ? >> >>Thanks, Mike >> >> >>_______________________________________________ >>Flow-tools mailing list >>flow-tools@splintered.net >>http://mailman.splintered.net/mailman/listinfo/flow-tools > >_______________________________________________ >Flow-tools mailing list >flow-tools@splintered.net >http://mailman.splintered.net/mailman/listinfo/flow-tools From splintered-flow-tools-owner@splintered.net Tue Dec 09 22:36:23 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 47409 invoked by uid 4001); 9 Dec 2003 22:36:23 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 47407 invoked by alias); 9 Dec 2003 22:36:22 -0000 Received: from unknown (HELO cmail.caspian.com) (63.108.173.139) by 66.250.216.131 with SMTP; 9 Dec 2003 22:36:22 -0000 Received: from VMW2KNLUGHMAN ([192.168.1.105]) by cmail.caspian.com (Mirapoint Messaging Server MOS 3.2.4-GA) with ESMTP id AKS03812; Tue, 9 Dec 2003 14:36:20 -0800 (PST) From: "Nadeem Lughmani" To: Date: Tue, 9 Dec 2003 14:36:22 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.6604 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Importance: Normal Cc: Subject: [Flow-tools] installing flow-tools X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Dec 2003 22:36:23 -0000 Hi All, Is it absolute must to have zlib, tcp_wrappers and gnu_make on a machine before installing flow-tools. Thanks Nadeem From nlughman@caspiannetworks.com Tue Dec 09 22:39:46 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 48044 invoked by alias); 9 Dec 2003 22:39:46 -0000 Received: from unknown (HELO cmail.caspian.com) (63.108.173.139) by 66.250.216.131 with SMTP; 9 Dec 2003 22:39:46 -0000 Received: from VMW2KNLUGHMAN ([192.168.1.105]) by cmail.caspian.com (Mirapoint Messaging Server MOS 3.2.4-GA) with ESMTP id AKS03833; Tue, 9 Dec 2003 14:39:44 -0800 (PST) From: "Nadeem Lughmani" To: Date: Tue, 9 Dec 2003 14:39:46 -0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.6604 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Importance: Normal Subject: [Flow-tools] installing flow-tools X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Dec 2003 22:39:47 -0000 Hi All, Is it absolute must to have zlib, tcp_wrappers and gnu_make on a machine before installing flow-tools Thanks Nadeem From lex@sci-nnov.ru Wed Dec 10 10:52:05 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 51854 invoked by alias); 10 Dec 2003 10:52:05 -0000 Received: from adm.sci-nnov.ru (195.122.226.2) by 66.250.216.131 with SMTP; 10 Dec 2003 10:52:05 -0000 Received: from vista1 (vista.sci-nnov.ru [195.122.226.25]) by adm.sci-nnov.ru (Postfix) with SMTP id B035819F064 for ; Wed, 10 Dec 2003 13:52:03 +0300 (MSK) Message-ID: <00d901c3bf0b$a562b1c0$19e27ac3@corporate.sandy.ru> From: "Alex A. Pavlenko" To: Date: Wed, 10 Dec 2003 13:52:03 +0300 MIME-Version: 1.0 Content-Type: text/plain; charset="koi8-r" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1158 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165 Subject: [Flow-tools] Tag symbolyc names. X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Dec 2003 10:52:05 -0000 Hello, everybody! I've read man pages but still it is not clear for me. How to make flow-stat to put symbolic names for tags into reports? flow-stat -n -f30 command which I used to generate report, prints WEB_SERVERS for tag=0x0 and numbers for other tags. I've searched all files on my hard drive to find out where WEB_SERVERS tag defined as 0x0 but found nothing. Where should I put tag definitions and in which format? Is it possible to get symbols as tags in flow-report? Flow-tools package works on FreeBSD 4.9 Here is my /usr/local/etc/flow-tools/tag.cfg -------------cut---------- # 0x00 EXTERNAL # 0x0001 LOCAL # 0x0002 PROXY # 0x0003 ATLAS # 0x0004 KIS # 0x0005 INFORIS # 0x0006 MTS # 0x0007 NIS # 0x0008 NTS # 0x0009 UNN tag-action ACT_SRC type src-prefix # LOCAL match x.x.x.0/23 set-src 0x0001 .... tag-definition DEF_SRC term action ACT_SRC -------------cut---------- Thanks. -- Alex Pavlenko From maf@eng.oar.net Thu Dec 11 17:30:25 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 58445 invoked by alias); 11 Dec 2003 17:30:25 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 11 Dec 2003 17:30:25 -0000 In-Reply-To: <00d901c3bf0b$a562b1c0$19e27ac3@corporate.sandy.ru> References: <00d901c3bf0b$a562b1c0$19e27ac3@corporate.sandy.ru> Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <48E350CB-2BF7-11D8-A1DC-000A95DA1C38@eng.oar.net> Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [Flow-tools] Tag symbolyc names. Date: Thu, 11 Dec 2003 11:30:08 -0500 To: "Alex A. Pavlenko" X-Mailer: Apple Mail (2.606) Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Dec 2003 17:30:25 -0000 By default flow-tools will look for a tag.sym file in /usr/local/netflow/var/sym If you're using flow-tools-0.66 this file got swapped with tag.cfg which would explain the 0x0 symbol. If you can't find it look in lib/ftpaths.h. Depending on how flow-tools was installed this file may be elsewhere. mark On Dec 10, 2003, at 5:52 AM, Alex A. Pavlenko wrote: > Hello, everybody! > > I've read man pages but still it is not clear for me. > How to make flow-stat to put symbolic names for tags into reports? > flow-stat -n -f30 command which I > used to generate report, prints WEB_SERVERS for tag=0x0 and numbers > for other > tags. I've searched > all files on my hard drive to find out where WEB_SERVERS tag defined > as 0x0 but > found nothing. > Where should I put tag definitions and in which format? Is it possible > to get > symbols as tags in > flow-report? > Flow-tools package works on FreeBSD 4.9 > Here is my /usr/local/etc/flow-tools/tag.cfg > -------------cut---------- > # 0x00 EXTERNAL > # 0x0001 LOCAL > # 0x0002 PROXY > # 0x0003 ATLAS > # 0x0004 KIS > # 0x0005 INFORIS > # 0x0006 MTS > # 0x0007 NIS > # 0x0008 NTS > # 0x0009 UNN > tag-action ACT_SRC > type src-prefix > # LOCAL > match x.x.x.0/23 set-src 0x0001 > .... > tag-definition DEF_SRC > term > action ACT_SRC > -------------cut---------- > > Thanks. > > -- > Alex Pavlenko > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From splintered-flow-tools-owner@splintered.net Thu Dec 11 17:31:08 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 58666 invoked by uid 4001); 11 Dec 2003 17:31:08 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 58664 invoked by alias); 11 Dec 2003 17:31:08 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 11 Dec 2003 17:31:08 -0000 In-Reply-To: References: Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <640A4267-2BF7-11D8-A1DC-000A95DA1C38@eng.oar.net> Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [Flow-tools] installing flow-tools Date: Thu, 11 Dec 2003 11:30:54 -0500 To: "Nadeem Lughmani" X-Mailer: Apple Mail (2.606) Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Dec 2003 17:31:08 -0000 You need zlib. tcp_wrappers is optional. gnu make may not be required. mark On Dec 9, 2003, at 5:36 PM, Nadeem Lughmani wrote: > Hi All, > > Is it absolute must to have zlib, tcp_wrappers and gnu_make on a > machine > before installing flow-tools. > > Thanks > > Nadeem > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From splintered-flow-tools-owner@splintered.net Fri Dec 12 05:28:05 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 62702 invoked by uid 4001); 12 Dec 2003 05:28:05 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 62700 invoked by alias); 12 Dec 2003 05:28:05 -0000 Received: from mail3.panix.com (166.84.1.74) by 66.250.216.131 with SMTP; 12 Dec 2003 05:28:05 -0000 Received: from panix5.panix.com (panix5.panix.com [166.84.1.5]) by mail3.panix.com (Postfix) with ESMTP id 5F2D39822D for ; Fri, 12 Dec 2003 00:28:04 -0500 (EST) Received: (from eravin@localhost) by panix5.panix.com (8.11.6p2-a/8.8.8/PanixN1.1) id hBC5S4g22095 for flow-tools@splintered.net; Fri, 12 Dec 2003 00:28:04 -0500 (EST) Date: Fri, 12 Dec 2003 00:28:04 -0500 From: Ed Ravin To: flow-tools@splintered.net Message-ID: <20031212052804.GA21584@panix.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.1i X-Y-Z: 1, 2, 3? Cc: Subject: [Flow-tools] Checking for DoS or portscanning traffic X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Dec 2003 05:28:05 -0000 I recently cobbled together the script below as a cheap way to identify suspicious traffic, like DoS attacks from non-spoofed hosts or portscans. The logic is simple - run "flow-stat -f9 -S1" (collate by source IP address, sort descending on number of flows) and only print out the hosts whose ratio of packets / flows is less than or equal to 2, i.e., there is only one or two packets in a flow. The top entries on the list are almost always attackers or machines with Nachi or Nimda-like worms busy scanning our IP space. I see, after looking at the man page for flow-report, that I really need to upgrade to 0.67 to do this stuff properly. Is anybody else using flow-tools for detecting portscanning or malicious traffic? ------------------------ #!/bin/sh # look for source IP addresses that are sending us many, many packets # which are getting classified into flows with 2 or less packets per flow # pipe the output of this into a pager, since the output is sorted # by number of packets, the offenders will be at the top of the list. set -u USAGE="Usage: $0 [flow-data file] # defaults to current flow" flowtop=/YOUR/FLOW/DIRECTORY/HERE currentflowdir=$(date +$flowtop/%Y/%Y-%m/%Y-%m-%d/) file=${1:-$currentflowdir/tmp*} flow-cat $file | flow-stat -f9 -S1 | awk '{ if ( ($2 * 2) >= $4 || /^#/) {print}}' From splintered-flow-tools-owner@splintered.net Sun Dec 14 07:45:32 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 71770 invoked by uid 4001); 14 Dec 2003 07:45:32 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 71768 invoked by alias); 14 Dec 2003 07:45:32 -0000 Received: from xena.bway.net (216.220.96.26) by 66.250.216.131 with SMTP; 14 Dec 2003 07:45:32 -0000 Received: (qmail 40600 invoked by uid 0); 14 Dec 2003 07:45:31 -0000 Received: from unknown (HELO green.nat.fasttrackmonkey.com) (66.92.108.110) by xena.bway.net with AES256-SHA encrypted SMTP; 14 Dec 2003 07:45:31 -0000 Date: Sun, 14 Dec 2003 02:45:31 -0500 (EST) From: Charles Sprickman X-X-Sender: spork@green.nat.fasttrackmonkey.com To: flow-tools@splintered.net Message-ID: <20031214024126.I32249@green.nat.fasttrackmonkey.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: Subject: [Flow-tools] sharing examples X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Dec 2003 07:45:33 -0000 Hello all, I'm mighty new to flow-tools; new enough that I see it as a very valuable set of tools that I don't know how to use. Since flow-report seems to be (relatively) new, I'm wondering if we could perhaps start a thread on the list here where the more seasoned flow-tools users could share some of their favorite real-world example of DoS detection using flow-report (or flow-stat, flow-nfilter, whatever works for you). If we could get a nice sampling of examples, I would gladly sort/categorize and make a nice little web page for inclusion on the flow-tools homepage, and also an EXAMPLES file for distribution with the flow-tools package. Thanks, Charles ___ Charles Sprickman NetEng/SysAdmin Bway.net - New York's Best Internet - www.bway.net spork@bway.net - 212.655.9344 From splintered-flow-tools-owner@splintered.net Sun Dec 14 11:56:09 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 73309 invoked by uid 4001); 14 Dec 2003 11:56:09 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 73307 invoked by alias); 14 Dec 2003 11:56:09 -0000 Received: from mail.ateneo.edu (202.138.180.10) by 66.250.216.131 with SMTP; 14 Dec 2003 11:56:09 -0000 Received: from mail.ateneo.edu (localhost [127.0.0.1]) by localhost (Postfix) with ESMTP id 6ADF03F06; Sun, 14 Dec 2003 19:56:07 +0800 (PHT) Received: from sysads.ateneo.net (jabber.admu.edu.ph [10.2.10.70]) by mail.ateneo.edu (Postfix) with ESMTP id 512FF3EA8; Sun, 14 Dec 2003 19:56:07 +0800 (PHT) Received: (from wyy@localhost) by sysads.ateneo.net (8.11.6/8.11.6) id hBECEpV13219; Sun, 14 Dec 2003 20:14:51 +0800 Date: Sun, 14 Dec 2003 20:14:51 +0800 From: "Horatio B. Bogbindero" To: Charles Sprickman Subject: Re: [Flow-tools] sharing examples Message-ID: <20031214201451.A13204@admu.edu.ph> References: <20031214024126.I32249@green.nat.fasttrackmonkey.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-md5; protocol="application/pgp-signature"; boundary="RnlQjJ0d97Da+TV1" Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <20031214024126.I32249@green.nat.fasttrackmonkey.com>; from spork@bway.net on Sun, Dec 14, 2003 at 02:45:31AM -0500 Organization: Ateneo Cervini-Eliazo Networks X-ACENT-Conspiracy: Where is the conspiracy? Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list Reply-To: wyu@ateneo.edu List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Dec 2003 11:56:09 -0000 --RnlQjJ0d97Da+TV1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable question is were to start. well one place to start is this mailing list. there are a number of examples posted already throughout the months. maybe you can start from there. On Sun, Dec 14, 2003 at 02:45:31AM -0500, Charles Sprickman wrote (wyy sez): > Hello all, >=20 > I'm mighty new to flow-tools; new enough that I see it as a very valuable > set of tools that I don't know how to use. >=20 > Since flow-report seems to be (relatively) new, I'm wondering if we could > perhaps start a thread on the list here where the more seasoned flow-tools > users could share some of their favorite real-world example of DoS > detection using flow-report (or flow-stat, flow-nfilter, whatever works > for you). >=20 > If we could get a nice sampling of examples, I would gladly > sort/categorize and make a nice little web page for inclusion on the > flow-tools homepage, and also an EXAMPLES file for distribution with the > flow-tools package. >=20 > Thanks, >=20 > Charles >=20 > ___ > Charles Sprickman > NetEng/SysAdmin > Bway.net - New York's Best Internet - www.bway.net > spork@bway.net - 212.655.9344 >=20 > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools --=20 =20 ------------------------------------------- William Emmanuel S. Yu Ateneo Campus Network Group (AteneoCNG) email : wyy at admu dot edu dot ph web : http://CNG.ateneo.net/wyu/ phone : +63(2)4266001-4186 GPG : http://CNG.ateneo.net/wyu/wyy.pgp =20 War spares not the brave, but the cowardly. -- Anacreon =20 --RnlQjJ0d97Da+TV1 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE/3FQ6OgIOlr0CsAERAiM3AJ0RQHgikjaG1Q6sOuTFKtQNM7T4TgCfcUt0 7263dx+vMoj2lO/mtQ+jxtg= =mckr -----END PGP SIGNATURE----- --RnlQjJ0d97Da+TV1-- From splintered-flow-tools-owner@splintered.net Mon Dec 15 06:53:56 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 76797 invoked by uid 4001); 15 Dec 2003 06:53:56 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 76791 invoked by alias); 15 Dec 2003 06:53:55 -0000 Received: from eoa.ecoweb.co.zw (193.220.40.7) by 66.250.216.131 with SMTP; 15 Dec 2003 06:53:55 -0000 Received: from eoa.ecoweb.co.zw (viruswall.ecoweb.co.zw [193.220.40.15]) by eoa.ecoweb.co.zw (8.9.0/8.9.0) with ESMTP id IAA01148 for ; Mon, 15 Dec 2003 08:54:50 +0200 (GMT) Received: from CHIGWENDE (mathew.ecoweb.co.zw [217.15.123.5] (may be forged)) by eoa.ecoweb.co.zw (8.9.0/8.9.0) with SMTP id IAA01143 for ; Mon, 15 Dec 2003 08:54:49 +0200 (GMT) Message-ID: <004f01c3c2d7$f0a358e0$057b0fd9@ecoweb.co.zw> From: "Mathew Chigwende" To: Subject: [Flow-tools] Unsubscribe Date: Mon, 15 Dec 2003 08:51:29 +0200 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0045_01C3C2E8.A1BBDA70" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1158 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165 Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2003 06:53:56 -0000 This is a multi-part message in MIME format. ------=_NextPart_000_0045_01C3C2E8.A1BBDA70 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hello, Will you help me to unsubscribe to the mailing list. I have sent lots = of emails trying to unsubscribe but in vain. Your assistance will be greatly appreciated. Mathew Chigwende chigwende@ecoweb.co.zw mathew@ecoweb.co.zw matthew@ecoweb.co.zw Please help! ------=_NextPart_000_0045_01C3C2E8.A1BBDA70 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hello,
 
Will you help me to unsubscribe to the = mailing=20 list.  I have sent lots of emails trying to unsubscribe but in=20 vain.
 
Your assistance will be greatly=20 appreciated.
 
Mathew Chigwende
 
chigwende@ecoweb.co.zw<= /DIV>
mathew@ecoweb.co.zw
matthew@ecoweb.co.zw
 
Please help!
 
------=_NextPart_000_0045_01C3C2E8.A1BBDA70-- From splintered-flow-tools-owner@splintered.net Mon Dec 15 13:20:04 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 78982 invoked by uid 4001); 15 Dec 2003 13:20:04 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 78980 invoked by alias); 15 Dec 2003 13:20:04 -0000 Received: from mx3.versus.com (HELO mx1.versus.com) (205.205.118.15) by 66.250.216.131 with SMTP; 15 Dec 2003 13:20:04 -0000 Received: from versus.com (boyle-new.versus.com [206.83.39.36]) by mx1.versus.com (Postfix) with ESMTP id 963D77466 for ; Mon, 15 Dec 2003 08:20:03 -0500 (EST) X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C3C30E.263662E4" Subject: RE: [Flow-tools] Unsubscribe Date: Mon, 15 Dec 2003 08:20:03 -0500 Message-ID: <5B38BEB87ED6C449BD0CC4770F69857F016581D0@boyle.versus.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Flow-tools] Unsubscribe Thread-Index: AcPC2H1ClFr9IGlLRvymH3dw6I/SFwANWtiQ From: "Eric Rousse" To: "Mathew Chigwende" , Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2003 13:20:04 -0000 This is a multi-part message in MIME format. ------_=_NextPart_001_01C3C30E.263662E4 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Always check in the full e-mail headers, all the info is there... Anyway you have to send an e-mail to that address from what it says in = the headers. =20 flow-tools-request@list.splintered.net subject=3Dunsubscribe =20 Or go to that address : = http://mailman.splintered.net/mailman/listinfo/flow-tools =20 _____ =20 From: Mathew Chigwende [mailto:chigwende@ecoweb.co.zw]=20 Sent: 15 d=E9cembre 2003 01:51 To: flow-tools@splintered.net Subject: [Flow-tools] Unsubscribe Hello, =20 Will you help me to unsubscribe to the mailing list. I have sent lots = of emails trying to unsubscribe but in vain. =20 Your assistance will be greatly appreciated. =20 Mathew Chigwende =20 chigwende@ecoweb.co.zw mathew@ecoweb.co.zw matthew@ecoweb.co.zw =20 Please help! =20 ------_=_NextPart_001_01C3C30E.263662E4 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Always check in the full e-mail headers, all = the info is=20 there...
Anyway you have to send an e-mail to that = address from=20 what it says in the headers.
 
flow-tools-request= @list.splintered.net
subject=3Dunsubscribe
 
Or go=20 to that address : http:/= /mailman.splintered.net/mailman/listinfo/flow-tools
 

From: Mathew Chigwende=20 [mailto:chigwende@ecoweb.co.zw]
Sent: 15 d=E9cembre 2003=20 01:51
To: flow-tools@splintered.net
Subject: = [Flow-tools]=20 Unsubscribe

Hello,
 
Will you help me to unsubscribe to the = mailing=20 list.  I have sent lots of emails trying to unsubscribe but in=20 vain.
 
Your assistance will be greatly=20 appreciated.
 
Mathew Chigwende
 
chigwende@ecoweb.co.zw<= /DIV>
mathew@ecoweb.co.zw
matthew@ecoweb.co.zw
 
Please help!
 
------_=_NextPart_001_01C3C30E.263662E4-- From splintered-flow-tools-owner@splintered.net Mon Dec 15 16:20:58 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 80495 invoked by uid 4001); 15 Dec 2003 16:20:58 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 80493 invoked by alias); 15 Dec 2003 16:20:57 -0000 Received: from igw1.br.ibm.com (HELO mailgw1.br.ibm.com) (32.104.18.24) by 66.250.216.131 with SMTP; 15 Dec 2003 16:20:57 -0000 Received: from mailhub3.br.ibm.com (mailhub3.br.ibm.com [9.179.63.32]) by mailgw1.br.ibm.com (8.12.9/8.12.3) with ESMTP id hBFGHtNW071352 for ; Mon, 15 Dec 2003 14:17:58 -0200 Received: from d24bml05.br.ibm.com (d24av01.tsc.br.ibm.com [9.179.5.241]) by mailhub3.br.ibm.com (8.12.10/NCO/VER6.6) with ESMTP id hBFGJnYp013634 for ; Mon, 15 Dec 2003 14:19:50 -0200 Importance: Normal MIME-Version: 1.0 Sensitivity: To: flow-tools@splintered.net X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 Message-ID: From: "Alaerte Gladston Vidali" Date: Mon, 15 Dec 2003 14:19:37 -0200 X-MIMETrack: Serialize by Router on d24bml05/24/M/IBM(Release 5.0.9a |January 7, 2002) at 15/12/2003 02:19:38 PM, Serialize complete at 15/12/2003 02:19:38 PM Content-Type: multipart/alternative; boundary="=_alternative 0059B65883256DFD_=" Cc: Subject: [Flow-tools] Installing 0.67 on Solaris X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2003 16:20:58 -0000 This is a multipart message in MIME format. --=_alternative 0059B65883256DFD_= Content-Type: text/plain; charset="us-ascii" Is there any known issue with Solaris 2.6? I am trying to upgrade to Flow-Tools 0.67 but there is the following message: configure: error: Link with "-lz" (zlib >= 1.0.2) failed! I installed Zlib 1-2-1 again but it did not help. Cordially ------------------------------------------------------------------ Alaerte --=_alternative 0059B65883256DFD_= Content-Type: text/html; charset="us-ascii"
Is there any known issue with Solaris 2.6?

I am trying to upgrade to Flow-Tools 0.67 but there is the following message:

configure: error: Link with "-lz" (zlib >= 1.0.2) failed!

I installed Zlib 1-2-1 again but it did not help.

Cordially
------------------------------------------------------------------
Alaerte --=_alternative 0059B65883256DFD_=-- From splintered-flow-tools-owner@splintered.net Mon Dec 15 20:48:12 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 82332 invoked by uid 4001); 15 Dec 2003 20:48:12 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 82330 invoked by alias); 15 Dec 2003 20:48:12 -0000 Received: from garm.csv.cmich.edu (HELO garm.central.cmich.local) (141.209.15.48) by 66.250.216.131 with SMTP; 15 Dec 2003 20:48:12 -0000 Received: from cmail3.central.cmich.local ([141.209.15.83]) by egate1.central.cmich.local with Microsoft SMTPSVC(5.0.2195.6713); Mon, 15 Dec 2003 15:48:05 -0500 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPartTM-000-9f568350-4b2e-43e3-88d7-1e818a6106e3" X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 Date: Mon, 15 Dec 2003 15:48:04 -0500 Message-ID: <291B348BC59B47468C7824603C326082216834@cmail3.central.cmich.local> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: flow-cat limit? Thread-Index: AcPDTMXlMxPPZUYiS52VVJUnUGGsOw== From: "Liao, Kexiao" To: Return-Path: liao1k@cmich.edu X-OriginalArrivalTime: 15 Dec 2003 20:48:05.0456 (UTC) FILETIME=[BD220500:01C3C34C] Cc: Subject: [Flow-tools] flow-cat limit? X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2003 20:48:12 -0000 This is a multi-part message in MIME format. ------=_NextPartTM-000-9f568350-4b2e-43e3-88d7-1e818a6106e3 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C3C34C.BCD77AF8" ------_=_NextPart_001_01C3C34C.BCD77AF8 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, When I use flow-cat to concatenate 1440 flow data files (for example: ft-v05.2003-11-30.235800-0500), there are some error messages, and the result file has only 2.1GB data in it. I wonder whether the flow-cat has some limit when concatenate files. Thanks =20 =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Kexiao Liao=20 CMU Research Corporation=20 2625 Denison Dr. Mount Pleasant, MI 48858=20 Phone 989-774-2424 , Fax 989-774-2416 http://www.thecenter.cmich.edu/=20 liao1k@cmich.edu =20 =20 ------_=_NextPart_001_01C3C34C.BCD77AF8 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi,

   When I use flow-cat to concatenate 1440 = flow data files (for example: ft-v05.2003-11-30.235800-0500), there are some = error messages, and the result file has only 2.1GB data in it. I wonder = whether the flow-cat has some limit when concatenate files. Thanks

 

 

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D

Kexiao Liao

CMU Research Corporation

2625 Denison Dr.

Mount Pleasant, MI 48858

Phone 989-774-2424 , Fax 989-774-2416

http://www.thecenter.cmich.edu/<= /a>

liao1k@cmich.edu

 

 

=00 ------_=_NextPart_001_01C3C34C.BCD77AF8-- ------=_NextPartTM-000-9f568350-4b2e-43e3-88d7-1e818a6106e3-- From splintered-flow-tools-owner@splintered.net Mon Dec 15 20:52:20 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 83076 invoked by uid 4001); 15 Dec 2003 20:52:20 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 83074 invoked by alias); 15 Dec 2003 20:52:20 -0000 Received: from ack.berkeley.edu (128.32.206.66) by 66.250.216.131 with SMTP; 15 Dec 2003 20:52:20 -0000 Received: (from mhunter@localhost) by ack.Berkeley.EDU (8.11.3/8.11.3) id hBFKqIk27117; Mon, 15 Dec 2003 12:52:18 -0800 (PST) Date: Mon, 15 Dec 2003 12:52:18 -0800 From: Mike Hunter To: "Liao, Kexiao" Subject: Re: [Flow-tools] flow-cat limit? Message-ID: <20031215205218.GA26787@ack.Berkeley.EDU> References: <291B348BC59B47468C7824603C326082216834@cmail3.central.cmich.local> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <291B348BC59B47468C7824603C326082216834@cmail3.central.cmich.local> User-Agent: Mutt/1.4.1i Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2003 20:52:21 -0000 On Dec 15, "Liao, Kexiao" wrote: > Hi, > > When I use flow-cat to concatenate 1440 flow data files (for example: > ft-v05.2003-11-30.235800-0500), there are some error messages, and the > result file has only 2.1GB data in it. I wonder whether the flow-cat has > some limit when concatenate files. Thanks Can you please provide the error messages? Mike From splintered-flow-tools-owner@splintered.net Mon Dec 15 21:40:28 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 84028 invoked by uid 4001); 15 Dec 2003 21:40:28 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 84026 invoked by alias); 15 Dec 2003 21:40:28 -0000 Received: from garm.csv.cmich.edu (HELO garm.central.cmich.local) (141.209.15.48) by 66.250.216.131 with SMTP; 15 Dec 2003 21:40:28 -0000 Received: from cmail3.central.cmich.local ([141.209.15.83]) by egate1.central.cmich.local with Microsoft SMTPSVC(5.0.2195.6713); Mon, 15 Dec 2003 16:40:21 -0500 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 Subject: RE: [Flow-tools] flow-cat limit? Date: Mon, 15 Dec 2003 16:40:21 -0500 Message-ID: <291B348BC59B47468C7824603C326082216836@cmail3.central.cmich.local> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Flow-tools] flow-cat limit? Thread-Index: AcPDTVVAc8BagG3FTlyjj1nNZUQFtgABfl7g From: "Liao, Kexiao" To: "Mike Hunter" , =?iso-8859-1?Q?=22Perez=2C_Mart=EDn=22?= Return-Path: liao1k@cmich.edu X-OriginalArrivalTime: 15 Dec 2003 21:40:21.0695 (UTC) FILETIME=[0A7A1CF0:01C3C354] Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2003 21:40:28 -0000 Following are the error messages: bash-2.05a# flow-cat -o result.netflow ./test/ flow-cat: writen(): File too large flow-cat: ftio_write(): failed bash-2.05a# ./test directory has 1440 net flow files(each about 2.7MB data). The result.netflow file:=20 ls -l result.netflow -rw-r--r-- 1 root system 2147483647 Dec 15 16:21 result.netflow =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Kexiao Liao=20 CMU Research Corporation=20 2625 Denison Dr. Mount Pleasant, MI 48858=20 Phone 989-774-2424 , Fax 989-774-2416 http://www.thecenter.cmich.edu/=20 liao1k@cmich.edu =20 -----Original Message----- From: Mike Hunter [mailto:mhunter@ack.Berkeley.EDU]=20 Sent: Monday, December 15, 2003 3:52 PM To: Liao, Kexiao Cc: flow-tools@splintered.net Subject: Re: [Flow-tools] flow-cat limit? On Dec 15, "Liao, Kexiao" wrote: > Hi, >=20 > When I use flow-cat to concatenate 1440 flow data files (for = example: > ft-v05.2003-11-30.235800-0500), there are some error messages, and the > result file has only 2.1GB data in it. I wonder whether the flow-cat = has > some limit when concatenate files. Thanks Can you please provide the error messages? Mike From splintered-flow-tools-owner@splintered.net Mon Dec 15 21:48:30 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 84813 invoked by uid 4001); 15 Dec 2003 21:48:30 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 84811 invoked by alias); 15 Dec 2003 21:48:29 -0000 Received: from ack.berkeley.edu (128.32.206.66) by 66.250.216.131 with SMTP; 15 Dec 2003 21:48:29 -0000 Received: (from mhunter@localhost) by ack.Berkeley.EDU (8.11.3/8.11.3) id hBFLmSX04543; Mon, 15 Dec 2003 13:48:28 -0800 (PST) Date: Mon, 15 Dec 2003 13:48:28 -0800 From: Mike Hunter To: "Liao, Kexiao" Subject: Re: [Flow-tools] flow-cat limit? Message-ID: <20031215214828.GA3975@ack.Berkeley.EDU> References: <291B348BC59B47468C7824603C326082216836@cmail3.central.cmich.local> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <291B348BC59B47468C7824603C326082216836@cmail3.central.cmich.local> User-Agent: Mutt/1.4.1i Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2003 21:48:30 -0000 On Dec 15, "Liao, Kexiao" wrote: Do you mean to say "./test/*" ? Is the result the same if you do "flow-cat ./test/* > result.netflow"? > Following are the error messages: > > bash-2.05a# flow-cat -o result.netflow ./test/ > > flow-cat: writen(): File too large > flow-cat: ftio_write(): failed > bash-2.05a# > > ./test directory has 1440 net flow files(each about 2.7MB data). > > The result.netflow file: > ls -l result.netflow > -rw-r--r-- 1 root system 2147483647 Dec 15 16:21 result.netflow > > > When I use flow-cat to concatenate 1440 flow data files (for example: > > ft-v05.2003-11-30.235800-0500), there are some error messages, and the > > result file has only 2.1GB data in it. I wonder whether the flow-cat has > > some limit when concatenate files. Thanks > > Can you please provide the error messages? > > Mike > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From splintered-flow-tools-owner@splintered.net Mon Dec 15 22:11:47 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 85667 invoked by uid 4001); 15 Dec 2003 22:11:47 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 85665 invoked by alias); 15 Dec 2003 22:11:47 -0000 Received: from garm.csv.cmich.edu (HELO garm.central.cmich.local) (141.209.15.48) by 66.250.216.131 with SMTP; 15 Dec 2003 22:11:47 -0000 Received: from cmail3.central.cmich.local ([141.209.15.83]) by egate1.central.cmich.local with Microsoft SMTPSVC(5.0.2195.6713); Mon, 15 Dec 2003 17:11:40 -0500 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 Subject: RE: [Flow-tools] flow-cat limit? Date: Mon, 15 Dec 2003 17:11:40 -0500 Message-ID: <291B348BC59B47468C7824603C326082216837@cmail3.central.cmich.local> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Flow-tools] flow-cat limit? Thread-Index: AcPDVx4BMaP2wQdTSGejfb5KmgOuhwAAPcTJ From: "Liao, Kexiao" To: "Paul Schmidt" , Return-Path: liao1k@cmich.edu X-OriginalArrivalTime: 15 Dec 2003 22:11:40.0785 (UTC) FILETIME=[6A807E10:01C3C358] Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2003 22:11:47 -0000 VGggT1MgaXMgQUlYIFZlcnNpb24gNS4yLCBhbmQgdGhlIHdvcmtpbmcgZmlsZSBzeXN0ZW0gaXMg amZzMi4gV2UgYWxyZWFkeSBjcmVhdGVkIGEgZmlsZSB3aGljaCBpcyA2NEdCIGluIHNpemUuDQog DQpLZXhpYW8NCg0KCS0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tIA0KCUZyb206IFBhdWwgU2No bWlkdCBbbWFpbHRvOmRhd3NvbnNAYXVnc2J1cmcuZWR1XSANCglTZW50OiBNb24gMTIvMTUvMjAw MyA1OjAyIFBNIA0KCVRvOiBMaWFvLCBLZXhpYW8gDQoJQ2M6IA0KCVN1YmplY3Q6IFJlOiBbRmxv dy10b29sc10gZmxvdy1jYXQgbGltaXQ/DQoJDQoJDQoNCglNeSBpbml0aWFsIHJlYWN0aW9uIGlz IHRoYXQgdGhpcyBpc24ndCBhIGNhc2Ugb2YgZmxvdy1jYXQgbm90IHdvcmtpbmcsDQoJYnV0IHJh dGhlciB5b3VyIE9TIGRvZXNuJ3Qgc3VwcG9ydCBmaWxlcyBsYXJnZXIgdGhhbiAyR0IuICBJdCdz IGF0IGxlYXN0DQoJd29ydGggaW52ZXN0aWdhdGluZyB0byBzZWUgaWYgeW91IE9TIHN1cHBvcnRz IGxhcmdlIGZpbGVzLg0KCQ0KCS1QYXVsIFNjaG1pZHQNCglkYXdzb25zQGF1Z3NidXJnLmVkdQ0K CQ0KCUxpYW8sIEtleGlhbyB3cm90ZToNCgkNCgk+Rm9sbG93aW5nIGFyZSB0aGUgZXJyb3IgbWVz c2FnZXM6DQoJPg0KCT5iYXNoLTIuMDVhIyBmbG93LWNhdCAtbyByZXN1bHQubmV0ZmxvdyAuL3Rl c3QvDQoJPmZsb3ctY2F0OiB3cml0ZW4oKTogRmlsZSB0b28gbGFyZ2UNCgk+Zmxvdy1jYXQ6IGZ0 aW9fd3JpdGUoKTogZmFpbGVkDQoJPmJhc2gtMi4wNWEjDQoJPg0KCT4uL3Rlc3QgZGlyZWN0b3J5 IGhhcyAxNDQwIG5ldCBmbG93IGZpbGVzKGVhY2ggYWJvdXQgMi43TUIgZGF0YSkuDQoJPg0KCT5U aGUgcmVzdWx0Lm5ldGZsb3cgZmlsZToNCgk+bHMgLWwgcmVzdWx0Lm5ldGZsb3cNCgk+LXJ3LXIt LXItLSAgIDEgcm9vdCAgICAgc3lzdGVtICAgMjE0NzQ4MzY0NyBEZWMgMTUgMTY6MjEgcmVzdWx0 Lm5ldGZsb3cNCgk+DQoJPj09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KCT5LZXhpYW8gTGlh bw0KCT5DTVUgUmVzZWFyY2ggQ29ycG9yYXRpb24NCgk+MjYyNSBEZW5pc29uIERyLg0KCT5Nb3Vu dCBQbGVhc2FudCwgTUkgNDg4NTgNCgk+UGhvbmUgOTg5LTc3NC0yNDI0ICwgRmF4IDk4OS03NzQt MjQxNg0KCT5odHRwOi8vd3d3LnRoZWNlbnRlci5jbWljaC5lZHUvDQoJPmxpYW8xa0BjbWljaC5l ZHUNCgk+DQoJPg0KCT4tLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KCT5Gcm9tOiBNaWtlIEh1 bnRlciBbbWFpbHRvOm1odW50ZXJAYWNrLkJlcmtlbGV5LkVEVV0NCgk+U2VudDogTW9uZGF5LCBE ZWNlbWJlciAxNSwgMjAwMyAzOjUyIFBNDQoJPlRvOiBMaWFvLCBLZXhpYW8NCgk+Q2M6IGZsb3ct dG9vbHNAc3BsaW50ZXJlZC5uZXQNCgk+U3ViamVjdDogUmU6IFtGbG93LXRvb2xzXSBmbG93LWNh dCBsaW1pdD8NCgk+DQoJPk9uIERlYyAxNSwgIkxpYW8sIEtleGlhbyIgd3JvdGU6DQoJPg0KCT4g DQoJPg0KCT4+SGksDQoJPj4NCgk+PiAgIFdoZW4gSSB1c2UgZmxvdy1jYXQgdG8gY29uY2F0ZW5h dGUgMTQ0MCBmbG93IGRhdGEgZmlsZXMgKGZvciBleGFtcGxlOg0KCT4+ZnQtdjA1LjIwMDMtMTEt MzAuMjM1ODAwLTA1MDApLCB0aGVyZSBhcmUgc29tZSBlcnJvciBtZXNzYWdlcywgYW5kIHRoZQ0K CT4+cmVzdWx0IGZpbGUgaGFzIG9ubHkgMi4xR0IgZGF0YSBpbiBpdC4gSSB3b25kZXIgd2hldGhl ciB0aGUgZmxvdy1jYXQgaGFzDQoJPj5zb21lIGxpbWl0IHdoZW4gY29uY2F0ZW5hdGUgZmlsZXMu IFRoYW5rcw0KCT4+ICAgDQoJPj4NCgk+DQoJPkNhbiB5b3UgcGxlYXNlIHByb3ZpZGUgdGhlIGVy cm9yIG1lc3NhZ2VzPw0KCT4NCgk+TWlrZQ0KCT5fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fXw0KCT5GbG93LXRvb2xzIG1haWxpbmcgbGlzdA0KCT5mbG93LXRv b2xzQHNwbGludGVyZWQubmV0DQoJPmh0dHA6Ly9tYWlsbWFuLnNwbGludGVyZWQubmV0L21haWxt YW4vbGlzdGluZm8vZmxvdy10b29scw0KCT4gDQoJPg0KCQ0KCQ0KDQo= From splintered-flow-tools-owner@splintered.net Mon Dec 15 22:20:52 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 86464 invoked by uid 4001); 15 Dec 2003 22:20:52 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 86462 invoked by alias); 15 Dec 2003 22:20:52 -0000 Received: from mail1.panix.com (166.84.1.72) by 66.250.216.131 with SMTP; 15 Dec 2003 22:20:52 -0000 Received: from panix5.panix.com (panix5.panix.com [166.84.1.5]) by mail1.panix.com (Postfix) with ESMTP id 2EC5B48833; Mon, 15 Dec 2003 17:20:51 -0500 (EST) Received: (from eravin@localhost) by panix5.panix.com (8.11.6p2-a/8.8.8/PanixN1.1) id hBFMKp708000; Mon, 15 Dec 2003 17:20:51 -0500 (EST) Date: Mon, 15 Dec 2003 17:20:51 -0500 From: Ed Ravin To: "Liao, Kexiao" Subject: Re: [Flow-tools] flow-cat limit? Message-ID: <20031215222051.GB6271@panix.com> References: <291B348BC59B47468C7824603C326082216836@cmail3.central.cmich.local> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <291B348BC59B47468C7824603C326082216836@cmail3.central.cmich.local> User-Agent: Mutt/1.4.1i X-Y-Z: 1, 2, 3? Cc: flow-tools@splintered.net, Mike Hunter X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2003 22:20:52 -0000 On Mon, Dec 15, 2003 at 04:40:21PM -0500, Liao, Kexiao wrote: > Following are the error messages: > > bash-2.05a# flow-cat -o result.netflow ./test/ > flow-cat: writen(): File too large > flow-cat: ftio_write(): failed ... > ./test directory has 1440 net flow files(each about 2.7MB data). Which is around 3.9 or 4.0 gigabytes, but: > The result.netflow file: > ls -l result.netflow > -rw-r--r-- 1 root system 2147483647 Dec 15 16:21 result.netflow Only 2.1 gigabytes. Sounds like you are hitting the maximum file size limit in your environment. The problem is not the input, but the output file. What operating system and version are you running? I can think of a workaround or two, but we need to know more about your environment. From splintered-flow-tools-owner@splintered.net Mon Dec 15 22:30:07 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 87240 invoked by uid 4001); 15 Dec 2003 22:30:07 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 87238 invoked by alias); 15 Dec 2003 22:30:07 -0000 Received: from mail3.panix.com (166.84.1.74) by 66.250.216.131 with SMTP; 15 Dec 2003 22:30:07 -0000 Received: from panix5.panix.com (panix5.panix.com [166.84.1.5]) by mail3.panix.com (Postfix) with ESMTP id 352D4981B2; Mon, 15 Dec 2003 17:30:07 -0500 (EST) Received: (from eravin@localhost) by panix5.panix.com (8.11.6p2-a/8.8.8/PanixN1.1) id hBFMU7k08770; Mon, 15 Dec 2003 17:30:07 -0500 (EST) Date: Mon, 15 Dec 2003 17:30:07 -0500 From: Ed Ravin To: "Liao, Kexiao" Subject: Re: [Flow-tools] flow-cat limit? Message-ID: <20031215223007.GC6271@panix.com> References: <291B348BC59B47468C7824603C326082216837@cmail3.central.cmich.local> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <291B348BC59B47468C7824603C326082216837@cmail3.central.cmich.local> User-Agent: Mutt/1.4.1i X-Y-Z: 1, 2, 3? Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Dec 2003 22:30:07 -0000 On Mon, Dec 15, 2003 at 05:11:40PM -0500, Liao, Kexiao wrote: > Th[e] OS is AIX Version 5.2, and the working file system is jfs2. > We already created a file which is 64GB in size. Yes, but what programs did you use to create those files? Did you do it using one of the programs in the flow-tools suite? To create large files, most OS's require that the applications use a different interface to the I/O library, I think it's open2() in Solaris instead of open(), for example. I just peeked at flow-cat's source and I didn't see any hooks for alternate calls, so I suspect flow-tools might not support long filenames. From oliver.kurth@cyclades.de Tue Dec 16 11:31:26 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 90929 invoked by alias); 16 Dec 2003 11:31:26 -0000 Received: from mail.linux-router.org (HELO www.cyclades.de) (62.225.173.194) by 66.250.216.131 with SMTP; 16 Dec 2003 11:31:26 -0000 Received: from [192.168.10.32] (helo=oliver.cyclades.de) by www.cyclades.de with asmtp (Exim 3.35 #1 (Debian)) id 1AWDQC-0005CA-00 for ; Tue, 16 Dec 2003 12:31:24 +0100 Received: from 127.0.0.1 (ident=oku) by tammuz with esmtp (masqmail 0.2.20) id 1AWDMJ-8Eg-00 for ; Tue, 16 Dec 2003 12:27:23 +0100 From: Oliver Kurth To: Flow Tools ML Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-5fIoDUfs99A4drioPl90" Message-Id: <1071574043.26031.22.camel@tammuz> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.5 Date: Tue, 16 Dec 2003 12:27:23 +0100 X-MailScanner: Found to be clean X-MailScanner-SpamCheck: Subject: [Flow-tools] patch to build shared library X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2003 11:31:26 -0000 --=-5fIoDUfs99A4drioPl90 Content-Type: multipart/mixed; boundary="=-0hW0IET5RCZXGKlYyrvU" --=-0hW0IET5RCZXGKlYyrvU Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Hi, I have a patch here to use libtool to make a shared library, and link the binaries against it. Apply the patch and call libtoolize -a -c autoconf automake -a -c -f (I am not completely sure about the order) I tested this with the upstream 0.67 version. I makes quite a difference: static version: /tmp/ft-static/usr/local/netflow/bin: total 3636 -rwxr-xr-x 1 oku oku 241628 Dec 10 11:26 flow-capture -rwxr-xr-x 1 oku oku 134172 Dec 10 11:26 flow-cat ... -rwxr-xr-x 1 oku oku 613636 Dec 10 11:26 flow-report ... shared version: /tmp/ft-shared/usr/local/netflow/bin: total 436 -rwxr-xr-x 1 oku oku 26760 Dec 10 11:54 flow-capture -rwxr-xr-x 1 oku oku 13356 Dec 10 11:54 flow-cat ... -rwxr-xr-x 1 oku oku 8236 Dec 10 11:54 flow-report ... =20 But: I haven't tested it yet. Greetings, Oliver --=20 Cyclades GmbH - Everywhere with Linux Oliver Kurth Cyclades GmbH Research & Development Rennweg 33 Fon/Fax: +49 (0) 8122 90999-34/-33 D-85435 Erding --=-0hW0IET5RCZXGKlYyrvU Content-Disposition: attachment; filename=ft.patch Content-Type: text/x-patch; name=ft.patch; charset=ANSI_X3.4-1968 Content-Transfer-Encoding: base64 LS0tIGZsb3ctdG9vbHMtMC42Ny9saWIvTWFrZWZpbGUuYW0JMjAwMy0wNy0yOCAyMjoyNTozMC4w MDAwMDAwMDAgKzAyMDANCisrKyBmbG93LXRvb2xzLTAuNjcuc2hhcmVkL2xpYi9NYWtlZmlsZS5h bQkyMDAzLTEyLTEwIDExOjQyOjM2LjAwMDAwMDAwMCArMDEwMA0KQEAgLTcsMTIgKzcsMTIgQEAN CiBERUZTPS1JLiAtSSQoc3JjZGlyKS9saWINCiANCiBpbmNsdWRlX0hFQURFUlMgPSBmdGxpYi5o IGZ0cXVldWUuaCByYWRpeC5oIGZ0cGF0aHMuaCBmdGNvbmZpZy5oDQotbGliX0xJQlJBUklFUyA9 IGxpYmZ0LmENCitsaWJfTFRMSUJSQVJJRVMgPSBsaWJmdC5sYQ0KIA0KLWxpYmZ0X2FfU09VUkNF UyA9IGZ0aW8uYyBmdHN3YXAuYyBmdGVuY29kZS5jIGZ0ZGVjb2RlLmMgZnRwcm9mLmMgYml0MTAy NC5jIFwNCitsaWJmdF9sYV9TT1VSQ0VTID0gZnRpby5jIGZ0c3dhcC5jIGZ0ZW5jb2RlLmMgZnRk ZWNvZGUuYyBmdHByb2YuYyBiaXQxMDI0LmMgXA0KICBmbXQuYyBzdXBwb3J0LmMgZnRmaWxlLmMg ZnR0bHYuYyBmdG1hcC5jIGZ0cmVjLmMgZnRlcnIuYyBcDQogIGZ0Y2hhc2guYyBmdHN5bS5jIHJh ZGl4LmMgZnR0YWcuYyBmdGZpbC5jIGZ0c3RhdC5jIGdldGRhdGUuYyBmdHhmaWVsZC5jXA0KICBm dG1hc2suYyBmdHZhci5jIGZ0eGxhdGUuYw0KIA0KLWxpYmZ0X2FfTElCQUREID0gQExJQk9CSlNA IA0KK2xpYmZ0X2xhX0xJQkFERCA9IEBMSUJPQkpTQCANCiANCi0tLSBmbG93LXRvb2xzLTAuNjcv Y29uZmlndXJlLmluCTIwMDMtMTItMDQgMDc6MTg6MTUuMDAwMDAwMDAwICswMTAwDQorKysgZmxv dy10b29scy0wLjY3LnNoYXJlZC9jb25maWd1cmUuaW4JMjAwMy0xMi0xMCAxMTo0MzozMC4wMDAw MDAwMDAgKzAxMDANCkBAIC0xMCw2ICsxMCw3IEBADQogDQogZG5sIENoZWNrcyBmb3IgcHJvZ3Jh bXMuDQogQUNfUFJPR19DQw0KK0FDX1BST0dfTElCVE9PTA0KIEFDX1BST0dfSU5TVEFMTA0KIEFD X1BST0dfTUFLRV9TRVQNCiBBQ19QUk9HX1lBQ0MNCi0tLSBmbG93LXRvb2xzLTAuNjcvbGliL2Z0 ZmlsZS5jCTIwMDMtMDItMTMgMDM6Mzg6NDIuMDAwMDAwMDAwICswMTAwDQorKysgZmxvdy10b29s cy0wLjY3LnNoYXJlZC9saWIvZnRmaWxlLmMJMjAwMy0xMi0xMCAxMTo1MjoxNy4wMDAwMDAwMDAg KzAxMDANCkBAIC01Myw3ICs1Myw4IEBADQogDQogaW50IGxvYWRfZGlyKGNoYXIgKnByZWZpeCwg c3RydWN0IGZ0ZmlsZV9lbnRyaWVzICpmdGUsIGludCBmbGFncywgaW50ICpkZXB0aCk7DQogDQot ZXh0ZXJuIGludCBkZWJ1ZzsNCisvL2V4dGVybiBpbnQgZGVidWc7DQoraW50IGRlYnVnOw0KIA0K IC8qDQogICogZnVuY3Rpb246IGZ0ZmlsZV9lbnRyeV9uZXcNCg== --=-0hW0IET5RCZXGKlYyrvU-- --=-5fIoDUfs99A4drioPl90 Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux) iD8DBQA/3uwbUmVSJkUeqxsRAs7zAKCvBhp4dBq2Fhr0hw8CaLYPOoXagQCgsiz3 mDRzfhEP6CFdgcSq7oPRiBE= =QO+e -----END PGP SIGNATURE----- --=-5fIoDUfs99A4drioPl90-- From splintered-flow-tools-owner@splintered.net Tue Dec 16 16:27:46 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 92668 invoked by uid 4001); 16 Dec 2003 16:27:46 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 92666 invoked by alias); 16 Dec 2003 16:27:46 -0000 Received: from garm.csv.cmich.edu (HELO garm.central.cmich.local) (141.209.15.48) by 66.250.216.131 with SMTP; 16 Dec 2003 16:27:46 -0000 Received: from cmail3.central.cmich.local ([141.209.15.83]) by egate1.central.cmich.local with Microsoft SMTPSVC(5.0.2195.6713); Tue, 16 Dec 2003 11:27:38 -0500 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 Subject: RE: [Flow-tools] flow-cat limit? Date: Tue, 16 Dec 2003 11:27:38 -0500 Message-ID: <291B348BC59B47468C7824603C326082216838@cmail3.central.cmich.local> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Flow-tools] flow-cat limit? Thread-Index: AcPDWv7QASNP1TwfTECKNNqBbS+UCwAlg/uw From: "Liao, Kexiao" To: "Ed Ravin" Return-Path: liao1k@cmich.edu X-OriginalArrivalTime: 16 Dec 2003 16:27:38.0820 (UTC) FILETIME=[85585840:01C3C3F1] Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2003 16:27:46 -0000 Hi All, Does anyone has some ideas about the open2() function to create large file in C, I just check it in Linux and AIX, there is no such function? Any help is appreciated.=20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Kexiao Liao=20 CMU Research Corporation=20 2625 Denison Dr. Mount Pleasant, MI 48858=20 Phone 989-774-2424 , Fax 989-774-2416 http://www.thecenter.cmich.edu/=20 liao1k@cmich.edu =20 -----Original Message----- From: Ed Ravin [mailto:eravin@panix.com]=20 Sent: Monday, December 15, 2003 5:30 PM To: Liao, Kexiao Cc: Paul Schmidt; flow-tools@splintered.net Subject: Re: [Flow-tools] flow-cat limit? On Mon, Dec 15, 2003 at 05:11:40PM -0500, Liao, Kexiao wrote: > Th[e] OS is AIX Version 5.2, and the working file system is jfs2. > We already created a file which is 64GB in size. Yes, but what programs did you use to create those files? Did you do it using one of the programs in the flow-tools suite? To create large files, most OS's require that the applications use a different interface to the I/O library, I think it's open2() in Solaris instead of open(), for example. I just peeked at flow-cat's source and I didn't see any hooks for alternate calls, so I suspect flow-tools might not support long filenames. From splintered-flow-tools-owner@splintered.net Tue Dec 16 18:09:56 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 93967 invoked by uid 4001); 16 Dec 2003 18:09:56 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 93965 invoked by alias); 16 Dec 2003 18:09:56 -0000 Received: from verdi.nethelp.no (194.19.15.2) by 66.250.216.131 with SMTP; 16 Dec 2003 18:09:56 -0000 Received: (qmail 39525 invoked by uid 1001); 16 Dec 2003 18:09:54 +0000 (GMT) To: liao1k@cmich.edu Subject: RE: [Flow-tools] flow-cat limit? From: sthaug@nethelp.no In-Reply-To: Your message of "Tue, 16 Dec 2003 11:27:38 -0500" References: <291B348BC59B47468C7824603C326082216838@cmail3.central.cmich.local> X-Mailer: Mew version 1.05+ on Emacs 19.34.1 Mime-Version: 1.0 Content-Type: Text/Plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Date: Tue, 16 Dec 2003 19:09:54 +0100 Message-ID: <39523.1071598194@verdi.nethelp.no> Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2003 18:09:56 -0000 > Does anyone has some ideas about the open2() function to create large > file in C, I just check it in Linux and AIX, there is no such function? > Any help is appreciated. Some operating systems don't need it. FreeBSD natively supports large files - and as far as I know this also applies to NetBSD and OpenBSD. Steinar Haug, Nethelp consulting, sthaug@nethelp.no From splintered-flow-tools-owner@splintered.net Tue Dec 16 19:08:01 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 94968 invoked by uid 4001); 16 Dec 2003 19:08:01 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 94966 invoked by alias); 16 Dec 2003 19:08:00 -0000 Received: from igw2.br.ibm.com (HELO mailgw2.br.ibm.com) (32.104.18.25) by 66.250.216.131 with SMTP; 16 Dec 2003 19:08:00 -0000 Received: from mailhub1.br.ibm.com (mailhub1.br.ibm.com [9.179.63.14]) by mailgw2.br.ibm.com (8.12.9/8.12.3) with ESMTP id hBGIvuli036596 for ; Tue, 16 Dec 2003 16:57:56 -0200 Received: from d24bml05.br.ibm.com (d24av02.tsc.br.ibm.com [9.179.5.242]) by mailhub1.br.ibm.com (8.12.10/NCO/VER6.6) with ESMTP id hBGJ7vP8111836 for ; Tue, 16 Dec 2003 17:07:57 -0200 Importance: Normal MIME-Version: 1.0 Sensitivity: To: flow-tools@splintered.net X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 Message-ID: From: "Alaerte Gladston Vidali" Date: Tue, 16 Dec 2003 17:07:45 -0200 X-MIMETrack: Serialize by Router on d24bml05/24/M/IBM(Release 5.0.9a |January 7, 2002) at 16/12/2003 05:07:46 PM, Serialize complete at 16/12/2003 05:07:46 PM Content-Type: multipart/alternative; boundary="=_alternative 00691AEE83256DFE_=" Cc: Subject: [Flow-tools] Configure error on Solaris X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2003 19:08:01 -0000 This is a multipart message in MIME format. --=_alternative 00691AEE83256DFE_= Content-Type: text/plain; charset="us-ascii" Any clue on the following problem when trying to compile Flow-Tools 0.67 on Solaris? ./configure checking for a BSD-compatible install... ./install-sh -c checking whether build environment is sane... yes checking for gawk... no checking for mawk... no checking for nawk... nawk checking whether make sets $(MAKE)... yes checking for gcc... gcc . . checking for zlibVersion in -lz... no configure: error: Link with "-lz" (zlib >= 1.0.2) failed! I reinstalled zlib-1.2.1. Flow-Tools 0.59 is working fine on this machine, but I would like to try those 0.67 new commands. Cordially, ------------------------------------------------------------------ Alaerte Gladston Vidali IBM Global Services - SO Tel.55+11+2121-2879 Fax:55+11+2121-2449 Sent by: flow-tools-bounces@list.splintered.net To: liao1k@cmich.edu cc: flow-tools@splintered.net Subject: RE: [Flow-tools] flow-cat limit? > Does anyone has some ideas about the open2() function to create large > file in C, I just check it in Linux and AIX, there is no such function? > Any help is appreciated. Some operating systems don't need it. FreeBSD natively supports large files - and as far as I know this also applies to NetBSD and OpenBSD. Steinar Haug, Nethelp consulting, sthaug@nethelp.no _______________________________________________ Flow-tools mailing list flow-tools@splintered.net http://mailman.splintered.net/mailman/listinfo/flow-tools --=_alternative 00691AEE83256DFE_= Content-Type: text/html; charset="us-ascii"
Any clue on the following problem when trying to compile Flow-Tools 0.67 on Solaris?

./configure

checking for a BSD-compatible install... ./install-sh -c
checking whether build environment is sane... yes
checking for gawk... no
checking for mawk... no
checking for nawk... nawk
checking whether make sets $(MAKE)... yes
checking for gcc... gcc
.
.
checking for zlibVersion in -lz... no
configure: error: Link with "-lz" (zlib >= 1.0.2) failed!

I reinstalled zlib-1.2.1.  

Flow-Tools 0.59 is working fine on this machine, but I would like to try those 0.67 new commands.

Cordially,
------------------------------------------------------------------
Alaerte Gladston Vidali
IBM Global Services - SO
Tel.55+11+2121-2879   Fax:55+11+2121-2449

Sent by:        flow-tools-bounces@list.splintered.net

To:        liao1k@cmich.edu
cc:        flow-tools@splintered.net
Subject:        RE: [Flow-tools] flow-cat limit?



>   Does anyone has some ideas about the open2() function to create large
> file in C, I just check it in Linux and AIX, there is no such function?
> Any help is appreciated.
Some operating systems don't need it. FreeBSD natively supports large
files - and as far as I know this also applies to NetBSD and OpenBSD.
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
_______________________________________________
Flow-tools mailing list
flow-tools@splintered.net
http://mailman.splintered.net/mailman/listinfo/flow-tools

--=_alternative 00691AEE83256DFE_=-- From splintered-flow-tools-owner@splintered.net Tue Dec 16 19:25:25 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 95818 invoked by uid 4001); 16 Dec 2003 19:25:25 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 95816 invoked by alias); 16 Dec 2003 19:25:24 -0000 Received: from mail.datanet.ee (195.222.0.3) by 66.250.216.131 with SMTP; 16 Dec 2003 19:25:24 -0000 Received: from lost.data.ee (lost.data.ee [195.222.1.145]) by mail.datanet.ee (8.12.9/8.12.1) with ESMTP id hBGJPFbW000938 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Tue, 16 Dec 2003 21:25:15 +0200 Date: Tue, 16 Dec 2003 21:22:29 +0200 (EET) From: Cougar X-X-Sender: cougar@lost.data.ee To: Ed Ravin Subject: Re: [Flow-tools] flow-cat limit? In-Reply-To: <20031215223007.GC6271@panix.com> Message-ID: References: <291B348BC59B47468C7824603C326082216837@cmail3.central.cmich.local> <20031215223007.GC6271@panix.com> X-NCC-RegID: ee.data MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2003 19:25:25 -0000 On Mon, 15 Dec 2003, Ed Ravin wrote: > On Mon, Dec 15, 2003 at 05:11:40PM -0500, Liao, Kexiao wrote: > > Th[e] OS is AIX Version 5.2, and the working file system is jfs2. > > We already created a file which is 64GB in size. > > Yes, but what programs did you use to create those files? Did you do > it using one of the programs in the flow-tools suite? > > To create large files, most OS's require that the applications use a > different interface to the I/O library, I think it's open2() in Solaris > instead of open(), for example. I just peeked at flow-cat's source and > I didn't see any hooks for alternate calls, so I suspect flow-tools > might not support long filenames. Large files (larger than 2 GB) and long filenames are different things ;-) To create files larger than 2 GB your filesystem have to support that. Maximum filesize is usually 32 bit signed number which can be up to 2147483648 ie 2 GB. Some newer filesystems support 64 bit file size and thus can be up to 9 EB but usually is 1, 2 or 4 TB (which is most;ly big enough). As long you don't need to read/write bigger than 2 GB chunks of memory from/to disk you can still use 32 bit read() and write() syscalls. Only syscalls which have to use 64 bit file size/offset are seek() and stat(). As open() and close() don't have any relations with file size, they are the same for 32 and 64 bit systems. So, first check that your filesystem has 64 bit file size support and then check that your libc is 64 bit compatible. --- Cougar From splintered-flow-tools-owner@splintered.net Tue Dec 16 19:30:58 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 96411 invoked by uid 4001); 16 Dec 2003 19:30:58 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 96408 invoked by alias); 16 Dec 2003 19:30:58 -0000 Received: from ack.berkeley.edu (128.32.206.66) by 66.250.216.131 with SMTP; 16 Dec 2003 19:30:58 -0000 Received: (from mhunter@localhost) by ack.Berkeley.EDU (8.11.3/8.11.3) id hBGJUvq25224 for flow-tools@splintered.net; Tue, 16 Dec 2003 11:30:57 -0800 (PST) Date: Tue, 16 Dec 2003 11:30:57 -0800 From: Mike Hunter To: flow-tools@splintered.net Subject: RE: [Flow-tools] flow-cat limit? Message-ID: <20031216193057.GB24822@ack.Berkeley.EDU> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.1i Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Dec 2003 19:30:58 -0000 Forwarding for mail-archive completeness.... ----- Forwarded message from "Liao, Kexiao" ----- > Did you try what I suggested, using a redirect instead of -o? Hi Mike, That works (flow-cat ./test/ > result.netflow), it uses OS redirect function. Thanks From virgo@azcher.kharkov.ua Thu Dec 18 10:49:56 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 6334 invoked by alias); 18 Dec 2003 10:49:56 -0000 Received: from ns1.teleportsv.net (HELO relay.teleportsv.net) (193.41.48.230) by 66.250.216.131 with SMTP; 18 Dec 2003 10:49:56 -0000 Received: from virgo.teleportsv ([192.168.69.69] helo=azcher.kharkov.ua) by relay.teleportsv.net with asmtp (Exim 4.24; FreeBSD) id 1AWvj8-0002mP-Sz for flow-tools@list.splintered.net; Thu, 18 Dec 2003 12:49:54 +0200 Message-ID: <3FE18652.7000004@azcher.kharkov.ua> Date: Thu, 18 Dec 2003 12:49:54 +0200 From: Sergey Dolgopolov User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.5) Gecko/20031119 X-Accept-Language: ru, en-us, en MIME-Version: 1.0 To: flow-tools@list.splintered.net Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: [Flow-tools] NetFlow Aggregation X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2003 10:49:56 -0000 Hello! How i can use in flow-capture both aggregation 8.3 and 8.4? flow-capture -V 8.3 8.4 -w /flow 192.168.2.1/192.168.2.6/3000? Thanks. From splintered-flow-tools-owner@splintered.net Thu Dec 18 16:08:04 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 8454 invoked by uid 4001); 18 Dec 2003 16:08:04 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 8452 invoked by alias); 18 Dec 2003 16:08:04 -0000 Received: from igw1.br.ibm.com (HELO mailgw1.br.ibm.com) (32.104.18.24) by 66.250.216.131 with SMTP; 18 Dec 2003 16:08:04 -0000 Received: from mailhub1.br.ibm.com (mailhub1.br.ibm.com [9.179.63.14]) by mailgw1.br.ibm.com (8.12.9/8.12.3) with ESMTP id hBIG61NW075084 for ; Thu, 18 Dec 2003 14:06:01 -0200 Received: from d24bml05.br.ibm.com (d24av01.tsc.br.ibm.com [9.179.5.241]) by mailhub1.br.ibm.com (8.12.10/NCO/VER6.6) with ESMTP id hBIG7wk1154194 for ; Thu, 18 Dec 2003 14:07:58 -0200 Importance: Normal MIME-Version: 1.0 Sensitivity: To: flow-tools@splintered.net X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 Message-ID: From: "Alaerte Gladston Vidali" Date: Thu, 18 Dec 2003 14:07:44 -0200 X-MIMETrack: Serialize by Router on d24bml05/24/M/IBM(Release 5.0.9a |January 7, 2002) at 18/12/2003 02:07:46 PM, Serialize complete at 18/12/2003 02:07:46 PM Content-Type: multipart/alternative; boundary="=_alternative 00589FA683256E00_=" Cc: Subject: [Flow-tools] Upgrading to 0.67 X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Dec 2003 16:08:04 -0000 This is a multipart message in MIME format. --=_alternative 00589FA683256E00_= Content-Type: text/plain; charset="us-ascii" Sorry if it is a basic question, but I can not figure out why the upgrade from 0.59 to 0.67 is resulting in error messages. I though it was Zlib, but after reinstalling it there was no change. This is the error message: ./configure . . checking for zlibVersion in -lz... no configure: error: Link with "-lz" (zlib >= 1.0.2) failed! It is a Sparc20 station, running Solaris 2.6. Any Though? Thanks in Advance, ------------------------------------------------------------------ Alaerte Gladston Vidali IBM Global Services - SO Tel.55+11+2121-2879 Fax:55+11+2121-2449 --=_alternative 00589FA683256E00_= Content-Type: text/html; charset="us-ascii"
Sorry if it is a basic question, but I can not figure out why the upgrade from 0.59 to 0.67 is resulting in error messages.

I though it was Zlib, but after reinstalling it there was no change.

This is the error message:

./configure

.
.
checking for zlibVersion in -lz... no
configure: error: Link with "-lz" (zlib >= 1.0.2) failed!

It is a Sparc20 station, running Solaris 2.6.

Any Though?

Thanks in Advance,
------------------------------------------------------------------
Alaerte Gladston Vidali
IBM Global Services - SO
Tel.55+11+2121-2879   Fax:55+11+2121-2449 --=_alternative 00589FA683256E00_=-- From splintered-flow-tools-owner@splintered.net Fri Dec 19 03:02:15 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 11782 invoked by uid 4001); 19 Dec 2003 03:02:15 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 11779 invoked by alias); 19 Dec 2003 03:02:15 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 19 Dec 2003 03:02:15 -0000 In-Reply-To: <20031204155458.86623.qmail@web21504.mail.yahoo.com> References: <20031204155458.86623.qmail@web21504.mail.yahoo.com> Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=ISO-8859-1; format=flowed Message-Id: Content-Transfer-Encoding: quoted-printable From: Mark Fullmer Subject: Re: [Flow-tools] Performance / compile question Date: Thu, 18 Dec 2003 22:02:15 -0500 To: johann lafer X-Mailer: Apple Mail (2.606) Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2003 03:02:15 -0000 In src/Makefile.am and lib/Makefile.am change AM_CFLAGS=3D-g -Wall to AM_CFLAGS=3D-O2 Reports will generally run faster without compression. Flow-capture has compression enabled by default. It may be possible to pre-compute some of your reports with flow-report, then aggregate the summaries on the fly... One other idea is to use the inline filtering option with flow-report. You may not be using flow-export correctly. To compress a bunch of flow files: flow-cat -z9

> all.flows.best.compressed or flow-cat -z0 > all.flows.no.compression mark On Dec 4, 2003, at 10:54 AM, johann lafer wrote: > Hm, > =A0 > that wasn't so easy. First I had to install autoconf and automake.=20 > Then I changed Makefile.am in both dirs. Running make, there is still=20= > a "-g -Wall" visible. So I am not really shure, if the optimization=20 > happend. I also appended -mpentium. But i do not notice any=20 > performance optimization. > =A0 > I don't know if it is really a performance problem, but the whole=20 > story is, that i have written a php frontend with flow-tools=20 > (flow-cat,flow-nfilter,flow-stat) in the background. If a web-user=20 > wants to generate a 1,3,12 or 24 hour report, it takes up to 5=20 > minutes. But creating a "week-report" or "month-report"=A0 takes more=20= > than 25 minutes, depending on the generated data. The size of 5 minute=20= > flow-file differs between 1 and 4 Mbytes. Maybe i am impatient?! It=20 > looks like a hanging application if you have to wait to long. > =A0 > Yesterday i tried flow-export -z to compress all files generated the=20= > day before to single compressed file. The volume seems to be compress=20= > 10% and a report was generated 30% faster, but i do not know if this=20= > is an accident. > =A0 > Another idea is to run more than 1 process for a report (up to 31=20 > process for a month report, bottleneck competing CPU and/or HD=20 > access?!) > =A0 > Is there a possibility to make the data volume smaller without losing=20= > important information? > =A0 > Thanks > Janno > > Mark Fullmer wrote: > src/Makefile.am and lib/Makefile.am. You'll need to have automake > installed. Haven't tried any of the processor options to gcc. > > Are you having performance problems? > > mark > > On Dec 3, 2003, at 4:55 PM, johann lafer wrote: > > > Hello Mark, > > =A0 > > in the mailing list i found > > =A0 > > = http://www.pairlist.net/pipermail/flow-tools/2003-January/001007.html > > =A0 > > where you wrote, that compiling flow-tools with -O or -O2 increases > > the performance. > > =A0 > > I tried to use the cflags, but running "make" still shows -g -Wall. > > Which modifications do i have to do where? Have you ever tried to > > compile flow-tools with the processor option ?(i know this is more a > > linux question).=A0 > > =A0 > > Thanks > > =A0 > > Janno > > > > Do you Yahoo!? > > Free Pop-Up Blocker - Get it > > now_______________________________________________ > > Flow-tools mailing list > > flow-tools@splintered.net > > http://mailman.splintered.net/mailman/listinfo/flow-tools > > Do you Yahoo!? > Free Pop-Up Blocker - Get it=20 > now_______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools= From splintered-flow-tools-owner@splintered.net Fri Dec 19 03:03:52 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 12173 invoked by uid 4001); 19 Dec 2003 03:03:52 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 12170 invoked by alias); 19 Dec 2003 03:03:51 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 19 Dec 2003 03:03:51 -0000 In-Reply-To: References: Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=ISO-8859-1; format=flowed Message-Id: Content-Transfer-Encoding: quoted-printable From: Mark Fullmer Subject: Re: [Flow-tools] Upgrading to 0.67 Date: Thu, 18 Dec 2003 22:03:51 -0500 To: "Alaerte Gladston Vidali" X-Mailer: Apple Mail (2.606) Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2003 03:03:52 -0000 zlib is not getting installed where configure can find it. Check where zlib is installing files. mark On Dec 18, 2003, at 11:07 AM, Alaerte Gladston Vidali wrote: > > Sorry if it is a basic question, but I can not figure out why the=20 > upgrade from 0.59 to 0.67 is resulting in error messages. > > I though it was Zlib, but after reinstalling it there was no change. > > This is the error message: > > ./configure > > . > . > checking for zlibVersion in -lz... no > configure: error: Link with "-lz" (zlib >=3D 1.0.2) failed! > > It is a Sparc20 station, running Solaris 2.6. > > Any Though? > > Thanks in Advance, > ------------------------------------------------------------------ > Alaerte Gladston Vidali > IBM Global Services - SO > Tel.55+11+2121-2879 =A0=20 > Fax:55+11+2121-2449_______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools= From maf@splintered.net Fri Dec 19 03:04:46 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 12324 invoked by alias); 19 Dec 2003 03:04:46 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 19 Dec 2003 03:04:46 -0000 In-Reply-To: <3FE18652.7000004@azcher.kharkov.ua> References: <3FE18652.7000004@azcher.kharkov.ua> Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <1A5D1BD1-31D0-11D8-960D-000A95DA1C38@splintered.net> Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [Flow-tools] NetFlow Aggregation Date: Thu, 18 Dec 2003 22:04:47 -0500 To: Sergey Dolgopolov X-Mailer: Apple Mail (2.606) Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2003 03:04:47 -0000 You'll need to run multiple instances of flow-capture. mark On Dec 18, 2003, at 5:49 AM, Sergey Dolgopolov wrote: > Hello! > How i can use in flow-capture both aggregation 8.3 and 8.4? > flow-capture -V 8.3 8.4 -w /flow 192.168.2.1/192.168.2.6/3000? > Thanks. > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From splintered-flow-tools-owner@splintered.net Fri Dec 19 03:06:37 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 12710 invoked by uid 4001); 19 Dec 2003 03:06:37 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 12708 invoked by alias); 19 Dec 2003 03:06:37 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 19 Dec 2003 03:06:37 -0000 In-Reply-To: <000e01c3c4e9$9744d300$b964b1d5@IldarGabdulline> References: <000e01c3c4e9$9744d300$b964b1d5@IldarGabdulline> Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=ISO-8859-1; format=flowed Message-Id: <5C3328B8-31D0-11D8-960D-000A95DA1C38@splintered.net> Content-Transfer-Encoding: quoted-printable From: Mark Fullmer Date: Thu, 18 Dec 2003 22:06:37 -0500 To: "Ildar Gabdulline" X-Mailer: Apple Mail (2.606) Cc: flow list Subject: [Flow-tools] Re: RealEast Networks - flow-gen utility usage X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2003 03:06:37 -0000 An easy way to do this is generate random flow data in ASCII that can be parsed by the ASCII option of flow-import. You can use flow-gen | flow-export -f2 to get an idea of what the data should look like. mark On Dec 17, 2003, at 5:03 PM, Ildar Gabdulline wrote: > Hi Mark, > =A0 > My name is Ildar, I am from RealEast Networks. > We use some parts of flow-tools in our environment. > There is a question on flow-gen: is it possibleto change > content of generated flows (srd ip, dst, ip, protocol) ? > Probably according to some script or configuration file ? > =A0 > Ildar From splintered-flow-tools-owner@splintered.net Fri Dec 19 05:19:24 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 15458 invoked by uid 4001); 19 Dec 2003 05:19:24 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 15455 invoked by alias); 19 Dec 2003 05:19:24 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 19 Dec 2003 05:19:24 -0000 In-Reply-To: <20031214024126.I32249@green.nat.fasttrackmonkey.com> References: <20031214024126.I32249@green.nat.fasttrackmonkey.com> Mime-Version: 1.0 (Apple Message framework v606) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [Flow-tools] sharing examples Date: Fri, 19 Dec 2003 00:19:23 -0500 To: Charles Sprickman X-Mailer: Apple Mail (2.606) Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Dec 2003 05:19:24 -0000 I think a lot of people would find a flow-tools FAQ useful. There are a lot of examples and information in the mailing list archive. The flow-tools-examples man page has some hints too. Many of the examples use flow-stat and flow-filter instead of flow-report and flow-nfilter. I'd like to drop the former from the distribution since they have a lot less functionality than their newer counterparts. The one piece that has been missing is an easy way to use flow-report from the command line. I have a not too kludgy to do this coming up in the next snapshot using variable substitution. The default stat.cfg will look like: stat-report default type @{TYPE:-summary-counters} output format ascii sort @{SORT:-+} fields @{FIELDS:-+} options @{OPTIONS:-+header,+xheader,+totals} stat-definition default report default Then you can do flow-cat | flow-report -v TYPE=ip-protocol -v SORT=+octets | flow-rptfmt Where flow-rptfmt is a small python script which will make the output of flow-report look like like flow-stat. Flow-report also accepts -hh like flow-stat to list the available reports. A better set of defaults for flow-nfilter with variables in the right places should help it too. mark On Dec 14, 2003, at 2:45 AM, Charles Sprickman wrote: > Hello all, > > I'm mighty new to flow-tools; new enough that I see it as a very > valuable > set of tools that I don't know how to use. > > Since flow-report seems to be (relatively) new, I'm wondering if we > could > perhaps start a thread on the list here where the more seasoned > flow-tools > users could share some of their favorite real-world example of DoS > detection using flow-report (or flow-stat, flow-nfilter, whatever works > for you). > > If we could get a nice sampling of examples, I would gladly > sort/categorize and make a nice little web page for inclusion on the > flow-tools homepage, and also an EXAMPLES file for distribution with > the > flow-tools package. > > Thanks, > > Charles > > ___ > Charles Sprickman > NetEng/SysAdmin > Bway.net - New York's Best Internet - www.bway.net > spork@bway.net - 212.655.9344 > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From Patrice@ensyst.com.au Mon Dec 22 00:56:23 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 49400 invoked by alias); 22 Dec 2003 00:56:22 -0000 Received: from smtp1.adl2.internode.on.net (203.16.214.181) by 66.250.216.131 with SMTP; 22 Dec 2003 00:56:22 -0000 Received: from nt2-temp.ensyst.com.au (ppp127-213.lns1.syd3.internode.on.net [150.101.127.213]) by smtp1.adl2.internode.on.net (8.12.9/8.12.9) with ESMTP id hBM0uJcK053272 for ; Mon, 22 Dec 2003 11:26:20 +1030 (CST) X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Content-class: urn:content-classes:message Date: Mon, 22 Dec 2003 11:56:07 +1100 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C3C826.61D2EC12" Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: storing flow-capture information to mySQL Thread-Index: AcPIJmG8WbnWu4qPT6+F/6fDftisqg== From: "Patrice Empeigne" To: Subject: [Flow-tools] storing flow-capture information to mySQL X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2003 00:56:23 -0000 This is a multi-part message in MIME format. ------_=_NextPart_001_01C3C826.61D2EC12 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi! im in storing all flow-capture information to a mySQL database. The means for doing so is to: flow-cat ft-v05* | flow-print -f5 > data this will give me a file called 'data' containing all information that = the 5 format of flow-print provides. i would then use mysqlimport to import the textfile into the db. But my dilemma is, the date format from the flow-print statement is not = in the proper mysql datatime format. Would anyone know how maybe i could: 1) change the date format in the ft files so i can do this import = seamlessly 2) other methods of pushing the information to the db Thanks in advance, Pat ------_=_NextPart_001_01C3C826.61D2EC12 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable storing flow-capture information to mySQL

Hi!
im in storing all flow-capture = information to a mySQL database.
The means for doing so is to:
flow-cat ft-v05* | flow-print -f5 > = data

this will give me a file called 'data' = containing all information that the 5 format of flow-print = provides.
i would then use mysqlimport to import = the textfile into the db.

But my dilemma is, the date format from = the flow-print statement is not in the proper mysql datatime = format.

Would anyone know how maybe i = could:
1) change the date format in the ft = files so i can do this import seamlessly
2) other methods of pushing the = information to the db

Thanks in advance,
Pat

------_=_NextPart_001_01C3C826.61D2EC12-- From Patrice@ensyst.com.au Mon Dec 22 01:01:32 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 50142 invoked by alias); 22 Dec 2003 01:01:31 -0000 Received: from smtp0.adl1.internode.on.net (203.16.214.194) by 66.250.216.131 with SMTP; 22 Dec 2003 01:01:31 -0000 Received: from nt2-temp.ensyst.com.au (ppp127-213.lns1.syd3.internode.on.net [150.101.127.213]) by smtp0.adl1.internode.on.net (8.12.4/8.12.9) with ESMTP id hBM11FHx047941 for ; Mon, 22 Dec 2003 11:31:23 +1030 (CST) X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Content-class: urn:content-classes:message Subject: RE: [Flow-tools] storing flow-capture information to mySQL Date: Mon, 22 Dec 2003 12:01:00 +1100 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C3C827.10E0140A" Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: storing flow-capture information to mySQL Thread-Index: AcPIJmG8WbnWu4qPT6+F/6fDftisqgAAGXmg From: "Patrice Empeigne" To: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2003 01:01:32 -0000 This is a multi-part message in MIME format. ------_=_NextPart_001_01C3C827.10E0140A Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable i've found we can use flow-export but will that give me the ability to = query the database based on time, because i dont see any time fields in = flow-export. =20 Thanks again, Pat -----Original Message----- From: flow-tools-bounces@list.splintered.net = [mailto:flow-tools-bounces@list.splintered.net]On Behalf Of Patrice = Empeigne Sent: Monday, 22 December 2003 11:56 AM To: flow-tools@list.splintered.net Subject: [Flow-tools] storing flow-capture information to mySQL Hi!=20 im in storing all flow-capture information to a mySQL database.=20 The means for doing so is to:=20 flow-cat ft-v05* | flow-print -f5 > data=20 this will give me a file called 'data' containing all information that = the 5 format of flow-print provides.=20 i would then use mysqlimport to import the textfile into the db.=20 But my dilemma is, the date format from the flow-print statement is not = in the proper mysql datatime format.=20 Would anyone know how maybe i could:=20 1) change the date format in the ft files so i can do this import = seamlessly=20 2) other methods of pushing the information to the db=20 Thanks in advance,=20 Pat=20 ------_=_NextPart_001_01C3C827.10E0140A Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable storing flow-capture information to mySQL
i've=20 found we can use flow-export but will that give me the ability to query = the=20 database based on time, because i dont see any time fields in=20 flow-export.
 
Thanks=20 again,
Pat
-----Original Message-----
From:=20 flow-tools-bounces@list.splintered.net=20 [mailto:flow-tools-bounces@list.splintered.net]On Behalf Of = Patrice=20 Empeigne
Sent: Monday, 22 December 2003 11:56 = AM
To:=20 flow-tools@list.splintered.net
Subject: [Flow-tools] storing = flow-capture information to mySQL

Hi!
im in=20 storing all flow-capture information to a mySQL database. =
The means for doing so is to:
flow-cat ft-v05* | flow-print -f5 > data

this will give me a file called 'data' = containing=20 all information that the 5 format of flow-print provides. =
i would then use mysqlimport to import the = textfile into the=20 db.

But my dilemma is, the date format from = the=20 flow-print statement is not in the proper mysql datatime = format.

Would anyone know how maybe i = could:=20
1) change the date format in the ft = files so i can=20 do this import seamlessly
2) = other methods=20 of pushing the information to the db

Thanks in advance,
Pat

------_=_NextPart_001_01C3C827.10E0140A-- From lsharpe@pacificwireless.com.au Mon Dec 22 01:05:32 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 50883 invoked by alias); 22 Dec 2003 01:05:31 -0000 Received: from ns1.pacificwireless.com.au (203.166.40.26) by 66.250.216.131 with SMTP; 22 Dec 2003 01:05:31 -0000 Received: from gandalf ([172.16.1.29]) by ns1.pacificwireless.com.au (8.11.6/8.11.6) with SMTP id hBLMnQR21244; Mon, 22 Dec 2003 09:49:27 +1100 Message-ID: <004201c3c827$af479690$1d0110ac@pacwire.local> From: "Leigh Sharpe" To: "Patrice Empeigne" , References: Subject: Re: [Flow-tools] storing flow-capture information to mySQL Date: Mon, 22 Dec 2003 12:05:23 +1100 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_003F_01C3C883.E1030B90" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2600.0000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2003 01:05:32 -0000 This is a multi-part message in MIME format. ------=_NextPart_000_003F_01C3C883.E1030B90 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable storing flow-capture information to mySQLIf you have compiled flow-tools = with mySQL support, flow-export can do the job for you. It will even = load the SQL all in one hit. ----- Original Message -----=20 From: Patrice Empeigne=20 To: flow-tools@list.splintered.net=20 Sent: Monday, December 22, 2003 11:56 AM Subject: [Flow-tools] storing flow-capture information to mySQL Hi!=20 im in storing all flow-capture information to a mySQL database.=20 The means for doing so is to:=20 flow-cat ft-v05* | flow-print -f5 > data=20 this will give me a file called 'data' containing all information that = the 5 format of flow-print provides.=20 i would then use mysqlimport to import the textfile into the db.=20 But my dilemma is, the date format from the flow-print statement is = not in the proper mysql datatime format.=20 Would anyone know how maybe i could:=20 1) change the date format in the ft files so i can do this import = seamlessly=20 2) other methods of pushing the information to the db=20 Thanks in advance,=20 Pat=20 -------------------------------------------------------------------------= ----- _______________________________________________ Flow-tools mailing list flow-tools@splintered.net http://mailman.splintered.net/mailman/listinfo/flow-tools ------=_NextPart_000_003F_01C3C883.E1030B90 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable storing flow-capture information to mySQL
If you have compiled flow-tools with = mySQL support,=20 flow-export can do the job for you. It will even load the SQL all in one = hit.
 
----- Original Message -----
From:=20 Patrice=20 Empeigne
To: flow-tools@list.splintered= .net=20
Sent: Monday, December 22, 2003 = 11:56=20 AM
Subject: [Flow-tools] storing=20 flow-capture information to mySQL

Hi!
im in=20 storing all flow-capture information to a mySQL database. =
The means for doing so is to:
flow-cat ft-v05* | flow-print -f5 > data

this will give me a file called 'data' = containing=20 all information that the 5 format of flow-print provides. =
i would then use mysqlimport to import the = textfile into the=20 db.

But my dilemma is, the date format from = the=20 flow-print statement is not in the proper mysql datatime = format.

Would anyone know how maybe i = could:=20
1) change the date format in the ft = files so i can=20 do this import seamlessly
2) = other methods=20 of pushing the information to the db

Thanks in advance,
Pat

_______________________________________________
Flow-tools = mailing=20 = list
flow-tools@splintered.net
http://mailman.splintered.net/mailma= n/listinfo/flow-tools ------=_NextPart_000_003F_01C3C883.E1030B90-- From Patrice@ensyst.com.au Mon Dec 22 01:12:18 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 51644 invoked by alias); 22 Dec 2003 01:12:17 -0000 Received: from smtp1.adl2.internode.on.net (203.16.214.181) by 66.250.216.131 with SMTP; 22 Dec 2003 01:12:17 -0000 Received: from nt2-temp.ensyst.com.au (ppp127-213.lns1.syd3.internode.on.net [150.101.127.213]) by smtp1.adl2.internode.on.net (8.12.9/8.12.9) with ESMTP id hBM1CFcK084140 for ; Mon, 22 Dec 2003 11:42:15 +1030 (CST) X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Content-class: urn:content-classes:message Subject: RE: [Flow-tools] storing flow-capture information to mySQL Date: Mon, 22 Dec 2003 12:12:02 +1100 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C3C828.9B39A840" Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: storing flow-capture information to mySQL Thread-Index: AcPIJmG8WbnWu4qPT6+F/6fDftisqgAAGXmgAABif/A= From: "Patrice Empeigne" To: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2003 01:12:18 -0000 This is a multi-part message in MIME format. ------_=_NextPart_001_01C3C828.9B39A840 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable this is all sorted! unix seconds field has done this for me :D =20 Have a great Christmas and a Happy New Year all! =20 Thanx! Pat =20 p.s> i wonder what the unix_nsecs field is for =20 -----Original Message----- From: flow-tools-bounces@list.splintered.net = [mailto:flow-tools-bounces@list.splintered.net]On Behalf Of Patrice = Empeigne Sent: Monday, 22 December 2003 12:01 PM To: flow-tools@list.splintered.net Subject: RE: [Flow-tools] storing flow-capture information to mySQL i've found we can use flow-export but will that give me the ability to = query the database based on time, because i dont see any time fields in = flow-export. =20 Thanks again, Pat -----Original Message----- From: flow-tools-bounces@list.splintered.net = [mailto:flow-tools-bounces@list.splintered.net]On Behalf Of Patrice = Empeigne Sent: Monday, 22 December 2003 11:56 AM To: flow-tools@list.splintered.net Subject: [Flow-tools] storing flow-capture information to mySQL Hi!=20 im in storing all flow-capture information to a mySQL database.=20 The means for doing so is to:=20 flow-cat ft-v05* | flow-print -f5 > data=20 this will give me a file called 'data' containing all information that = the 5 format of flow-print provides.=20 i would then use mysqlimport to import the textfile into the db.=20 But my dilemma is, the date format from the flow-print statement is not = in the proper mysql datatime format.=20 Would anyone know how maybe i could:=20 1) change the date format in the ft files so i can do this import = seamlessly=20 2) other methods of pushing the information to the db=20 Thanks in advance,=20 Pat=20 ------_=_NextPart_001_01C3C828.9B39A840 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable storing flow-capture information to mySQL
this=20 is all sorted!
unix=20 seconds field has done this for me :D
 
Have a=20 great Christmas and a Happy New Year all!
 
Thanx!
Pat
 
p.s> i wonder what the unix_nsecs field is = for
 
-----Original Message-----
From:=20 flow-tools-bounces@list.splintered.net=20 [mailto:flow-tools-bounces@list.splintered.net]On Behalf Of = Patrice=20 Empeigne
Sent: Monday, 22 December 2003 12:01 = PM
To:=20 flow-tools@list.splintered.net
Subject: RE: [Flow-tools] = storing=20 flow-capture information to mySQL

i've=20 found we can use flow-export but will that give me the ability to = query the=20 database based on time, because i dont see any time fields in=20 flow-export.
 
Thanks again,
Pat
-----Original Message-----
From:=20 flow-tools-bounces@list.splintered.net=20 [mailto:flow-tools-bounces@list.splintered.net]On Behalf Of = Patrice=20 Empeigne
Sent: Monday, 22 December 2003 11:56 = AM
To:=20 flow-tools@list.splintered.net
Subject: [Flow-tools] = storing=20 flow-capture information to mySQL

Hi!
im in=20 storing all flow-capture information to a mySQL database. =
The means for doing so is to:
flow-cat ft-v05* | flow-print -f5 > data

this will give me a file called = 'data' containing=20 all information that the 5 format of flow-print provides. =
i would then use mysqlimport to import the = textfile into=20 the db.

But my dilemma is, the date format = from the=20 flow-print statement is not in the proper mysql datatime = format.

Would anyone know how maybe i = could:=20
1) change the date format in the ft = files so i=20 can do this import seamlessly
2) other=20 methods of pushing the information to the db

Thanks in advance,
Pat

------_=_NextPart_001_01C3C828.9B39A840-- From splintered-flow-tools-owner@splintered.net Mon Dec 22 14:50:20 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 56279 invoked by uid 4001); 22 Dec 2003 14:50:20 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 56277 invoked by alias); 22 Dec 2003 14:50:19 -0000 Received: from sebra.eyp.ee (HELO inlook.eyp.ee) (62.65.41.134) by 66.250.216.131 with SMTP; 22 Dec 2003 14:50:19 -0000 Received: from internal by inlook.eyp.ee; for ; Mon, 22 Dec 2003 16:50:18 +0200 Message-ID: <3FE704A9.708@eyp.ee> Date: Mon, 22 Dec 2003 16:50:17 +0200 From: Risto Vaarandi User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20020830 X-Accept-Language: en-us, en MIME-Version: 1.0 To: flow-tools@splintered.net Subject: Re: [flow-tools] Software NetFlow probes References: <3FC205D1.7030502@eyp.ee> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2003 14:50:20 -0000 hi all, in order to answer my own question, the fprobe package (http://sourceforge.net/projects/fprobe/) seems to be a good software probe solution. It is able to export flow records for long-lasting connections, and it is also very small in size (around 2000 lines of C) which reduces the risk of hard-to-spot bugs. I've found that it works well with flow-tools, and by combining fprobe with flow-tools you will get a cost effective and efficient solution for gathering network statistics. br, risto Risto Vaarandi wrote: > hi all, > > first of all please accept my apologies since this question is slightly > off-topic. I am looking for a free software NetFlow probe that could be > used together with flow-tools. The most popular choice seems to be ntop > (http://www.ntop.org) which also has a support for NetFlow. I have made > some tests with ntop+flow-tools and my setup works quite fine. However, > ntop has some downsides - it generates a NetFlow record only when a TCP > connection is closed, therefore you can't see connections that are still > active among the statistics. The other thing is that aside from NetFlow > support ntop implements a large set of other features which can't be > switched off and which all consume some CPU time. > > Therefore, I started to look for another software implementation of > NetFlow probe and found some relevant projects: > fprobe (http://sourceforge.net/projects/fprobe), softflowd > (http://www.mindrot.org/softflowd.html), and Flow Probe > (http://psi.home.ro/flow/). > > However, the homepages of these tools contain no information how many > sites are actually using them and if they have been successfully used > with flow-tools. Does anyone in this list have any experience with the > tools mentioned above (either positive or negative)? > > Also, if you know some other good software implementation of NetFlow > probe, please let me know. > > tia, > risto > > > _______________________________________________ > flow-tools@splintered.net > http://www.splintered.net/sw/flow-tools > From splintered-flow-tools-owner@splintered.net Mon Dec 22 15:30:39 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 57244 invoked by uid 4001); 22 Dec 2003 15:30:39 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 57241 invoked by alias); 22 Dec 2003 15:30:38 -0000 Received: from igw3.br.ibm.com (HELO mailgw3.br.ibm.com) (32.104.18.26) by 66.250.216.131 with SMTP; 22 Dec 2003 15:30:38 -0000 Received: from mailhub3.br.ibm.com (mailhub3.br.ibm.com [9.179.63.32]) by mailgw3.br.ibm.com (8.12.9/8.12.3) with ESMTP id hBMFVrw6233652; Mon, 22 Dec 2003 13:31:54 -0200 Received: from d24bml05.br.ibm.com (d24av01.tsc.br.ibm.com [9.179.5.241]) by mailhub3.br.ibm.com (8.12.10/NCO/VER6.6) with ESMTP id hBMFUaNj059420; Mon, 22 Dec 2003 13:30:36 -0200 Importance: Normal MIME-Version: 1.0 Sensitivity: To: flow-tools@splintered.net Subject: Re: [Flow-tools] Upgrading to 0.67 X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 Message-ID: From: "Alaerte Gladston Vidali" Date: Mon, 22 Dec 2003 13:30:20 -0200 X-MIMETrack: Serialize by Router on d24bml05/24/M/IBM(Release 5.0.9a |January 7, 2002) at 22/12/2003 01:30:21 PM, Serialize complete at 22/12/2003 01:30:21 PM Content-Type: multipart/alternative; boundary="=_alternative 005533CB83256E04_=" Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2003 15:30:39 -0000 This is a multipart message in MIME format. --=_alternative 005533CB83256E04_= Content-Type: text/plain; charset="us-ascii" Thanks, Zlib is installing files on /usr/local/include, /usr/local/lib and /usr/local/share/man/man3. Where does Flow-Tools require Zlib to be installed? # make install cp zlib.h zconf.h /usr/local/include chmod 644 /usr/local/include/zlib.h /usr/local/include/zconf.h cp libz.so.1.2.1 /usr/local/lib cd /usr/local/lib; chmod 755 libz.so.1.2.1 cd /usr/local/lib; if test -f libz.so.1.2.1; then \ rm -f libz.so libz.so.1; \ ln -s libz.so.1.2.1 libz.so; \ ln -s libz.so.1.2.1 libz.so.1; \ (ldconfig || true) >/dev/null 2>&1; \ fi cp zlib.3 /usr/local/share/man/man3 chmod 644 /usr/local/share/man/man3/zlib.3 Cordially, ------------------------------------------------------------------ Alaerte Gladston Vidali IBM Global Services - SO Tel.55+11+2121-2879 Fax:55+11+2121-2449 To: Alaerte Gladston Vidali/Brazil/IBM@IBMBR cc: flow-tools@splintered.net Subject: Re: [Flow-tools] Upgrading to 0.67 zlib is not getting installed where configure can find it. Check where zlib is installing files. mark On Dec 18, 2003, at 11:07 AM, Alaerte Gladston Vidali wrote: > > Sorry if it is a basic question, but I can not figure out why the > upgrade from 0.59 to 0.67 is resulting in error messages. > > I though it was Zlib, but after reinstalling it there was no change. > > This is the error message: > > ./configure > > . > . > checking for zlibVersion in -lz... no > configure: error: Link with "-lz" (zlib >= 1.0.2) failed! > > It is a Sparc20 station, running Solaris 2.6. > > Any Though? > > Thanks in Advance, > ------------------------------------------------------------------ > Alaerte Gladston Vidali > IBM Global Services - SO > Tel.55+11+2121-2879 > Fax:55+11+2121-2449_______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools --=_alternative 005533CB83256E04_= Content-Type: text/html; charset="us-ascii"
Thanks,

Zlib is installing files on /usr/local/include, /usr/local/lib and /usr/local/share/man/man3.

Where does Flow-Tools require Zlib to be installed?


# make install
cp zlib.h zconf.h /usr/local/include
chmod 644 /usr/local/include/zlib.h /usr/local/include/zconf.h
cp libz.so.1.2.1 /usr/local/lib
cd /usr/local/lib; chmod 755 libz.so.1.2.1
cd /usr/local/lib; if test -f libz.so.1.2.1; then \
  rm -f libz.so libz.so.1; \
  ln -s libz.so.1.2.1 libz.so; \
  ln -s libz.so.1.2.1 libz.so.1; \
  (ldconfig || true)  >/dev/null 2>&1; \
fi
cp zlib.3 /usr/local/share/man/man3
chmod 644 /usr/local/share/man/man3/zlib.3


Cordially,
------------------------------------------------------------------
Alaerte Gladston Vidali
IBM Global Services - SO
Tel.55+11+2121-2879   Fax:55+11+2121-2449



To:        Alaerte Gladston Vidali/Brazil/IBM@IBMBR
cc:        flow-tools@splintered.net
Subject:        Re: [Flow-tools] Upgrading to 0.67



zlib is not getting installed where configure can find it.
Check where zlib is installing files.
mark

On Dec 18, 2003, at 11:07 AM, Alaerte Gladston Vidali wrote:
>
> Sorry if it is a basic question, but I can not figure out why the
> upgrade from 0.59 to 0.67 is resulting in error messages.
>
> I though it was Zlib, but after reinstalling it there was no change.
>
> This is the error message:
>
> ./configure
>
> .
> .
> checking for zlibVersion in -lz... no
> configure: error: Link with "-lz" (zlib >= 1.0.2) failed!
>
> It is a Sparc20 station, running Solaris 2.6.
>
> Any Though?
>
> Thanks in Advance,
> ------------------------------------------------------------------
> Alaerte Gladston Vidali
> IBM Global Services - SO
> Tel.55+11+2121-2879
> Fax:55+11+2121-2449_______________________________________________
> Flow-tools mailing list
> flow-tools@splintered.net
> http://mailman.splintered.net/mailman/listinfo/flow-tools

--=_alternative 005533CB83256E04_=-- From andywhite@ntlworld.ie Mon Dec 22 21:04:59 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 59320 invoked by alias); 22 Dec 2003 21:04:59 -0000 Received: from mta05-svc.ntlworld.com (62.253.162.45) by 66.250.216.131 with SMTP; 22 Dec 2003 21:04:59 -0000 Received: from deskgx ([81.98.90.226]) by mta05-svc.ntlworld.com (InterMail vM.4.01.03.37 201-229-121-137-20020806) with ESMTP id <20031222210448.YKUJ19387.mta05-svc.ntlworld.com@deskgx> for ; Mon, 22 Dec 2003 21:04:48 +0000 From: "Andrew White" To: Date: Mon, 22 Dec 2003 21:04:54 -0000 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook, Build 11.0.5510 Thread-Index: AcPIzz8f/c9oReDQTP2FnyZ7HVe0tw== X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165 Message-Id: <20031222210448.YKUJ19387.mta05-svc.ntlworld.com@deskgx> Subject: [Flow-tools] Newbie: Flow-capture file format X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2003 21:04:59 -0000 Is there any documentation on the format of the files saved by flow-capture ? I can see the flow data in there, but also header information and other data. Any faq's or links would be appreciated Tks Andrew From billf@elvis.mu.org Mon Dec 22 21:45:43 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 60214 invoked by alias); 22 Dec 2003 21:45:43 -0000 Received: from elvis.mu.org (192.203.228.196) by 66.250.216.131 with SMTP; 22 Dec 2003 21:45:43 -0000 Received: by elvis.mu.org (Postfix, from userid 1098) id 009895C78A; Mon, 22 Dec 2003 13:45:42 -0800 (PST) Date: Mon, 22 Dec 2003 13:45:42 -0800 From: Bill Fumerola To: Andrew White Subject: Re: [Flow-tools] Newbie: Flow-capture file format Message-ID: <20031222214542.GJ82802@elvis.mu.org> References: <20031222210448.YKUJ19387.mta05-svc.ntlworld.com@deskgx> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20031222210448.YKUJ19387.mta05-svc.ntlworld.com@deskgx> User-Agent: Mutt/1.4.1i X-Operating-System: FreeBSD 4.9-MUORG-20031210 i386 Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2003 21:45:43 -0000 On Mon, Dec 22, 2003 at 09:04:54PM -0000, Andrew White wrote: > Is there any documentation on the format of the files saved by flow-capture > ? I can see the flow data in there, but also header information and other > data. Any faq's or links would be appreciated there is an entire api you can use "documented" in ftlib.h -- - bill fumerola / fumerola@yahoo-inc.com / billf@FreeBSD.org From kirill@comstar.ru Mon Dec 29 14:28:47 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 90050 invoked by alias); 29 Dec 2003 14:28:46 -0000 Received: from hut.comstar.ru (195.210.128.8) by 66.250.216.131 with SMTP; 29 Dec 2003 14:28:46 -0000 Received: from comstar.ru (som1.bcd.comstar.ru [195.210.131.101]) by hut.comstar.ru (8.12.6/8.12.9) with ESMTP id hBTEShmb045432 for ; Mon, 29 Dec 2003 17:28:44 +0300 (MSK) X-Deliver-To: Message-ID: <3FF02C73.9060300@comstar.ru> Date: Mon, 29 Dec 2003 17:30:27 +0400 From: Kirill Kuvshinnikov User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20020830 X-Accept-Language: ru, en-us, en MIME-Version: 1.0 To: flow-tools@list.splintered.net Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Subject: [Flow-tools] Flow-tags and v8 X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Dec 2003 14:28:47 -0000 I wonder, is it possible to use flow-tags with v8 (8.5) of netflow. After configuring tag definitions, I try to use flow-tag and receive message: "Flow record missing required field for tagging" (from flow-tag.c) Regards, Kirillium. From splintered-flow-tools-owner@splintered.net Wed Dec 31 02:49:39 2003 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 97443 invoked by uid 4001); 31 Dec 2003 02:49:39 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 97441 invoked by alias); 31 Dec 2003 02:49:38 -0000 Received: from 202-127-99-4.triplegate.net.id (HELO oracle) (202.127.99.4) by 66.250.216.131 with SMTP; 31 Dec 2003 02:49:38 -0000 Received: (qmail 28728 invoked by uid 105); 31 Dec 2003 02:55:15 -0000 Received: from anang@csmcom.com by oracle by uid 118 with qmail-scanner-1.15 (avpdaemon: ???. spamassassin: 2.43. Clear:. Processed in 0.115207 secs); 31 Dec 2003 02:55:15 -0000 Received: from csmwks-52-9.csmcom.com (HELO csmcom.com) (172.18.52.9) by oracle with SMTP; 31 Dec 2003 02:55:14 -0000 Message-ID: <3FF238D4.5060706@csmcom.com> Date: Wed, 31 Dec 2003 09:47:48 +0700 From: Anang Syarifudin Organization: PT. CItra Sari Makmur User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031202 X-Accept-Language: en-us, en MIME-Version: 1.0 To: flow-tools@splintered.net Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: [Flow-tools] MySQL support failed on RH7.3 X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Dec 2003 02:49:39 -0000 Hi, I've tried to compile flow-tools with with mysql support but failed. I've tried : ./configure --with-mysql=/usr I got error : flow-export.c:52:19: mysql.h: No such file or directory make[1]: *** [flow-export.o] Error 1 I also tried : ./configure --with-mysql=/usr/include/myql I got no error, but no myql also. I checked config.log : configure:1467: gcc -o conftest -g -Wall conftest.c -lmysqlclient -L/usr/include/mysql/lib/mysql 1>&5 /usr/bin/ld: cannot find -lmysqlclient I think it should be -L/usr/lib/mysql , maybe something wrong with configure, please help My env : flow-tools-0.66 SRC.RPM mysql-server-3.23.58-1.73 mysql-3.23.58-1.73 mysql-devel-3.23.58-1.73 gcc-2.96-113 make-3.79.1-8 autoconf-2.13-17 TIA Anang Syarifudin From splintered-flow-tools-owner@splintered.net Thu Jan 01 02:58:50 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 2199 invoked by uid 4001); 1 Jan 2004 02:58:50 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 2197 invoked by alias); 1 Jan 2004 02:58:50 -0000 Received: from mail.ateneo.edu (202.138.180.10) by 66.250.216.131 with SMTP; 1 Jan 2004 02:58:50 -0000 Received: from mail.ateneo.edu (localhost [127.0.0.1]) by localhost (Postfix) with ESMTP id 255453F46; Thu, 1 Jan 2004 10:58:48 +0800 (PHT) Received: from sysads.ateneo.net (jabber.admu.edu.ph [10.2.10.70]) by mail.ateneo.edu (Postfix) with ESMTP id 0EE933F42; Thu, 1 Jan 2004 10:58:48 +0800 (PHT) Received: (from wyy@localhost) by sysads.ateneo.net (8.11.6/8.11.6) id i013JHi16440; Thu, 1 Jan 2004 11:19:17 +0800 Date: Thu, 1 Jan 2004 11:19:17 +0800 From: "Horatio B. Bogbindero" To: Anang Syarifudin Subject: re: [Flow-tools] MySQL support failed on RH7.3 Message-ID: <20040101111917.A16426@admu.edu.ph> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-md5; protocol="application/pgp-signature"; boundary="fUYQa+Pmc3FrFX/N" Content-Disposition: inline User-Agent: Mutt/1.2.5.1i Organization: Ateneo Cervini-Eliazo Networks X-ACENT-Conspiracy: Where is the conspiracy? Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list Reply-To: wyu@ateneo.edu List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Jan 2004 02:58:50 -0000 --fUYQa+Pmc3FrFX/N Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable do not forget to install the mysqlclient rpm. this is required since it contains the MySQL Client libraries. --=20 =20 ------------------------------------------- William Emmanuel S. Yu Ateneo Campus Network Group (AteneoCNG) email : wyy at admu dot edu dot ph web : http://CNG.ateneo.net/wyu/ phone : +63(2)4266001-4186 GPG : http://CNG.ateneo.net/wyu/wyy.pgp =20 War spares not the brave, but the cowardly. -- Anacreon Old message: Hi, I've tried to compile flow-tools with with mysql support but failed. I've tried : =2E/configure --with-mysql=3D/usr I got error : flow-export.c:52:19: mysql.h: No such file or directory make[1]: *** [flow-export.o] Error 1 I also tried : =2E/configure --with-mysql=3D/usr/include/myql I got no error, but no myql also. I checked config.log : configure:1467: gcc -o conftest -g -Wall conftest.c -lmysqlclient =20 -L/usr/include/mysql/lib/mysql 1>&5 /usr/bin/ld: cannot find -lmysqlclient I think it should be -L/usr/lib/mysql , maybe something wrong with=20 configure, please help My env : flow-tools-0.66 SRC.RPM mysql-server-3.23.58-1.73 mysql-3.23.58-1.73 mysql-devel-3.23.58-1.73 gcc-2.96-113 make-3.79.1-8 autoconf-2.13-17 TIA Anang Syarifudin =20 --fUYQa+Pmc3FrFX/N Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE/85GuOgIOlr0CsAERAmZvAJ407gAbrjJdltWX+jKy99wlHYsf4ACeLO1Q B0r9AiPBs5wlyJdJFlVUtyw= =EG/E -----END PGP SIGNATURE----- --fUYQa+Pmc3FrFX/N-- From splintered-flow-tools-owner@splintered.net Fri Jan 02 01:55:52 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 8044 invoked by uid 4001); 2 Jan 2004 01:55:52 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 8042 invoked by alias); 2 Jan 2004 01:55:51 -0000 Received: from 202-127-99-4.triplegate.net.id (HELO oracle) (202.127.99.4) by 66.250.216.131 with SMTP; 2 Jan 2004 01:55:51 -0000 Received: (qmail 26278 invoked by uid 105); 2 Jan 2004 01:55:59 -0000 Received: from anang@csmcom.com by oracle by uid 118 with qmail-scanner-1.15 (avpdaemon: ???. spamassassin: 2.43. Clear:. Processed in 0.103733 secs); 02 Jan 2004 01:55:59 -0000 Received: from csmwks-52-9.csmcom.com (HELO csmcom.com) (172.18.52.9) by oracle with SMTP; 2 Jan 2004 01:55:58 -0000 Message-ID: <3FF4CF93.8090109@csmcom.com> Date: Fri, 02 Jan 2004 08:55:31 +0700 From: Anang Syarifudin Organization: PT. CItra Sari Makmur User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031202 X-Accept-Language: en-us, en MIME-Version: 1.0 To: wyu@ateneo.edu Subject: Re: [Flow-tools] MySQL support failed on RH7.3 References: <20040101111917.A16426@admu.edu.ph> In-Reply-To: <20040101111917.A16426@admu.edu.ph> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jan 2004 01:55:52 -0000 I don't think so. I've installed all msql related rpm. Actually I solved the problem by manually editing src/Makefile, I added : DEFS = -I$(srcdir)/../lib -I/usr/include/mysql -L$(srcdir)/../lib but I'm looking for better solution. thanks anyway. Anang Horatio B. Bogbindero wrote: >do not forget to install the mysqlclient rpm. this is required >since it contains the MySQL Client libraries. > > > From splintered-flow-tools-owner@splintered.net Mon Jan 05 15:24:22 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 21936 invoked by uid 4001); 5 Jan 2004 15:24:22 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 21934 invoked by alias); 5 Jan 2004 15:24:22 -0000 Received: from realprod.acns.fsu.edu (146.201.3.22) by 66.250.216.131 with SMTP; 5 Jan 2004 15:24:22 -0000 Received: from realprod.acns.fsu.edu (localhost.localdomain [127.0.0.1]) by realprod.acns.fsu.edu (8.12.8/8.12.8) with ESMTP id i05FOCRs013091; Mon, 5 Jan 2004 10:24:12 -0500 Received: (from emanners@localhost) by realprod.acns.fsu.edu (8.12.8/8.12.8/Submit) id i05FO5kj013089; Mon, 5 Jan 2004 10:24:05 -0500 Subject: Re: [Flow-tools] MySQL support failed on RH7.3 From: Edson Manners To: Anang Syarifudin In-Reply-To: <3FF238D4.5060706@csmcom.com> References: <3FF238D4.5060706@csmcom.com> Content-Type: text/plain Content-Transfer-Encoding: 7bit Organization: Florida State University (ACNS) Message-Id: <1073316244.22295.20.camel@realprod.acns.fsu.edu> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.5 Date: Mon, 05 Jan 2004 10:24:05 -0500 Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jan 2004 15:24:23 -0000 Looks like you may need the devel rpm packages installed. On Tue, 2003-12-30 at 21:47, Anang Syarifudin wrote: > Hi, > > I've tried to compile flow-tools with with mysql support but failed. > I've tried : > ./configure --with-mysql=/usr > I got error : > flow-export.c:52:19: mysql.h: No such file or directory > make[1]: *** [flow-export.o] Error 1 > > I also tried : > ./configure --with-mysql=/usr/include/myql > I got no error, but no myql also. I checked config.log : > configure:1467: gcc -o conftest -g -Wall conftest.c -lmysqlclient > -L/usr/include/mysql/lib/mysql 1>&5 > /usr/bin/ld: cannot find -lmysqlclient > > I think it should be -L/usr/lib/mysql , maybe something wrong with > configure, please help > > My env : > flow-tools-0.66 SRC.RPM > mysql-server-3.23.58-1.73 > mysql-3.23.58-1.73 > mysql-devel-3.23.58-1.73 > gcc-2.96-113 > make-3.79.1-8 > autoconf-2.13-17 > > TIA > > Anang Syarifudin > > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools -- Edson Manners Academic Computing & Networking Services Florida State University (850)644-2591-ext 125 From Patrice@ensyst.com.au Tue Jan 06 23:04:28 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 27978 invoked by alias); 6 Jan 2004 23:04:27 -0000 Received: from ppp127-213.lns1.syd3.internode.on.net (HELO NT2.corporate.ensyst.com.au) (150.101.127.213) by 66.250.216.131 with SMTP; 6 Jan 2004 23:04:27 -0000 Content-class: urn:content-classes:message Date: Wed, 7 Jan 2004 10:04:22 +1100 Message-ID: <52E510BAD83DAD4BB429B4C53C269FD2064BC8@NT2.corporate.ensyst.com.au> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C3D4A9.6C1E260B" X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Billing System Thread-Index: AcPUqWwbbLGP5AspQNCPBPupHPBcpg== X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 From: "Patrice Empeigne" To: Subject: [Flow-tools] Billing System X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jan 2004 23:04:28 -0000 This is a multi-part message in MIME format. ------_=_NextPart_001_01C3D4A9.6C1E260B Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi All, Im currently writing a billing system using flow-tools. Im piping all information from flow-capture straight into a mySQL db. Ive written the frontend and all works fine. What advice im looking for is in regards to performance, currently im = storing everything in one table. Im currently changing that and creating a table per day to shorten the = response times of queries. Before i do so, i was just wondering if anyone else has done something = similar and could suggest a data model or even other hints for performance. Thanks team Pat ------_=_NextPart_001_01C3D4A9.6C1E260B Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Billing System

Hi All,
Im currently writing a billing system = using flow-tools.
Im piping all information from = flow-capture straight into a mySQL db.
Ive written the frontend and all works = fine.
What advice im looking for is in = regards to performance, currently im storing everything in one = table.
Im currently changing that and = creating a table per day to shorten the response times of = queries.

Before i do so, i was just wondering if = anyone else has done something similar and could suggest a data = model
or even other hints for = performance.

Thanks team

Pat

------_=_NextPart_001_01C3D4A9.6C1E260B-- From maf@eng.oar.net Wed Jan 07 06:40:56 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 30320 invoked by alias); 7 Jan 2004 06:40:56 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 7 Jan 2004 06:40:56 -0000 In-Reply-To: <52E510BAD83DAD4BB429B4C53C269FD2064BC8@NT2.corporate.ensyst.com.au> References: <52E510BAD83DAD4BB429B4C53C269FD2064BC8@NT2.corporate.ensyst.com.au> Mime-Version: 1.0 (Apple Message framework v609) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <71BD2FC8-40DC-11D8-BCCB-000A95DA1C38@eng.oar.net> Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [Flow-tools] Billing System Date: Wed, 7 Jan 2004 01:40:55 -0500 To: "Patrice Empeigne" X-Mailer: Apple Mail (2.609) Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 06:40:56 -0000 Have you considered using flow-report to summarize the data before storing it to mySQL? As an example lets say you're billing for every IP address in a /16. With summarized flows this is 65535 (inbound) or 65535*2 (inbound+outbound) records per day vs potentially millions of records per day by storing the flows. mark On Jan 6, 2004, at 6:04 PM, Patrice Empeigne wrote: > Hi All, > Im currently writing a billing system using flow-tools. > Im piping all information from flow-capture straight into a mySQL db. > Ive written the frontend and all works fine. > What advice im looking for is in regards to performance, currently im > storing everything in one table. > Im currently changing that and creating a table per day to shorten the > response times of queries. > > Before i do so, i was just wondering if anyone else has done something > similar and could suggest a data model > or even other hints for performance. > > Thanks team > > Pat > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From Patrice@ensyst.com.au Wed Jan 07 06:54:27 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 31091 invoked by alias); 7 Jan 2004 06:54:26 -0000 Received: from ppp127-213.lns1.syd3.internode.on.net (HELO NT2.corporate.ensyst.com.au) (150.101.127.213) by 66.250.216.131 with SMTP; 7 Jan 2004 06:54:26 -0000 Content-class: urn:content-classes:message Subject: RE: [Flow-tools] Billing System Date: Wed, 7 Jan 2004 17:54:24 +1100 Message-ID: <52E510BAD83DAD4BB429B4C53C269FD2064BCD@NT2.corporate.ensyst.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Flow-tools] Billing System X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Thread-Index: AcPU6Tc7vDn3Z+ZAQ4CmAPPepDqVZwAAGP4w From: "Patrice Empeigne" To: "Mark Fullmer" Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 06:54:27 -0000 Hi Mark, I have considered flow-report but there is a need for granularity, thus = storage of all data. This need for granularity is from the incorporation of an investigation = tool that allows, for example, to check if Chuck is streaming any porn. = ;) Therefore the approach, for performance, I have taken is pretty much = through summarisation in the data model as suggested by Geoffrey = Bradford above and intelligent queries. Thanks Mark, Pat -----Original Message----- From: Mark Fullmer [mailto:maf@eng.oar.net] Sent: Wednesday, 7 January 2004 5:41 PM To: Patrice Empeigne Cc: flow-tools@list.splintered.net Subject: Re: [Flow-tools] Billing System Have you considered using flow-report to summarize the data before=20 storing it to mySQL? As an example lets say you're billing for every IP address in a /16. With summarized flows this is 65535 (inbound) or 65535*2 (inbound+outbound) records per day vs potentially millions of records=20 per day by storing the flows. mark On Jan 6, 2004, at 6:04 PM, Patrice Empeigne wrote: > Hi All, > Im currently writing a billing system using flow-tools. > Im piping all information from flow-capture straight into a mySQL db. > Ive written the frontend and all works fine. > What advice im looking for is in regards to performance, currently im=20 > storing everything in one table. > Im currently changing that and creating a table per day to shorten the = > response times of queries. > > Before i do so, i was just wondering if anyone else has done something = > similar and could suggest a data model > or even other hints for performance. > > Thanks team > > Pat > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From splintered-flow-tools-owner@splintered.net Wed Jan 07 07:54:43 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 32003 invoked by uid 4001); 7 Jan 2004 07:54:43 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 32001 invoked by alias); 7 Jan 2004 07:54:42 -0000 Received: from mail.ateneo.edu (202.138.180.10) by 66.250.216.131 with SMTP; 7 Jan 2004 07:54:42 -0000 Received: from mail.ateneo.edu (localhost [127.0.0.1]) by localhost (Postfix) with ESMTP id A80C73F35 for ; Wed, 7 Jan 2004 15:54:38 +0800 (PHT) Received: by mail.ateneo.edu (Postfix, from userid 48) id 864D83F2B; Wed, 7 Jan 2004 15:54:38 +0800 (PHT) Received: from 203.82.45.150 ([203.82.45.150]) by mail.ateneo.edu (IMP) with HTTP for ; Wed, 7 Jan 2004 15:54:38 +0800 Message-ID: <1073462078.3ffbbb3e6d55c@mail.ateneo.edu> Date: Wed, 7 Jan 2004 15:54:38 +0800 From: "Horatio B. Bogbindero" To: flow-tools@splintered.net MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.2 Cc: Subject: [Flow-tools] Flow-tools 0.67 RPM available X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 07:54:43 -0000 i would just like to announce that the flow-tools 0.67 is already available. it can be downloaded at the usual place http://cng.ateneo.net/cng/wyu/software/flow-tools.php. ----------------------------------------------- William Emmanuel S. Yu Ateneo Campus Network Group (AteneoCNG) email : wyu at ateneo dot edu web : http://CNG.ateneo.net/cng/wyu/ phone : +63(2)4266001-4186 GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp From JohnWong@crimsonlogic.com Wed Jan 07 08:19:51 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 33151 invoked by alias); 7 Jan 2004 08:19:50 -0000 Received: from mail1.crimsonlogic.com (HELO cronus.crimsonlogic.com) (210.56.134.80) by 66.250.216.131 with SMTP; 7 Jan 2004 08:19:50 -0000 Received: from SG01PVEXCH00.snshub.org ([172.29.8.14]) by cronus.crimsonlogic.com (8.12.10/8.12.10) with ESMTP id i078Jm11001333 for ; Wed, 7 Jan 2004 16:19:48 +0800 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Wed, 7 Jan 2004 16:19:48 +0800 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: flow-capture & flow-fanout with filer option Thread-Index: AcPU9wPXV7XzeVEESdqG2zL5Q/0TVw== From: "John Wong" To: Subject: [Flow-tools] flow-capture & flow-fanout with filer option X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 08:19:51 -0000 Hi, I'm trying to configure flow-capture to only capture to file netflows with specific SNMP ifindex. From the router, i'm exporting the flows as V5. I run flow-capture as follow :- /opt/flow-tools/bin/flow-capture -p- \ -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ x.x.x.x/y.y.y.y/2055 Content of /opt/flow-tools/etc/filter :- ----------- BEGIN ------------------ filter-primitive if1 type ifindex permit 3 filter-definition default match input-interface if1 or match output-interface if1 ----------- END ------------------ Somehow, i am still getting flows from other interfaces on that router. Any idea if what i want can be done and what is the purpose of the "-f" option for flow-capture and flow-fanout if it doesn't? Thanks alot. From wyu@ateneo.edu Wed Jan 07 08:45:59 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 33983 invoked by alias); 7 Jan 2004 08:45:59 -0000 Received: from mail.ateneo.edu (202.138.180.10) by 66.250.216.131 with SMTP; 7 Jan 2004 08:45:59 -0000 Received: from mail.ateneo.edu (localhost [127.0.0.1]) by localhost (Postfix) with ESMTP id 4668D3F07; Wed, 7 Jan 2004 16:45:54 +0800 (PHT) Received: by mail.ateneo.edu (Postfix, from userid 48) id 30D573F00; Wed, 7 Jan 2004 16:45:54 +0800 (PHT) Received: from 203.82.45.150 ([203.82.45.150]) by mail.ateneo.edu (IMP) with HTTP for ; Wed, 7 Jan 2004 16:45:54 +0800 Message-ID: <1073465154.3ffbc74215737@mail.ateneo.edu> Date: Wed, 7 Jan 2004 16:45:54 +0800 From: "Horatio B. Bogbindero" To: John Wong Subject: Re: [Flow-tools] flow-capture & flow-fanout with filer option References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.2 Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 08:45:59 -0000 Quoting John Wong : > Hi, > > I'm trying to configure flow-capture to only capture > to file netflows with specific SNMP ifindex. From the > router, i'm exporting the flows as V5. I run flow-capture > as follow :- > > /opt/flow-tools/bin/flow-capture -p- \ > -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ > x.x.x.x/y.y.y.y/2055 > /opt/flow-tools/bin/flow-capture -p- \ -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ x.x.x.x/y.y.y.y/2055 -F default you forgot to tell flow-capture which filter definition to use. > Content of /opt/flow-tools/etc/filter :- > > ----------- BEGIN ------------------ > filter-primitive if1 > type ifindex > permit 3 > > filter-definition default > match input-interface if1 > or > match output-interface if1 > ----------- END ------------------ > > Somehow, i am still getting flows from other interfaces > on that router. Any idea if what i want can be done and > what is the purpose of the "-f" option for flow-capture > and flow-fanout if it doesn't? > > Thanks alot. > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools > ----------------------------------------------- William Emmanuel S. Yu Ateneo Campus Network Group (AteneoCNG) email : wyu at ateneo dot edu web : http://CNG.ateneo.net/cng/wyu/ phone : +63(2)4266001-4186 GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp From JohnWong@crimsonlogic.com Wed Jan 07 08:53:17 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 34754 invoked by alias); 7 Jan 2004 08:53:16 -0000 Received: from mail1.crimsonlogic.com (HELO cronus.crimsonlogic.com) (210.56.134.80) by 66.250.216.131 with SMTP; 7 Jan 2004 08:53:16 -0000 Received: from SG01PVEXCH00.snshub.org ([172.29.8.14]) by cronus.crimsonlogic.com (8.12.10/8.12.10) with ESMTP id i078r111002401; Wed, 7 Jan 2004 16:53:01 +0800 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [Flow-tools] flow-capture & flow-fanout with filer option Date: Wed, 7 Jan 2004 16:53:01 +0800 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Flow-tools] flow-capture & flow-fanout with filer option Thread-Index: AcPU+q1r6dTI2joHSsyNj/FfsBfkTAAAGJow From: "John Wong" To: "Horatio B. Bogbindero" Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 08:53:17 -0000 Hi, I thought the "default" definition is the one to be used if i do not specify the "-F" option. Anyway, I did try putting in a specific "-F" option but got the same results. The thing is, when i used flow-nfilter with the same filter file & definition, i get the correct result i.e. only interfaces matching ifindex 3. So i figure it could be something with flow-capture or flow-fanout that i'm missing. Thanks. > -----Original Message----- > From: Horatio B. Bogbindero [mailto:wyu@ateneo.edu]=20 > Sent: Wednesday, January 07, 2004 4:46 PM > To: John Wong > Cc: flow-tools@list.splintered.net > Subject: Re: [Flow-tools] flow-capture & flow-fanout with filer option >=20 >=20 > Quoting John Wong : >=20 > > Hi, > >=20 > > I'm trying to configure flow-capture to only capture > > to file netflows with specific SNMP ifindex. From the > > router, i'm exporting the flows as V5. I run flow-capture > > as follow :- > >=20 > > /opt/flow-tools/bin/flow-capture -p- \ > > -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ > > x.x.x.x/y.y.y.y/2055 > >=20 > /opt/flow-tools/bin/flow-capture -p- \ > -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ > x.x.x.x/y.y.y.y/2055 -F default >=20 > you forgot to tell flow-capture which filter definition > to use. >=20 > > Content of /opt/flow-tools/etc/filter :- > >=20 > > ----------- BEGIN ------------------ > > filter-primitive if1 > > type ifindex > > permit 3 > >=20 > > filter-definition default > > match input-interface if1 > > or > > match output-interface if1 > > ----------- END ------------------ > >=20 > > Somehow, i am still getting flows from other interfaces > > on that router. Any idea if what i want can be done and > > what is the purpose of the "-f" option for flow-capture > > and flow-fanout if it doesn't? > >=20 > > Thanks alot. > > _______________________________________________ > > Flow-tools mailing list > > flow-tools@splintered.net > > http://mailman.splintered.net/mailman/listinfo/flow-tools > >=20 >=20 >=20 >=20 > ----------------------------------------------- > William Emmanuel S. Yu > Ateneo Campus Network Group (AteneoCNG) > email : wyu at ateneo dot edu > web : http://CNG.ateneo.net/cng/wyu/ > phone : +63(2)4266001-4186 > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp >=20 >=20 From warren.daly@heanet.ie Wed Jan 07 09:18:36 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 35650 invoked by alias); 7 Jan 2004 09:18:36 -0000 Received: from byron.heanet.ie (193.1.219.90) by 66.250.216.131 with SMTP; 7 Jan 2004 09:18:36 -0000 Received: from dhcp178.heanet.ie ([193.1.219.178] helo=VENUS) by byron.heanet.ie with esmtp (Exim 4.22) id 1Ae9pi-0007Rk-9i; Wed, 07 Jan 2004 09:18:34 +0000 From: "Warren Daly" To: "'Patrice Empeigne'" , Subject: RE: [Flow-tools] Billing System Date: Wed, 7 Jan 2004 09:20:09 -0000 Message-ID: <1948D86456DFD511883900306E1C5B9740A380@exchange.heanet.ie> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0013_01C3D4FF.725D1E00" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.2627 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165 In-Reply-To: <1948D86456DFD511883900306E1C5B978997ED@exchange.heanet.ie> Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 09:18:37 -0000 This is a multi-part message in MIME format. ------=_NextPart_000_0013_01C3D4FF.725D1E00 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Pat, I am using my own flow collector and I pipe directly into a Mysql db. I use a table per day. I see certain days a single table can contain upto 5,000,000 entries. The machine has 2 Gb of RAM, I used some of the simple settings to optimize the server as much as possible http://www.mysql.com/doc/en/MySQL_Optimisation.html Hope this helps. Warren Warren Daly - Network Security Expert HEAnet Limited Brooklawn House, Crampton Ave, Shelbourne Rd, Ballsbridge, Dublin 4 Phone: +353 1 6609040; Fax: +353 1 6603666 email: warren.daly@heanet.ie This message may be digitally signed or encrypted using a (PKI) certificate issued by the Irish Academic & Research Authority. -----Original Message----- From: flow-tools-bounces@list.splintered.net [mailto:flow-tools-bounces@list.splintered.net] On Behalf Of Patrice Empeigne Sent: 06 January 2004 23:04 To: flow-tools@list.splintered.net Subject: [Flow-tools] Billing System Hi All, Im currently writing a billing system using flow-tools. Im piping all information from flow-capture straight into a mySQL db. Ive written the frontend and all works fine. What advice im looking for is in regards to performance, currently im storing everything in one table. Im currently changing that and creating a table per day to shorten the response times of queries. Before i do so, i was just wondering if anyone else has done something similar and could suggest a data model or even other hints for performance. Thanks team Pat ------=_NextPart_000_0013_01C3D4FF.725D1E00 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Billing System

Pat,

I am using my own flow collector = and I pipe directly into a Mysql db. I use a table per day. =

I see certain days a single table = can contain upto 5,000,000 entries.

The machine has 2 Gb of RAM, I used some of the simple settings to optimize the server as = much as possible

http://www.m= ysql.com/doc/en/MySQL_Optimisation.html

Hope this = helps.

Warren

 

 

Warren Daly - = Network Security Expert

 <= /span>

HEAnet Limited =
Brooklawn House, Crampton Ave,
Shelbourne Rd, Ballsbridge, Dublin 4
Phone: +353 1 6609040; Fax: +353 1 6603666
email: warren.daly@heanet.ie

 <= /span>

This message may = be digitally signed or encrypted using a (PKI) certificate issued by the Irish = Academic & Research Authority.

-----Original = Message-----
From: flow-tools-bounces@list.splintered.net = [mailto:flow-tools-bounces@list.splintered.net] On Behalf Of Patrice = Empeigne
Sent: 06 January 2004 = 23:04
To: = flow-tools@list.splintered.net
Subject: [Flow-tools] = Billing System

 

Hi All,
Im currently writing a billing system using flow-tools.
Im piping all information from flow-capture straight into a mySQL = db.
Ive written the frontend and all works fine.
What advice im looking for is in regards to performance, currently im storing everything in one table.
Im currently changing that and creating a table per day to shorten the = response times of queries.

Before i do so, i was just wondering if anyone = else has done something similar and could suggest a data model =
or even other hints for performance.

Thanks team

Pat

------=_NextPart_000_0013_01C3D4FF.725D1E00-- From wyu@ateneo.edu Wed Jan 07 09:18:38 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 35656 invoked by alias); 7 Jan 2004 09:18:37 -0000 Received: from mail.ateneo.edu (202.138.180.10) by 66.250.216.131 with SMTP; 7 Jan 2004 09:18:37 -0000 Received: from mail.ateneo.edu (localhost [127.0.0.1]) by localhost (Postfix) with ESMTP id 3DBC23F02; Wed, 7 Jan 2004 17:18:28 +0800 (PHT) Received: by mail.ateneo.edu (Postfix, from userid 48) id 22E193F00; Wed, 7 Jan 2004 17:18:28 +0800 (PHT) Received: from 203.82.45.150 ([203.82.45.150]) by mail.ateneo.edu (IMP) with HTTP for ; Wed, 7 Jan 2004 17:18:28 +0800 Message-ID: <1073467108.3ffbcee40575d@mail.ateneo.edu> Date: Wed, 7 Jan 2004 17:18:28 +0800 From: "Horatio B. Bogbindero" To: John Wong Subject: RE: [Flow-tools] flow-capture & flow-fanout with filer option References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.2 Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 09:18:38 -0000 Quoting John Wong : > Hi, > > I thought the "default" definition is the one > to be used if i do not specify the "-F" option. > Anyway, I did try putting in a specific "-F" > option but got the same results. > oh? anyway, i checked the flow-capture and if you do not specify the '-F' arguement explicity the filtering code is not invoked at all. > The thing is, when i used flow-nfilter with the > same filter file & definition, i get the correct > result i.e. only interfaces matching ifindex 3. > > So i figure it could be something with flow-capture > or flow-fanout that i'm missing. > probably. i will have to take a closer look first. can you email me the error message return? or there is none? try running it again with a higher debug level. > Thanks. > > > > -----Original Message----- > > From: Horatio B. Bogbindero [mailto:wyu@ateneo.edu] > > Sent: Wednesday, January 07, 2004 4:46 PM > > To: John Wong > > Cc: flow-tools@list.splintered.net > > Subject: Re: [Flow-tools] flow-capture & flow-fanout with filer option > > > > > > Quoting John Wong : > > > > > Hi, > > > > > > I'm trying to configure flow-capture to only capture > > > to file netflows with specific SNMP ifindex. From the > > > router, i'm exporting the flows as V5. I run flow-capture > > > as follow :- > > > > > > /opt/flow-tools/bin/flow-capture -p- \ > > > -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ > > > x.x.x.x/y.y.y.y/2055 > > > > > /opt/flow-tools/bin/flow-capture -p- \ > > -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ > > x.x.x.x/y.y.y.y/2055 -F default > > > > you forgot to tell flow-capture which filter definition > > to use. > > > > > Content of /opt/flow-tools/etc/filter :- > > > > > > ----------- BEGIN ------------------ > > > filter-primitive if1 > > > type ifindex > > > permit 3 > > > > > > filter-definition default > > > match input-interface if1 > > > or > > > match output-interface if1 > > > ----------- END ------------------ > > > > > > Somehow, i am still getting flows from other interfaces > > > on that router. Any idea if what i want can be done and > > > what is the purpose of the "-f" option for flow-capture > > > and flow-fanout if it doesn't? > > > > > > Thanks alot. > > > _______________________________________________ > > > Flow-tools mailing list > > > flow-tools@splintered.net > > > http://mailman.splintered.net/mailman/listinfo/flow-tools > > > > > > > > > > > ----------------------------------------------- > > William Emmanuel S. Yu > > Ateneo Campus Network Group (AteneoCNG) > > email : wyu at ateneo dot edu > > web : http://CNG.ateneo.net/cng/wyu/ > > phone : +63(2)4266001-4186 > > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp > > > > > ----------------------------------------------- William Emmanuel S. Yu Ateneo Campus Network Group (AteneoCNG) email : wyu at ateneo dot edu web : http://CNG.ateneo.net/cng/wyu/ phone : +63(2)4266001-4186 GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp From wyu@ateneo.edu Wed Jan 07 09:45:21 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 37325 invoked by alias); 7 Jan 2004 09:45:20 -0000 Received: from mail.ateneo.edu (202.138.180.10) by 66.250.216.131 with SMTP; 7 Jan 2004 09:45:20 -0000 Received: from mail.ateneo.edu (localhost [127.0.0.1]) by localhost (Postfix) with ESMTP id 797E33F02; Wed, 7 Jan 2004 17:45:17 +0800 (PHT) Received: by mail.ateneo.edu (Postfix, from userid 48) id 634B93F00; Wed, 7 Jan 2004 17:45:17 +0800 (PHT) Received: from 203.82.45.150 ([203.82.45.150]) by mail.ateneo.edu (IMP) with HTTP for ; Wed, 7 Jan 2004 17:45:17 +0800 Message-ID: <1073468717.3ffbd52d48864@mail.ateneo.edu> Date: Wed, 7 Jan 2004 17:45:17 +0800 From: "Horatio B. Bogbindero" To: Patrice Empeigne Subject: RE: [Flow-tools] Billing System References: <52E510BAD83DAD4BB429B4C53C269FD2064BCD@NT2.corporate.ensyst.com.au> In-Reply-To: <52E510BAD83DAD4BB429B4C53C269FD2064BCD@NT2.corporate.ensyst.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.2 Cc: Mark Fullmer , flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 09:45:21 -0000 speaking of flow-report and exporting. i would really like to write a "plugin" or some code for flow-report that will allow it to export to different things like databases. unfortunately, i took a look at the flow-report/ftlib-relevant code and cannot find an elegant way of doing it. the solution i initially had in mind (about a year ago) was to make a new set of reports for each! maybe somebody has a better way i would like to hear it. thanks. Quoting Patrice Empeigne : > Hi Mark, > I have considered flow-report but there is a need for granularity, thus storage of all data. > This need for granularity is from the incorporation of an investigation tool that allows, for > example, to check if Chuck is streaming any porn. ;) Therefore the approach, for performance, I > have taken is pretty much through summarisation in the data model as suggested by Geoffrey > Bradford above and intelligent queries. > > Thanks Mark, > Pat > > -----Original Message----- > From: Mark Fullmer [mailto:maf@eng.oar.net] > Sent: Wednesday, 7 January 2004 5:41 PM > To: Patrice Empeigne > Cc: flow-tools@list.splintered.net > Subject: Re: [Flow-tools] Billing System > > > Have you considered using flow-report to summarize the data before > storing > it to mySQL? As an example lets say you're billing for every IP address > in a /16. With summarized flows this is 65535 (inbound) or 65535*2 > (inbound+outbound) records per day vs potentially millions of records > per > day by storing the flows. > > mark > > On Jan 6, 2004, at 6:04 PM, Patrice Empeigne wrote: > > > Hi All, > > Im currently writing a billing system using flow-tools. > > Im piping all information from flow-capture straight into a mySQL db. > > Ive written the frontend and all works fine. > > What advice im looking for is in regards to performance, currently im > > storing everything in one table. > > Im currently changing that and creating a table per day to shorten the > > response times of queries. > > > > Before i do so, i was just wondering if anyone else has done something > > similar and could suggest a data model > > or even other hints for performance. > > > > Thanks team > > > > Pat > > _______________________________________________ > > Flow-tools mailing list > > flow-tools@splintered.net > > http://mailman.splintered.net/mailman/listinfo/flow-tools > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools > ----------------------------------------------- William Emmanuel S. Yu Ateneo Campus Network Group (AteneoCNG) email : wyu at ateneo dot edu web : http://CNG.ateneo.net/cng/wyu/ phone : +63(2)4266001-4186 GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp From CihanS@garanti.com.tr Wed Jan 07 09:53:03 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 38096 invoked by alias); 7 Jan 2004 09:52:54 -0000 Received: from mailser7.garanti.com.tr (HELO dmzexc5.garanti.com.tr) (194.29.208.15) by 66.250.216.131 with SMTP; 7 Jan 2004 09:52:54 -0000 Received: from copcon1.fw.garanti.com.tr ([10.145.0.197]) by copcon2.fw.garanti.com.tr with Microsoft SMTPSVC(5.0.2195.6713); Wed, 7 Jan 2004 11:52:02 +0200 Received: from gtiexc5.fw.teknoloji.com.tr ([10.129.0.122]) by copcon1.fw.garanti.com.tr with Microsoft SMTPSVC(5.0.2195.6713); Wed, 7 Jan 2004 11:52:02 +0200 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C3D503.E613EEB0" X-MimeOLE: Produced By Microsoft Exchange V6.0.6487.1 Subject: RE: [Flow-tools] Billing System Date: Wed, 7 Jan 2004 11:52:01 +0200 Message-ID: <5FA6F2E719F6C04FAD76837E84929C34047DDC3D@gtiexc5.fw.teknoloji.com.tr> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Flow-tools] Billing System Thread-Index: AcPU/6hVh+SWG05dRfmHXKNW/yxZtwABAFiQ From: To: , , Return-Path: CihanS@garanti.com.tr X-OriginalArrivalTime: 07 Jan 2004 09:52:02.0289 (UTC) FILETIME=[E65BA610:01C3D503] Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 09:53:03 -0000 This is a multi-part message in MIME format. ------_=_NextPart_001_01C3D503.E613EEB0 Content-Type: text/plain; charset="windows-1254" Content-Transfer-Encoding: quoted-printable Have you tried this? =0D http://www.tekyazilim.com/ =0D Seems like they have traffic analysis tools for netflow and ip= accounting..I tried the ip accounting version, it is fast and gives you a= good information about the ASes -----Original Message----- From: flow-tools-bounces@list.splintered.net= [mailto:flow-tools-bounces@list.splintered.net]On Behalf Of Warren Daly Sent: Wednesday, January 07, 2004 11:20 AM To: 'Patrice Empeigne'; flow-tools@list.splintered.net Subject: RE: [Flow-tools] Billing System Pat, I am using my own flow collector and I pipe directly into a Mysql db. I use= a table per day.=0D I see certain days a single table can contain upto 5,000,000 entries.=0D The machine has 2 Gb of RAM, I used some of the simple settings to optimize= the server as much as possible http://www.mysql.com/doc/en/MySQL_Optimisation.html Hope this helps. Warren=0D =0D =0D Warren Daly - Network Security Expert =0D HEAnet Limited=0D Brooklawn House, Crampton Ave, Shelbourne Rd, Ballsbridge, Dublin 4 Phone: +353 1 6609040; Fax: +353 1 6603666 email: warren.daly@heanet.ie =0D This message may be digitally signed or encrypted using a (PKI) certificate= issued by the Irish Academic & Research Authority. -----Original Message----- From: flow-tools-bounces@list.splintered.net= [mailto:flow-tools-bounces@list.splintered.net] On Behalf Of Patrice= Empeigne Sent: 06 January 2004 23:04 To: flow-tools@list.splintered.net Subject: [Flow-tools] Billing System =0D Hi All,=0D Im currently writing a billing system using flow-tools.=0D Im piping all information from flow-capture straight into a mySQL db.=0D Ive written the frontend and all works fine.=0D What advice im looking for is in regards to performance, currently im= storing everything in one table.=0D Im currently changing that and creating a table per day to shorten the= response times of queries.=0D Before i do so, i was just wondering if anyone else has done something= similar and could suggest a data model=0D or even other hints for performance.=0D Thanks team=0D Pat=0D This message and attachments are confidential and intended solely for the= individual(s) stated in this message.If you received this message although you are not the addressee you= are responsible to keep the message confidential .The sender has no responsibility for the accuracy or= correctness of the information in the message and its attachments.Our company shall have no= liability for any changes or late receiving,loss of integrity and confidentiality,viruses and any= damages caused in anyway to your computer system. Bu mesaj ve ekleri mesajda gonderildigi belirtilen kisi/kisilere ozeldir ve= gizlidir.Bu mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis olmasi halinde mesaj iceriginin= gizliligi ve bu gizlilik yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur.Mesaj ve eklerinde yer= alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da sirketimizin herhangi bir sorumlulugu= bulunmamaktadir.Sirketimiz mesajin ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan,= butunlugunun ve gizliliginin korunamamasindan, virus icermesinden ve bilgisayar sisteminize verebilecegi= herhangi bir zarardan sorumlu tutulamaz. ------_=_NextPart_001_01C3D503.E613EEB0 Content-Type: text/html; charset="windows-1254" Content-Transfer-Encoding: quoted-printable Billing System
Have=0D you tried this?
 
http://www.tekyazilim.com/
 
Seems=0D like they have traffic analysis tools for netflow and ip accounting..I= tried the=0D ip accounting version, it is fast and gives you a good information about= the=0D ASes
-----Original Message-----
From:=0D flow-tools-bounces@list.splintered.net=0D [mailto:flow-tools-bounces@list.splintered.net]On Behalf Of Warren= =0D Daly
Sent: Wednesday, January 07, 2004 11:20 AM
To:=0D 'Patrice Empeigne'; flow-tools@list.splintered.net
Subject: RE:= =0D [Flow-tools] Billing System

Pat,

I am using my= own=0D flow collector and I pipe directly into a Mysql db. I use a table per= day.=0D

I see certain= days a=0D single table can contain upto 5,000,000 entries.=

The machine= has 2=0D Gb of RAM, I used some of the simple settings= to=0D optimize the server as much as possible

http://www.mysql.c= om/doc/en/MySQL_Optimisation.html

Hope this=0D helps.

Warren=0D

 

 

Warren=0D Daly - Network Security Expert

 

HEAnet=0D Limited
Brooklawn House, Crampton Ave,
Shelbourne Rd, Ballsbridge,= =0D Dublin 4
Phone: +353 1 6609040; Fax: +353 1 6603666
email: warren.daly@heanet.ie

 

This=0D message may be digitally signed or encrypted using a (PKI) certificate= issued=0D by the Irish Academic & Research=0D Authority.

-----Original=0D Message-----
From:=0D flow-tools-bounces@list.splintered.net=0D [mailto:flow-tools-bounces@list.splintered.net] On Behalf Of Patrice= Empeigne
Sent: 06 January 2004= 23:04
To:=0D flow-tools@list.splintered.net
Subject: [Flow-tools] Billing=0D System

 

Hi All,=
Im=0D currently writing a billing system using flow-tools.=
Im piping=0D all information from flow-capture straight into a mySQL db.= =0D
Ive written the frontend= and all=0D works fine.
What advice im looking for= is in=0D regards to performance, currently im storing everything in one=0D table.
Im currently changing that= and=0D creating a table per day to shorten the response times of=0D queries.

Before i do so, i was just= =0D wondering if anyone else has done something similar and could suggest a= data=0D model
or even other hints for=0D performance.

Thanks team=0D

Pat=0D 

This message=
 and attachments are confidential and intended solely for the individual(s)=
 stated in this
message.If you received this message although you are not the addressee you=
 are responsible to keep the
message confidential .The sender has no responsibility for the accuracy or=
 correctness of the
information in the message and its attachments.Our company shall have no=
 liability for any changes
or late receiving,loss of integrity and confidentiality,viruses and any=
 damages caused in
anyway to your computer system.

Bu mesaj ve=
 ekleri mesajda gonderildigi belirtilen kisi/kisilere ozeldir ve=
 gizlidir.Bu mesajin muhatabi
olmamaniza ragmen tarafiniza ulasmis olmasi halinde mesaj iceriginin=
 gizliligi ve bu gizlilik yukumlulugune
uyulmasi zorunlulugu tarafiniz icin de soz konusudur.Mesaj ve eklerinde yer=
 alan bilgilerin dogrulugu ve
guncelligi konusunda gonderenin ya da sirketimizin herhangi bir sorumlulugu=
 bulunmamaktadir.Sirketimiz
mesajin ve bilgilerinin size degisiklige ugrayarak veya gec ulasmasindan,=
 butunlugunun ve gizliliginin
korunamamasindan, virus icermesinden ve bilgisayar sisteminize verebilecegi=
 herhangi bir zarardan
sorumlu tutulamaz.
------_=_NextPart_001_01C3D503.E613EEB0-- From JohnWong@crimsonlogic.com Wed Jan 07 10:02:28 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 38908 invoked by alias); 7 Jan 2004 10:02:27 -0000 Received: from mail1.crimsonlogic.com (HELO cronus.crimsonlogic.com) (210.56.134.80) by 66.250.216.131 with SMTP; 7 Jan 2004 10:02:27 -0000 Received: from SG01PVEXCH00.snshub.org ([172.29.8.14]) by cronus.crimsonlogic.com (8.12.10/8.12.10) with ESMTP id i07A2N11004478; Wed, 7 Jan 2004 18:02:23 +0800 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [Flow-tools] flow-capture & flow-fanout with filer option Date: Wed, 7 Jan 2004 18:02:23 +0800 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Flow-tools] flow-capture & flow-fanout with filer option Thread-Index: AcPU/z5O3pBhyAWUSamFpYbz0wz+0QABUjsw From: "John Wong" To: "Horatio B. Bogbindero" Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 10:02:28 -0000 Hi, There wasn't any errors, just that it didn't do what i thought it was supposed to. You're right that if i do not specify any definition it defaults to not filtering. The manpages need to be updated. I tried explicitly setting the "-F" option again to flow-fanout but am getting the same result. I supposed it will be the same if I set it for flow-capture too. I tried setting "-d 1" but where does the logs go to? I don't see any at the stdout. Thanks. > -----Original Message----- > From: Horatio B. Bogbindero [mailto:wyu@ateneo.edu]=20 > Sent: Wednesday, January 07, 2004 5:18 PM > To: John Wong > Cc: flow-tools@list.splintered.net > Subject: RE: [Flow-tools] flow-capture & flow-fanout with filer option >=20 >=20 > Quoting John Wong : >=20 > > Hi, > >=20 > > I thought the "default" definition is the one > > to be used if i do not specify the "-F" option. > > Anyway, I did try putting in a specific "-F" > > option but got the same results. > >=20 > oh? anyway, i checked the flow-capture and if you > do not specify the '-F' arguement explicity > the filtering code is not invoked at all. >=20 > > The thing is, when i used flow-nfilter with the > > same filter file & definition, i get the correct > > result i.e. only interfaces matching ifindex 3. > >=20 > > So i figure it could be something with flow-capture > > or flow-fanout that i'm missing. > >=20 > probably. i will have to take a closer look first. > can you email me the error message return? or there is > none? try running it again with a higher debug level. >=20 > > Thanks. > >=20 > >=20 > > > -----Original Message----- > > > From: Horatio B. Bogbindero [mailto:wyu@ateneo.edu]=20 > > > Sent: Wednesday, January 07, 2004 4:46 PM > > > To: John Wong > > > Cc: flow-tools@list.splintered.net > > > Subject: Re: [Flow-tools] flow-capture & flow-fanout with=20 > filer option > > >=20 > > >=20 > > > Quoting John Wong : > > >=20 > > > > Hi, > > > >=20 > > > > I'm trying to configure flow-capture to only capture > > > > to file netflows with specific SNMP ifindex. From the > > > > router, i'm exporting the flows as V5. I run flow-capture > > > > as follow :- > > > >=20 > > > > /opt/flow-tools/bin/flow-capture -p- \ > > > > -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ > > > > x.x.x.x/y.y.y.y/2055 > > > >=20 > > > /opt/flow-tools/bin/flow-capture -p- \ > > > -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ > > > x.x.x.x/y.y.y.y/2055 -F default > > >=20 > > > you forgot to tell flow-capture which filter definition > > > to use. > > >=20 > > > > Content of /opt/flow-tools/etc/filter :- > > > >=20 > > > > ----------- BEGIN ------------------ > > > > filter-primitive if1 > > > > type ifindex > > > > permit 3 > > > >=20 > > > > filter-definition default > > > > match input-interface if1 > > > > or > > > > match output-interface if1 > > > > ----------- END ------------------ > > > >=20 > > > > Somehow, i am still getting flows from other interfaces > > > > on that router. Any idea if what i want can be done and > > > > what is the purpose of the "-f" option for flow-capture > > > > and flow-fanout if it doesn't? > > > >=20 > > > > Thanks alot. > > > > _______________________________________________ > > > > Flow-tools mailing list > > > > flow-tools@splintered.net > > > > http://mailman.splintered.net/mailman/listinfo/flow-tools > > > >=20 > > >=20 > > >=20 > > >=20 > > > ----------------------------------------------- > > > William Emmanuel S. Yu > > > Ateneo Campus Network Group (AteneoCNG) > > > email : wyu at ateneo dot edu > > > web : http://CNG.ateneo.net/cng/wyu/ > > > phone : +63(2)4266001-4186 > > > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp > > >=20 > > >=20 > >=20 >=20 >=20 >=20 > ----------------------------------------------- > William Emmanuel S. Yu > Ateneo Campus Network Group (AteneoCNG) > email : wyu at ateneo dot edu > web : http://CNG.ateneo.net/cng/wyu/ > phone : +63(2)4266001-4186 > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp >=20 >=20 From anang@csmcom.com Wed Jan 07 10:05:37 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 39704 invoked by alias); 7 Jan 2004 10:05:37 -0000 Received: from 202-127-99-4.triplegate.net.id (HELO oracle) (202.127.99.4) by 66.250.216.131 with SMTP; 7 Jan 2004 10:05:37 -0000 Received: (qmail 17360 invoked by uid 105); 7 Jan 2004 10:06:57 -0000 Received: from anang@csmcom.com by oracle by uid 118 with qmail-scanner-1.15 (avpdaemon: ???. spamassassin: 2.43. Clear:. Processed in 0.104317 secs); 07 Jan 2004 10:06:57 -0000 Received: from csmwks-52-9.csmcom.com (HELO csmcom.com) (172.18.52.9) by oracle with SMTP; 7 Jan 2004 10:06:57 -0000 Message-ID: <3FFBD9DC.4060301@csmcom.com> Date: Wed, 07 Jan 2004 17:05:16 +0700 From: Anang Syarifudin Organization: PT. CItra Sari Makmur User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031202 X-Accept-Language: en-us, en MIME-Version: 1.0 To: flow-tools@list.splintered.net Subject: Re: [Flow-tools] Billing System References: <1948D86456DFD511883900306E1C5B9740A380@exchange.heanet.ie> In-Reply-To: <1948D86456DFD511883900306E1C5B9740A380@exchange.heanet.ie> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 10:05:37 -0000 I tried to make my own flow collector flow-capture -R /u02/flowdata/rotate.sh -E600M -n 288 -z5 -d 0 -N 0 -w /u02/flowdata 0/0/9901 #!/bin/sh mv -f 2.flw 3.flw 2>/dev/null mv -f 1.flw 2.flw 2>/dev/null mv -f $1 1.flw 2>/dev/null flow-export -f 3 -mUNIX_SECS,EXADDR,DPKTS,DOCTETS,FIRST,LAST,SRCADDR,DSTADDR,NEXTHOP,INPUT,OUTPUT,SRCPORT,DSTPORT,PROT -u "flowtools:flowtoolsdb:localhost:3306:flowtools:flow_raw" < 1.flw >/dev/null /usr/local/bin/php -q /usr/local/flowcol/engine/analyze.php 2>/dev/null >&- <&- >/dev/null & first I tried plain table without index, table insertion is reasonably fast. but analyze.php run very slow, it took more than 20 minutes to be completely executed, for 5 minutes flow analysis it's unaceptable. then I do mysql optimisation, I add some index, now analyze.php run faster, less than 2 minutes (with forking), but table insertion is very slow, rotate.sh is overlapping 10 instances. can you share how your collector works ? or you can give me some clue, how to improve mine ? my goal is 5 minutes sampling statistics per interface: - top 10 incoming/outgoing source/destination port - top 10 incoming/outgoing source/destination address Anang Warren Daly wrote: > Pat, > > I am using my own flow collector and I pipe directly into a Mysql db. > I use a table per day. > > I see certain days a single table can contain upto 5,000,000 entries. > > The machine has 2 Gb of RAM, I used some of the simple settings to > optimize the server as much as possible > > http://www.mysql.com/doc/en/MySQL_Optimisation.html > > Hope this helps. > > Warren > > > From wyu@ateneo.edu Wed Jan 07 10:16:12 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 40571 invoked by alias); 7 Jan 2004 10:16:10 -0000 Received: from mail.ateneo.edu (202.138.180.10) by 66.250.216.131 with SMTP; 7 Jan 2004 10:16:10 -0000 Received: from mail.ateneo.edu (localhost [127.0.0.1]) by localhost (Postfix) with ESMTP id 7446F3F24; Wed, 7 Jan 2004 18:16:06 +0800 (PHT) Received: by mail.ateneo.edu (Postfix, from userid 48) id 5F05F3F07; Wed, 7 Jan 2004 18:16:06 +0800 (PHT) Received: from 203.82.45.150 ([203.82.45.150]) by mail.ateneo.edu (IMP) with HTTP for ; Wed, 7 Jan 2004 18:16:06 +0800 Message-ID: <1073470566.3ffbdc6640d70@mail.ateneo.edu> Date: Wed, 7 Jan 2004 18:16:06 +0800 From: "Horatio B. Bogbindero" To: John Wong Subject: RE: [Flow-tools] flow-capture & flow-fanout with filer option References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.2 Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 10:16:12 -0000 Quoting John Wong : > Hi, > > There wasn't any errors, just that it didn't do > what i thought it was supposed to. You're right > that if i do not specify any definition it defaults > to not filtering. The manpages need to be updated. > > I tried explicitly setting the "-F" option again > to flow-fanout but am getting the same result. I > supposed it will be the same if I set it for > flow-capture too. I tried setting "-d 1" but where > does the logs go to? I don't see any at the stdout. > for flow-fanout, there is no code that invokes flow-filter at all. hmmm. something is not right here. well, i will check it out tomorrow. need to get home now. > > > > -----Original Message----- > > From: Horatio B. Bogbindero [mailto:wyu@ateneo.edu] > > Sent: Wednesday, January 07, 2004 5:18 PM > > To: John Wong > > Cc: flow-tools@list.splintered.net > > Subject: RE: [Flow-tools] flow-capture & flow-fanout with filer option > > > > > > Quoting John Wong : > > > > > Hi, > > > > > > I thought the "default" definition is the one > > > to be used if i do not specify the "-F" option. > > > Anyway, I did try putting in a specific "-F" > > > option but got the same results. > > > > > oh? anyway, i checked the flow-capture and if you > > do not specify the '-F' arguement explicity > > the filtering code is not invoked at all. > > > > > The thing is, when i used flow-nfilter with the > > > same filter file & definition, i get the correct > > > result i.e. only interfaces matching ifindex 3. > > > > > > So i figure it could be something with flow-capture > > > or flow-fanout that i'm missing. > > > > > probably. i will have to take a closer look first. > > can you email me the error message return? or there is > > none? try running it again with a higher debug level. > > > > > Thanks. > > > > > > > > > > -----Original Message----- > > > > From: Horatio B. Bogbindero [mailto:wyu@ateneo.edu] > > > > Sent: Wednesday, January 07, 2004 4:46 PM > > > > To: John Wong > > > > Cc: flow-tools@list.splintered.net > > > > Subject: Re: [Flow-tools] flow-capture & flow-fanout with > > filer option > > > > > > > > > > > > Quoting John Wong : > > > > > > > > > Hi, > > > > > > > > > > I'm trying to configure flow-capture to only capture > > > > > to file netflows with specific SNMP ifindex. From the > > > > > router, i'm exporting the flows as V5. I run flow-capture > > > > > as follow :- > > > > > > > > > > /opt/flow-tools/bin/flow-capture -p- \ > > > > > -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ > > > > > x.x.x.x/y.y.y.y/2055 > > > > > > > > > /opt/flow-tools/bin/flow-capture -p- \ > > > > -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ > > > > x.x.x.x/y.y.y.y/2055 -F default > > > > > > > > you forgot to tell flow-capture which filter definition > > > > to use. > > > > > > > > > Content of /opt/flow-tools/etc/filter :- > > > > > > > > > > ----------- BEGIN ------------------ > > > > > filter-primitive if1 > > > > > type ifindex > > > > > permit 3 > > > > > > > > > > filter-definition default > > > > > match input-interface if1 > > > > > or > > > > > match output-interface if1 > > > > > ----------- END ------------------ > > > > > > > > > > Somehow, i am still getting flows from other interfaces > > > > > on that router. Any idea if what i want can be done and > > > > > what is the purpose of the "-f" option for flow-capture > > > > > and flow-fanout if it doesn't? > > > > > > > > > > Thanks alot. > > > > > _______________________________________________ > > > > > Flow-tools mailing list > > > > > flow-tools@splintered.net > > > > > http://mailman.splintered.net/mailman/listinfo/flow-tools > > > > > > > > > > > > > > > > > > > > > ----------------------------------------------- > > > > William Emmanuel S. Yu > > > > Ateneo Campus Network Group (AteneoCNG) > > > > email : wyu at ateneo dot edu > > > > web : http://CNG.ateneo.net/cng/wyu/ > > > > phone : +63(2)4266001-4186 > > > > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp > > > > > > > > > > > > > > > > > > > ----------------------------------------------- > > William Emmanuel S. Yu > > Ateneo Campus Network Group (AteneoCNG) > > email : wyu at ateneo dot edu > > web : http://CNG.ateneo.net/cng/wyu/ > > phone : +63(2)4266001-4186 > > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp > > > > > ----------------------------------------------- William Emmanuel S. Yu Ateneo Campus Network Group (AteneoCNG) email : wyu at ateneo dot edu web : http://CNG.ateneo.net/cng/wyu/ phone : +63(2)4266001-4186 GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp From JohnWong@crimsonlogic.com Wed Jan 07 10:26:03 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 41367 invoked by alias); 7 Jan 2004 10:26:02 -0000 Received: from mail2.crimsonlogic.com (HELO rhea.crimsonlogic.com) (210.56.134.81) by 66.250.216.131 with SMTP; 7 Jan 2004 10:26:02 -0000 Received: from SG01PVEXCH00.snshub.org ([172.29.8.14]) by rhea.crimsonlogic.com (8.12.10/8.12.10) with ESMTP id i07APwAf014612; Wed, 7 Jan 2004 18:25:59 +0800 X-MimeOLE: Produced By Microsoft Exchange V6.0.4712.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [Flow-tools] flow-capture & flow-fanout with filer option Date: Wed, 7 Jan 2004 18:25:58 +0800 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Flow-tools] flow-capture & flow-fanout with filer option Thread-Index: AcPVB0XiwGOoXeNKRJmihiWnqBWmmAAAHjrA From: "John Wong" To: "Horatio B. Bogbindero" Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 10:26:03 -0000 Hi, Finally got it to work. Looks like it works on version 0.67. I did a diff of flow-fanout on version 0.66 vs 0.67. A chunk of code on filtering seems to be missing in 0.66. I guess that's the problem. Thanks for your help... flow-tools rocks... Thanks. > -----Original Message----- > From: Horatio B. Bogbindero [mailto:wyu@ateneo.edu]=20 > Sent: Wednesday, January 07, 2004 6:16 PM > To: John Wong > Cc: flow-tools@list.splintered.net > Subject: RE: [Flow-tools] flow-capture & flow-fanout with filer option >=20 >=20 > Quoting John Wong : >=20 > > Hi, > >=20 > > There wasn't any errors, just that it didn't do > > what i thought it was supposed to. You're right > > that if i do not specify any definition it defaults > > to not filtering. The manpages need to be updated. > >=20 > > I tried explicitly setting the "-F" option again > > to flow-fanout but am getting the same result. I > > supposed it will be the same if I set it for > > flow-capture too. I tried setting "-d 1" but where > > does the logs go to? I don't see any at the stdout. > >=20 > for flow-fanout, there is no code that invokes flow-filter > at all. hmmm. something is not right here. >=20 > well, i will check it out tomorrow. need to get home now. >=20 >=20 >=20 > >=20 > >=20 > > > -----Original Message----- > > > From: Horatio B. Bogbindero [mailto:wyu@ateneo.edu]=20 > > > Sent: Wednesday, January 07, 2004 5:18 PM > > > To: John Wong > > > Cc: flow-tools@list.splintered.net > > > Subject: RE: [Flow-tools] flow-capture & flow-fanout with=20 > filer option > > >=20 > > >=20 > > > Quoting John Wong : > > >=20 > > > > Hi, > > > >=20 > > > > I thought the "default" definition is the one > > > > to be used if i do not specify the "-F" option. > > > > Anyway, I did try putting in a specific "-F" > > > > option but got the same results. > > > >=20 > > > oh? anyway, i checked the flow-capture and if you > > > do not specify the '-F' arguement explicity > > > the filtering code is not invoked at all. > > >=20 > > > > The thing is, when i used flow-nfilter with the > > > > same filter file & definition, i get the correct > > > > result i.e. only interfaces matching ifindex 3. > > > >=20 > > > > So i figure it could be something with flow-capture > > > > or flow-fanout that i'm missing. > > > >=20 > > > probably. i will have to take a closer look first. > > > can you email me the error message return? or there is > > > none? try running it again with a higher debug level. > > >=20 > > > > Thanks. > > > >=20 > > > >=20 > > > > > -----Original Message----- > > > > > From: Horatio B. Bogbindero [mailto:wyu@ateneo.edu]=20 > > > > > Sent: Wednesday, January 07, 2004 4:46 PM > > > > > To: John Wong > > > > > Cc: flow-tools@list.splintered.net > > > > > Subject: Re: [Flow-tools] flow-capture & flow-fanout with=20 > > > filer option > > > > >=20 > > > > >=20 > > > > > Quoting John Wong : > > > > >=20 > > > > > > Hi, > > > > > >=20 > > > > > > I'm trying to configure flow-capture to only capture > > > > > > to file netflows with specific SNMP ifindex. From the > > > > > > router, i'm exporting the flows as V5. I run flow-capture > > > > > > as follow :- > > > > > >=20 > > > > > > /opt/flow-tools/bin/flow-capture -p- \ > > > > > > -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ > > > > > > x.x.x.x/y.y.y.y/2055 > > > > > >=20 > > > > > /opt/flow-tools/bin/flow-capture -p- \ > > > > > -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ > > > > > x.x.x.x/y.y.y.y/2055 -F default > > > > >=20 > > > > > you forgot to tell flow-capture which filter definition > > > > > to use. > > > > >=20 > > > > > > Content of /opt/flow-tools/etc/filter :- > > > > > >=20 > > > > > > ----------- BEGIN ------------------ > > > > > > filter-primitive if1 > > > > > > type ifindex > > > > > > permit 3 > > > > > >=20 > > > > > > filter-definition default > > > > > > match input-interface if1 > > > > > > or > > > > > > match output-interface if1 > > > > > > ----------- END ------------------ > > > > > >=20 > > > > > > Somehow, i am still getting flows from other interfaces > > > > > > on that router. Any idea if what i want can be done and > > > > > > what is the purpose of the "-f" option for flow-capture > > > > > > and flow-fanout if it doesn't? > > > > > >=20 > > > > > > Thanks alot. > > > > > > _______________________________________________ > > > > > > Flow-tools mailing list > > > > > > flow-tools@splintered.net > > > > > > http://mailman.splintered.net/mailman/listinfo/flow-tools > > > > > >=20 > > > > >=20 > > > > >=20 > > > > >=20 > > > > > ----------------------------------------------- > > > > > William Emmanuel S. Yu > > > > > Ateneo Campus Network Group (AteneoCNG) > > > > > email : wyu at ateneo dot edu > > > > > web : http://CNG.ateneo.net/cng/wyu/ > > > > > phone : +63(2)4266001-4186 > > > > > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp > > > > >=20 > > > > >=20 > > > >=20 > > >=20 > > >=20 > > >=20 > > > ----------------------------------------------- > > > William Emmanuel S. Yu > > > Ateneo Campus Network Group (AteneoCNG) > > > email : wyu at ateneo dot edu > > > web : http://CNG.ateneo.net/cng/wyu/ > > > phone : +63(2)4266001-4186 > > > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp > > >=20 > > >=20 > >=20 >=20 >=20 >=20 > ----------------------------------------------- > William Emmanuel S. Yu > Ateneo Campus Network Group (AteneoCNG) > email : wyu at ateneo dot edu > web : http://CNG.ateneo.net/cng/wyu/ > phone : +63(2)4266001-4186 > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp >=20 >=20 From lorins@assist.ro Wed Jan 07 13:50:10 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 43255 invoked by alias); 7 Jan 2004 13:50:10 -0000 Received: from mail.e-suceava.com (HELO users.assist.ro) (194.102.130.2) by 66.250.216.131 with SMTP; 7 Jan 2004 13:50:10 -0000 Received: from arrakis.assist.ro (arrakis.assist.ro [194.102.130.6]) by users.assist.ro (8.11.6/) with ESMTP id i07Dn5Q14343 for ; Wed, 7 Jan 2004 15:49:05 +0200 From: Lorin Scraba Organization: s.c. ASSIST Software s.r.l. To: flow-tools@list.splintered.net Subject: Re: [Flow-tools] Billing System Date: Wed, 7 Jan 2004 15:50:30 +0000 User-Agent: KMail/1.5.3 References: <52E510BAD83DAD4BB429B4C53C269FD2064BC8@NT2.corporate.ensyst.com.au> In-Reply-To: <52E510BAD83DAD4BB429B4C53C269FD2064BC8@NT2.corporate.ensyst.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200401071550.30784.lorins@assist.ro> X-RAVMilter-Version: 8.4.2(snapshot 20021217) (users.assist.ro) X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list Reply-To: lorins@assist.ro List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 13:50:10 -0000 On Tuesday 06 January 2004 23:04, Patrice Empeigne wrote: > Hi All, > Im currently writing a billing system using flow-tools. > Im piping all information from flow-capture straight into a mySQL db. > Ive written the frontend and all works fine. > What advice im looking for is in regards to performance, currently im > storing everything in one table. Im currently changing that and creating a > table per day to shorten the response times of queries. > > Before i do so, i was just wondering if anyone else has done something > similar and could suggest a data model or even other hints for performance. > > Thanks team > > Pat Hi ! Well i pipe the data into a mysql database using flow-export and i use 2 temporary _Heap_ tables for speed. Table flow_import, which is continously feed with netflow data and table flow, which contains flow information for the last 5 minutes. I use a little trick to rotate those 2 from a script every 5 minutes so i can have small tables, easy to process: mysql -h localhost -u root -p"blabla" -n -N -B -q accounting < Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 44349 invoked by alias); 7 Jan 2004 14:59:16 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 7 Jan 2004 14:59:16 -0000 In-Reply-To: <1073468717.3ffbd52d48864@mail.ateneo.edu> References: <52E510BAD83DAD4BB429B4C53C269FD2064BCD@NT2.corporate.ensyst.com.au> <1073468717.3ffbd52d48864@mail.ateneo.edu> Mime-Version: 1.0 (Apple Message framework v609) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <0F84F8DE-4122-11D8-BCCB-000A95DA1C38@eng.oar.net> Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [Flow-tools] Billing System Date: Wed, 7 Jan 2004 09:59:15 -0500 To: Horatio B. Bogbindero X-Mailer: Apple Mail (2.609) Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 14:59:17 -0000 The easiest way to do this is work with the ASCII output. There will be two scripts, flow-rptfmt and flow-rpt2rrd included in 0.68. Flow-rptfmt will generate flow-stat type output from flow-report with the options to display in percent total form, add symbol lookups, and select fields. Flow-rpt2rrd will convert flow-report output to RRD files. You could potentially do this internally, but it's a lot of work because each data-structure has its own display/output function, so you would need to write code for each of bucket_dump1(), chash_c64_dump(), chash_c32_dump(), etc. Looks like about 20....At least that's how it works now. mark On Jan 7, 2004, at 4:45 AM, Horatio B. Bogbindero wrote: > > speaking of flow-report and exporting. i would really like to write a > "plugin" or some > code for flow-report that will allow it to export to different things > like databases. > unfortunately, i took a look at the flow-report/ftlib-relevant code > and cannot find an > elegant way of doing it. the solution i initially had in mind (about a > year ago) was > to make a new set of reports for each! > > maybe somebody has a better way i would like to hear it. > > thanks. > > Quoting Patrice Empeigne : > >> Hi Mark, >> I have considered flow-report but there is a need for granularity, >> thus storage of all data. >> This need for granularity is from the incorporation of an >> investigation tool that allows, for >> example, to check if Chuck is streaming any porn. ;) Therefore the >> approach, for performance, I >> have taken is pretty much through summarisation in the data model as >> suggested by Geoffrey >> Bradford above and intelligent queries. >> >> Thanks Mark, >> Pat >> >> -----Original Message----- >> From: Mark Fullmer [mailto:maf@eng.oar.net] >> Sent: Wednesday, 7 January 2004 5:41 PM >> To: Patrice Empeigne >> Cc: flow-tools@list.splintered.net >> Subject: Re: [Flow-tools] Billing System >> >> >> Have you considered using flow-report to summarize the data before >> storing >> it to mySQL? As an example lets say you're billing for every IP >> address >> in a /16. With summarized flows this is 65535 (inbound) or 65535*2 >> (inbound+outbound) records per day vs potentially millions of records >> per >> day by storing the flows. >> >> mark >> >> On Jan 6, 2004, at 6:04 PM, Patrice Empeigne wrote: >> >>> Hi All, >>> Im currently writing a billing system using flow-tools. >>> Im piping all information from flow-capture straight into a mySQL db. >>> Ive written the frontend and all works fine. >>> What advice im looking for is in regards to performance, currently im >>> storing everything in one table. >>> Im currently changing that and creating a table per day to shorten >>> the >>> response times of queries. >>> >>> Before i do so, i was just wondering if anyone else has done >>> something >>> similar and could suggest a data model >>> or even other hints for performance. >>> >>> Thanks team >>> >>> Pat >>> _______________________________________________ >>> Flow-tools mailing list >>> flow-tools@splintered.net >>> http://mailman.splintered.net/mailman/listinfo/flow-tools >> >> _______________________________________________ >> Flow-tools mailing list >> flow-tools@splintered.net >> http://mailman.splintered.net/mailman/listinfo/flow-tools >> > > > > ----------------------------------------------- > William Emmanuel S. Yu > Ateneo Campus Network Group (AteneoCNG) > email : wyu at ateneo dot edu > web : http://CNG.ateneo.net/cng/wyu/ > phone : +63(2)4266001-4186 > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From liao1k@cmich.edu Wed Jan 07 15:15:15 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 45195 invoked by alias); 7 Jan 2004 15:15:15 -0000 Received: from garm.csv.cmich.edu (HELO garm.central.cmich.local) (141.209.15.48) by 66.250.216.131 with SMTP; 7 Jan 2004 15:15:15 -0000 Received: from cmail3.central.cmich.local ([141.209.15.83]) by egate1.central.cmich.local with Microsoft SMTPSVC(5.0.2195.6713); Wed, 7 Jan 2004 10:15:09 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPartTM-000-da32696d-fba7-498f-9092-12dad6b88939" Date: Wed, 7 Jan 2004 10:15:09 -0500 Message-ID: <291B348BC59B47468C7824603C326082216845@cmail3.central.cmich.local> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Meaning of the Net Flow data Thread-Index: AcPVMRSaVy/qQfneTS6tJslRe+J86g== From: "Liao, Kexiao" To: Return-Path: liao1k@cmich.edu X-OriginalArrivalTime: 07 Jan 2004 15:15:09.0903 (UTC) FILETIME=[0A46C5F0:01C3D531] Subject: [Flow-tools] Meaning of the Net Flow data X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 15:15:15 -0000 This is a multi-part message in MIME format. ------=_NextPartTM-000-da32696d-fba7-498f-9092-12dad6b88939 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C3D531.0A218546" ------_=_NextPart_001_01C3D531.0A218546 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Does anybody know the actual meaning of the net flow data? For example what is the meaning of UNIX_SECS, SYSUPTIME, EXADDR, DPKTS,DOCTETS, FIRST, LAST etc. Where can I find the document for these fields' meaning? Thanks =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Kexiao Liao=20 CMU Research Corporation=20 2625 Denison Dr. Mount Pleasant, MI 48858=20 Phone 989-774-2424 , Fax 989-774-2416 http://www.thecenter.cmich.edu/=20 liao1k@cmich.edu =20 =20 ------_=_NextPart_001_01C3D531.0A218546 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Does anybody know the actual meaning of the net flow = data? For example what is the meaning of UNIX_SECS, SYSUPTIME, EXADDR, = DPKTS,DOCTETS, FIRST, LAST etc. Where can I find the document for these fields’ = meaning? Thanks

 

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D

Kexiao Liao

CMU Research Corporation

2625 Denison Dr.

Mount Pleasant, MI 48858

Phone 989-774-2424 , Fax 989-774-2416

http://www.thecenter.cmich.edu/<= /a>

liao1k@cmich.edu

 

 

=00 ------_=_NextPart_001_01C3D531.0A218546-- ------=_NextPartTM-000-da32696d-fba7-498f-9092-12dad6b88939-- From splintered-flow-tools-owner@splintered.net Wed Jan 07 15:16:28 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 45550 invoked by uid 4001); 7 Jan 2004 15:16:28 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 45548 invoked by alias); 7 Jan 2004 15:16:28 -0000 Received: from quake1.xnet.com (198.147.221.67) by 66.250.216.131 with SMTP; 7 Jan 2004 15:16:28 -0000 Received: from chntmailgate.tradingtechnologies.com (nat-111.tradingtechnologies.com [204.248.60.111]) by quake1.xnet.com (Postfix) with ESMTP id 1C82A7AFE for ; Wed, 7 Jan 2004 09:16:28 -0600 (CST) Received: from evnbridgehead-vlan1.tradingtechnologies (unverified) by chntmailgate.tradingtechnologies.com (Content Technologies SMTPRS 4.2.5) with ESMTP id ; Wed, 7 Jan 2004 09:16:27 -0600 Received: from 172.17.8.244 ([172.17.8.244]) by evnbridgehead-vlan1.tradingtechnologies with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id ZCLTZFSV; Wed, 7 Jan 2004 09:16:26 -0600 Subject: Re: [Flow-tools] Flow-tools 0.67 RPM available From: jeff vier To: "Horatio B. Bogbindero" In-Reply-To: <1073462078.3ffbbb3e6d55c@mail.ateneo.edu> References: <1073462078.3ffbbb3e6d55c@mail.ateneo.edu> Content-Type: text/plain Organization: TT Message-Id: <1073488567.4513.77.camel@localhost> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.5-2mdk Date: Wed, 07 Jan 2004 09:16:07 -0600 Content-Transfer-Encoding: 7bit Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 15:16:28 -0000 Is there no source tarball? On Wed, 2004-01-07 at 01:54, Horatio B. Bogbindero wrote: > i would just like to announce that the flow-tools 0.67 is already available. it can be > downloaded at the usual place http://cng.ateneo.net/cng/wyu/software/flow-tools.php. > > ----------------------------------------------- > William Emmanuel S. Yu > Ateneo Campus Network Group (AteneoCNG) > email : wyu at ateneo dot edu > web : http://CNG.ateneo.net/cng/wyu/ > phone : +63(2)4266001-4186 > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From maf@eng.oar.net Wed Jan 07 15:23:56 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 46720 invoked by alias); 7 Jan 2004 15:23:56 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 7 Jan 2004 15:23:56 -0000 In-Reply-To: <52E510BAD83DAD4BB429B4C53C269FD2064BCD@NT2.corporate.ensyst.com.au> References: <52E510BAD83DAD4BB429B4C53C269FD2064BCD@NT2.corporate.ensyst.com.au> Mime-Version: 1.0 (Apple Message framework v609) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <81CCDB08-4125-11D8-BCCB-000A95DA1C38@eng.oar.net> Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [Flow-tools] Billing System Date: Wed, 7 Jan 2004 10:23:55 -0500 To: "Patrice Empeigne" X-Mailer: Apple Mail (2.609) Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 15:23:56 -0000 I guess it depends on how much data you're working with. You could leave the raw flows in flow-tools format and then do fine-grained queries with flow-nfilter and flow-report. With the Abilene flow data we're moving around 700,000,000 flows/day and about 50 or so reports/router plus providing data feeds to 14 research groups with a 3+ month archive of raw data. Trying to do this with raw flow data in an SQL database would be cost prohibitive to say the least. mark On Jan 7, 2004, at 1:54 AM, Patrice Empeigne wrote: > Hi Mark, > I have considered flow-report but there is a need for granularity, > thus storage of all data. > This need for granularity is from the incorporation of an > investigation tool that allows, for example, to check if Chuck is > streaming any porn. ;) Therefore the approach, for performance, I have > taken is pretty much through summarisation in the data model as > suggested by Geoffrey Bradford above and intelligent queries. > > Thanks Mark, > Pat > > -----Original Message----- > From: Mark Fullmer [mailto:maf@eng.oar.net] > Sent: Wednesday, 7 January 2004 5:41 PM > To: Patrice Empeigne > Cc: flow-tools@list.splintered.net > Subject: Re: [Flow-tools] Billing System > > > Have you considered using flow-report to summarize the data before > storing > it to mySQL? As an example lets say you're billing for every IP > address > in a /16. With summarized flows this is 65535 (inbound) or 65535*2 > (inbound+outbound) records per day vs potentially millions of records > per > day by storing the flows. > > mark > > On Jan 6, 2004, at 6:04 PM, Patrice Empeigne wrote: > >> Hi All, >> Im currently writing a billing system using flow-tools. >> Im piping all information from flow-capture straight into a mySQL db. >> Ive written the frontend and all works fine. >> What advice im looking for is in regards to performance, currently im >> storing everything in one table. >> Im currently changing that and creating a table per day to shorten the >> response times of queries. >> >> Before i do so, i was just wondering if anyone else has done something >> similar and could suggest a data model >> or even other hints for performance. >> >> Thanks team >> >> Pat >> _______________________________________________ >> Flow-tools mailing list >> flow-tools@splintered.net >> http://mailman.splintered.net/mailman/listinfo/flow-tools > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From maf@eng.oar.net Wed Jan 07 15:31:40 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 47558 invoked by alias); 7 Jan 2004 15:31:40 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 7 Jan 2004 15:31:40 -0000 In-Reply-To: <291B348BC59B47468C7824603C326082216845@cmail3.central.cmich.local> References: <291B348BC59B47468C7824603C326082216845@cmail3.central.cmich.local> Mime-Version: 1.0 (Apple Message framework v609) Content-Type: text/plain; charset=WINDOWS-1252; format=flowed Message-Id: <964444B0-4126-11D8-BCCB-000A95DA1C38@eng.oar.net> Content-Transfer-Encoding: quoted-printable From: Mark Fullmer Subject: Re: [Flow-tools] Meaning of the Net Flow data Date: Wed, 7 Jan 2004 10:31:39 -0500 To: "Liao, Kexiao" X-Mailer: Apple Mail (2.609) Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 15:31:40 -0000 Lookin lib/ftlib.h struct ftpdu_v5 { /* 24 byte header */ u_int16 version; /* 5 */ u_int16 count; /* The number of records in the PDU */ u_int32 sysUpTime; /* Current time in millisecs since router=20 booted */ u_int32 unix_secs; /* Current seconds since 0000 UTC 1970 */ u_int32 unix_nsecs; /* Residual nanoseconds since 0000 UTC 1970 */ u_int32 flow_sequence; /* Seq counter of total flows seen */ u_int8 engine_type; /* Type of flow switching engine (RP,VIP,etc.)=20= */ u_int8 engine_id; /* Slot number of the flow switching engine */ u_int16 reserved; /* 48 byte payload */ struct ftrec_v5 { u_int32 srcaddr; /* Source IP Address */ u_int32 dstaddr; /* Destination IP Address */ u_int32 nexthop; /* Next hop router's IP Address */ u_int16 input; /* Input interface index */ u_int16 output; /* Output interface index */ u_int32 dPkts; /* Packets sent in Duration */ u_int32 dOctets; /* Octets sent in Duration. */ u_int32 First; /* SysUptime at start of flow */ u_int32 Last; /* and of last packet of flow */ u_int16 srcport; /* TCP/UDP source port number or equivalent */ u_int16 dstport; /* TCP/UDP destination port number or equiv */ u_int8 pad; u_int8 tcp_flags; /* Cumulative OR of tcp flags */ u_int8 prot; /* IP protocol, e.g., 6=3DTCP, 17=3DUDP, ... */ u_int8 tos; /* IP Type-of-Service */ u_int16 src_as; /* originating AS of source address */ u_int16 dst_as; /* originating AS of destination address */ u_int8 src_mask; /* source address prefix mask bits */ u_int8 dst_mask; /* destination address prefix mask bits */ u_int16 drops; } records[FT_PDU_V5_MAXFLOWS]; }; On Jan 7, 2004, at 10:15 AM, Liao, Kexiao wrote: > Does anybody know the actual meaning of the net flow data? For example=20= > what is the meaning of UNIX_SECS, SYSUPTIME, EXADDR, DPKTS,DOCTETS,=20 > FIRST, LAST etc. Where can I find the document for these fields=92=20 > meaning? Thanks > > =A0 > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D > > Kexiao Liao > > CMU Research Corporation > > 2625 Denison Dr. > > Mount Pleasant, MI 48858 > > Phone 989-774-2424 , Fax 989-774-2416 > > http://www.thecenter.cmich.edu/ > > liao1k@cmich.edu > > =A0 > > =A0 > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools= From liao1k@cmich.edu Wed Jan 07 16:27:39 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 48728 invoked by alias); 7 Jan 2004 16:27:39 -0000 Received: from garm.csv.cmich.edu (HELO garm.central.cmich.local) (141.209.15.48) by 66.250.216.131 with SMTP; 7 Jan 2004 16:27:39 -0000 Received: from cmail3.central.cmich.local ([141.209.15.83]) by egate1.central.cmich.local with Microsoft SMTPSVC(5.0.2195.6713); Wed, 7 Jan 2004 11:27:34 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPartTM-000-6ea75c45-61aa-4a85-bec4-25a8785a1cfc" Date: Wed, 7 Jan 2004 11:27:34 -0500 Message-ID: <291B348BC59B47468C7824603C326082216846@cmail3.central.cmich.local> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Meaning of the Net Flow data again Thread-Index: AcPVNGK4oHMNSilHQV+AZf2xBvl/JAABiOIw From: "Liao, Kexiao" To: "Mark Fullmer" Return-Path: liao1k@cmich.edu X-OriginalArrivalTime: 07 Jan 2004 16:27:34.0381 (UTC) FILETIME=[27C981D0:01C3D53B] Cc: flow-tools@list.splintered.net Subject: [Flow-tools] Meaning of the Net Flow data again X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 16:27:40 -0000 This is a multi-part message in MIME format. ------=_NextPartTM-000-6ea75c45-61aa-4a85-bec4-25a8785a1cfc Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C3D53B.279E1C28" ------_=_NextPart_001_01C3D53B.279E1C28 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable If I want to know the size of payload data in each TCP/UDP packet, which one I need to choose? Thanks =20 =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Kexiao Liao=20 CMU Research Corporation=20 2625 Denison Dr. Mount Pleasant, MI 48858=20 Phone 989-774-2424 , Fax 989-774-2416 http://www.thecenter.cmich.edu/=20 liao1k@cmich.edu =20 =20 -----Original Message----- From: Mark Fullmer [mailto:maf@eng.oar.net]=20 Sent: Wednesday, January 07, 2004 10:32 AM To: Liao, Kexiao Cc: flow-tools@list.splintered.net Subject: Re: [Flow-tools] Meaning of the Net Flow data =20 Lookin lib/ftlib.h =20 struct ftpdu_v5 { /* 24 byte header */ u_int16 version; /* 5 */ u_int16 count; /* The number of records in the PDU */ u_int32 sysUpTime; /* Current time in millisecs since router=20 booted */ u_int32 unix_secs; /* Current seconds since 0000 UTC 1970 */ u_int32 unix_nsecs; /* Residual nanoseconds since 0000 UTC 1970 */ u_int32 flow_sequence; /* Seq counter of total flows seen */ u_int8 engine_type; /* Type of flow switching engine (RP,VIP,etc.) */ u_int8 engine_id; /* Slot number of the flow switching engine */ u_int16 reserved; /* 48 byte payload */ struct ftrec_v5 { u_int32 srcaddr; /* Source IP Address */ u_int32 dstaddr; /* Destination IP Address */ u_int32 nexthop; /* Next hop router's IP Address */ u_int16 input; /* Input interface index */ u_int16 output; /* Output interface index */ u_int32 dPkts; /* Packets sent in Duration */ u_int32 dOctets; /* Octets sent in Duration. */ u_int32 First; /* SysUptime at start of flow */ u_int32 Last; /* and of last packet of flow */ u_int16 srcport; /* TCP/UDP source port number or equivalent */ u_int16 dstport; /* TCP/UDP destination port number or equiv */ u_int8 pad; u_int8 tcp_flags; /* Cumulative OR of tcp flags */ u_int8 prot; /* IP protocol, e.g., 6=3DTCP, 17=3DUDP, ... */ u_int8 tos; /* IP Type-of-Service */ u_int16 src_as; /* originating AS of source address */ u_int16 dst_as; /* originating AS of destination address */ u_int8 src_mask; /* source address prefix mask bits */ u_int8 dst_mask; /* destination address prefix mask bits */ u_int16 drops; } records[FT_PDU_V5_MAXFLOWS]; }; =20 On Jan 7, 2004, at 10:15 AM, Liao, Kexiao wrote: =20 > Does anybody know the actual meaning of the net flow data? For example > what is the meaning of UNIX_SECS, SYSUPTIME, EXADDR, DPKTS,DOCTETS,=20 > FIRST, LAST etc. Where can I find the document for these fields'=20 > meaning? Thanks > > =20 > > = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > > Kexiao Liao > > CMU Research Corporation > > 2625 Denison Dr. > > Mount Pleasant, MI 48858 > > Phone 989-774-2424 , Fax 989-774-2416 > > http://www.thecenter.cmich.edu/ > > liao1k@cmich.edu > > =20 > > =20 > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools ------_=_NextPart_001_01C3D53B.279E1C28 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

If I want to know the size of payload data in each TCP/UDP = packet, which one I need to choose? Thanks

 

 

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D

Kexiao Liao

CMU Research Corporation

2625 Denison Dr.

Mount Pleasant, MI 48858

Phone 989-774-2424 , Fax 989-774-2416

http://www.thecenter.cmich.edu/

liao1k@cmich.edu

 

 

-----Original Message-----
From: Mark Fullmer [mailto:maf@eng.oar.net]
Sent: Wednesday, January 07, 2004 10:32 AM
To: Liao, Kexiao
Cc: flow-tools@list.splintered.net
Subject: Re: [Flow-tools] Meaning of the Net Flow data

 

Lookin lib/ftlib.h

 

struct ftpdu_v5 {

   /* 24 byte header */

   u_int16 = version;       /* 5 */

   u_int16 count;         /* The number of = records in the PDU */

   u_int32 sysUpTime;     /* = Current time in millisecs since router

booted */

   u_int32 unix_secs;     /* = Current seconds since 0000 UTC 1970 */

   u_int32 unix_nsecs;    /* Residual nanoseconds since 0000 UTC 1970 */

   u_int32 flow_sequence; /* Seq counter of total = flows seen */

   u_int8  engine_type;   /* Type of = flow switching engine (RP,VIP,etc.)

*/

   u_int8  engine_id;     /* = Slot number of the flow switching engine */

   u_int16 reserved;

   /* 48 byte payload */

   struct ftrec_v5 {

     u_int32 srcaddr;    /* = Source IP Address */

     u_int32 dstaddr;    /* Destination IP Address */

     u_int32 nexthop;    /* = Next hop router's IP Address */

     u_int16 = input;      /* Input interface index */

     u_int16 output;     = /* Output interface index */

     u_int32 = dPkts;      /* Packets sent in Duration */

     u_int32 dOctets;    /* = Octets sent in Duration. */

     u_int32 = First;      /* SysUptime at start of flow */

     u_int32 Last;       /* and of last packet of flow = */

     u_int16 srcport;    /* = TCP/UDP source port number or equivalent */

     u_int16 dstport;    /* = TCP/UDP destination port number or equiv */

     u_int8  pad;

     u_int8  tcp_flags;  /* = Cumulative OR of tcp flags */

     u_int8  prot;       /* IP protocol, e.g., 6=3DTCP, = 17=3DUDP, ... */

     u_int8  tos;   =      /* IP Type-of-Service */

     u_int16 src_as;     = /* originating AS of source address */

     u_int16 dst_as;     = /* originating AS of destination address */

     u_int8  src_mask;   /* = source address prefix mask bits */

     u_int8  dst_mask;   /* = destination address prefix mask bits */

     u_int16 drops;

   } records[FT_PDU_V5_MAXFLOWS];

};

 

On Jan 7, 2004, at 10:15 AM, Liao, Kexiao = wrote:

 

> Does anybody know the actual meaning of the net flow data? = For example

> what is the meaning of UNIX_SECS, SYSUPTIME, EXADDR, DPKTS,DOCTETS,

> FIRST, LAST etc. Where can I find the document for these fields’

> meaning? Thanks

>

>  

>

> = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D

>

> Kexiao Liao

>

> CMU Research Corporation

>

> 2625 Denison Dr.

>

> Mount Pleasant, MI 48858

>

> Phone 989-774-2424 , Fax 989-774-2416

>

> http://www.thecenter.cmich.edu/

>

> liao1k@cmich.edu

>

>  

>

>  

> = _______________________________________________

> Flow-tools mailing list

> flow-tools@splintered.net

> = http://mailman.splintered.net/mailman/listinfo/flow-tools

=00 ------_=_NextPart_001_01C3D53B.279E1C28-- ------=_NextPartTM-000-6ea75c45-61aa-4a85-bec4-25a8785a1cfc-- From mhunter@ack.Berkeley.EDU Wed Jan 07 17:29:31 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 49902 invoked by alias); 7 Jan 2004 17:29:31 -0000 Received: from ack.berkeley.edu (128.32.206.66) by 66.250.216.131 with SMTP; 7 Jan 2004 17:29:31 -0000 Received: (from mhunter@localhost) by ack.Berkeley.EDU (8.11.3/8.11.3) id i07HTSG19243; Wed, 7 Jan 2004 09:29:28 -0800 (PST) Date: Wed, 7 Jan 2004 09:29:28 -0800 From: Mike Hunter To: "Liao, Kexiao" Subject: Re: [Flow-tools] Meaning of the Net Flow data again Message-ID: <20040107172928.GA18832@ack.Berkeley.EDU> References: <291B348BC59B47468C7824603C326082216846@cmail3.central.cmich.local> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <291B348BC59B47468C7824603C326082216846@cmail3.central.cmich.local> User-Agent: Mutt/1.4i Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 17:29:31 -0000 On Jan 07, "Liao, Kexiao" wrote: > If I want to know the size of payload data in each TCP/UDP packet, which > one I need to choose? Thanks The netflow data cannot tell you the size of each packet, because usually only one netflow PDU is generated for a group of TCP/UDP packets; you can only know the average bytes per packet. The number of packets is dPkts, the number of octets ("octet" is a synonym for an 8-bit byte) is dOctets. It might help you to look at the output of flow-print -f 5 to get an idea of what the field are available and what they mean in terms of "real-world" data. Mike > u_int16 version; /* 5 */ > > u_int16 count; /* The number of records in the PDU */ > > u_int32 sysUpTime; /* Current time in millisecs since router > > booted */ > > u_int32 unix_secs; /* Current seconds since 0000 UTC 1970 */ > > u_int32 unix_nsecs; /* Residual nanoseconds since 0000 UTC 1970 */ > > u_int32 flow_sequence; /* Seq counter of total flows seen */ > > u_int8 engine_type; /* Type of flow switching engine (RP,VIP,etc.) > > > */ > > u_int8 engine_id; /* Slot number of the flow switching engine */ > > u_int16 reserved; > > /* 48 byte payload */ > > struct ftrec_v5 { > > u_int32 srcaddr; /* Source IP Address */ > > u_int32 dstaddr; /* Destination IP Address */ > > u_int32 nexthop; /* Next hop router's IP Address */ > > u_int16 input; /* Input interface index */ > > u_int16 output; /* Output interface index */ > > u_int32 dPkts; /* Packets sent in Duration */ > > u_int32 dOctets; /* Octets sent in Duration. */ > > u_int32 First; /* SysUptime at start of flow */ > > u_int32 Last; /* and of last packet of flow */ > > u_int16 srcport; /* TCP/UDP source port number or equivalent */ > > u_int16 dstport; /* TCP/UDP destination port number or equiv */ > > u_int8 pad; > > u_int8 tcp_flags; /* Cumulative OR of tcp flags */ > > u_int8 prot; /* IP protocol, e.g., 6=TCP, 17=UDP, ... */ > > u_int8 tos; /* IP Type-of-Service */ > > u_int16 src_as; /* originating AS of source address */ > > u_int16 dst_as; /* originating AS of destination address */ > > u_int8 src_mask; /* source address prefix mask bits */ > > u_int8 dst_mask; /* destination address prefix mask bits */ > > u_int16 drops; > > } records[FT_PDU_V5_MAXFLOWS]; > > }; From jrockwell@ChoiceOneCom.Com Wed Jan 07 17:32:14 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 50523 invoked by alias); 7 Jan 2004 17:32:14 -0000 Received: from rochexc1.choiceonecom.com (HELO rochexc1.ad.choiceonecom.com) (66.202.28.8) by 66.250.216.131 with SMTP; 7 Jan 2004 17:32:14 -0000 Received: by ROCHEXC1.ad.choiceonecom.com with Internet Mail Service (5.5.2657.72) id ; Wed, 7 Jan 2004 12:32:14 -0500 Message-ID: <990A175D26546C4A95A6715F25CAB97D05689CD0@ROCHEXC3.choiceonecom.com> From: "Rockwell, John" To: "'Liao, Kexiao'" , Mark Fullmer Subject: RE: [Flow-tools] Meaning of the Net Flow data again Date: Wed, 7 Jan 2004 12:32:13 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2657.72) Content-Type: text/plain; charset="iso-8859-1" Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 17:32:14 -0000 I don't believe there is any such option to get the size of each TCP/UDP packet. There is an option which covers the number of packets in a flow and the total size of the flow, which would allow you to compute a quasi-average, but it would be difficult to ascertain the size of each individual packet as the Netflow packet is a summary of the entire flow. John E. Rockwell Network Engineer, Data Level II Operations Team Leader, Network Security Choice One Communications 100 Chestnut Street Rochester, New York 14604 585-697-2162 (Office) 585-452-3604 (Pager) -----Original Message----- From: Liao, Kexiao [mailto:liao1k@cmich.edu] Sent: Wednesday, January 07, 2004 11:28 AM To: Mark Fullmer Cc: flow-tools@list.splintered.net Subject: [Flow-tools] Meaning of the Net Flow data again If I want to know the size of payload data in each TCP/UDP packet, which one I need to choose? Thanks =========================== Kexiao Liao CMU Research Corporation 2625 Denison Dr. Mount Pleasant, MI 48858 Phone 989-774-2424 , Fax 989-774-2416 http://www.thecenter.cmich.edu/ liao1k@cmich.edu -----Original Message----- From: Mark Fullmer [mailto:maf@eng.oar.net] Sent: Wednesday, January 07, 2004 10:32 AM To: Liao, Kexiao Cc: flow-tools@list.splintered.net Subject: Re: [Flow-tools] Meaning of the Net Flow data Lookin lib/ftlib.h struct ftpdu_v5 { /* 24 byte header */ u_int16 version; /* 5 */ u_int16 count; /* The number of records in the PDU */ u_int32 sysUpTime; /* Current time in millisecs since router booted */ u_int32 unix_secs; /* Current seconds since 0000 UTC 1970 */ u_int32 unix_nsecs; /* Residual nanoseconds since 0000 UTC 1970 */ u_int32 flow_sequence; /* Seq counter of total flows seen */ u_int8 engine_type; /* Type of flow switching engine (RP,VIP,etc.) */ u_int8 engine_id; /* Slot number of the flow switching engine */ u_int16 reserved; /* 48 byte payload */ struct ftrec_v5 { u_int32 srcaddr; /* Source IP Address */ u_int32 dstaddr; /* Destination IP Address */ u_int32 nexthop; /* Next hop router's IP Address */ u_int16 input; /* Input interface index */ u_int16 output; /* Output interface index */ u_int32 dPkts; /* Packets sent in Duration */ u_int32 dOctets; /* Octets sent in Duration. */ u_int32 First; /* SysUptime at start of flow */ u_int32 Last; /* and of last packet of flow */ u_int16 srcport; /* TCP/UDP source port number or equivalent */ u_int16 dstport; /* TCP/UDP destination port number or equiv */ u_int8 pad; u_int8 tcp_flags; /* Cumulative OR of tcp flags */ u_int8 prot; /* IP protocol, e.g., 6=TCP, 17=UDP, ... */ u_int8 tos; /* IP Type-of-Service */ u_int16 src_as; /* originating AS of source address */ u_int16 dst_as; /* originating AS of destination address */ u_int8 src_mask; /* source address prefix mask bits */ u_int8 dst_mask; /* destination address prefix mask bits */ u_int16 drops; } records[FT_PDU_V5_MAXFLOWS]; }; On Jan 7, 2004, at 10:15 AM, Liao, Kexiao wrote: > Does anybody know the actual meaning of the net flow data? For example > what is the meaning of UNIX_SECS, SYSUPTIME, EXADDR, DPKTS,DOCTETS, > FIRST, LAST etc. Where can I find the document for these fields' > meaning? Thanks > > > > =========================== > > Kexiao Liao > > CMU Research Corporation > > 2625 Denison Dr. > > Mount Pleasant, MI 48858 > > Phone 989-774-2424 , Fax 989-774-2416 > > http://www.thecenter.cmich.edu/ > > liao1k@cmich.edu > > > > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From Patrice@ensyst.com.au Wed Jan 07 21:55:36 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 52975 invoked by alias); 7 Jan 2004 21:55:35 -0000 Received: from ppp127-213.lns1.syd3.internode.on.net (HELO NT2.corporate.ensyst.com.au) (150.101.127.213) by 66.250.216.131 with SMTP; 7 Jan 2004 21:55:35 -0000 Content-class: urn:content-classes:message Subject: RE: [Flow-tools] Billing System Date: Thu, 8 Jan 2004 08:55:32 +1100 Message-ID: <52E510BAD83DAD4BB429B4C53C269FD2064BCE@NT2.corporate.ensyst.com.au> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C3D568.F915E10E" X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Flow-tools] Billing System X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Thread-Index: AcPU/zxVvWGh0I5eQWC4PH8inPfG6gAaafNg From: "Patrice Empeigne" To: "Warren Daly" , Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 21:55:36 -0000 This is a multi-part message in MIME format. ------_=_NextPart_001_01C3D568.F915E10E Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Great! Thanks for the link Warren, Ill definately be looking into it. =20 Thanks again, Pat -----Original Message----- From: Warren Daly [mailto:warren.daly@heanet.ie] Sent: Wednesday, 7 January 2004 8:20 PM To: Patrice Empeigne; flow-tools@list.splintered.net Subject: RE: [Flow-tools] Billing System Pat, I am using my own flow collector and I pipe directly into a Mysql db. I = use a table per day.=20 I see certain days a single table can contain upto 5,000,000 entries.=20 The machine has 2 Gb of RAM, I used some of the simple settings to = optimize the server as much as possible http://www.mysql.com/doc/en/MySQL_Optimisation.html Hope this helps. Warren=20 =20 =20 Warren Daly - Network Security Expert =20 HEAnet Limited=20 Brooklawn House, Crampton Ave, Shelbourne Rd, Ballsbridge, Dublin 4 Phone: +353 1 6609040; Fax: +353 1 6603666 email: warren.daly@heanet.ie =20 This message may be digitally signed or encrypted using a (PKI) = certificate issued by the Irish Academic & Research Authority. -----Original Message----- From: flow-tools-bounces@list.splintered.net = [mailto:flow-tools-bounces@list.splintered.net] On Behalf Of Patrice = Empeigne Sent: 06 January 2004 23:04 To: flow-tools@list.splintered.net Subject: [Flow-tools] Billing System =20 Hi All,=20 Im currently writing a billing system using flow-tools.=20 Im piping all information from flow-capture straight into a mySQL db.=20 Ive written the frontend and all works fine.=20 What advice im looking for is in regards to performance, currently im = storing everything in one table.=20 Im currently changing that and creating a table per day to shorten the = response times of queries.=20 Before i do so, i was just wondering if anyone else has done something = similar and could suggest a data model=20 or even other hints for performance.=20 Thanks team=20 Pat=20 ------_=_NextPart_001_01C3D568.F915E10E Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Billing System
Great!
Thanks=20 for the link Warren, Ill definately be looking into = it.
 
Thanks=20 again,
Pat
-----Original Message-----
From: Warren Daly=20 [mailto:warren.daly@heanet.ie]
Sent: Wednesday, 7 January = 2004 8:20=20 PM
To: Patrice Empeigne;=20 flow-tools@list.splintered.net
Subject: RE: [Flow-tools] = Billing=20 System

Pat,

I am using = my own=20 flow collector and I pipe directly into a Mysql db. I use a table per = day.=20

I see = certain days a=20 single table can contain upto 5,000,000 entries. =

The machine = has 2=20 Gb of RAM, I used some of the simple = settings to=20 optimize the server as much as possible

http://www.m= ysql.com/doc/en/MySQL_Optimisation.html

Hope this=20 helps.

Warren=20

 

 

Warren=20 Daly - Network Security Expert

 

HEAnet=20 Limited
Brooklawn House, Crampton Ave,
Shelbourne Rd, = Ballsbridge,=20 Dublin 4
Phone: +353 1 6609040; Fax: +353 1 6603666
email: warren.daly@heanet.ie

 

This=20 message may be digitally signed or encrypted using a (PKI) certificate = issued=20 by the Irish Academic & Research=20 Authority.

-----Original=20 Message-----
From:=20 flow-tools-bounces@list.splintered.net=20 [mailto:flow-tools-bounces@list.splintered.net] On Behalf Of Patrice = Empeigne
Sent: 06 January 2004 = 23:04
To:=20 flow-tools@list.splintered.net
Subject: [Flow-tools] Billing=20 System

 

Hi All, =
Im=20 currently writing a billing system using flow-tools. =
Im piping=20 all information from flow-capture straight into a mySQL = db.=20
Ive written the frontend = and all=20 works fine.
What advice im looking = for is in=20 regards to performance, currently im storing everything in one=20 table.
Im currently changing = that and=20 creating a table per day to shorten the response times of=20 queries.

Before i do so, i was = just=20 wondering if anyone else has done something similar and could suggest = a data=20 model
or even other hints for=20 performance.

Thanks = team=20

Pat=20

------_=_NextPart_001_01C3D568.F915E10E-- From Patrice@ensyst.com.au Wed Jan 07 22:31:58 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 53924 invoked by alias); 7 Jan 2004 22:31:57 -0000 Received: from ppp127-213.lns1.syd3.internode.on.net (HELO NT2.corporate.ensyst.com.au) (150.101.127.213) by 66.250.216.131 with SMTP; 7 Jan 2004 22:31:57 -0000 Content-class: urn:content-classes:message Subject: RE: [Flow-tools] Billing System Date: Thu, 8 Jan 2004 09:31:22 +1100 Message-ID: <52E510BAD83DAD4BB429B4C53C269FD2064BCF@NT2.corporate.ensyst.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Flow-tools] Billing System X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Thread-Index: AcPVJWrtONR+8aBkTU2sM7ZCgfQ1kAARgOzw From: "Patrice Empeigne" To: , Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 22:31:58 -0000 Hi Lorin, Thanks for the reply. Our main issue is definately how we will sculpture = our data model for this system. This is certainly achieved by = understanding the types of queries clients will be requesting and = summarising your tables corresponding to those queries. The system you = described does so, and I will build mine in accordance with the basic = needs of my clients. Thanks again, Patrice -----Original Message----- From: flow-tools-bounces@list.splintered.net [mailto:flow-tools-bounces@list.splintered.net]On Behalf Of Lorin Scraba Sent: Thursday, 8 January 2004 2:51 AM To: flow-tools@list.splintered.net Subject: Re: [Flow-tools] Billing System On Tuesday 06 January 2004 23:04, Patrice Empeigne wrote: > Hi All, > Im currently writing a billing system using flow-tools. > Im piping all information from flow-capture straight into a mySQL db. > Ive written the frontend and all works fine. > What advice im looking for is in regards to performance, currently im > storing everything in one table. Im currently changing that and = creating a > table per day to shorten the response times of queries. > > Before i do so, i was just wondering if anyone else has done something > similar and could suggest a data model or even other hints for = performance. > > Thanks team > > Pat Hi ! Well i pipe the data into a mysql database using flow-export and = i use=20 2 temporary _Heap_ tables for speed. Table flow_import, which is = continously=20 feed with netflow data and table flow, which contains flow information = for=20 the last 5 minutes. I use a little trick to rotate those 2 from a = script=20 every 5 minutes so i can have small tables, easy to process: mysql -h localhost -u root -p"blabla" -n -N -B -q accounting = < Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 54665 invoked by alias); 7 Jan 2004 22:38:15 -0000 Received: from ppp127-213.lns1.syd3.internode.on.net (HELO NT2.corporate.ensyst.com.au) (150.101.127.213) by 66.250.216.131 with SMTP; 7 Jan 2004 22:38:15 -0000 Content-class: urn:content-classes:message Subject: RE: [Flow-tools] Billing System Date: Thu, 8 Jan 2004 09:38:13 +1100 Message-ID: <52E510BAD83DAD4BB429B4C53C269FD2064BD0@NT2.corporate.ensyst.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Flow-tools] Billing System X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Thread-Index: AcPVMkb7Oa7aN20uT+SzrtocqpFVkQAO/8tg From: "Patrice Empeigne" To: "Mark Fullmer" Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 22:38:15 -0000 Hi Mark, Absolutely to say the least! A fine-grained investigation tool would be = out of the question when moving around so much data. I am definately in = a simple trade-off between performance and functionality. Thanks for your help Mark, Patrice -----Original Message----- From: Mark Fullmer [mailto:maf@eng.oar.net] Sent: Thursday, 8 January 2004 2:24 AM To: Patrice Empeigne Cc: flow-tools@list.splintered.net Subject: Re: [Flow-tools] Billing System I guess it depends on how much data you're working with. You could=20 leave the raw flows in flow-tools format and then do fine-grained queries with flow-nfilter and flow-report. With the Abilene flow data we're moving around 700,000,000 flows/day and about 50 or so reports/router plus providing data feeds to 14 research groups with a 3+ month archive of raw data. Trying to do this with raw flow data in an SQL database would be cost prohibitive to say the least. mark On Jan 7, 2004, at 1:54 AM, Patrice Empeigne wrote: > Hi Mark, > I have considered flow-report but there is a need for granularity,=20 > thus storage of all data. > This need for granularity is from the incorporation of an=20 > investigation tool that allows, for example, to check if Chuck is=20 > streaming any porn. ;) Therefore the approach, for performance, I have = > taken is pretty much through summarisation in the data model as=20 > suggested by Geoffrey Bradford above and intelligent queries. > > Thanks Mark, > Pat > > -----Original Message----- > From: Mark Fullmer [mailto:maf@eng.oar.net] > Sent: Wednesday, 7 January 2004 5:41 PM > To: Patrice Empeigne > Cc: flow-tools@list.splintered.net > Subject: Re: [Flow-tools] Billing System > > > Have you considered using flow-report to summarize the data before > storing > it to mySQL? As an example lets say you're billing for every IP=20 > address > in a /16. With summarized flows this is 65535 (inbound) or 65535*2 > (inbound+outbound) records per day vs potentially millions of records > per > day by storing the flows. > > mark > > On Jan 6, 2004, at 6:04 PM, Patrice Empeigne wrote: > >> Hi All, >> Im currently writing a billing system using flow-tools. >> Im piping all information from flow-capture straight into a mySQL db. >> Ive written the frontend and all works fine. >> What advice im looking for is in regards to performance, currently im >> storing everything in one table. >> Im currently changing that and creating a table per day to shorten = the >> response times of queries. >> >> Before i do so, i was just wondering if anyone else has done = something >> similar and could suggest a data model >> or even other hints for performance. >> >> Thanks team >> >> Pat >> _______________________________________________ >> Flow-tools mailing list >> flow-tools@splintered.net >> http://mailman.splintered.net/mailman/listinfo/flow-tools > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From maf@splintered.net Wed Jan 07 23:45:37 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 64098 invoked by alias); 7 Jan 2004 23:45:37 -0000 Received: from unknown (HELO ?IPv6:::1?) (66.250.216.130) by 66.250.216.130 with SMTP; 7 Jan 2004 23:45:37 -0000 In-Reply-To: <3FF02C73.9060300@comstar.ru> References: <3FF02C73.9060300@comstar.ru> Mime-Version: 1.0 (Apple Message framework v609) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <976CC3C8-416B-11D8-BCCB-000A95DA1C38@splintered.net> Content-Transfer-Encoding: 7bit From: Mark Fullmer Subject: Re: [Flow-tools] Flow-tags and v8 Date: Wed, 7 Jan 2004 18:45:36 -0500 To: Kirill Kuvshinnikov X-Mailer: Apple Mail (2.609) Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jan 2004 23:45:37 -0000 Tags are currently only a V5 feature. mark On Dec 29, 2003, at 8:30 AM, Kirill Kuvshinnikov wrote: > I wonder, is it possible to use flow-tags with v8 (8.5) of netflow. > > After configuring tag definitions, I try to use flow-tag and receive > message: > "Flow record missing required field for tagging" (from flow-tag.c) > > Regards, > Kirillium. > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From wyu@ateneo.edu Thu Jan 08 00:38:02 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 65207 invoked by alias); 8 Jan 2004 00:38:02 -0000 Received: from mail.ateneo.edu (202.138.180.10) by 66.250.216.131 with SMTP; 8 Jan 2004 00:38:02 -0000 Received: from mail.ateneo.edu (localhost [127.0.0.1]) by localhost (Postfix) with ESMTP id 7DEF53ECA; Thu, 8 Jan 2004 08:38:00 +0800 (PHT) Received: by mail.ateneo.edu (Postfix, from userid 48) id 65E1C3EB3; Thu, 8 Jan 2004 08:38:00 +0800 (PHT) Received: from 203.82.45.150 ([203.82.45.150]) by mail.ateneo.edu (IMP) with HTTP for ; Thu, 8 Jan 2004 08:38:00 +0800 Message-ID: <1073522280.3ffca66840c91@mail.ateneo.edu> Date: Thu, 8 Jan 2004 08:38:00 +0800 From: "Horatio B. Bogbindero" To: Mark Fullmer Subject: Re: [Flow-tools] Billing System References: <52E510BAD83DAD4BB429B4C53C269FD2064BCD@NT2.corporate.ensyst.com.au> <1073468717.3ffbd52d48864@mail.ateneo.edu> <0F84F8DE-4122-11D8-BCCB-000A95DA1C38@eng.oar.net> In-Reply-To: <0F84F8DE-4122-11D8-BCCB-000A95DA1C38@eng.oar.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.2 Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jan 2004 00:38:02 -0000 Quoting Mark Fullmer : > The easiest way to do this is work with the ASCII output. There will be > two scripts, flow-rptfmt and flow-rpt2rrd included in 0.68. Flow-rptfmt > will generate flow-stat type output from flow-report with the options > to display in percent total form, add symbol lookups, and select fields. > Flow-rpt2rrd will convert flow-report output to RRD files. > for now, this would seem to be the most appropriate way of doing things. right now, i use flow-report to output ascii that dumps data to an ETL tool which we wrote internally to push data to our DSS system. > You could potentially do this internally, but it's a lot of work because > each data-structure has its own display/output function, so you would > need > to write code for each of bucket_dump1(), chash_c64_dump(), > chash_c32_dump(), etc. Looks like about 20....At least that's how it > works > now. > re-writing those functions would be a lot of work. plus code maintenance will be difficult. sigh. of course, the potential performance improvement when removing the translation to ASCII and using the STDIN and STDOUT would really be one major reason why i would like to write directly to a database. but for now, this will have to do. back to the drawing board then. good luck guys! > mark > > On Jan 7, 2004, at 4:45 AM, Horatio B. Bogbindero wrote: > > > > > speaking of flow-report and exporting. i would really like to write a > > "plugin" or some > > code for flow-report that will allow it to export to different things > > like databases. > > unfortunately, i took a look at the flow-report/ftlib-relevant code > > and cannot find an > > elegant way of doing it. the solution i initially had in mind (about a > > year ago) was > > to make a new set of reports for each! > > > > maybe somebody has a better way i would like to hear it. > > > > thanks. > > > > Quoting Patrice Empeigne : > > > >> Hi Mark, > >> I have considered flow-report but there is a need for granularity, > >> thus storage of all data. > >> This need for granularity is from the incorporation of an > >> investigation tool that allows, for > >> example, to check if Chuck is streaming any porn. ;) Therefore the > >> approach, for performance, I > >> have taken is pretty much through summarisation in the data model as > >> suggested by Geoffrey > >> Bradford above and intelligent queries. > >> > >> Thanks Mark, > >> Pat > >> > >> -----Original Message----- > >> From: Mark Fullmer [mailto:maf@eng.oar.net] > >> Sent: Wednesday, 7 January 2004 5:41 PM > >> To: Patrice Empeigne > >> Cc: flow-tools@list.splintered.net > >> Subject: Re: [Flow-tools] Billing System > >> > >> > >> Have you considered using flow-report to summarize the data before > >> storing > >> it to mySQL? As an example lets say you're billing for every IP > >> address > >> in a /16. With summarized flows this is 65535 (inbound) or 65535*2 > >> (inbound+outbound) records per day vs potentially millions of records > >> per > >> day by storing the flows. > >> > >> mark > >> > >> On Jan 6, 2004, at 6:04 PM, Patrice Empeigne wrote: > >> > >>> Hi All, > >>> Im currently writing a billing system using flow-tools. > >>> Im piping all information from flow-capture straight into a mySQL db. > >>> Ive written the frontend and all works fine. > >>> What advice im looking for is in regards to performance, currently im > >>> storing everything in one table. > >>> Im currently changing that and creating a table per day to shorten > >>> the > >>> response times of queries. > >>> > >>> Before i do so, i was just wondering if anyone else has done > >>> something > >>> similar and could suggest a data model > >>> or even other hints for performance. > >>> > >>> Thanks team > >>> > >>> Pat > >>> _______________________________________________ > >>> Flow-tools mailing list > >>> flow-tools@splintered.net > >>> http://mailman.splintered.net/mailman/listinfo/flow-tools > >> > >> _______________________________________________ > >> Flow-tools mailing list > >> flow-tools@splintered.net > >> http://mailman.splintered.net/mailman/listinfo/flow-tools > >> > > > > > > > > ----------------------------------------------- > > William Emmanuel S. Yu > > Ateneo Campus Network Group (AteneoCNG) > > email : wyu at ateneo dot edu > > web : http://CNG.ateneo.net/cng/wyu/ > > phone : +63(2)4266001-4186 > > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp > > > > _______________________________________________ > > Flow-tools mailing list > > flow-tools@splintered.net > > http://mailman.splintered.net/mailman/listinfo/flow-tools > > ----------------------------------------------- William Emmanuel S. Yu Ateneo Campus Network Group (AteneoCNG) email : wyu at ateneo dot edu web : http://CNG.ateneo.net/cng/wyu/ phone : +63(2)4266001-4186 GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp From splintered-flow-tools-owner@splintered.net Thu Jan 08 00:39:33 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 65605 invoked by uid 4001); 8 Jan 2004 00:39:33 -0000 Delivered-To: splintered-flow-tools@splintered.net Received: (qmail 65603 invoked by alias); 8 Jan 2004 00:39:32 -0000 Received: from mail.ateneo.edu (202.138.180.10) by 66.250.216.131 with SMTP; 8 Jan 2004 00:39:32 -0000 Received: from mail.ateneo.edu (localhost [127.0.0.1]) by localhost (Postfix) with ESMTP id 3FD123EBE; Thu, 8 Jan 2004 08:39:31 +0800 (PHT) Received: by mail.ateneo.edu (Postfix, from userid 48) id 29FC53EB3; Thu, 8 Jan 2004 08:39:31 +0800 (PHT) Received: from 203.82.45.150 ([203.82.45.150]) by mail.ateneo.edu (IMP) with HTTP for ; Thu, 8 Jan 2004 08:39:31 +0800 Message-ID: <1073522371.3ffca6c3114a8@mail.ateneo.edu> Date: Thu, 8 Jan 2004 08:39:31 +0800 From: "Horatio B. Bogbindero" To: jeff vier Subject: Re: [Flow-tools] Flow-tools 0.67 RPM available References: <1073462078.3ffbbb3e6d55c@mail.ateneo.edu> <1073488567.4513.77.camel@localhost> In-Reply-To: <1073488567.4513.77.camel@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.2 Cc: flow-tools@splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jan 2004 00:39:33 -0000 Quoting jeff vier : > Is there no source tarball? > you can grab the SRPM from the link below. or you can download the "official" sources from Mark's site at http://splintered.net/sw/flow-tools/ have a good day! > On Wed, 2004-01-07 at 01:54, Horatio B. Bogbindero wrote: > > i would just like to announce that the flow-tools 0.67 is already available. it can be > > downloaded at the usual place http://cng.ateneo.net/cng/wyu/software/flow-tools.php. > > ----------------------------------------------- William Emmanuel S. Yu Ateneo Campus Network Group (AteneoCNG) email : wyu at ateneo dot edu web : http://CNG.ateneo.net/cng/wyu/ phone : +63(2)4266001-4186 GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp From wyu@ateneo.edu Thu Jan 08 00:44:56 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 66678 invoked by alias); 8 Jan 2004 00:44:55 -0000 Received: from mail.ateneo.edu (202.138.180.10) by 66.250.216.131 with SMTP; 8 Jan 2004 00:44:55 -0000 Received: from mail.ateneo.edu (localhost [127.0.0.1]) by localhost (Postfix) with ESMTP id 029983EBE; Thu, 8 Jan 2004 08:44:54 +0800 (PHT) Received: by mail.ateneo.edu (Postfix, from userid 48) id DFEAC3EB3; Thu, 8 Jan 2004 08:44:53 +0800 (PHT) Received: from 203.82.45.150 ([203.82.45.150]) by mail.ateneo.edu (IMP) with HTTP for ; Thu, 8 Jan 2004 08:44:53 +0800 Message-ID: <1073522693.3ffca805c5c21@mail.ateneo.edu> Date: Thu, 8 Jan 2004 08:44:53 +0800 From: "Horatio B. Bogbindero" To: John Wong Subject: RE: [Flow-tools] flow-capture & flow-fanout with filer option References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.2 Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jan 2004 00:44:56 -0000 Quoting John Wong : > Hi, > > Finally got it to work. Looks like it works on version 0.67. > I did a diff of flow-fanout on version 0.66 vs 0.67. A chunk > of code on filtering seems to be missing in 0.66. I guess that's > the problem. Thanks for your help... flow-tools rocks... > yup. that seems to be the case. i was looking at the 0.66 code when trying to troubleshoot the problem. i checked 0.67 this morning and saw that the code was already included. somebody must have reported it before as a bug and it got fixed. have a nice day! > > > > -----Original Message----- > > From: Horatio B. Bogbindero [mailto:wyu@ateneo.edu] > > Sent: Wednesday, January 07, 2004 6:16 PM > > To: John Wong > > Cc: flow-tools@list.splintered.net > > Subject: RE: [Flow-tools] flow-capture & flow-fanout with filer option > > > > > > Quoting John Wong : > > > > > Hi, > > > > > > There wasn't any errors, just that it didn't do > > > what i thought it was supposed to. You're right > > > that if i do not specify any definition it defaults > > > to not filtering. The manpages need to be updated. > > > > > > I tried explicitly setting the "-F" option again > > > to flow-fanout but am getting the same result. I > > > supposed it will be the same if I set it for > > > flow-capture too. I tried setting "-d 1" but where > > > does the logs go to? I don't see any at the stdout. > > > > > for flow-fanout, there is no code that invokes flow-filter > > at all. hmmm. something is not right here. > > > > well, i will check it out tomorrow. need to get home now. > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Horatio B. Bogbindero [mailto:wyu@ateneo.edu] > > > > Sent: Wednesday, January 07, 2004 5:18 PM > > > > To: John Wong > > > > Cc: flow-tools@list.splintered.net > > > > Subject: RE: [Flow-tools] flow-capture & flow-fanout with > > filer option > > > > > > > > > > > > Quoting John Wong : > > > > > > > > > Hi, > > > > > > > > > > I thought the "default" definition is the one > > > > > to be used if i do not specify the "-F" option. > > > > > Anyway, I did try putting in a specific "-F" > > > > > option but got the same results. > > > > > > > > > oh? anyway, i checked the flow-capture and if you > > > > do not specify the '-F' arguement explicity > > > > the filtering code is not invoked at all. > > > > > > > > > The thing is, when i used flow-nfilter with the > > > > > same filter file & definition, i get the correct > > > > > result i.e. only interfaces matching ifindex 3. > > > > > > > > > > So i figure it could be something with flow-capture > > > > > or flow-fanout that i'm missing. > > > > > > > > > probably. i will have to take a closer look first. > > > > can you email me the error message return? or there is > > > > none? try running it again with a higher debug level. > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Horatio B. Bogbindero [mailto:wyu@ateneo.edu] > > > > > > Sent: Wednesday, January 07, 2004 4:46 PM > > > > > > To: John Wong > > > > > > Cc: flow-tools@list.splintered.net > > > > > > Subject: Re: [Flow-tools] flow-capture & flow-fanout with > > > > filer option > > > > > > > > > > > > > > > > > > Quoting John Wong : > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I'm trying to configure flow-capture to only capture > > > > > > > to file netflows with specific SNMP ifindex. From the > > > > > > > router, i'm exporting the flows as V5. I run flow-capture > > > > > > > as follow :- > > > > > > > > > > > > > > /opt/flow-tools/bin/flow-capture -p- \ > > > > > > > -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ > > > > > > > x.x.x.x/y.y.y.y/2055 > > > > > > > > > > > > > /opt/flow-tools/bin/flow-capture -p- \ > > > > > > -w /opt/NetFlow -f /opt/flow-tools/etc/filter \ > > > > > > x.x.x.x/y.y.y.y/2055 -F default > > > > > > > > > > > > you forgot to tell flow-capture which filter definition > > > > > > to use. > > > > > > > > > > > > > Content of /opt/flow-tools/etc/filter :- > > > > > > > > > > > > > > ----------- BEGIN ------------------ > > > > > > > filter-primitive if1 > > > > > > > type ifindex > > > > > > > permit 3 > > > > > > > > > > > > > > filter-definition default > > > > > > > match input-interface if1 > > > > > > > or > > > > > > > match output-interface if1 > > > > > > > ----------- END ------------------ > > > > > > > > > > > > > > Somehow, i am still getting flows from other interfaces > > > > > > > on that router. Any idea if what i want can be done and > > > > > > > what is the purpose of the "-f" option for flow-capture > > > > > > > and flow-fanout if it doesn't? > > > > > > > > > > > > > > Thanks alot. > > > > > > > _______________________________________________ > > > > > > > Flow-tools mailing list > > > > > > > flow-tools@splintered.net > > > > > > > http://mailman.splintered.net/mailman/listinfo/flow-tools > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ----------------------------------------------- > > > > > > William Emmanuel S. Yu > > > > > > Ateneo Campus Network Group (AteneoCNG) > > > > > > email : wyu at ateneo dot edu > > > > > > web : http://CNG.ateneo.net/cng/wyu/ > > > > > > phone : +63(2)4266001-4186 > > > > > > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ----------------------------------------------- > > > > William Emmanuel S. Yu > > > > Ateneo Campus Network Group (AteneoCNG) > > > > email : wyu at ateneo dot edu > > > > web : http://CNG.ateneo.net/cng/wyu/ > > > > phone : +63(2)4266001-4186 > > > > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp > > > > > > > > > > > > > > > > > > > ----------------------------------------------- > > William Emmanuel S. Yu > > Ateneo Campus Network Group (AteneoCNG) > > email : wyu at ateneo dot edu > > web : http://CNG.ateneo.net/cng/wyu/ > > phone : +63(2)4266001-4186 > > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp > > > > > ----------------------------------------------- William Emmanuel S. Yu Ateneo Campus Network Group (AteneoCNG) email : wyu at ateneo dot edu web : http://CNG.ateneo.net/cng/wyu/ phone : +63(2)4266001-4186 GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp From anang@csmcom.com Thu Jan 08 03:07:44 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 68587 invoked by alias); 8 Jan 2004 03:07:43 -0000 Received: from 202-127-99-4.triplegate.net.id (HELO oracle) (202.127.99.4) by 66.250.216.131 with SMTP; 8 Jan 2004 03:07:43 -0000 Received: (qmail 11455 invoked by uid 105); 8 Jan 2004 03:09:17 -0000 Received: from anang@csmcom.com by oracle by uid 118 with qmail-scanner-1.15 (avpdaemon: ???. spamassassin: 2.43. Clear:. Processed in 0.105822 secs); 08 Jan 2004 03:09:17 -0000 Received: from csmwks-52-9.csmcom.com (HELO csmcom.com) (172.18.52.9) by oracle with SMTP; 8 Jan 2004 03:09:17 -0000 Message-ID: <3FFCC96C.5040004@csmcom.com> Date: Thu, 08 Jan 2004 10:07:24 +0700 From: Anang Syarifudin Organization: PT. CItra Sari Makmur User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031202 X-Accept-Language: en-us, en MIME-Version: 1.0 To: lorins@assist.ro Subject: Re: [Flow-tools] Billing System References: <52E510BAD83DAD4BB429B4C53C269FD2064BC8@NT2.corporate.ensyst.com.au> <200401071550.30784.lorins@assist.ro> In-Reply-To: <200401071550.30784.lorins@assist.ro> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jan 2004 03:07:44 -0000 >Hi ! > Well i pipe the data into a mysql database using flow-export and i use >2 temporary _Heap_ tables for speed. Table flow_import, which is continously >feed with netflow data and table flow, which contains flow information for >the last 5 minutes. I use a little trick to rotate those 2 from a script >every 5 minutes so i can have small tables, easy to process: > >mysql -h localhost -u root -p"blabla" -n -N -B -q accounting <delete from flow; >alter table flow rename flow_tmp; >alter table flow_import rename flow; >alter table flow_tmp rename flow_import; >QUERY_INPUT > >I capture the flows like this: >/usr/local/netflow/bin/flow-receive 0/0/13000 | >/usr/local/netflow/bin/flow-export -m 0x0000000000383060LL -f3 -u >root:blabla:localhost:3306:accounting:flow_import > > So for billing you could use a third table (non Heap), i named it >clients, >which gets it's data from table flow and it's relatively small. My clients >table has the following fields: name, ip, packets, bytes, download_rate_base, >download_rate_peak, upload_rate_base, upload_rate_peak, quantity, time_range >which are self explanatory (i think;). But for around 150 clients i currently >monitor on a p4,1.4GHz,256 RAM, i sometimes get "no more connections allowed" >error... I also run a apache server, and draw rrd graphs (150 graphs) so perl >or c would do a better job than slow bash scripts. >Hope i helped! > >p.s. Please excuse my bad english... > > thanks, I'll try I do optimisation as directed by these links : http://www.linux-mag.com/2001-12/mysql_01.html http://www.linux-mag.com/2001-06/mysql_01.html p.s. my english is worse than yours Anang From rheber@nawias.pl Thu Jan 08 09:56:02 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 72041 invoked by alias); 8 Jan 2004 09:56:01 -0000 Received: from kwadratowy.nawias.pl (62.111.251.94) by 66.250.216.131 with SMTP; 8 Jan 2004 09:56:01 -0000 Received: (qmail 26769 invoked from network); 8 Jan 2004 10:04:35 -0000 Received: from host254.webinn.pl (HELO nawias.pl) (robert@193.108.34.254) by kwadratowy.nawias.pl with SMTP; 8 Jan 2004 10:04:35 -0000 Message-ID: <3FFD2945.6020507@nawias.pl> Date: Thu, 08 Jan 2004 10:56:21 +0100 From: Robert Heber User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; PL; rv:1.4) Gecko/20030624 X-Accept-Language: pl, en-us, en MIME-Version: 1.0 To: flow-tools@list.splintered.net Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: [Flow-tools] Collector for Windows machine X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jan 2004 09:56:02 -0000 Hi, is there way to run flow-tools collector on Windows-based servers ? My question is dictate by other monitoring tools which demand this kind of OS, and the fact that customer agree to host only one server. I thought about VMvare or Cygwin - but I look for proven solution - do you have any suggestion for this case ? Best regards, Robert From tunde.adebayo@email.matrixng.com Fri Jan 09 14:13:21 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 80978 invoked by alias); 9 Jan 2004 14:13:20 -0000 Received: from host-64-110-79-50.interpacket.net (HELO email.matrixng.com) (64.110.79.50) by 66.250.216.131 with SMTP; 9 Jan 2004 14:13:20 -0000 Received: from email.matrixng.com (localhost [127.0.0.1]) by email.matrixng.com (8.12.8/8.12.5) with ESMTP id i09EEe87021839 for ; Fri, 9 Jan 2004 15:14:40 +0100 From: "tunde.adebayo" To: flow-tools@list.splintered.net Date: Fri, 9 Jan 2004 09:14:40 -0500 Message-Id: <20040109140808.M81471@email.matrixng.com> In-Reply-To: References: X-Mailer: Open WebMail 1.90 20031127 X-OriginatingIP: 64.110.79.59 (tunde.adebayo) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Subject: [Flow-tools] Ever sleeping flowscan X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2004 14:13:21 -0000 Hi Guys, i am a newbie to flow-tools i got everything installed and yet all my flowscan does is to "sleep 30..." it is yet to do any processing. tcpdump shows me udp packets are entering from my router,netstat -lnp told me my flow-capturing is good,and there are ft-v05* files(rapidly growing!)in the /var/netflow/ft folder. Please help in directing me to what i am missing out. Many Thanks ====================== Babatunde Adebayo Matrix Control Ltd. Tel:234 1 4971440 Mobile:234 802 3243728 ----------------------------------------- "Imagination is more important than knowledge" (A. Einstein) From liao1k@cmich.edu Fri Jan 09 14:26:02 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 81751 invoked by alias); 9 Jan 2004 14:26:02 -0000 Received: from garm.csv.cmich.edu (HELO garm.central.cmich.local) (141.209.15.48) by 66.250.216.131 with SMTP; 9 Jan 2004 14:26:02 -0000 Received: from cmail3.central.cmich.local ([141.209.15.83]) by egate1.central.cmich.local with Microsoft SMTPSVC(5.0.2195.6713); Fri, 9 Jan 2004 09:25:57 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPartTM-000-03e9b7a2-0c65-4340-a360-78c06688d9b5" Date: Fri, 9 Jan 2004 09:25:57 -0500 Message-ID: <291B348BC59B47468C7824603C326082216849@cmail3.central.cmich.local> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: TCP/UDP Packet size Thread-Index: AcPWvIm/ZXwGHardSt6wvZLod378dA== From: "Liao, Kexiao" To: Return-Path: liao1k@cmich.edu X-OriginalArrivalTime: 09 Jan 2004 14:25:57.0422 (UTC) FILETIME=[7F496CE0:01C3D6BC] Subject: [Flow-tools] TCP/UDP Packet size X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2004 14:26:03 -0000 This is a multi-part message in MIME format. ------=_NextPartTM-000-03e9b7a2-0c65-4340-a360-78c06688d9b5 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C3D6BC.7F24B949" ------_=_NextPart_001_01C3D6BC.7F24B949 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, Can I use the product of dpkts and doctets as the size of current TCP/UDP packet byte length? Thanks. =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Kexiao Liao=20 CMU Research Corporation=20 2625 Denison Dr. Mount Pleasant, MI 48858=20 Phone 989-774-2424 , Fax 989-774-2416 http://www.thecenter.cmich.edu/=20 liao1k@cmich.edu =20 =20 ------_=_NextPart_001_01C3D6BC.7F24B949 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi,

   Can I use the product of dpkts and = doctets as the size of current TCP/UDP packet byte length? = Thanks.

 

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D

Kexiao Liao

CMU Research Corporation

2625 Denison Dr.

Mount Pleasant, MI 48858

Phone 989-774-2424 , Fax 989-774-2416

http://www.thecenter.cmich.edu/<= /a>

liao1k@cmich.edu

 

 

=00 ------_=_NextPart_001_01C3D6BC.7F24B949-- ------=_NextPartTM-000-03e9b7a2-0c65-4340-a360-78c06688d9b5-- From awebster@illinois.net Fri Jan 09 17:00:01 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 83243 invoked by alias); 9 Jan 2004 17:00:01 -0000 Received: from mailgw.illinois.net (HELO mailgw.linc2icn.net) (206.166.96.53) by 66.250.216.131 with SMTP; 9 Jan 2004 17:00:01 -0000 Received: from ASSP-nospam.illinois.net ([206.166.96.53]) by mailgw.linc2icn.net with Microsoft SMTPSVC(5.0.2195.6713); Fri, 9 Jan 2004 11:00:01 -0600 Received: from 206.166.96.50 ([206.166.96.50] helo=MAILSERV.linc2icn.net) by ASSP-nospam.illinois.net ; 9 Jan 04 17:00:01 -0000 X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Fri, 9 Jan 2004 11:00:00 -0600 Message-ID: <7CD4CD9D537C294D9ED9E5CE2F01910601E7369E@MAILSERV.linc2icn.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: flow-report questions about port report Thread-Index: AcPW0gRTgt2BH4ceTjKxnemlDdMDEg== From: "Andy Webster" To: Return-Path: awebster@illinois.net X-OriginalArrivalTime: 09 Jan 2004 17:00:01.0082 (UTC) FILETIME=[04F001A0:01C3D6D2] Subject: [Flow-tools] flow-report questions about port report X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2004 17:00:02 -0000 hi, my apologies if my questions are answered in the docs somewhere. When I look at my ip-port flow-report, it looks like the values for the flows,octets,packets, & duration columns are listed as percentages, am I correct? =20 Also, can anyone shed some light on this entry in my report? 0,70.822172,5.261527,23.158241,1.548590 Based on the huge number of flows & small number of octets I guess this is ICMP? My question is what will show up listed as port 0 in this report? thanks, Andy $ more port-all=20 # --- ---- ---- Report Information --- --- --- # build-version: flow-tools 0.66 # name: port # type: ip-port # options: +percent-total,+names,+header,+xheader,+totals # sort_field: +packets # fields: +key,+flows,+octets,+packets,+duration,+other # pre-filter: port # records: 57571 # first-flow: 730954248 Sun Feb 28 20:50:48 1993 # last-flow: 1073244516 Sun Jan 4 13:28:36 2004 # now: 1073665748 Fri Jan 9 10:29:08 2004 # # mode: streaming # compress: off # byte order: little # stream version: 3 # export version: 5 # # rec1: ignores,flows,octets,packets,duration 0,7294018,10000414114,24893160,6860989828 # recn: ip-port,flows,octets,packets,duration http,8.265636,49.300643,30.233783,18.451914 0,70.822172,5.261527,23.158241,1.548590 dls-monitor,67.068165,4.971677,21.689167,0.536616 loc-srv,5.593419,1.007924,8.113574,5.071544 6667,4.835360,0.615305,4.870691,7.596625 5049,0.001508,11.192760,4.732842,0.070990 domain,1.958865,0.539575,1.781851,11.969300 6881,0.114011,3.259280,1.753048,3.078963 filter-primitive ICMP type ip-protocol permit 1 default deny filter-primitive TCP type ip-protocol permit TCP default deny filter-primitive UDP type ip-protocol permit 17 default deny filter-definition port match ip-protocol TCP or match ip-protocol UDP or=20 match ip-protocol ICMP From ellidz@eridu.uchicago.edu Fri Jan 09 17:05:22 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 84023 invoked by alias); 9 Jan 2004 17:05:22 -0000 Received: from eridu.uchicago.edu (128.135.0.118) by 66.250.216.131 with SMTP; 9 Jan 2004 17:05:22 -0000 Received: from localhost (localhost [127.0.0.1]) by eridu.uchicago.edu (Postfix) with ESMTP id 13B011AD for ; Fri, 9 Jan 2004 11:05:05 -0600 (CST) Received: from eridu.uchicago.edu ([127.0.0.1]) by localhost (eridu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 06330-02 for ; Fri, 9 Jan 2004 11:05:04 -0600 (CST) Received: from eridu.uchicago.edu (localhost [127.0.0.1]) by eridu.uchicago.edu (Postfix) with ESMTP id D96E91AC for ; Fri, 9 Jan 2004 11:05:04 -0600 (CST) From: "E. Larry Lidz" To: flow-tools@list.splintered.net Date: Fri, 09 Jan 2004 11:05:04 -0600 Sender: ellidz@eridu.uchicago.edu Message-Id: <20040109170504.D96E91AC@eridu.uchicago.edu> Subject: [Flow-tools] c6509s, msfcs, hybrid mode and flows X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2004 17:05:22 -0000 Hello, Does anyone have any experience sending flows from a 6509 running in hybrid mode? We're wanting to capture all of the flows through the device, and we're not sure whether we'll need to send it only from the 6509 itself or if we'll need to send them from the MSFC to get all of the flows. If we do need to send it from the MSFC it appears that we'll likely have an issue as we would like to aggregate the flows and flow-merge won't merge disparate flow versions... Any suggestions/advice? -Larry --- E. Larry Lidz Phone: +1 773 702-2208 Sr. Network Security Officer Fax: +1 773 834-8444 Network Security Center, The University of Chicago PGP: http://security.uchicago.edu/centerinfo/pgpkeys.shtml From mhunter@ack.Berkeley.EDU Fri Jan 09 17:35:21 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 84957 invoked by alias); 9 Jan 2004 17:35:21 -0000 Received: from ack.berkeley.edu (128.32.206.66) by 66.250.216.131 with SMTP; 9 Jan 2004 17:35:21 -0000 Received: (from mhunter@localhost) by ack.Berkeley.EDU (8.11.3/8.11.3) id i09HZDH28026; Fri, 9 Jan 2004 09:35:13 -0800 (PST) Date: Fri, 9 Jan 2004 09:35:13 -0800 From: Mike Hunter To: "E. Larry Lidz" Subject: Re: [Flow-tools] c6509s, msfcs, hybrid mode and flows Message-ID: <20040109173512.GA27553@ack.Berkeley.EDU> References: <20040109170504.D96E91AC@eridu.uchicago.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20040109170504.D96E91AC@eridu.uchicago.edu> User-Agent: Mutt/1.4i Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2004 17:35:21 -0000 On Jan 09, "E. Larry Lidz" wrote: > > Hello, > > Does anyone have any experience sending flows from a 6509 running in > hybrid mode? We're wanting to capture all of the flows through the > device, and we're not sure whether we'll need to send it only from the > 6509 itself or if we'll need to send them from the MSFC to get all of > the flows. > > If we do need to send it from the MSFC it appears that we'll likely > have an issue as we would like to aggregate the flows and flow-merge > won't merge disparate flow versions... > > Any suggestions/advice? Did you see this in the flow-capture manpage? ---- Multiple non aggregated PDU ver- sions may be accepted at once to support Cisco's Catalyst 6500 NetFlow implementation which exports from both the supervisor and MSFC with the same IP address and same port but different export versions. In this case the exports will be stored in the format specified by pdu_version or whichever export type is received first. ---- Mike From rheber@nawias.pl Fri Jan 09 20:17:54 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 86557 invoked by alias); 9 Jan 2004 20:17:54 -0000 Received: from kwadratowy.nawias.pl (62.111.251.94) by 66.250.216.131 with SMTP; 9 Jan 2004 20:17:54 -0000 Received: (qmail 29944 invoked from network); 9 Jan 2004 20:26:56 -0000 Received: from wi162.internetdsl.tpnet.pl (HELO nawias.pl) (robert@80.55.190.162) by kwadratowy.nawias.pl with SMTP; 9 Jan 2004 20:26:56 -0000 Message-ID: <3FFF0C8B.6070508@nawias.pl> Date: Fri, 09 Jan 2004 21:18:19 +0100 From: Robert Heber User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; PL; rv:1.4) Gecko/20030624 X-Accept-Language: pl, en-us, en MIME-Version: 1.0 To: "Liao, Kexiao" Subject: Re: [Flow-tools] TCP/UDP Packet size References: <291B348BC59B47468C7824603C326082216849@cmail3.central.cmich.local> In-Reply-To: <291B348BC59B47468C7824603C326082216849@cmail3.central.cmich.local> Content-Type: text/plain; charset=ISO-8859-2; format=flowed Content-Transfer-Encoding: 8bit Cc: flow-tools@list.splintered.net X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2004 20:17:54 -0000 Hello, maybe I'm wrong but as far as I understand a flow has dpkts number of packets and doctets of summary lenght of bytes, so you do not have to times dpkts by doctets. The value you are looking for is doctets itself. Regards, Robert Użytkownik Liao, Kexiao napisał: > Hi, > > Can I use the product of dpkts and doctets as the size of current > TCP/UDP packet byte length? Thanks. > > > > =========================== > > Kexiao Liao > > CMU Research Corporation > > 2625 Denison Dr. > > Mount Pleasant, MI 48858 > > Phone 989-774-2424 , Fax 989-774-2416 > > http://www.thecenter.cmich.edu/ > > liao1k@cmich.edu > > > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools From lsharpe@pacificwireless.com.au Sun Jan 11 22:12:45 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 182 invoked by alias); 11 Jan 2004 22:12:44 -0000 Received: from ns1.pacificwireless.com.au (203.166.40.26) by 66.250.216.131 with SMTP; 11 Jan 2004 22:12:44 -0000 Received: from gandalf ([172.16.1.29]) by ns1.pacificwireless.com.au (8.11.6/8.11.6) with SMTP id i0BJsuY11048; Mon, 12 Jan 2004 06:54:57 +1100 Message-ID: <000601c3d88f$fc733f00$1d0110ac@pacwire.local> From: "Leigh Sharpe" To: "tunde.adebayo" , References: <20040109140808.M81471@email.matrixng.com> Subject: Re: [Flow-tools] Ever sleeping flowscan Date: Mon, 12 Jan 2004 09:12:20 +1100 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2600.0000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Cc: X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jan 2004 22:12:45 -0000 Check the flowFileGlob directive in your flowscan configuration. Try the flowscan list. ----- Original Message ----- From: "tunde.adebayo" To: Sent: Saturday, January 10, 2004 1:14 AM Subject: [Flow-tools] Ever sleeping flowscan > Hi Guys, > i am a newbie to flow-tools i got everything installed and yet all my > flowscan does is to "sleep 30..." it is yet to do any processing. > tcpdump shows me udp packets are entering from my router,netstat -lnp told me > my flow-capturing is good,and there are ft-v05* files(rapidly growing!)in > the /var/netflow/ft folder. > Please help in directing me to what i am missing out. > > Many Thanks > ====================== > Babatunde Adebayo > Matrix Control Ltd. > Tel:234 1 4971440 > Mobile:234 802 3243728 > ----------------------------------------- > "Imagination is more important than knowledge" (A. Einstein) > > > > _______________________________________________ > Flow-tools mailing list > flow-tools@splintered.net > http://mailman.splintered.net/mailman/listinfo/flow-tools > From afort@choqolat.org Mon Jan 12 13:55:14 2004 Return-Path: Delivered-To: splinteredlist-flow-tools@list.splintered.net Received: (qmail 4641 invoked by alias); 12 Jan 2004 13:55:14 -0000 Received: from gizmo03ps.bigpond.com (144.140.71.13) by 66.250.216.131 with SMTP; 12 Jan 2004 13:55:14 -0000 Received: (qmail 3912 invoked from network); 12 Jan 2004 13:52:33 -0000 Received: from unknown (HELO psmam01.bigpond.com) (144.135.25.69) by gizmo03ps.bigpond.com with SMTP; 12 Jan 2004 13:52:33 -0000 Received: from cpe-144-132-104-134.vic.bigpond.net.au ([144.132.104.134]) by psmam01.bigpond.com(MAM REL_3_4_2 71/18166611) with SMTP id 18166611; Mon, 12 Jan 2004 23:55:11 +1000 Message-ID: <4002A73E.6000502@choqolat.org> Date: Tue, 13 Jan 2004 00:55:10 +1100 From: Andrew Fort User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.5) Gecko/20031013 Thunderbird/0.3 X-Accept-Language: en-us, en MIME-Version: 1.0 CC: flow-tools@list.splintered.net Subject: Re: [Flow-tools] c6509s, msfcs, hybrid mode and flows References: <20040109170504.D96E91AC@eridu.uchicago.edu> <20040109173512.GA27553@ack.Berkeley.EDU> In-Reply-To: <20040109173512.GA27553@ack.Berkeley.EDU> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: flow-tools@list.splintered.net X-Mailman-Version: 2.1.3 Precedence: list List-Id: Discussion of flow-tools software List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jan 2004 13:55:15 -0000 Mike Hunter wrote: >On Jan 09, "E. Larry Lidz" wrote: > > > >>Hello, >> >>Does anyone have any experience sending flows from a 6509 running in >>hybrid mode? We're wanting to capture all of the flows through the >>device, and we're not sure whether we'll need to send it only from the >>6509 itself or if we'll need to send them from the MSFC to get all of >>the flows. >> >>If we do need to send it from the MSFC it appears that we'll likely >>have an issue as we would like to aggregate the flows and flow-merge >>won't merge disparate flow versions... >> >>Any suggestions/advice? >> >> > >Did you see this in the flow-capture manpage? > >---- >Multiple non aggregated PDU ver- sions may be accepted at once to support >Cisco's Catalyst 6500 NetFlow implementation which exports from both the >supervisor and MSFC with the same IP address and same port but different >export versions. In this case the exports will be stored in the format >specified by pdu_version or whichever export type is received first. >---- > >Mike > > This feature works (I remember this feature fondly back in the days I was exporting flows from Sup1a 6509's - thanks Mark!), but it requires the exporter PFC/Supervisor to export with the same source IP address as the MSFC, which may not be the case in Hybrid mode software (it is in Native mode). I think in later relases of CatOS this is configurable, but I cannot speak authoritatively there. As to whether you need flows from both the MSFC and the PFC(Supervisor), it depends on your Supervisor's architecture. Supervisor 1/1A has a MLS switching cache, so all flows will hit the MSFC (usually a flow export of 1 packet which is the flow which populates the PFC's MLS switching cache). The remaining packets in the flow (if handled by the PFC) will be exported from the PFC as a separate flow with the same 4-tuple of header info (in PDU version 7 as opposed to 1, 5 or 6 from the MSFC). If you have a Supervisor 2, you can probably do without the MSFC flows - the flows exported by the MSFC are only those handled by the software (e.g. ACL entries requiring a "log" keyword, or other things the PFC hardware cannot natively CEF switch without MSFC "IP Input" process). If you have a Supervisor 720/720-3BXL, the situation is very similar to the Sup2. Both these architectures have a Netflow TCAM which is used to store the recent flows for data export (it's 128k entries on the Sup2 and Sup720, and 256k entries on the Sup720-3BXL). When this cache overflows, netflow data is not exported to the MSFC for UDP packet generation and export, so be careful to watch your cache size if you rely on this data for something important (like billing). I don't know how the cache is expired (LRU or FIFO). Exports are done in the "router" format (v5) and the "switch" format (v7) is deprecated on Native IOS for this hardware platform. commands to check on this stuff on the Sup720s here are (ymmv, this command has changed between this release and the previous one). This box is working as an internet gateway (about 50% of an stm-1's worth of transit through it). agg1.mil100#sh mls netflow table detailed Earl in Module 1 Detailed Netflow CAM (TCAM and ICAM) Utilization ================================================ TCAM Utilization : 15% ICAM Utilization : 0% Netflow TCAM count : 20322 Netflow ICAM count : 0 Netflow Creation Failures : 0 Netflow CAM aliases : 0 Earl in Module 5 Detailed Netflow CAM (TCAM and ICAM) Utilization ================================================ TCAM Utilization : 81% ICAM Utilization : 0% Netflow TCAM count : 107127 Netflow ICAM count : 0 Netflow Creation Failures : 0 Netflow CAM aliases : 0 agg1.mil100#sh mls netflow table agg Earl in Module 1 Aggregate Netflow CAM Contention Information ============================================= Netflow Creation Failures : 7691477 Netflow Hash Aliases : 2 Earl in Module 5 Aggregate Netflow CAM Contention Information ============================================= Netflow Creation Failures : 31010874 Netflow Hash Aliases : 0 Earl in Module 8 -afort From splintered-flow-tools-owner@splintered.net Tue Jan 13 0