#Ntopng export flows how to#
#Ntopng export flows driver#
#Ntopng export flows full#
#Ntopng export flows software#

ntopng allows you to create up to 32 virtual flow collection interfaces that can be used to avoid merging collected flows: unfortunately they are not enough when collecting flows from 100+ routers. Beside the number of flows to be collected, another key point is to be able to visualize the informations in a simple yet effective way. In a second follow-on article, a real-time dashboard for Network Flows data will be presented.Collecting flows on large networks with hundred of routers can be challenging. Databricks also have a relatively expensive but very good intensive online training program. It also provides a convenient JDBC connector out of the box.

Given the plethora of attributes in the NetFlow format provided by modern collectors it is easy to imagine the power of this approach in giving network visibility to the security teams, and in perspective as a foundation for an AIops system as well.ĭatabricks is a mature platform that removes the difficulties associated with integrating libraries and deploying a Spark application in a cluster.

#Ntopng export flows how to#

The examples above are representative of how to provide real-time detection, network traffic analysis, and data aggregates using a big data approach with streaming data. This proves to be very useful for NetFlows, and to provide visibility of network traffic particularly when used with modern flow collectors that generate metadata as result of deep-packets inspection, such as the nProbe collector from NTOP².

While this is a fairly small cluster, it was able to handle a sustained traffic of 15k flows/sec and to query more than 100 million flows in the data lake in less than 15 seconds.Īpache Spark has a steep learning curve, but it is a very powerful tool that allows writing pipelines that combine data ingestion, real-time analysis and data aggregation.

#Ntopng export flows driver#

The system used for data ingestion and analysis is composed of one or more nProbe flow collectors, a Kafka message broker in AWS, and a Spark cluster composed of one driver and four workers in Databricks, all r5.large EC2 instances with 122GB of RAM.

#Ntopng export flows software#

This posts describe an architecture and the software used to perform real-time analysis of network flows from a number of nProbe collectors, while storing data on a data lake in S3 in such a way to allow for fast retrieval. Modern network flow collectors, such as the highly performing and inexpensive nProbe from Italian company NTOP², can inspect packets as an inline tool and extract Layer-4 to Layer-7 metadata as network flows, that is where it counts, and at speed of up to 100 Gbit using commodity hardware. Network flows composed of only a few attributes are a thing of the past.

#Ntopng export flows full#

The metadata is carrying the information actually needed for analysis, while the full content of a packet payload may become relevant in forensic analysis, for instance to find out the extent of a breach. While packet capture has been the tool of choice for a long time, it is obvious that it is too costly to use as the preferred way of performing data analysis. These days it is not uncommon for enterprise networks to carry very large amount of data in the network traffic. Network visibility and analysis relies on data from the network.