Updated documention

robcowart · Feb 18, 2019 · 5d6ccad · 5d6ccad
1 parent edae8b3
commit 5d6ccad
Show file tree

Hide file tree

Showing 3 changed files with 52 additions and 13 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,24 @@
+## v3.4.0
+### Breaking Changes
+v3.4.0 adds custom field definitions for the Netflow codec. While greatly expanding the number of supported vendor-specific fields, many existing vendor-specific fields have been renamed. The ElastiFlow dashboards in previous releases were based on its normalized `flow` schema, or other standard Netflow and IPFIX fields, all of which are unchanged. However it may be necessary to update any Dashboards you created for the old vendor-specific field names to use the new names.
+
+### New Features
+- Add a new Threats dashboard, based on IP reputation tags
+- Netflow and IPFIX now default to included field definitions
+- Provide a `sysctl.d` file to set `net.core.rmem_max`
+- Added application ID support for Sophos, Sonicwall, Citrix Netscaler, IXIA IxFlow and Palo Alto
+- Added support for Ziften ZFlow IPFIX host agents
+- Added enrichment of enumerated values for many vendor-specific fields.
+
+### Updates
+- Updated GeoLite2-City and GeoLite2-ASN DBs
+- Updated IP Reputation dictionary
+- Set all `translate` filters to use the new option `refresh_behaviour`, setting it to `replace`
+- Updated FortiOS 5.6 Application IDs
+- Disabled name lookups for connections to the `tcp` input
+- Kibana index pattern now contains many new vendor-specific fields
+
+---
 ## v3.3.0
 v3.3.0 is a minor release. No migration of data from v3.0.x or later to v3.3.0 is required.
 

diff --git a/INSTALL.md b/INSTALL.md
@@ -1,5 +1,5 @@
 # ElastiFlow&trade; Installation
-[![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.me/robcowart) [![Tweet](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/intent/tweet?text=ElastiFlow%E2%84%A2%20provides%20Netflow%20v5%2Fv9%2C%20sFlow%20and%20IPFIX%20data%20collection%20and%20visualization%20using%20the%20Elastic%20Stack.&url=https://github.com/robcowart/elastiflow&hashtags=elastiflow,netflow,sflow,ipfix)
+[![patreon](https://user-images.githubusercontent.com/10326954/52966127-c9847680-33a6-11e9-8640-10dd7abc3af0.png)](https://www.patreon.com/elastiflow) [![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.me/robcowart) [![Tweet](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/intent/tweet?text=ElastiFlow%E2%84%A2%20provides%20Netflow%20v5%2Fv9%2C%20sFlow%20and%20IPFIX%20data%20collection%20and%20visualization%20using%20the%20Elastic%20Stack.&url=https://github.com/robcowart/elastiflow&hashtags=elastiflow,netflow,sflow,ipfix)
 
 ElastiFlow&trade; is built using the Elastic Stack, including Elasticsearch, Logstash and Kibana. To install and configure ElastiFlow&trade;, you must first have a working Elastic Stack environment.
 
@@ -9,6 +9,8 @@ Refer to the following compatibility chart to choose a release of ElastiFlow&tra
 
 Elastic Stack | ElastiFlow&trade; 1.x | ElastiFlow&trade; 2.x | ElastiFlow&trade; 3.x
 :---:|:---:|:---:|:---:
+6.6 |  |  | &#10003;
+6.5 |  |  | &#10003;
 6.4 |  |  | &#10003;
 6.3 |  |  | &#10003;
 6.2 | &#10003; | &#10003; | &#10003;
@@ -267,12 +269,18 @@ If using Netflow v9 or IPFIX you will likely see warning messages related to the
 Logstash setup is now complete. If you are receiving flow data, you should have an `elastiflow-` daily index in Elasticsearch.
 
 ## Setting up Kibana
+Kibana 6.5 introduced the ability to export and import Index Patterns through the UI. This greatly simplifies the setup of Kibana.
+
+### Kibana 6.5.x and Later
+The Index Patterns, vizualizations and dashboards can be loaded into Kibana by importing the `elastiflow.kibana.<VER>.json` file from within the Kibana UI. This is done from the `Management -> Saved Objects` page.
+
+### Kibana 6.4.x and Earlier
 An API (yet undocumented) is available to import and export Index Patterns. The JSON file which contains the Index Pattern configuration is `kibana/elastiflow.index_pattern.json`. To setup the `elastiflow-*` Index Pattern run the following command:
 ```
 curl -X POST -u USERNAME:PASSWORD http://KIBANASERVER:5601/api/saved_objects/index-pattern/elastiflow-* -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @/PATH/TO/elastiflow.index_pattern.json
 ```
 
-Finally the vizualizations and dashboards can be loaded into Kibana by importing the `elastiflow.dashboards.<VER>.json` file from within the Kibana UI. This is done from the Management - > Saved Objects page. There are separate dashboard import files for version 6.2.x, 6.3.x and 6.4.x of Kibana. Select the file that corresponds to your version of Kibana.
+Finally the vizualizations and dashboards can be loaded into Kibana by importing the `elastiflow.dashboards.<VER>.json` file from within the Kibana UI. This is done from the `Management -> Saved Objects` page. There are separate dashboard import files for version 6.2.x, 6.3.x and 6.4.x of Kibana. Select the file that corresponds to your version of Kibana.
 
 ### Recommended Kibana Advanced Settings
 You may find that modifying a few of the Kibana advanced settings will produce a more user-friendly experience while using ElastiFlow&trade;. These settings are made in Kibana, under `Management -> Advanced Settings`.

diff --git a/README.md b/README.md
@@ -1,9 +1,11 @@
 # ElastiFlow&trade;
-[![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.me/robcowart) [![Tweet](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/intent/tweet?text=ElastiFlow%E2%84%A2%20provides%20Netflow%20v5%2Fv9%2C%20sFlow%20and%20IPFIX%20data%20collection%20and%20visualization%20using%20the%20Elastic%20Stack.&url=https://github.com/robcowart/elastiflow&hashtags=elastiflow,netflow,sflow,ipfix)
+[![patreon](https://user-images.githubusercontent.com/10326954/52966127-c9847680-33a6-11e9-8640-10dd7abc3af0.png)](https://www.patreon.com/elastiflow) [![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.me/robcowart) [![Tweet](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/intent/tweet?text=ElastiFlow%E2%84%A2%20provides%20Netflow%20v5%2Fv9%2C%20sFlow%20and%20IPFIX%20data%20collection%20and%20visualization%20using%20the%20Elastic%20Stack.&url=https://github.com/robcowart/elastiflow&hashtags=elastiflow,netflow,sflow,ipfix)
+
+> SUPPORTING ElastiFlow&trade; - Today literally 1000s of users leverage ElastiFlow&trade; As a powerful alternative to expensive commercial flow collecting solutions. As its popularity has increased, so has the time commitment necessary to support users and provide further enhancments. If you are one of the organizations who appreciate the value of ElastiFlow&trade;, I would like to ask you to consider bcoming a sponsor. The support from sponsors allows me dedicate more time and energy to the project. To become a sponsor, please visit ElastiFlow on [![patreon](https://user-images.githubusercontent.com/10326954/52966127-c9847680-33a6-11e9-8640-10dd7abc3af0.png)](https://www.patreon.com/elastiflow).
 
 ElastiFlow&trade; provides network flow data collection and visualization using the Elastic Stack (Elasticsearch, Logstash and Kibana). It supports Netflow v5/v9, sFlow and IPFIX flow types (1.x versions support only Netflow v5/v9).
 
-![ElastiFlow&trade;](https://user-images.githubusercontent.com/10326954/39966506-0934e198-56ad-11e8-9f40-c6454b6c6ea7.png)
+![ElastiFlow&trade;](https://user-images.githubusercontent.com/10326954/52973891-f42ef900-33bf-11e9-8243-aed047decf3b.png)
 
 I was inspired to create ElastiFlow&trade; following the overwhelmingly positive feedback received to an article I posted on Linkedin... [WTFlow?! Are you really still paying for commercial solutions to collect and analyze network flow data?](https://www.linkedin.com/pulse/wtflow-you-really-still-paying-commercial-solutions-collect-cowart)
 
@@ -24,32 +26,40 @@ The following dashboards are provided.
 > NOTE: The dashboards are optimized for a monitor resolution of 1920x1080.
 
 ### Overview
-![Overview](https://user-images.githubusercontent.com/10326954/39966471-9b0a40dc-56ac-11e8-8962-78b928c7971f.png)
+![Overview](https://user-images.githubusercontent.com/10326954/52973920-158fe500-33c0-11e9-96ed-606c01aca7c8.png)
 
 ### Top-N
 There are separate Top-N dashboards for Top Talkers, Services, Conversations and Applciations.
-![Top-N](https://user-images.githubusercontent.com/10326954/39966477-b52ee92c-56ac-11e8-84eb-4688ddff7754.png)
+![Top-N](https://user-images.githubusercontent.com/10326954/52973927-19bc0280-33c0-11e9-9352-76c483738c24.png)
+
+### Threats
+The Threats dashboard uses IP Reputation tags to help you identify possible threats and risky traffic.
+![Threats](https://user-images.githubusercontent.com/10326954/52973930-1c1e5c80-33c0-11e9-8aa8-87252461336c.png)
 
-### Sankey
+### Flows
 There are separate Sankey dashboards for Client/Server, Source/Destination and Autonomous System perspectives. The sankey visualizations are built using the new Vega visualization plugin.
-![Sankey](https://user-images.githubusercontent.com/10326954/39966483-c14a3aa4-56ac-11e8-9319-a56b2bf60d9f.png)
+![Flows](https://user-images.githubusercontent.com/10326954/52973933-204a7a00-33c0-11e9-91d8-7b194bd978eb.png)
 
 ### Geo IP
 There are separate Geo Loacation dashboards for Client/Server and Source/Destination perspectives.
-![Geo IP](https://user-images.githubusercontent.com/10326954/39966487-cd06acf6-56ac-11e8-9da7-1bff5e822d8d.png)
+![Geo IP](https://user-images.githubusercontent.com/10326954/52973940-27718800-33c0-11e9-88d9-466396e080e6.png)
 
 ### AS Traffic
 Provides a view of traffic to and from Autonomous Systems (public IP ranges)
-![AS Traffic](https://user-images.githubusercontent.com/10326954/39966490-d8d6032e-56ac-11e8-8784-b9903855d4f3.png)
+![AS Traffic](https://user-images.githubusercontent.com/10326954/52973944-2cced280-33c0-11e9-9e95-e2f17fbb7ea6.png)
 
 ### Exporters
-![Flow Exporters](https://user-images.githubusercontent.com/10326954/39966495-e42c14f2-56ac-11e8-8c0e-b4275bfb32eb.png)
+![Flow Exporters](https://user-images.githubusercontent.com/10326954/52973950-322c1d00-33c0-11e9-954d-7446f0bc2e23.png)
 
 ### Traffic Details
-![Traffic Details](https://user-images.githubusercontent.com/10326954/39966499-ecfa036e-56ac-11e8-98fc-bde7cbbea787.png)
+![Traffic Details](https://user-images.githubusercontent.com/10326954/52973955-35bfa400-33c0-11e9-89db-74e8754a7c25.png)
 
 ### Flow Records
-![Flow Records](https://user-images.githubusercontent.com/10326954/39966504-fafe1446-56ac-11e8-96f3-0f01a01811ca.png)
+![Flow Records](https://user-images.githubusercontent.com/10326954/52973958-38ba9480-33c0-11e9-96b3-9de9f2dceca6.png)
+
+### Ziften ZFlow
+ElastiFlow&trade; v3.4.0 added support for IPFIX records from Ziften's ZFlow agent. In addition to being fully integrated with the standard dashboards, a stand-alone ZFlow dashboards displays network traffic based on user and command data provided by ZFlow.
+![Ziften ZFlow](https://user-images.githubusercontent.com/10326954/52973968-3ce6b200-33c0-11e9-98c5-20179ae80db3.png)
 
 ## Attribution
 This product includes GeoLite2 data created by MaxMind, available from (http://www.maxmind.com)