Docs
test1test2
6.5
6.5
  • ElastiFlow Documentation
  • Unified Flow Collector
    • General Configuration
    • Changelog
    • Maxmind GeoIP2 and GeoLite2
    • RiskIQ PassiveTotal
    • Network Interfaces
    • User-Defined Metadata
    • Docker
    • Linux
    • Unified Flow Collector Introduction
    • System Requirements
    • Supported IEs
    • AWS VPC Flow Log IEs
    • IPFIX IEs
    • Netflow IEs
    • sFlow IEs
  • Unified SNMP Collector
    • Device Groups
    • Changelog
    • Devices
    • Downloading Definitions
    • Enumerations
    • Objects
    • Object Groups
    • User-Defined Metadata
    • Docker
    • Network Interfaces
    • United SNMP Collector Introduction
    • Linux
    • Scheduling Rediscovery
  • Monitoring ElastiFlow
    • Liveness & Readiness
    • Metrics
    • Prometheus & Grafana
  • Configuration Reference
    • YAML Configuration Files
    • Configuration Reference Overview
    • Common
      • API
      • Licensing
      • Overview
      • Logging
      • HTTP output
      • Elasticsearch output
      • Kafka output
      • Monitor output
      • OpenSearch output
      • Splunk output
      • stdout output
      • Processor
    • Unified Flow Collector
      • Overview
      • Community/Conversation IDs
      • EF_PROCESSOR_ENRICH_TOTALS_IF_NO_DELTAS
      • Overview
      • RiskIQ PassiveTotal
      • Maxmind
      • User-Defined Metadata
      • Overview
      • Overview
      • User-Defined Metadata
      • Overview
      • Benchmark Input
      • Netflow/IPFIX/sFlow (UDP)
      • Licensing
      • Decoder/Processor
      • Sample Rate
      • Configuration Changes
    • Unified SNMP Collector
      • User-Defined Metadata
      • Overview
      • Licensing
      • SNMP Poller
      • EF_PROCESSOR_SNMP_ENUM_DEFINITIONS_DIRECTORY_PATH
  • API Reference
    • API Reference Overview
    • SNMP Operations
  • Data Platforms
    • Elastic
      • Basic Cluster
      • Advanced Cluster
      • Single Server
      • Multi-Tier Cluster
      • Single "Lab" Server
      • Elasticsearch
      • ElastiFlow vs. Filebeat and Logstash
      • RHEL/CentOS
      • Ubuntu/Debian
      • Kibana
      • ML
        • Network Security
        • Machine Learning
        • Availability
          • Network Availability
          • DHCP
          • LDAP
          • DNS
          • NTP
          • RADIUS
          • TCP Sessions
        • Network Security Activity
          • Rare Autonomous System
          • Network Activity
          • Rare Conversation
          • Rare Geolocation
        • Network Security Brute Force
          • Brute Force CLI Access
          • Brute Force Remote Desktop Access
          • Brute Force Attacks
        • Network Security DDoS
          • Denial-of-Service
          • ICMP Flood Attack
          • SYN Flood Attack
          • TCP DDoS Attack
          • UDP Amplification Attack
        • Network Security Recon
          • ICMP Scan
          • Reconnaissance
          • Port Scan
        • Performance
          • Unusual ASN Traffic Volume
          • Unusual Network Interface Traffic Volume
          • Network Performance
    • Opensearch
      • Dashboards
      • Auth Sig V4
    • Splunk
      • Default Search Macro
      • Configuring Data Input & Index
      • Splunk App Installation
    • Output Configuration
  • Additional Guides
    • Catalyst (sFlow)
    • FortiGate
    • hsflowd
    • Configuring Flow Sampling on Juniper Routers
    • Junos OS (sFlow)
    • MikroTik RouterOS
    • OpenWRT (softflowd)
    • Ubiquiti EdgeRouter
    • SonicWall
    • Junos OS
    • Extending SNMP Device Support
    • Flow Device Support Overview
    • SNMP Device Support Overview
    • Generating A Support Bundle
  • FAQ
    • Flows stopped showing up in Kibana (Disk(s) Full)
    • Common reasons why you have discrepancies between ElastiFlow data & reality
    • What Are Snapshots?
    • Importing the wrong dashboards (No data)
  • Knowledge Base
    • Config
      • Elasticsearch Authentication Failure
      • CA Certificate Path Incorrect
      • license/error Invalid Segments
    • Flow
      • Bidirectional Flow Support
      • Configure the UDP Input
      • Flow Records Not Received
      • Netflow v9/IPFIX Template Not Receieved
      • Unsupported sFlow Structures
    • General
      • License Has Expired
      • License Agreement Not Accepted
    • Install
      • .deb Upgrade Fails File Overwrite
    • Operation
      • Flow Collector Queues 90% Full
      • Dashboard Updates
      • Change elastiflow-* Index Name?
  • Elastic Stack Deployment
  • Download Links
Powered by GitBook
On this page
  • Problem
  • Reason #1: Sample Rate
  • Reason #2: Your flow exporter is counting the same flow(s) multiple times
  1. FAQ

Common reasons why you have discrepancies between ElastiFlow data & reality

PreviousFlows stopped showing up in Kibana (Disk(s) Full)NextWhat Are Snapshots?

Problem

When visualizing ElastiFlow data, it is much higher or lower than expected.

image

Reason #1: Sample Rate

How sample rates impact ElastiFlow data accuracy

The collector must adjust the calculation of bytes and packets based on the sampling rate used. Usually, devices will inform the collector of the sampling rate either within the flow record itself or as option data sent periodically by the device. This setting specifies the size of the cache to be used to hold sample rate information learned from option data.

Confirming a devices sample rate

If you go to:

After filtering on the exporter in question

You can look at a record and focus on the following field to see what the flow collector thinks the current sample rate is:

In the above example you can see the flow collector believes the sample rate is 1:1; however, as you can see below the device is configured with a sample rate of 1:512 thus the discrepancy in packets and bytes:

This is because the sampler table is not configured to be sent by the flow exporter (In this example a Cisco Nexus switch). On some flow exporters you must specify a 'sample options template' to be sent, in this case Cisco refers to this as a 'sampler-table'. Configuring the network device to send this is as easy as adding it to the flow configuration on the flow exporter (a Cisco Nexus switch in this example):

Now (after the timeout period has elapsed and the flow exporter sends the sampler table), when we go back into Kibana & check the flow.meter.packet_select.interval.packets field values in flow records coming from the Nexus switch, we can see the flow collector knows about the correct sample rate of 1:512:

Reason #2: Your flow exporter is counting the same flow(s) multiple times

When configuring Flow exporters (routers, switches, firewalls, etc...) It is typical to configure flow collection on specific interface(s). If misconfigured, this can lead to flows being counted two or more times and then added together.

Example of a flow exporter counting a single flow multiple times

Why this happens (configuration of R1)

As you can see below, not only is R1 configured to collect flows from Ethernet 0/3, Ethernet 0/2, and Vlan200, It is collecting flows from these interfaces in both the input and output directions.

In the example speed test above, flows are counted twice and achieve a download speed of ~213mbps. The flow is counted as it enters interface ethernet 0/3 and as it comes out of vlan 200 toward the destination Win11-2 computer. In both cases, the ingress interface is Ethernet 0/3, and the egress interface is Vlan200; thus, counting a single flow two times resulted in the above-measured throughput of 506.2Mbps when in reality, it was only ~213mbps.

We will only apply the monitor in the input direction on all these interfaces to correct this and remove the output configuration.

Now when we run another speed test, we see the correct throughput:

A sample rate is the average ratio of packets incoming on a sFlow-enabled port to the number of flow samples taken from those packets. sFlow sampling can affect performance on some network equipment.

Reference:

What is a "good" sampling rate?:

image

Detailed Explanation:

image
image
image
image
image
image
image
image
image
image
https://blog.sflow.com/2009/06/sampling-rates.html
https://docs.elastiflow.com/config_ref_sampling/#sample-rate
https://blog.sflow.com/2009/05/scalability-and-accuracy-of-packet.html
https://sflow.org/packetSamplingBasics/index.htm