Modern Telemetry - Streaming vs Traditional
🎯 Learning Objective: Understand how telecom OSS moved from traditional SNMP polling to streaming telemetry using gNMI, gRPC, and message buses like Kafka. Also learn why batch telemetry and Data Lakes are still important for long-term analytics.
The Evolution of Network Telemetry
1990s-2010s
SNMP Polling
Request-Response (Pull Model)
2015-Present
Streaming Telemetry
Push Model
The industry moved from periodically asking devices for data to letting devices continuously send data to OSS systems.
Traditional Approach: SNMP Polling
SNMP (Simple Network Management Protocol) has been used for network monitoring for many years. In this model, OSS or NMS asks devices for data at fixed intervals such as every 5 minutes or every 15 minutes.
How SNMP Polling Works
- OSS requests data from devices
- Devices return the current value
- Collection happens at fixed intervals
- Data is a sample, not a continuous feed
- Information is organized using MIBs
Main Limitations
- Sampling gaps - Short spikes may be missed
- No true real-time view - Delay depends on poll interval
- Scaling challenges - Large networks create heavy polling load
- Overhead - Frequent polling uses device and OSS resources
- Traps are not enough - Traps help with alarms, but not with continuous metrics
Traditional SNMP Polling Pattern
OSS: "What is ifInOctets on port 1?" Device: "1972837462 bytes" -- wait 5 minutes -- OSS: "What is ifInOctets on port 1?" Device: "1985000000 bytes"
Modern Approach: Streaming Telemetry
Streaming telemetry uses a push model. Instead of waiting for OSS to ask for data, the device sends updates continuously or whenever values change.
Protocol: gNMI streaming telemetry sends PRB utilization every 500 ms.
OSS Action: Analytics detects the spike and triggers automated scaling.
Result: Users get better service and the operator reduces the risk of SLA penalties.
BGP: Routing systems detect route loss and instability.
TL1: Optical equipment reports Loss of Signal alarms.
NMS Correlation: The OSS correlates routing symptoms and transport alarms to identify the failing segment faster.
Result: Engineers are sent to the right location much sooner.
How Streaming Works
- Devices push data to OSS
- Updates can be periodic or on-change
- Telemetry arrives with very low delay
- Common technologies include gNMI and gRPC
- Data may use JSON or Protobuf encoding
Key Advantages
- Better visibility - Far fewer sampling gaps
- Faster detection - Problems are seen sooner
- Lower repetitive polling - Less request-response overhead
- Better automation - Structured data works well with analytics and AIOps
Streaming Telemetry Pattern (gNMI/gRPC)
Device: "ifInOctets updated: 1972837462 bytes" -- milliseconds later -- Device: "ifInOctets updated: 1973000000 bytes" -- continues every 100ms or when value changes --
On-change telemetry: Device sends updates only when values change or cross a threshold.
Many operators use both models depending on the metric and the business need.
Protocol → Ingestion Method & Latency
The protocol used determines how data is ingested into OSS and how fast it arrives.
| Protocol / Source | Ingestion Method | Typical Latency |
|---|---|---|
| gNMI / Streaming Telemetry | Continuous push → Kafka → Data Lake | Sub-second to seconds |
| SNMP Traps | Device push → Trap Receiver → OSS | Few seconds |
| Syslog | Device push → Log Collector → OSS | Few seconds |
| SNMP Polling | OSS pulls every N minutes → Database | N minutes (5-15 min typical) |
| PM Files (FTP/SFTP) | Batch files every 5-15 min → Collector | Minutes to hour |
| CDR / Usage Files | Batch or near-real-time streams | Minutes to hours |
| NETCONF / RESTCONF | Pull (RPC) or push notifications | Seconds to minutes |
Streaming telemetry protocols such as gNMI provide near real-time model-driven telemetry, while traps and syslog provide near real-time events and logs. File and polling-based protocols are periodic and slower by design. Use streaming for real-time automation, batch for historical analysis.
Key Telemetry Protocols Compared
| Protocol | Model | Data Format | Use Case |
|---|---|---|---|
| SNMP | Poll (request-response) | ASN.1 / MIB | Traditional monitoring, traps, legacy devices |
| NETCONF | RPC-based management | XML / YANG | Configuration and state management |
| gNMI | Streaming + RPC | Protobuf / JSON + YANG | Modern telemetry and configuration |
| gRPC | RPC framework / transport | Protobuf | Transport framework used by gNMI and similar services |
| RESTCONF | REST-based | JSON or XML + YANG models | HTTP-based configuration and state access |
gNMI - A Common Modern Telemetry Interface
gNMI stands for gRPC Network Management Interface. It is widely used in modern networks for streaming telemetry and can also be used for retrieving state and pushing configuration.
gNMI Capabilities
- CAPABILITIES: Learn what the device supports
- GET: Read current state
- SET: Change configuration
- SUBSCRIBE: Receive streaming telemetry
Sample gNMI SUBSCRIBE
Subscribe Request: path: /interfaces/interface/state/counters mode: STREAM sample_interval: 100ms
Modern OSS uses telemetry not only for dashboards, but also for automation, anomaly detection, AI-based assurance, and closed-loop actions.
Which Devices Don't Support gNMI / Modern Telemetry?
Not all network devices support modern streaming telemetry. Knowing which devices lack support helps with migration planning.
| Device Type / Age | gNMI Support | What to Use Instead |
|---|---|---|
| Modern routers/switches (2020+) | ✅ Usually yes | gNMI, NETCONF/YANG, vendor gRPC |
| Legacy routers/switches (pre-2015) | ❌ No | SNMP, CLI, or replace |
| Old 2G/3G RAN equipment | ❌ No | EMS proprietary interfaces, SNMP |
| Modern 5G gNBs (2020+) | ✅ Usually yes | gNMI, NETCONF |
| Legacy optical transport (SDH/SONET) | ❌ No | TL1, proprietary EMS APIs |
| Modern optical transport (2020+) | ⚠️ Sometimes | Check vendor support; gNMI emerging |
| White-box switches with SONiC | ✅ Yes | gNMI, OpenConfig YANG |
In every new RFP, explicitly ask: "Does this device support NETCONF/YANG and gNMI/OpenConfig for telemetry and configuration?" Then verify it in a lab PoC before purchasing.
Workaround for Legacy Devices
- Use protocol adapters (gNMI → SNMP gateway)
- Keep EMS as abstraction layer
- Plan phased replacement
- Use mediation to normalize data
Detection Methods
- SNMP polling for basic health
- Syslog for events
- Traps for alarms
- Manual config backups
The Data Highway: Kafka & Pulsar
Streaming telemetry produces a large volume of events. Kafka and Pulsar help OSS receive, buffer, and distribute this data reliably to many downstream systems.
Apache Kafka
Kafka is a popular event streaming platform. It stores data in ordered logs and allows many consumers to read the same data stream.
- High throughput: Handles very large telemetry volumes
- Strong ecosystem: Works with analytics, AIOps, and data platforms
Apache Pulsar
Pulsar is another modern streaming platform. It is known for multi-tenancy and separation of compute and storage.
- Multi-tenancy: Useful for separating teams, domains, or customers
- Flexible usage: Supports both streaming and queue-like workloads
A message bus decouples the network from the OSS applications. If one consumer is slow or temporarily unavailable, the data can still be buffered and processed later.
YANG Data Models - The Structure Behind Modern Telemetry
YANG is a data modeling language. It defines how configuration and telemetry data should be structured so systems and vendors can exchange information more consistently.
• Machine-readable - Good for automation and validation
• Hierarchical structure - Useful for complex network data
• Versioned - Easier to manage model changes over time
Real-World Example: SNMP vs Streaming
Scenario: A 5G gNB experiences a 2-second latency spike every 3 minutes.
SNMP (polling every 5 minutes)
- May miss the spike completely
- Can report normal-looking average latency
- NOC may not see the problem quickly
Streaming Telemetry (100ms intervals)
- Captures the short spike
- NOC can see the recurring pattern
- Root cause analysis becomes faster
Modern Telemetry Pipeline
Network Device
gNB, Router, Switch
gNMI/gRPC
Streaming telemetry
Kafka / Pulsar
Buffer and event distribution
Streaming Analytics
Real-time processing
AIOps / Orchestration
Automation and dashboards
A simple way to remember the flow: device → stream → message bus → analytics → automation.
Batch Telemetry - Still Important
Streaming telemetry is important for real-time monitoring, but batch telemetry is still needed for history, planning, reporting, and analytics over long time periods.
Common Batch Data Sources
- PM Counters - Aggregated performance metrics
- CDRs - Usage records for billing
- Alarm History - Historical fault records
- Configuration Backups - Periodic snapshots
- Inventory Exports - Asset and network reports
Batch Collection Methods
- File-based transfers: FTP, SFTP, S3
- SNMP Bulk Polling: Large table collection
- Database Exports: Scheduled SQL jobs
- API Polling: REST calls at fixed times
Streaming helps with fast alerts and automation.
Batch helps with long-term analysis, ML training, and compliance.
Modern OSS usually needs both.
Data Lake Integration
A Data Lake is a large centralized repository where raw and processed telemetry can be stored at scale. It supports both streaming and batch data.
Data Lake Zones
- Raw Zone: Data as it arrives
- Processed Zone: Cleaned and enriched data
- Curated Zone: Analytics-ready datasets
Common Data Formats
- Streaming: Avro, Protobuf
- Batch: Parquet, ORC
- Legacy: CSV, XML, JSON
Batch: PM files are loaded later for long-term trend analysis and ML.
Result: The same Data Lake supports both real-time actions and long-term planning.
Batch Integration Architecture
Batch pipelines usually run on schedules. They collect files or database extracts, transform them, and then load them into a Data Lake.
1. Source
EMS exports PM files
2. Scheduling
Airflow or similar tool starts the job
3. Processing
Normalize and enrich the data
4. Storage
Write to the Data Lake
# Example: Airflow DAG for PM Batch Processing
from airflow import DAG
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
default_args = {
'owner': 'oss_team',
'schedule_interval': '0 1 * * *',
'retries': 3,
}
with DAG(dag_id='pm_batch_etl', default_args=default_args) as dag:
extract = SparkSubmitOperator(task_id='extract_pm_files', application='jobs/extract_pm.py')
transform = SparkSubmitOperator(task_id='transform_normalize', application='jobs/normalize_pm.py')
load = SparkSubmitOperator(task_id='load_to_datalake', application='jobs/write_parquet.py')
extract >> transform >> load
Key Terms You Must Know
Simple Network Management Protocol - traditional polling, widely used for traditional monitoring and traps
OSS asks devices for data at fixed intervals (pull model)
Devices push data continuously or on change (push model)
Device sends data without waiting for a request
OSS requests data from the device
gRPC Network Management Interface - modern telemetry and configuration protocol
High-performance RPC framework used by gNMI
Data modeling language for network configuration and telemetry
Management Information Base - SNMP data structure
Google Protocol Buffers - compact binary format for efficient messaging
Data sent at fixed intervals (e.g., 100ms, 1 second)
Data sent when values change or cross a threshold
Distributed log-based event streaming platform for high-volume telemetry
Streaming platform with strong multi-tenancy support
Middleware that buffers and distributes event data between systems
Large central storage for raw and processed telemetry data at scale
Landing area for original, unmodified telemetry data
Cleaned, normalized, and enriched data area
Business-ready datasets for analytics, BI, and ML
Columnar file format for efficient batch analytics
Compact row-based format often used in streaming systems
Combining both batch and streaming pipelines for complete data processing
AI-driven operations and assurance using telemetry data
Automatic action triggered by detected network conditions
Virtual representation of the network for simulation and predictive analysis
Border Gateway Protocol - routing protocol that detects global connectivity changes
Transaction Language 1 - protocol for optical transport networks
Migration Challenges: SNMP to Streaming Telemetry
Legacy Devices
Some devices do not support modern streaming interfaces.
Hybrid Operations
Operators often run SNMP and streaming together.
Data Volume
Streaming produces a much larger amount of data.
Skills Gap
Teams need knowledge of gNMI, YANG, Protobuf, and Kafka.
Mediation Layers
Legacy EMS platforms may need adapters for modern telemetry.
Storage Platforms
Traditional databases may struggle with very high telemetry volume.
Real-time telemetry is becoming a major input for digital twins, predictive maintenance, and simulation-based operations.
Connection to BSS
SLA Dashboards
Near real-time metrics support service commitments and reporting.
Usage-Based Billing
More detailed usage data can support modern charging models.
Customer Experience
Real-time performance visibility helps improve QoE.
Predictive Maintenance
Streaming data helps identify issues before customers are impacted.
Common Questions
Q1. Why is a message bus like Kafka or Pulsar needed for streaming telemetry?
Because telemetry arrives very quickly and in large volume. The message bus buffers the data and allows many downstream applications to consume it safely.
Q2. What are the two common streaming telemetry models?
Sampled telemetry and on-change telemetry.
Q3. What is the role of Protobuf in modern telemetry?
Protobuf is a compact binary format. It is smaller and faster than text-heavy formats like XML and often more efficient than JSON for high-volume telemetry.
Q4. What are YANG models?
YANG models define the structure of network data so telemetry and configuration can be handled more consistently.
Q5. Why are Data Lakes useful for telemetry?
They store very large volumes of raw and processed telemetry and support long-term analytics, reporting, and ML.
Q6. What is the difference between streaming and batch telemetry?
Streaming is used for fast, near real-time visibility. Batch is used for scheduled collection, historical analysis, and reporting.
Q7. Why do operators still keep SNMP?
Because many legacy devices still rely on SNMP, and traps remain useful for alarms.
Q8. What is a simple modern OSS telemetry pipeline?
Device → gNMI/gRPC → Kafka/Pulsar → analytics → automation or dashboards.
Q9. Which devices don't support gNMI?
Legacy routers (pre-2015), old 2G/3G equipment, legacy optical transport (SDH/SONET), and some older switches. Always check vendor specifications before assuming support.
Q10. How does protocol choice affect ingestion latency?
Streaming protocols (gNMI, traps, syslog) give sub-second to seconds latency. Polling protocols (SNMP) give minutes latency based on poll interval. Batch files give minutes to hours latency.
📌 Key Takeaways:
- SNMP polling works, but it may miss short events between polls
- Streaming telemetry gives faster and more detailed visibility
- Protocol determines ingestion: Streaming = push (sub-second), SNMP polling = pull (minutes), Batch files = scheduled (minutes to hours)
- gNMI and gRPC are important technologies for modern telemetry
- Legacy devices (pre-2015, old 2G/3G, SDH/SONET) do not support gNMI. Ask for YANG/gNMI support in RFPs
- Kafka and Pulsar help carry and buffer high-volume event streams
- YANG provides structure and consistency for data models
- Batch telemetry is still needed for historical analytics and reporting
- Data Lakes support large-scale storage for both streaming and batch data
- Modern OSS usually combines legacy and modern approaches together