Monitoring

The hopit Edge System Monitoring service traces many resource parameters like CPU usage, memory usage, disk usage, network details or service details. The monitoring solution is split in two services: the hopit Edge Service Monitoring and the System Monitoring. The corresponding Targets which can be used in the router are named System and System Monitor. The signals can be collected and stored with the Insights Collector and Data Lake services or used in the Router.

hopit Edge Service Monitoring

The Service Monitoring is always enabled and monitors the state of the internal hopit Edge services (Data Lake, VPN, ...), the Target connections and some internal parameters like service up-time or version.

The signals can be used in the Router with the Target System. When sending the signals via VPN to the hopit Portal, they are stored in the long-term Data Lake.

Available signals are:

hopit_edge_service_uptime{data_lake="long-term,edge"}: hopit Edge serivce up-time in seconds
hopit_edge_version_major{data_lake="long-term,edge"}: hopit Edge major version (the version structure is [major].[minor].[patch] with an optional -[qualifier] for pre-releases)
hopit_edge_version_minor{data_lake="long-term,edge"}: hopit Edge minor version
hopit_edge_version_patch{data_lake="long-term,edge",Qualifier="[qualifier]"}: hopit Edge patch vrsion with the qualifier
hopit_edge_service_up{data_lake="long-term,edge",service="Insights Collector"}
hopit_edge_service_up{data_lake="long-term,edge",service="Deployment"}
hopit_edge_service_up{data_lake="long-term,edge",service="Dashboard"}
hopit_edge_service_up{data_lake="long-term,edge",service="ADS-Beckhoff"}
hopit_edge_service_up{data_lake="long-term,edge",service="Data Lake"}
hopit_edge_service_up{data_lake="long-term,edge",service="Siemens-Siemens"}
hopit_edge_service_up{data_lake="long-term,edge",service="OpcUa-S7"}
hopit_edge_service_up{data_lake="long-term,edge",service="Azure-IoT Hub"}
hopit_edge_service_up{data_lake="long-term,edge",service="Tasks"}
hopit_edge_service_up{data_lake="long-term,edge",service="ADS Router"}
hopit_edge_service_up{data_lake="long-term,edge",service="System Time Sync"}
hopit_edge_service_up{data_lake="long-term,edge",service="Router"}
hopit_edge_service_up{data_lake="long-term,edge",service="System Monitor"}
hopit_edge_service_up{data_lake="long-term,edge",service="Log Collector"}
hopit_edge_service_up{data_lake="long-term,edge",service="Device Twin"}
hopit_edge_service_up{data_lake="long-term,edge",service="VPN"}
hopit_edge_service_up{data_lake="long-term,edge",service="Alerting"}
hopit_edge_service_up{data_lake="long-term,edge",service="[TargetType]-[Target Name]"}: This signal is available for every Target.
hopit_edge_signals_write_count{data_lake="long-term,edge",target="[Target Name]",route="[Route Name]"}: Amount of signals sent by the Target. This signal is available for every Destination Target for every Route.
hopit_edge_signals_read_count{data_lake="long-term,edge",target="[Target Name]",route="[Route Name]"}: Amount of signals read by the Target. This signal is available for every Destination Target for every Route.
hopit_edge_signals_write_time_s{data_lake="long-term,edge",target="[Target Name]",route="[Route]"}: Time to write all signals from a Destination Target in a Route. This signal is available for all Targets in all Routes.
hopit_edge_signals_read_time_s{data_lake="long-term,edge",target="[Target Name]",route="[Route]"}: Time to read all signals from all Sources in a Route. This signal is available for all Routes.
hopit_edge_signals_read_time_s{data_lake="long-term,edge",target="All",route="[Route]"}: Time to read all signals from a Sources Target in a Route. This signal is available for all Targets in all Routes.

The hopit_edge_service_up signals can have following values:

0: Service down
1: Service stopping
2: Service starting
3: Service failed
4: Service has no valid license
5: Service up and healthy
6: Service has missing required settings
7: Service disabled
8: Dependent service is not running

System Monitor

The System Monitor is a service, which monitors many IPC resource like CPU usage, memory usage, disk usage or network details.

The signals can be used in the Router with the Target System Monitor. When sending the signals via VPN to the hopit Portal, they are stored in the short-term Data Lake.

Corresponding Edge configuration and Device Twin definition to activate the System Monitoring service:

Edge-UI
Device Twin

https://localhost:5050/Settings

{
    "SystemMonitor": {
        "Enabled": true
    }
}

All available signal names are listed in the following. For many signals, there are multiple instances with details listed in curly brackets: [signal-name]{[lable-name]="[label-value]"}. For example, there are windows_logical_disk_free_bytes for each available partition:

windows_logical_disk_free_bytes{volume="C:"}
windows_logical_disk_free_bytes{volume="D:"}
...

CPU signals:

windows_cpu_cstate_seconds_total: Time spent in low-power idle states
windows_cpu_time_total: Time that processor spent in different modes (idle, user, system, ...)
windows_cpu_interrupts_total: Total number of received and serviced hardware interrupts
windows_cpu_dpcs_total: Total number of received and serviced deferred procedure calls (DPCs)
windows_cpu_clock_interrupts_total: Total number of received and serviced clock tick interrupts
windows_cpu_idle_break_events_total: Total number of time processor was woken from idle
windows_cpu_parking_status: Parking Status represents whether a processor is parked or not
windows_cpu_core_frequency_mhz: Core frequency in megahertz
windows_cpu_processor_performance: Processor Performance is the average performance of the processor while it is executing instructions, as a percentage of the nominal performance of the processor. On some processors, Processor Performance may exceed 100%
windows_cpu_info: Labeled CPU information

Computer System signals:

windows_cs_logical_processors: Number of installed logical processors
windows_cs_physical_memory_bytes: Total installed physical memory
windows_cs_hostname: Labeled system hostname information

Disk signals:

windows_logical_disk_requests_queued: Number of requests outstanding on the disk at the time the performance data is collected
windows_logical_disk_read_bytes_total: Rate at which bytes are transferred from the disk during read operations
windows_logical_disk_reads_total: Rate of read operations on the disk
windows_logical_disk_write_bytes_total: Rate at which bytes are transferred to the disk during write operations
windows_logical_disk_writes_total: Rate of write operations on the disk
windows_logical_disk_read_seconds_total: Seconds the disk was busy servicing read requests
windows_logical_disk_write_seconds_total: Seconds the disk was busy servicing write requests
windows_logical_disk_free_bytes: Unused space of the disk in bytes (not real time, updates every 10-15 min)
windows_logical_disk_size_bytes: Total size of the disk in bytes (not real time, updates every 10-15 min)
windows_logical_disk_idle_seconds_total: Seconds the disk was idle (not servicing read/write requests)
windows_logical_disk_split_ios_total: Number of I/Os to the disk split into multiple I/Os

Memory signals:

windows_memory_available_bytes: The amount of physical memory immediately available for allocation to a process or for system use. It is equal to the sum of memory assigned to the standby (cached), free and zero page lists
windows_memory_cache_bytes: Number of bytes currently being used by the file system cache
windows_memory_cache_bytes_peak: Maximum number of CacheBytes after the system was last restarted
windows_memory_cache_faults_total: Number of faults which occur when a page sought in the file system cache is not found there and must be retrieved from elsewhere in memory (soft fault) or from disk (hard fault)
windows_memory_commit_limit: Amount of virtual memory, in bytes, that can be committed without having to extend the paging file(s)
windows_memory_committed_bytes: Amount of committed virtual memory, in bytes
windows_memory_demand_zero_faults_total: The number of zeroed pages required to satisfy faults. Zeroed pages, pages emptied of previously stored data and filled with zeros, are a security feature of Windows that prevent processes from seeing data stored by earlier processes that used the memory space
windows_memory_free_and_zero_page_list_bytes: Not yet documented
windows_memory_free_system_page_table_entries: Number of page table entries not being used by the system
windows_memory_modified_page_list_bytes: Not yet documented
windows_memory_page_faults_total: Overall rate at which faulted pages are handled by the processor
windows_memory_swap_page_reads_total: Number of disk page reads (a single read operation reading several pages is still only counted once)
windows_memory_swap_pages_read_total: Number of pages read across all page reads (ie counting all pages read even if they are read in a single operation)
windows_memory_swap_pages_written_total: Number of pages written across all page writes (ie counting all pages written even if they are written in a single operation)
windows_memory_swap_page_operations_total: Total number of swap page read and writes (PagesPersec)
windows_memory_swap_page_writes_total: Number of disk page writes (a single write operation writing several pages is still only counted once)
windows_memory_pool_nonpaged_allocs_total: The number of calls to allocate space in the nonpaged pool. The nonpaged pool is an area of system memory area for objects that cannot be written to disk, and must remain in physical memory as long as they are allocated
windows_memory_pool_nonpaged_bytes_total: Number of bytes in the non-paged pool
windows_memory_pool_paged_allocs_total: Number of calls to allocate space in the paged pool, regardless of the amount of space allocated in each call
windows_memory_pool_paged_bytes: Number of bytes in the paged pool
windows_memory_pool_paged_resident_bytes: Not yet documented
windows_memory_standby_cache_core_bytes: Not yet documented
windows_memory_standby_cache_normal_priority_bytes: Not yet documented
windows_memory_standby_cache_reserve_bytes: Not yet documented
windows_memory_system_cache_resident_bytes: Not yet documented
windows_memory_system_code_resident_bytes: Not yet documented
windows_memory_system_code_total_bytes: Not yet documented
windows_memory_system_driver_resident_bytes: Not yet documented
windows_memory_system_driver_total_bytes: Not yet documented
windows_memory_transition_faults_total: Not yet documented
windows_memory_transition_pages_repurposed_total: Not yet documented
windows_memory_write_copies_total: The number of page faults caused by attempting to write that were satisfied by copying the page from elsewhere in physical memory

Network signals:

windows_net_bytes_received_total: Total bytes received by interface
windows_net_bytes_sent_total: Total bytes transmitted by interface
windows_net_bytes_total: Total bytes received and transmitted by interface
windows_net_packets_outbound_discarded_total: Total outbound packets that were chosen to be discarded even though no errors had been detected to prevent transmission
windows_net_packets_outbound_errors_total: Total packets that could not be transmitted due to errors
windows_net_packets_received_discarded_total: Total inbound packets that were chosen to be discarded even though no errors had been detected to prevent delivery
windows_net_packets_received_errors_total: Total packets that could not be received due to errors
windows_net_packets_received_total: Total packets received by interface
windows_net_packets_received_unknown_total: Total packets received by interface that were discarded because of an unknown or unsupported protocol
windows_net_packets_total: Total packets received and transmitted by interface
windows_net_packets_sent_total: Total packets transmitted by interface
windows_net_current_bandwidth_bytes: Estimate of the interface's current bandwidth in bytes per second

Operating System signals:

windows_os_info: Contains full product name & version in labels
windows_os_paging_limit_bytes: Total number of bytes that can be stored in the operating system paging files. 0 (zero) indicates that there are no paging files
windows_os_paging_free_bytes: Number of bytes that can be mapped into the operating system paging files without causing any other pages to be swapped out
windows_os_physical_memory_free_bytes: Bytes of physical memory currently unused and available
windows_os_time: Current time as reported by the operating system, in Unix time. See time.Unix() for details
windows_os_timezone: Current timezone as reported by the operating system. See time.Zone() for details
windows_os_processes: Number of process contexts currently loaded or running on the operating system
windows_os_processes_limit: Maximum number of process contexts the operating system can support. The default value set by the provider is 4294967295 (0xFFFFFFFF)
windows_os_process_memory_limit_bytes: Maximum number of bytes of memory that can be allocated to a process
windows_os_users: Number of user sessions for which the operating system is storing state information currently. For a list of current active logon sessions, see logon
windows_os_virtual_memory_bytes: Bytes of virtual memory
windows_os_visible_memory_bytes: Total bytes of physical memory available to the operating system. This value does not necessarily indicate the true amount of physical memory, but what is reported to the operating system as available to it
windows_os_virtual_memory_free_bytes: Bytes of virtual memory currently unused and available

Processes signals:

The Processes signals are available for the Windows processes:

hopit Edge proces
hopit Edge Updater process
hopit Edge sub processes:

Grafana Server
Log Collector
Data Lake
System Monitor
Insights Collector
VPN process

TwinCAT processes with names starting with TC, Tc or TF

Signal list:

windows_process_start_time: Time of process start
windows_process_cpu_time_total: Returns elapsed time that all of the threads of this process used the processor to execute instructions by mode (privileged, user). An instruction is the basic unit of execution in a computer, a thread is the object that executes instructions, and a process is the object created when a program is run. Code executed to handle some hardware interrupts and trap conditions is included in this count.
windows_process_handles: Total number of handles the process has open. This number is the sum of the handles currently open by each thread in the process.
windows_process_io_bytes_total: Bytes issued to I/O operations in different modes (read, write, other). This property counts all I/O activity generated by the process to include file, network, and device I/Os. Read and write mode includes data operations; other mode includes those that do not involve data, such as control operations.
windows_process_io_operations_total: I/O operations issued in different modes (read, write, other). This property counts all I/O activity generated by the process to include file, network, and device I/Os. Read and write mode includes data operations; other mode includes those that do not involve data, such as control operations.
windows_process_page_faults_total: Page faults by the threads executing in this process. A page fault occurs when a thread refers to a virtual memory page that is not in its working set in main memory. This can cause the page not to be fetched from disk if it is on the standby list and hence already in main memory, or if it is in use by another process with which the page is shared.
windows_process_page_file_bytes: Current number of bytes this process has used in the paging file(s). Paging files are used to store pages of memory used by the process that are not contained in other files. Paging files are shared by all processes, and lack of space in paging files can prevent other processes from allocating memory.
windows_process_pool_bytes: Pool Bytes is the last observed number of bytes in the paged or nonpaged pool. The nonpaged pool is an area of system memory (physical memory used by the operating system) for objects that cannot be written to disk, but must remain in physical memory as long as they are allocated. The paged pool is an area of system memory (physical memory used by the operating system) for objects that can be written to disk when they are not being used. Nonpaged pool bytes is calculated differently than paged pool bytes, so it might not equal the total of paged pool bytes.
windows_process_priority_base: Current base priority of this process. Threads within a process can raise and lower their own base priority relative to the process base priority of the process.
windows_process_private_bytes: Current number of bytes this process has allocated that cannot be shared with other processes.
windows_process_threads: Number of threads currently active in this process. An instruction is the basic unit of execution in a processor, and a thread is the object that executes instructions. Every running process has at least one thread.
windows_process_virtual_bytes: Current size, in bytes, of the virtual address space that the process is using. Use of virtual address space does not necessarily imply corresponding use of either disk or main memory pages. Virtual space is finite and, by using too much, the process can limit its ability to load libraries.
windows_process_working_set_private_bytes: Size of the working set, in bytes, that is use for this process only and not shared nor shareable by other processes.
windows_process_working_set_peak_bytes: Maximum size, in bytes, of the Working Set of this process at any point in time. The Working Set is the set of memory pages touched recently by the threads in the process. If free memory in the computer is above a threshold, pages are left in the Working Set of a process even if they are not in use. When free memory falls below a threshold, pages are trimmed from Working Sets. If they are needed they will then be soft-faulted back into the Working Set before they leave main memory.
windows_process_working_set_bytes: Maximum number of bytes in the working set of this process at any point in time. The working set is the set of memory pages touched recently by the threads in the process. If free memory in the computer is above a threshold, pages are left in the working set of a process even if they are not in use. When free memory falls below a threshold, pages are trimmed from working sets. If they are needed, they are then soft-faulted back into the working set before they leave main memory.

Services signals:

The Services signals are available for the Windows services:

hopit Edge service
hopit Edge Updater service
TwinCAT services with names starting with TwinCAT
VPN service

Signal list:

windows_service_info: Contains service information in labels, constant 1
windows_service_state: The state of the service, 1 if the current state, 0 otherwise
windows_service_start_mode: The start mode of the service, 1 if the current start mode, 0 otherwise
windows_service_status: The status of the service, 1 if the current status, 0 otherwiseStates

System signals:

windows_system_context_switches_total: Total number of context switches
windows_system_exception_dispatches_total: Total exceptions dispatched by the system
windows_system_processor_queue_length: Number of threads in the processor queue. There is a single queue for processor time even on computers with multiple processors.
windows_system_system_calls_total: Total combined calls to Windows NT system service routines by all processes running on the computer
windows_system_system_up_time: Time of last boot of system
windows_system_threads: Number of Windows system threads

Temperature signals:

windows_thermalzone_percent_passive_limit: % Passive Limit is the current limit this thermal zone is placing on the devices it controls. A limit of 100% indicates the devices are unconstrained. A limit of 0% indicates the devices are fully constrained.
windows_thermalzone_temperature_celsius: Temperature of the thermal zone, in degrees Celsius.
windows_thermalzone_throttle_reasons: Throttle Reasons indicate reasons why the thermal zone is limiting performance of the devices it controls. 0x0 - The zone is not throttled. 0x1 - The zone is throttled for thermal reasons. 0x2 - The zone is throttled to limit electrical current.

Remote Desktop signals:

windows_remote_fx_net_base_udp_rtt_seconds: Base UDP round-trip time (RTT) detected in seconds.
windows_remote_fx_net_base_tcp_rtt_seconds: Base TCP round-trip time (RTT) detected in seconds.
windows_remote_fx_net_current_tcp_bandwidth: TCP Bandwidth detected in bytes per second.
windows_remote_fx_net_current_tcp_rtt_seconds: Average TCP round-trip time (RTT) detected in seconds.
windows_remote_fx_net_current_udp_bandwidth: UDP Bandwidth detected in bytes per second.
windows_remote_fx_net_current_udp_rtt_seconds: Average UDP round-trip time (RTT) detected in seconds.
windows_remote_fx_net_received_bytes_total: Not yet documented
windows_remote_fx_net_sent_bytes_total: Not yet documented
windows_remote_fx_net_udp_packets_received_total: Rate in packets per second at which packets are received over UDP.
windows_remote_fx_net_udp_packets_sent_total: Rate in packets per second at which packets are sent over UDP.

hopit Edge Service Monitoring​

System Monitor​

hopit Edge Service Monitoring

System Monitor