Geneos Gateway Publishing Data User Guide

Geneos Gateway Publishing Data User Guide

Geneos Gateway Publishing Data User Guide

Geneos Gateway Publishing Data User Guide Functional Area: Geneos Gateway Publishing Data Geneos Release: v4.8 Document Version: v1.0.0 Date Published: 22 August 2018

Copyright 2018. ITRS Group Ltd. All rights reserved. Information in this document is subject to change without notice. The software described in this document is furnished under a license agreement or nondisclosure agreement. The software may be used or copied only in accordance with the terms of those agreements. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or any means electronic or mechanical, including photocopying and recording for any purpose other than the purchaser's personal use without the written permission of ITRS Group Ltd.

ITRS Group Ltd 6th Floor, The Bonhill Building, 15 Bonhill Street, London, EC2A 4DN, UK t: +44 (0)20 7638 6700

Table of Contents Publishing To Kafka And Nanomsg - Technical Reference 6 Publishing 6 Introduction 6 Configuration 7 Publishing 7 Publishing > Enabled 7 Publishing > Adapter 7 Publishing > Adapter > Nanomsg 7 Publishing > Adapter > Nanomsg > PublishPort 7 Publishing > Adapter > Nanomsg > RequestPort 7 Publishing > Adapter > Nanomsg > SecondaryPorts 7 Publishing > Adapter > Nanomsg > SecondaryPorts > PublishPort 8 Publishing > Adapter > Nanomsg > SecondaryPorts > RequestPort 8 Publishing > Adapter > Kafka 8 Publishing > Adapter > Kafka > TopicPrefix 8 Publishing > Adapter > Kafka > BrokerList 8 Publishing > AdditionalSettings 8 SSL Encryption 9 SSL Encryption With SSL Authentication 9 Publishing > Strategies 10 Publishing > Strategies > StrategyGroup 11 Publishing > Strategies > StrategyGroup > Name 11 Publishing > Strategies > Strategy 11 Publishing > Strategies > Strategy > Name 11 Publishing > Strategies > Strategy > Targets 12 Publishing > Strategies > Strategy > Targets > Target 12 Publishing > Strategies > Strategy > PivotDataview 12 Publishing > Strategies > Strategy > Schedule 12 Publishing > PublishPort 12 Publishing > RequestPort 12 Publishing > SecondaryPorts 12 Publishing > SecondaryPorts > PublishPort 13 Publishing > SecondaryPorts > RequestPort 13 Standardised Formatting 13

System Supplied Formatting 13 User Specified Formatting 15 Configuration 16 Samplers > Sampler > StandardisedFormatting 16 Samplers > Sampler > StandardisedFormatting > Dataviews > Dataview 16 Samplers > Sampler > StandardisedFormatting > Dataviews > Dataview > Variables > Variable 16 Samplers > Sampler > StandardisedFormatting > Dataviews > Dataview > Variables > Variable > Type 16 Samplers > Sampler > StandardisedFormatting > Dataviews > Dataview > Variables > Variable > Type > DateTime 16 Samplers > Sampler > StandardisedFormatting > Dataviews > Dataview > Variables > Variable > Type > DateTime > Formats > Format 17 Samplers > Sampler > StandardisedFormatting > Dataviews > Dataview > Variables > Variable > Type > DateTime > IgnoreErrorsIfFormatsFail 17 Samplers > Sampler > StandardisedFormatting > Dataviews > Dataview > Variables > Variable > Type > DateTime > OverrideTimezone 17 Samplers > Sampler > StandardisedFormatting > Dataviews > Dataview > Variables > Variable > Applicability 17 Samplers > Sampler > StandardisedFormatting > Dataviews > Dataview > Variables > Variable > Applicability > Headlines 18 Samplers > Sampler > StandardisedFormatting > Dataviews > Dataview > Variables > Variable > Applicability > Headlines > Regex 18 Samplers > Sampler > StandardisedFormatting > Dataviews > Dataview > Variables > Variable > Applicability > Columns > Regex 18 Samplers > Sampler > StandardisedFormatting > Dataviews > Dataview > Variables > Variable > Applicability > Cells 19 OperatingEnvironment > Debug > DirectoryManager > ShowStandardisedFormattingErrors 19 Gateway Publishing Stats 20 Description 20 Output 20 Command Line Options 29 Publishing Message Formats Overview 30 Message Presentation 30 Kafka Topics, Keys And Messages 31 Nanomsg Topics And Message Payloads 31 Message Key Format (subject To Change) 32 Message Payloads 34

Directory Messages 34 Probes 34 Example Messages 34 Managed Entities 35 Example Message 35 Dataviews 35 Example Message 35 Metrics Messages 36 Raw And Enriched Forms Of Metrics Data 36 Headline Data 37 Table (cell) Data 38 Example Messages 38 Sample Times For Imported Data 40 Metadata Messages 40 Severity Messages 40 Example Messages 40 Snooze Messages 44 Example Messages 44 User Assignment Messages 46 Example Messages 46 Features Common To All Types Of Message 48 Timestamps 48 Operation 48 Request/reply Messages 49 Resend Directory Request 49 Snapshot Metrics Request 49 Target Syntax 50 Exact Vs Wildcard Matching 51 Snapshot Request For Events 51 System Information Request 52 Responses 52 Appendix: Payload Schema 54

Publishing to Kafka and Nanomsg - Technical Reference Publishing Introduction The Gateway can publish data to external systems via a dynamically loaded adapter. The 64-bit Gateway distributions include adapters for publishing via Nanomsg or Apache Kafka. These are provided as shared objects in the lib64 directory (lib64/geneos-adapter-*.so). If you need to run the Gateway from a directory which is not the parent of lib64, please ensure that the adapter you want to use can be located either via the LD_LIBRARY_PATH environment variable or via the adapter.library.path setting. (See publishing > additionalSettings below.) Gateway can use the Nanomsg adapter to provide a direct (peer-to-peer) link with a third party application.

In a Hot Standby configuration, the application needs to connect to both the primary and secondary gateways; the protocol used allows it to detect which is active. Gateway can use the Kafka adapter to publish data to a Kafka cluster (minimum version 0.9), from which it can be read by any system which implements a Kafka subscriber. In a Hot Standby configuration, only the active gateway will publish to the Kafka cluster; data consumers are isolated from any failover. The Kafka adapter supports SSL encryption and authentication; this needs to be configured both on the Kafka cluster and in the Gateway setup file.

Please see publishing > additionalSettings for more details.

Whichever adapter is used (a gateway can be configured to use either Nanomsg or Kafka, but not both at once), the gateway streams a fixed set of data: l directory information: data about each probe, managed entity and dataview known to the gateway l metrics: all dataview cells and headlines l metadata: severity, snooze and user assignment information This data is published using JSON-based formats which are described below. Note: The publishing of gateway data is only supported on 64-bit Linux and 64-bit Solaris x86. Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 6 of 65

Configuration publishing This section defines the parameters controlling publishing to external systems. publishing > enabled This setting allows publishing to be enabled/disabled. By default publishing is turned off, however if a publishing section exists then it is turned on. This setting allows that to be overridden, so publishing is disabled but there is configuration saved for a later date. Mandatory: No Default: true publishing > adapter The adapter section allows the adapter to be selected and holds settings specific to the selected adapter.

publishing > adapter > nanomsg Settings for the Nanomsg adapter.

publishing > adapter > nanomsg > publishPort This defines the port on which the Nanomsg adapter listens for connections from data subscribers. Mandatory: No Default: 7070 publishing > adapter > nanomsg > requestPort This defines the port on which the Nanomsg adapter listens for requests for status information, snapshots and retransmission of directory information. If this is not specified then it will listen on the next port after the publishing > adapter > nanomsg > publishPort.

Mandatory: No Default: (publishing > adapter > nanomsg > publishPort) + 1 publishing > adapter > nanomsg > secondaryPorts This section is used if the secondary gateway needs to use different publishing ports to the primary, for example if both gateways are running on the same host in a testing environment. If this section is not specified then the secondary gateway will use the values specified in publishing > adapter > nanomsg > publishPort and publishing > adapter > nanomsg > requestPort Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 7 of 65

publishing > adapter > nanomsg > secondaryPorts > publishPort This allows the value set in publishing > adapter > nanomsg > publishPort to be overridden on the secondary gateway. Mandatory: No Default: publishing > adapter > nanomsg > publishPort publishing > adapter > nanomsg > secondaryPorts > requestPort This allows the value set in publishing > adapter > nanomsg > requestPort to be overridden on the secondary gateway. Mandatory: No Default: publishing > adapter > nanomsg > requestPort publishing > adapter > kafka Settings for the Kafka adapter. publishing > adapter > kafka > topicPrefix This specifies the first part of the topic names under which data is published and the first part of the topic name to which the Kafka adapter will subscribe for requests, such as metrics snapshots.

By default, the publishing topics used include geneos-probes, geneos-raw.table, (and eight other names beginning geneos-) and the request topic is geneos-requests. This setting allows the topics to be changed to, for example, my-test-probes, my-test-raw.table and so on. Mandatory: No Default: geneospublishing > adapter > kafka > brokerList This specifies a comma-separated list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The adapter will make use of all servers irrespective of which servers are specified here for bootstrapping - this list only impacts the initial hosts used to discover the full set of brokers.

Please refer to the description of bootstrap.servers in the Kafka documentation for more information. Mandatory: No Default: localhost:9092 publishing > additionalSettings This advanced configuration option allows additional settings to be specified for an adapter. These settings either control the way the adapter is loaded or are interpreted by the adapter itself. Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 8 of 65

Settings are entered using the same syntax as a Java properties file: each line specifies a key-value pair. Settings with the prefix adapter. control the way the adapter is loaded. At present, the settings available are: l adapter.library.path, which allows you to specify the location of the shared library that implements the adapter. By default, this is loaded from the lib64 sub-directory of the Gateway's working directory; you can specify an alternate location, either as a path relative to the Gateways's working directory or as an absolute path (as in the example above.) l adapter.gateway.name, which is used, with the topic prefix, to construct a unique Kafka consumer group id for the gateway.

The default group id is , e.g. geneos-DevGateway. You can override the gateway name part with this setting, but generally will not need to do so.

The Kafka adapter uses the rdkafka library to connect to Kafka and supports transparent passthrough of settings to the library. Any global setting (other than callbacks) defined in the librdkafka documentation can be used by prefixing their names with kafka.. In the example above, the key kafka.client.id refers to the librdkafka option documented as client.id. Additionally, default topic settings can be specified by prefixing their names with kafka.topic:. (Note the use of a colon , here, since some global settings begin with the string topic..) Note: The Kafka adapter discards incoming messages when the librdkafka publishing queue is full.

When publishing is resumed, the adapter reports the number of discarded messages. The length of the queue is modified by including a kafka.queue.buffering.max.message setting here. SSL encryption To use SSL encryption for the connection, the SSL protocol must be enabled and the certificate used to sign the Kafka brokers' public keys must be trusted by the Kafka adapter. Use the following template, replacing the location with your credentials: kafka.security.protocol=ssl kafka.ssl.ca.location= SSL encryption with SSL authentication To use SSL encryption with SSL authentication, the Kafka adapter must be able to present a certificate which is signed using a certificate trusted by the broker(s).

Use the following template, replacing the locations with your credentials: kafka.security.protocol=ssl kafka.ssl.ca.location= kafka.ssl.certificate.location= kafka.ssl.key.location= Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 9 of 65

There are no additional settings for the Nanomsg adapter. Mandatory: No publishing > strategies Publishing strategies allow you to configure greater control over the publishing of Geneos dataviews. The current supported strategies are: l Pivoting Tables Some dataviews are published by default as a set of name-value pairs; 'pivoting' exchanges rows and columns to provide a more useful and SQL like row format. The templates directory of the Gateway distribution contains a gateway include file that shows how to pivot some of the more relevant Geneos dataviews.

Figure 29.2 - A dataview that should be published in pivoted form, so that the left-hand column becomes the table header Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 10 of 65

l Scheduled publishing of dataview You may also schedule publishing of dataviews that are updated infrequently to provide data points at fixed points in time. publishing > strategies > strategyGroup Strategies may be grouped for organisational purposes. This has no effect on their application but is useful for grouping categories and making them easier to find in setups. Figure 29.3 - Grouping publishing > strategies > strategyGroup > name Name of the strategy group Mandatory: Yes publishing > strategies > strategy A strategy is defined as follows: Figure 29.3 - Publishing strategy publishing > strategies > strategy > name Name to uniquely identify the strategy.

Mandatory: Yes See Publishing strategy definition Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 11 of 65

publishing > strategies > strategy > targets The set of dataviews that the strategy will apply to. Mandatory: No See Publishing strategy definition publishing > strategies > strategy > targets > target An XPath that identifies a dataview or dataviews that the strategy applies to. The target must be a dataview and this will be checked when setup is validated. Mandatory: No See Publishing strategy definition publishing > strategies > strategy > pivotDataview When selected this will pivot the target dataviews such that the rownames of the original dataview will become column names and the adjacent columns become rows.

Mandatory: No See Publishing strategy definition publishing > strategies > strategy > schedule Scheduling options for matching dataviews. Mandatory: No See Publishing strategy definition publishing > publishPort This setting has been deprecated. Please use the corresponding setting under publishing > adapter > nanomsg Mandatory: No Default: 7070 publishing > requestPort This setting has been deprecated. Please use the corresponding setting under publishing > adapter > nanomsg Mandatory: No Default: (publishing > adapter > nanomsg > publishPort) + 1 publishing > secondaryPorts These settings have been deprecated.

Please use the corresponding settings under publishing > adapter > nanomsg Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 12 of 65

publishing > secondaryPorts > publishPort This setting has been deprecated. Please use the corresponding setting under publishing > adapter > nanomsg Mandatory: No Default: publishing > adapter > nanomsg > publishPort publishing > secondaryPorts > requestPort This setting has been deprecated. Please use the corresponding setting under publishing > adapter > nanomsg Mandatory: No Default: publishing > adapter > nanomsg > requestPort Standardised Formatting Some data can vary in format from dataview to dataview or even from row to row or column to column. Standardised Formatting allows normalisation of this data to a standard format for downstream systems.

Currently dateTime is supported. The dateTime formatter can be applied as shown below: Date-times, dates and times are formatted to ISO 8601 format. Type Format Date-time 2012-07-27T19:12:00Z Date-time with micro seconds 2012-07-27T19:12:00.123456Z Date 2012-07-27 Note the adjustment to UTC. Cells containing only times are assumed to be for the current day and formatted as a date-time. There are two types of standardised formatting: System supplied formatting and User specified formatting. System Supplied Formatting The Gateway provides a set of converters for cells in Geneos plugins. These are defined in a file located in /standardisedformats/formats.json.

By default, the gateway resources directory is /resources but you can modify this using a command-line option.

This data in this file is in JavaScript Object Notation (JSON). Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 13 of 65

It is recommended that this file be left unchanged. However, in unusual circumstances it may be preferable to update the formats.json file rather than generate a set of user specified formatting. The file has a number of entries under the "formats" label at the start. Taking one as an example it breaks down as follows: { "formats" : [ { "type" : "datetime", "format " % a %b %d %H:%M:%S %Y", "region" : "gateway", "plugin" : "Gateway-gatewayData", "cell " row" : "licenseExpiryDate", "column" : "value"} }, ...

Element Description type The type field tells us the datatype of the data being formatted. Currently the only format is "datetime" format Specifies the expected format of the data. See the following section for a description of the time parsing codes. Where you need several of these to accommodate different servers that may generate dates in differing formats use the "formats" tag defined below. formats where a variable may be in a number of formats an array of alternative formats may be specified.

formats " % a %b %d %H:%M:%S %Y", "%s"] ... } region Timezone information for the formatter. For our purposes these should be "gateway" or "netprobe" depending on the location of the sampler. The formatter will use the timezone of the specified component. Where netprobe timezone isn't known it will fall back to the Gateway. plugin The name of the plugin this formatter applies to. The formatter applies to all dataviews created by the plugin unless you restrict this.

Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 14 of 65

Element Description dataview Only applies to variables within the named dataview.

dataview", "overview" ... } cell Targets the formatting of a specific row / column pair. For more flexibility you can target a range of columns. { ... "cell " row" : "licenseExpiryDate", "column" : "Value"} ... } cols An array or regular expressions used to target columns witin a dataview. { ... "cols [ " applied|changed"] ... } Will target all column headings containing "applied" or "changed" as part of the column name. i.e. "appliedDate" headlines An array or regular expressions used to target columns witin a dataview. Works as cols above and can be used in conjunction.

Note: Some "formats" entries may end with a special format called "raw". This prevents errors from being logged where none of the formats were able to process the input. User Specified Formatting Certain dataviews contain cells where the format of the cell cannot be determined. This may be because the data is generated by a user script, or it may be because a Geneos plugin is extracting data whose format is determied by the enviroment of the server on which the plugin is running. Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 15 of 65

The standardised formatting tab on samplers allows the user to describe the originating format of variables (cells and headlines).

This allows them to be transformed into standardised format. You can also specify which cells in the sampler the format is applied to by using the dataview name and the applicability sections. Configuration samplers > sampler > standardisedFormatting see Standardised Formatting samplers > sampler > standardisedFormatting > dataviews > dataview Provides a list of standardised formatting variables to apply to one or more dataviews in the sampler. When the name of the dataview is set the variable definitions are restricted to the named dataview. If the name is left unset the variable definitions applies to all dataviews belonging to the sampler.

Mandatory: No Default: Not set samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable Variable definition specifying type and applicability of variable. samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > type Specifies the type of the variable. Currently on dateTime is supported. samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > type > dateTime Specifies that the variable is a date-time (includes dates and times which are assumed to be for the current day).

Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 16 of 65

samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > type > dateTime > formats > format Specifies the expected format of the data. See the following section for a description of the time parsing codes. There can be several of these to accommodate that different servers may generate dates in differing formats. Mandatory: Yes samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > type > dateTime > ignoreErrorsIfFormatsFail If set to true and the none of the formats can translate the data then the errors generated are suppressed.

This can be used to specify a NULL format that overrides a system provided one with no formatting. This can be useful for exmaple if you don't care about formatting these and the output is in an unexpected format due to locale.

Mandatory: No Default: false samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > type > dateTime > overrideTimezone The default timezone date is that of the dataview. i.e. if the Probe has a timezone set the date would be interpretted from that timezone. If not the timezone will match the Gateway's timezone. This setting allows you to explicitly set the timezone from which the data is being received. Mandatory: No samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > applicability Allows matching of formats to dataview variables.

Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 17 of 65

samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > applicability > headlines Allows you to match variables to headline variables. samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > applicability > headlines > regex A regular expression to match against the headline names. Expression Effect date matches all headlines containing date date$ matches all headlines ending in date$ ^date$ matches the healdine date All values within the matching headlines will have the formatter applied for publishing. Mandatory: No samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > applicability > columns > regex A regular expression to match against the column names.

Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 18 of 65

Expression Effect date matches all columns containing date date$ matches all columns ending in date$ ^date$ matches the column date All values within the matching columns will have the formatter applied for publishing. Mandatory: No samplers > sampler > standardisedFormatting > dataviews > dataview > variables > variable > applicability > cells Maps to a specific row and column. These mappings are good for dataviews made from name, value pairs and take priority on lookup over column regular expressions.

As these are specific the row and columns are not regular expressions.

Mandatory: No operatingEnvironment > debug > DirectoryManager > showStandardisedFormattingErrors Turns on error reporting for errors found processing inputs to standardised formats. If this is not enabled and errors are found then a summary of the number of errors found is logged at 10 minute intervals. Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 19 of 65

Gateway Publishing stats Description The gateway package comes with a utility to help understand that amount of data that the gateway publishes.

This utility can be used to decide on the requirements of applications that consume gateway data from gateway publishing. The utility is a command-line application located in resources/nanomsg_stats and called nanomsg_ stats.. The utility connects to a gateway that is publishing data using nanomsg (See Publishing data to external systems). The utility consumes all the data sent by the gateway and periodically outputs statistics on the amount of data the gateway has sent on each nanomsg channel. The user can configure the following: l the nanomsg port on which the gateway to connect to is publishing.

l the file which will be used to store the statistics l how often to collate the statistics and write the data to the file l a filter to apply to the published messages before they are processed by the utility. Once started the utility keeps running until it is killed by the user. Every n seconds (where n is specified by the user), it will output a set of statistics. One for each topic and a summary row containing information about all the messages processed.

In the example below, you can see that the program collected data every 60 seconds and ran for 3 minutes. Statistics were written to the output file every minute. Output The utility produces a CSV file. The first line in the file are the column headings, while subsequent lines are summary lines. Summary lines summarise the amount of data seen over a set period of time either on a single topic or on all topics. An example output is shown below Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 20 of 65

Stat s Time stam p Dur atio n URL Topic Nu mb er of Fiel ds Num ber of Mes sage s Max Mes sag e Size Avg Mes sag e Size Min Mes sag e Size Total Size of Mes sage s Top ic 201 6-11- 15 10:3 8:57 60 tcp://127.0 .0.1:7070 enriched.headlines.Probes.Pro bes..Self.V.G1 9 1 473 473 473 473 Top ic 201 6-11- 15 10:3 8:57 60 tcp://127.0 .0.1:7070 enriched.table.CPU.CPU.Syste m.ME1.P1.G1 11 1160 515 511 505 592 578 Top ic 201 6-11- 15 10:3 8:57 60 tcp://127.0 .0.1:7070 enriched.table.CPU.CPU.Syste m.ME2.P2.G1 11 1200 516 511 505 613 140 Top ic 201 6-11- 15 10:3 8:57 60 tcp://127.0 .0.1:7070 enriched.table.Hardware.Hard ware.System.ME1.P1.G1 3 25 359 344 333 8612 Top ic 201 6-11- 15 10:3 8:57 60 tcp://127.0 .0.1:7070 enriched.table.Hardware.Hard ware.System.ME2.P2.G1 3 25 359 344 333 8612 Top ic 201 6-11- 15 10:3 8:57 60 tcp://127.0 .0.1:7070 enriched.table.Probes.Probes..

Self.V.G1 23 22 924 921 918 202 62 Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 21 of 65

Stat s Time stam p Dur atio n URL Topic Nu mb er of Fiel ds Num ber of Mes sage s Max Mes sag e Size Avg Mes sag e Size Min Mes sag e Size Total Size of Mes sage s Top ic 201 6-11- 15 10:3 8:57 60 tcp://127.0 .0.1:7070 raw.headlines.Probes.Probes.. Self.V.G1 9 1 468 468 468 468 Top ic 201 6-11- 15 10:3 8:57 60 tcp://127.0 .0.1:7070 raw.table.CPU.CPU.System.M E1.P1.G1 11 1161 510 506 500 587 284 Top ic 201 6-11- 15 10:3 8:57 60 tcp://127.0 .0.1:7070 raw.table.CPU.CPU.System.M E2.P2.G1 11 1200 511 506 500 607 140 Top ic 201 6-11- 15 10:3 8:57 60 tcp://127.0 .0.1:7070 raw.table.Hardware.Hardware.

System.ME1.P1.G1 3 25 354 339 328 8487 Top ic 201 6-11- 15 10:3 8:57 60 tcp://127.0 .0.1:7070 raw.table.Hardware.Hardware. System.ME2.P2.G1 3 25 354 339 328 8487 Top ic 201 6-11- 15 10:3 8:57 60 tcp://127.0 .0.1:7070 raw.table.Probes.Probes..Self. V.G1 23 22 919 916 913 201 52 Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 22 of 65

Stat s Time stam p Dur atio n URL Topic Nu mb er of Fiel ds Num ber of Mes sage s Max Mes sag e Size Avg Mes sag e Size Min Mes sag e Size Total Size of Mes sage s Gat ewa y 201 6-11- 15 10:3 8:57 60 tcp://127.0 .0.1:7070 4867 924 509 328 247 5695 Top ic 201 6-11- 15 10:3 9:57 60 tcp://127.0 .0.1:7070 enriched.headlines.Probes.Pro bes..Self.V.G1 9 4 473 473 473 1892 Top ic 201 6-11- 15 10:3 9:57 60 tcp://127.0 .0.1:7070 enriched.table.CPU.CPU.Syste m.ME1.P1.G1 11 1160 516 511 503 592 641 Top ic 201 6-11- 15 10:3 9:57 60 tcp://127.0 .0.1:7070 enriched.table.CPU.CPU.Syste m.ME2.P2.G1 11 1200 516 511 504 613 030 Top ic 201 6-11- 15 10:3 9:57 60 tcp://127.0 .0.1:7070 enriched.table.Hardware.Hard ware.System.ME1.P1.G1 3 13 359 345 332 4486 Top ic 201 6-11- 15 10:3 9:57 60 tcp://127.0 .0.1:7070 enriched.table.Hardware.Hard ware.System.ME2.P2.G1 3 13 359 345 332 4486 Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 23 of 65

Stat s Time stam p Dur atio n URL Topic Nu mb er of Fiel ds Num ber of Mes sage s Max Mes sag e Size Avg Mes sag e Size Min Mes sag e Size Total Size of Mes sage s Top ic 201 6-11- 15 10:3 9:57 60 tcp://127.0 .0.1:7070 enriched.table.Probes.Probes.. Self.V.G1 23 24 924 921 918 221 04 Top ic 201 6-11- 15 10:3 9:57 60 tcp://127.0 .0.1:7070 raw.headlines.Probes.Probes.. Self.V.G1 9 4 468 468 468 1872 Top ic 201 6-11- 15 10:3 9:57 60 tcp://127.0 .0.1:7070 raw.table.CPU.CPU.System.M E1.P1.G1 11 1160 511 506 498 586 841 Top ic 201 6-11- 15 10:3 9:57 60 tcp://127.0 .0.1:7070 raw.table.CPU.CPU.System.M E2.P2.G1 11 1200 511 506 499 607 030 Top ic 201 6-11- 15 10:3 9:57 60 tcp://127.0 .0.1:7070 raw.table.Hardware.Hardware.

System.ME1.P1.G1 3 13 354 340 327 4421 Top ic 201 6-11- 15 10:3 9:57 60 tcp://127.0 .0.1:7070 raw.table.Hardware.Hardware. System.ME2.P2.G1 3 13 354 340 327 4421 Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 24 of 65

Stat s Time stam p Dur atio n URL Topic Nu mb er of Fiel ds Num ber of Mes sage s Max Mes sag e Size Avg Mes sag e Size Min Mes sag e Size Total Size of Mes sage s Top ic 201 6-11- 15 10:3 9:57 60 tcp://127.0 .0.1:7070 raw.table.Probes.Probes..Self. V.G1 cw b MW Gat ewa y 201 6-11- 15 10:3 9:57 60 tcp://127.0 .0.1:7070 4828 924 511 327 246 5208 Top ic 201 6-11- 15 10:4 0:57 60 tcp://127.0 .0.1:7070 enriched.headlines.Probes.Pro bes..Self.V.G1 9 3 473 473 473 1419 Top ic 201 6-11- 15 10:4 0:57 60 tcp://127.0 .0.1:7070 enriched.table.CPU.CPU.Syste m.ME1.P1.G1 11 1160 516 511 502 592 529 Top ic 201 6-11- 15 10:4 0:57 60 tcp://127.0 .0.1:7070 enriched.table.CPU.CPU.Syste m.ME2.P2.G1 11 1200 516 511 504 613 062 Top ic 201 6-11- 15 10:4 0:57 60 tcp://127.0 .0.1:7070 enriched.table.Hardware.Hard ware.System.ME1.P1.G1 3 9 359 345 330 3108 Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 25 of 65

Stat s Time stam p Dur atio n URL Topic Nu mb er of Fiel ds Num ber of Mes sage s Max Mes sag e Size Avg Mes sag e Size Min Mes sag e Size Total Size of Mes sage s Top ic 201 6-11- 15 10:4 0:57 60 tcp://127.0 .0.1:7070 enriched.table.Hardware.Hard ware.System.ME2.P2.G1 3 8 359 347 330 2777 Top ic 201 6-11- 15 10:4 0:57 60 tcp://127.0 .0.1:7070 enriched.table.Probes.Probes.. Self.V.G1 23 24 924 921 918 221 04 Top ic 201 6-11- 15 10:4 0:57 60 tcp://127.0 .0.1:7070 raw.headlines.Probes.Probes.. Self.V.G1 9 3 468 468 468 1404 Top ic 201 6-11- 15 10:4 0:57 60 tcp://127.0 .0.1:7070 raw.table.CPU.CPU.System.M E1.P1.G1 11 1160 511 506 497 586 728 Top ic 201 6-11- 15 10:4 0:57 60 tcp://127.0 .0.1:7070 raw.table.CPU.CPU.System.M E2.P2.G1 11 1200 511 506 499 607 062 Top ic 201 6-11- 15 10:4 0:57 60 tcp://127.0 .0.1:7070 raw.table.Hardware.Hardware.

System.ME1.P1.G1 3 9 354 340 325 3063 Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 26 of 65

Stat s Time stam p Dur atio n URL Topic Nu mb er of Fiel ds Num ber of Mes sage s Max Mes sag e Size Avg Mes sag e Size Min Mes sag e Size Total Size of Mes sage s Top ic 201 6-11- 15 10:4 0:57 60 tcp://127.0 .0.1:7070 raw.table.Hardware.Hardware. System.ME2.P2.G1 3 8 354 342 325 2737 Gat ewa y 201 6-11- 15 10:4 0:57 60 tcp://127.0 .0.1:7070 4808 924 511 325 245 7977 The CSV file has the following columns; Column Description Stats This indicates if this row shows data from a single nanomsg topic or all the topics processed from the gateway. The value is either Topic or Gateway. Timestamp This is the time that the statistics were gathered.

It is the end of the period the stats were gathered over and not the start.

Duration This is the duration over which the stats were gathered. This value is measured in seconds URL This is the URL used to connect to the Gateway publishing. Topic This is the Nanomsg Topic name. (This column will be empty for Gateway rows) Number of Fields Number of fields in the data part of the message. (This column will be empty for Gateway rows) Number of Messages Number of messages received in the specified time period in bytes. Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 27 of 65

Column Description Max Message Size Maximum message size seen in the specified time period in bytes.

Avg Message Size Average message size over the specified time period in bytes. Min Message Size Minimum message size seen in the specified time period in bytes. Total Size of Messages Sum of the sizes of all messages seen in the time period. This is measured in bytes Gateway Name of the gateway that generated data summarised in this row. (This column will be empty for Gateway rows) Probe Name of the probe that generated data summarised in this row. (This column will be empty for Gateway rows) Managed Entity Name of the managed entity that generated data summarised in this row. (This column will be empty for Gateway rows) Sampler Name of the sampler that generated data summarised in this row.

(This column will be empty for Gateway rows) Sampler Type Type of the sampler that generated data summarised in this row. (This column will be empty for Gateway rows) Dataview Name of the dataview that generated data summarised in this row. (This column will be empty for Gateway rows) OS Type OS Type of the probe that generated data summarised in this row. (This column will be empty for Gateway rows) Plugin Type Plugin type of sampler that generated data summarised in this row. (This column will be empty for Gateway rows) Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 28 of 65

Command Line options The nanomsg_stats utility takes up to 5 command line parameters. l The first parameter is the url to connect to. This url contains the gateway host and the port on which the gateway is publishing. The argument is in the form tcp . So to connect to the default port on localhost you would use one of the following urls; tcp://locahost:7070, tcp://127.0.0.1:7070. This argument is mandatory. l The second parameter is a second url to connect to. This is the url that contains the gateway host and the port on which the gateway request response service is located. The argument is in the form tcp .

So to connect to the default port on localhost you would use one of the following urls; tcp://locahost:7071, tcp://127.0.0.1:7071. l The third argument is the name of the csv file in which to store the statistics acquired. This argument is mandatory.

l The fourth argument is the time period over which the utility gathers data before producing stats. This argument is a time in seconds. It is optional and if not supplied the system will produce stats every 60s. l The fifth argument is a filter that is applied to the data before it is processed. Any topic that does not start with the string provided will be ignored. So a value of "raw.table" will ignore the following topics: l Any topic starting with enriched.headline l Any topic starting with raw.headlines l Any topic starting with enriched.table It should be noted that the filter occurs before the stats are generated.

This means that any data that is filtered out will not contribute to the summary Gateway statistics.

Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 29 of 65

Publishing Message Formats Overview From Geneos version 3.4 onward, Gateway can be configured to publish metrics, directory data and metadata in JSON format. Initially this used the Nanomsg messaging library and the message formats were private to this use case and so were not documented. As of Geneos version 3.7, Gateway also supports publishing to a Kafka cluster, as a Kafka "producer client". As well as providing a more resilient mechanism for publishing, this allows other systems to consume Geneos data by implementing a Kafka consumer client.

Support for each publishing platform and operating system is provided by an adapter implemented as a shared library. At present, both Nanomsg and Kafka are supported only on 64-bit Linux and Solaris x86. The format of the JSON 'payload' of the published messages is determined by the Gateway, so it is the same whichever adapter is used. The configuration settings used to enable publishing using either adapter are described above. This remainder of this document describes the message formats used.

Message presentation Although the format of the message payload sent by the Gateway is the same in each case, the different publishing platforms support different mechanisms for selecting and identifying messages.

They also use different terminology to refer to similar concepts. In this document we use the following terms: l Topic A string identifying the category of message, for example "raw.headlines". Messages within a topic have similar schemas (in fact, apart from metrics messages, all messages in a topic have the same schema.) l Key A string identifying a subset of a topic, for example the key "gatewayLoad.gatewayLoad..Gateway.vp.Ad-hoc GW" might appear within the topic "raw.headlines". All messages with the same key will have the same schema, but there are better ways to infer the schema than by inspecting the key.

All messages which have a sequencing relationship to each other will have the same key: keys may be used to sub-divide topics for load balancing or to filter messages of interest for a particular application. Directory message topics are not subdivided in this way; they use an empty string as a key. Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 30 of 65

l Payload The actual data to be consumed by a client system. The payload is a JSON object, described in detail at Message payloads and Appendix: Payload schema below.

Kafka topics, keys and messages When the Kafka adapter is used: l The message topic is prefixed with a configurable string (the default is "geneos-") and used as the Kafka topic. l The message key is used to allocate messages to partitions within the Kafka topic and is available as the message key to Kafka clients. l The message payload becomes the message body. Nanomsg topics and message payloads The Nanomsg API simulates a limited form of topic-based subscription in that, if you subscribe to "Foo", you get all messages that begin with the string "Foo". When the Nanomsg adapter is used, the messages published consist of a modified topic string, followed by the payload.

l For messages within the metrics topics, the modified topic string is the message topic, followed by a period (\u002E), followed by the message key, followed by a pipe character (\u007C), so that the whole message is: .| l For messages in other topics, the modified topic string is the message topic followed by a pipe character (\u007C), so that the whole message is: | This message format allows Nanomsg clients to subscribe to individual topics, or to subsets of metrics topics identified by any leading substring of the key.

For example, with topic managedEntities, and payload { "data": { "timestamp": "2015-07-01T16:17:10.301Z", Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 31 of 65

name": "myEntity", "probe": "p", "gateway": "gw", "attributes": { "Purpose": "Realism" } }, "operation": "create" } { "data": { "timestamp": "2015-07-01T16:17:10.301Z", "name": "myEntity", "probe": "p", "gateway": "gw", "attributes": { "Purpose": "Realism" } }, "operation": "create" } the message sent via Nanomsg would be: managedEntities|{"data":{"timestamp":"2015-07-01T16:17:10.301Z","- name":"myEntity","probe":"p","gateway":"gw","attributes":{"Purpose":"Realism"}},"operation":"create"} Message key format (subject to change) In principle, the message key is an opaque string used to allocate messages to partitions (Kafka) or to support a limited form of low-level message filtering (Nanomsg).

Applications needing to filter messages should ignore the key and use the "target" object within the payload. In practice, "what is the format of the message key?" is a frequently asked question. At present, it is roughly equivalent to a reversed XPath delimited by dots: ...

For example (note that if a component name is not applicable, it will be omitted, but the following dot will not): Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 32 of 65

CPU.CPU..myEntiy.theProbe.Ad-hoc GW ...theProbe.Ad-hoc GW At present, the number of name components (and hence the number of dots) is the same for all metrics and (in the case of publishing via Kafka) metadata messages. This may change in future. Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 33 of 65

Message payloads A JSON schema for the publishing message payloads is provided as Appendix: Payload schema. The examples in this section are pretty-printed: the messages published by Gateway have no redundant whitespace. Directory messages Messages on the probes, managedEntities and dataviews topics provide information about the corresponding items in the Gateway directory hierarchy. They are published as these items are created and deleted and as their run-time attributes are updated. The Nanomsg adapter also allows clients to request retransmission of all messages in this category. Messages in this category are also retransmitted when publishing setup is reconfigured (for example to change the Kafka topic prefix) and the Nanomsg adapter is not being used.

Probes Example messages A message sent as a netprobe comes up: { "data": { "timestamp": "2015-07-01T16:18:23.263Z", "name": "theProbe", "gateway": "Ad-hoc GW", "osType": "Linux", "HostName": "linux-dev", "Port": "7036", "ConState": "Up", "OS": "Linux 2.6.18-371.4.1.el5", "Version": "GA3.1.1-150515" }, "operation": "update" } A message describing a virtual probe: { "data": { "timestamp": "2015-10-20T09:15:04.732Z", "name": "vp", "gateway": "Ad-hoc GW", Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 34 of 65

osType": "Virtual", "ConState": "Up" }, "operation": "update" } Points to Note: l The properties whose names start with capital letters are configuration ('HostName', 'Port') or run-time ('ConState', 'OS', 'Version') parameters. Of these, only 'ConState' is applicable to virtual probes. l When a connection is established to a probe, several update messages will be published, one for each run-time parameter.

Managed Entities Example message { "data": { "timestamp": "2016-05-25T15:02:18.255Z", "name": "theEntity", "probe": "theProbe", "gateway": "Ad-hoc GW", "attributes": { "Team": "Middleware", "Purpose": "Testing" } }, "operation": "update" } Note: l An update message will be published whenever a managed entity attribute is changed, added or deleted.

Dataviews Example message { "data": { "timestamp": "2016-05-27T12:51:16.009Z", Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 35 of 65

dataview": "CPU", "sampler": "CPU", "pluginName": "CPU", "type": "Default Samplers", "managedEntity": "basics", "probe": "theProbe", "gateway": "Ad-hoc GW", "topicSuffix": "CPU.CPU.Default Samplers.basics.theProbe.Ad-hoc GW", "availableTopics": [ "enriched.table.CPU.CPU.Default Samplers.basics.theProbe.Ad-hoc GW", "enriched.headlines.CPU.CPU.Default Samplers.basics.theProbe.Ad-hoc GW", "raw.table.CPU.CPU.Default Samplers.basics.theProbe.Ad-hoc GW", "raw.headlines.CPU.CPU.Default Samplers.basics.theProbe.Ad-hoc GW" ] }, "operation": "create" } Note: l Update messages will only be sent on the dataviews topic in response to a client request or a change in Publishing configuration.

Metrics messages Raw and enriched forms of metrics data Metrics data is available in two forms: l Raw form includes only the rows and cells provided by the data source (normally a Netprobe, but possibly a Gateway plugin, or, in the case of Gateway sharing, an exporting Gateway.) In this case the published sample time is provided by the data source. l Enriched form also includes additional rows and cells configured on the publishing Gateway and (usually) populated by rules. In this case the published sample time is the time on the publishing Gateway that the dataview was last updated (either by receiving an update or by processing a rule.) When new data for a row is provided by the data source and then values in that row are computed by Gateway rules, an update message will be published for each of these changes as they occur: first the change from the data source and then each change made by a rule.

Geneos Release v4.8 Geneos Gateway Publishing Data v1.0.0 - User Guide Published Date 22/08/2018 Page 36 of 65