Name

nstgeolocate — Geolocate Hosts obtained from an 'ntop' session, an 'ntopng' session or Geolocate IPv4 Address conversations from a network packet capture file on a Mercator World Map projection or Global imagery.

Synopsis

nstgeolocate [ -m ENTRY | --mode ENTRY ] [ --odir DIRECTORY ] [ --sname TEXT ] [ --src-type TEXT ] [ -url URL | --ntop-url URL ] [ -urlng URL | --ntopng-url URL ] [ --ntop-session-window-start-time [true]|false ] [ --ntop-netint-label TEXT ] [ --remove-uuid UUID ] [ --map-type TEXT ] [ --mercator-map-type TEXT ] [ --mercator-mark-type TEXT ] [ --mercator-mark-color TEXT ] [ --kml-map-type TEXT ] [ --webgl-map-type TEXT ] [ --pcap-file FILENAME ] [ --conv-cap-type TEXT ] [ --conv-cap-int TEXT ] [ --conv-xml FILENAME ] [ --conv-display-filter TEXT ] [ --conv-pref-opts TEXT ] [ --conv-annotation TEXT ] [ --conv-cap-host HOSTNAME|IPv4ADDR ] [ --conv-cap-dumpcap-ver TEXT ] [ --conv-connect-conversation [true]|false ] [ --conv-connection-line-color TEXT ] [ --conv-connection-line-width TEXT ] [ --conv-fqdn [true]|false ] [ --new-mercator [true]|false ] [ --crontab-user TEXT ] [ --cron-map-min INTEGER ] [ --cron-map-hr INTEGER ] [ --cron-map-other TEXT ] [ --rotate-interval-min INTEGER ] [ --rotate-interval-hr INTEGER ] [ --rotate-interval-day TEXT ] [ --cron-prune-min INTEGER ] [ --cron-prune-hr INTEGER ] [ --cron-prune-other TEXT ] [ --prune-min INTEGER ] [ --prune-day INTEGER ] [ --map-title TEXT ] [ --series-description TEXT ] [ --data-series-cnt INTEGER ] [ --quick-stats [true]|false ] [ --public-ipaddr-ref [true]|false ] [ -h [true]|false | --help [true]|false ] [ -H [true]|false | --help-long [true]|false ] [ -v [true]|false | --verbose [true]|false ] [ --version [true]|false ]

Overview

The nstgeolocate script is used to mark the origin of each Host and/or IPv4 Address conversation on either a Mercator World Map projection or Global imagery from the results of interrogating an active 'ntop' session, an active 'ntopng' session or from a network packet capture file. Additional network traffic information may be included with the rendered results.

Supported Modes

xml

If you specify "xml" this script will generate XML (Extensible Markup Language) formated text to standard output (display) from the results of interrogating a system running an active ntop session, an active ntopng session or from a network packet capture file. The XML output using ntop or ntopng as the data source will include the "IPv4 Address, "Latitude" and "Longitude" for each host that was successfully Geolocated. The XML output using a network packet capture file as the data source will include the "IPv4 Address, "Latitude" and "Longitude" for each host in the capture that was successfully Geolocated as well as all IPv4 host conversations.

mercator

Specify "mercator" to generate a Mercator Map image with results from interrogating a system running an active ntop session, an active ntopng session or from a network packet capture file.

kmz

Specify "kmz" to generate a KML (Keyhole Markup Language) formatted document from the results of interrogating a system running an active ntop session, an active ntopng session or from a network packet capture file. The KML document will be compressed using the zip utility producing a KMZ file. KML is an XML-based language schema for expressing geographic annotation and visualization on existing or future Web-based, two-dimensional maps and three-dimensional Earth browsers (Geocoding). KML was developed for use with Google Earth. Use the "kmz" mode when rendering nstgeolocate results with Google Earth.

webgl

Specify "webgl" to generate output for a single series NST WebGL Globe implementation in JSON format (Reference: "https://www.chromeexperiments.com/globe"). Both an active ntop session or an active ntopng session can be used with the "webgl" mode.

webgldataset

Specify "webgldataset" to generate a multi-series NST WebGL Globe JSON document from a list of existing single series WebGL JSON documents. Use the "--data-series-cnt" to limit the number of single series WebGL JSON documents used. Both the "--src-type" and the "--map-title" parameters must be set for this mode.

text

Specify "text" for similar output to the XML mode but with XML tags removed. The "text" mode is only available for data source type: "ntop" or "ntopng".

updateindicators

Specify "updateindicators" to update status indicators for both the state of the ntop or ntopng daemon and the state for automatic map generation using nstgeolocate under cron control. This mode is typically used by the NST WUI.

prunemap

Specify "prunemap" to remove older generated map imagery and associated XML status information files found within the "--odir" directory hierarchy. Use either the mandatory "--prune-min" or "--prune-day" option to establish the desired pruning window.

listprunemap

Specify "listprunemap" to list generated map imagery and associated XML status information files found within the "--odir" directory. Use the "--prune-min" or "--prune-day" option for the desired pruning window. This method is nondestructive and can be used to help determine your pruning time frame.

rmsessiondir

Specify "rmsessiondir" to remove an entire session name directory: "--sname". All "crontab" entries associated with the session name directory: "--sname" will be removed first.

rmcrontab

Specify "rmcrontab" to remove all "crontab" entries associated with the session name directory: "--sname".

bash

Specify "bash" to display internal 'Bash' variables specific to nstgeolocate for 'Bash' scripting integration.

Options

The following command line options are available:

[-m ENTRY] | [--mode ENTRY]

This option controls what the nstgeolocate script will do. The following modes are available: ("xml", "mercator", "kmz", "webgl", "webgldataset", "text", "updateindicators", "prunemap", "listprunemap", "rmsessiondir", "rmcrontab" and "bash"). The description for each supported mode is described above.

[--odir DIRECTORY]

This directory will contain a tree of all "nstgeolocate" session directories specified by the "--sname" session name directory option. If omitted, it defaults to directory: "/var/nst/nstgeolocate". The directory will be created if it does not already exist.

[--sname TEXT]

This script may need to generate intermediate files and map images before the final results are produced. This option enables one to specify a session name directory beneath the "--odir" directory for files produced by this script. The location of the session name directory will be based upon the "--mode", "--src-type" and "--map-type" type selected. If omitted, it defaults to a name representative of the default "--mode", "--src-type" and "--map-type". If the "--mode mercator", the "--src-type ntop" or "--src-type ntopng" and the "--mercator-map-type host" options were selected, then the default "--sname" session name directory will be: "mercator/host/wm_host". If the "--mode kmz", the "--src-type pcap2conv" and the "--mercator-map-type conv" options were selected, then the default "--sname" session name directory will be: "mercator/host/wm_conv". If the "--mode kmz" and the "--src-type ntop" options were selected, then the default "--sname" session name directory will be: "kml/host/kml_host". If the "--mode kmz" and the "--src-type pcap2conv" options were selected, then the default "--sname" session name directory will be: "kml/conv/kml_host". If the base output directory option: "--odir" is set to: "/var/nst/nstgeolocate" and the session name directory "--sname" is set to: "wm_host_5min" with options "--mode mercator", "--src-type ntop or ntopng" and "--mercator-map-type host" used, then the working directory for this 'nstgeolocate' session will be located in directory: "/var/nst/nstgeolocate/mercator/host/wm_host_5min". The session name directory will be created if it does not already exist.

[--src-type TEXT]

Use this option to select the data source type for Geolocating 'hosts' or plotting IPv4 host 'conversations'. Currently the valid data source types are: "ntop", "ntopng", "webgl" and "pcap2conv". The default data source type is "ntop". When choosing the "ntop" data source type, also specific which "ntop" service to use with the "--ntop-url" option. When choosing the "ntopng" data source type, also specific which "ntopng" service to use with the "--ntopng-url" option. The "webgl" data source must be used with mode: "--mode webgldataset". When choosing the "pcap2conv" data source type, also specific which "network packet capture" file to use with the "--pcap-file" option.

[-url URL] | [--ntop-url URL]

This option is used to specify the "URL" of an active ntop session. It is used with data source type: "ntop". The default value is an ntop service running on this local NST system: "https://127.0.0.1:3001"

[-urlng URL] | [--ntopng-url URL]

This option is used to specify the "URL" of an active ntopng session. It is used with data source type: "ntopng". The default value is an ntopng service running on this local NST system: "https://127.0.0.1:9943"

[--ntop-session-window-start-time [true]|false]

Use this option to set the "From Time" in the Map Title, Legend or Description area to the computed "ntop" or "ntopng" session collection window start time. The "From Time" is the begining "ntop" or "ntopng" time for the host collection window. ntop or ntopng will purge idle hosts prior to this time. This option is typically used when manually generating a single "ntop Hosts" or "ntopng Hosts" Mercator World Map projection, KML document or a WebGL JSON document.

[--ntop-netint-label TEXT]

This label text will overide the network interface value derived by interrogating either the active local or remote ntop or ntopng session specified by the "--ntop-url" or "--ntopng-url" option. The text will appear in the map legend or KML description area to identify the corresponding network interface names used by the ntop or ntopng session. When an ntop session is commenced by the setup_ntop script, it will generate the following information file: "/usr/share/ntop/html/ntopsessioninfo.xml" which contains one or more network interface names used by the ntop session. When an ntopng session is commenced by systemd using systemctl start ntopng.service, it will generate the following information file: "/usr/share/ntopng/httpdocs/ntopngsessioninfo.xml" which contains one or more network interface names used by the ntop session. Therefore, this option is typically not used unless a special overiding label is desired.

[--remove-uuid UUID]

This option may be used with the "--mode rmcrontab" mode to specify a targeted UUID for removal of all associated "crontab" entries. The UUID is a DCE compatible Universally Unique IDentifier produced by the uuid command. The nstgeolocate script uses a UUID to identify and bound a set of nstgeolocate script crontab entries for the automatic generation of map projections, global imagery, map rotation and pruning. Typically this option is not necessary because the UUID can be found in a XML session information file within each nstgeolocate session name directory. It may be necessary at times to use this option for "crontab" house cleaning if nstgeolocate session name directories are removed and leftover "crontab" entries remain.

[--map-type TEXT]

This option is used with modes: "updateindicators", "prunemap", "listprunemap", "rmsessiondir", "rmcrontab" and "bash" to specific either the "mercator" map type or the "kml" or "webgl" for the global imagery type. The "mercator" value is the default type.

[--mercator-map-type TEXT]

The following Mercator Map Types are available: ("host" and "conv"). Choose the "host" Mercator Map Type for geolocating hosts on the Mercator Map projection for source type: "ntop" or "ntopng". The "host" value is the default type for source type: "ntop". Choose the "conv" Mercator Map Type for geolocating conversations on the Mercator Map projection with source type: "pcap2conv". The "conv" value is the default type for source type: "pcap2conv".

[--mercator-mark-type TEXT]

Choose either a "point", "plus" sign or "star" as the mark on the Mercator Map image to represent the geolocation for each host. The "point" mark is the default type.

[--mercator-mark-color TEXT]

Choose the color for each Mercator mark type: 'point', 'plus' sign or 'star' that represents the geolocation for each host. The color "red" is the default value.

[--kml-map-type TEXT]

The following KML Map Types are available: ("host" and "conv"). Choose the "host" KML Map Type for geolocating hosts with source type: "ntop" or "ntopng". The "host" value is the default type for source type: "ntop" or "ntopng". Choose the "conv" KML Map Type for geolocating conversations with source type: "pcap2conv" and generating a KML Document. The "conv" value is the default type for source type: "pcap2conv".

[--webgl-map-type TEXT]

The following WebGL Map Types are available: ("host"). Choose the "host" WebGL Map Type for geolocating hosts with source type: "ntop" or "ntopng". The "host" value is the default type for source type: "ntop" or "ntopng".

[--pcap-file FILENAME]

Use this option with data source type: "pcap2conv" to specify the source network packet capture file from which the IPv4 Address conversation list will be generated. The file format can be in Wireshark's native capture file format: "libpcap" ('pcap' for short) or any other supported Wireshark formats (See the manual page for Wireshark for all currently supported network packet capture file formats).

[--conv-cap-type TEXT]

Use this option with IPv4 Address conversation generation to identify the NST network packet capture type associated with the 'pcap-file'. If the capture was produced using the NST Single-Tap Network Packet Capture implementation, then use type: "STYPE" which is the default value. If the capture was produced using the NST Multi-Tap Network Packet Capture implementation, then use type: "MTYPE". This option is also used for the creation of a 'Hyperlink' back to the appropriate NST WUI network packet capture page when rendered by "Google Earth".

[--conv-cap-int TEXT]

Use this option with IPv4 Address conversation generation to describe the name of each network interface used on the host system that actually performed the network packet capture. If more than one interface is specified, use a comma (,) delimiter character between each interface described. For example, one can enter a single network interface name: "eth1" for a Single-Tap Network Packet Capture. The following is an example for a Multi-Tap Network Packet Capture: "Tap0: eth1, Tap1: eth3, Tap3: eth6". Typically, this option is filled in by the NST WUI when used by either the "Single-Tap" or "Multi-Tap" Network Packet Capture page. This option is also used to provide linkage back to the appropriate NST WUI network packet capture page when rendered by "Google Earth".

[--conv-xml FILENAME]

Use this option with mode: "bash" data source type: "pcap2conv" to specify a previously generated IPv4 Address conversation XML file. Selective data from the XML file will be used in the sourced output.

[--conv-display-filter TEXT]

Apply a valid Wireshark display filter when generating the IPv4 Address conversation list. Only those packets that match the display filter will be used in the calculation. There is no default display filter, thus all packets will be used to generate the conversation list. This option only applies to source type: "pcap2conv".

[--conv-pref-opts TEXT]

Apply one or more valid Wireshark "Preference" options when generating the IPv4 Address conversation list. The preference option choices can be found in the global preference file: "/usr/share/wireshark/preferences". The applied option(s) will override the value(s) set in any preference file that is read in. Example 1 - Enable the lookup of IPv4 Addresses in each "GeoIP Database" that has been loaded: --conv-pref-opts "'ip.use_geoip:TRUE'". Example 2 - Set the "Name Resolution Concurrency" and the default "SSL" port: --conv-pref-opts "'name_resolve_concurrency:500' 'http.ssl.port:443'". ***Note: Enclose the entire option(s) parameter in double quotes with each preference option enclosed in a single quote.

[--conv-annotation TEXT]

Use this option to add an "Annotation" to the IPv4 Address conversation XML output and maps. This is used to document the results for historical review and analysis. Enclose the annotation in single or double quotes.

[--conv-cap-host HOSTNAME|IPv4ADDR]

Use this option with IPv4 Address conversation generation to set the name of the host system that actually performed the network packet capture. Enter either a Fully Qualified Domain Name (FQDN) or an IP Address for the host system.

[--conv-cap-dumpcap-ver TEXT]

Use this option to add the "dumpcap version" that was used to performed the network packet capture.

[--conv-connect-conversation [true]|false]

Use this option to connect IPv4 Address conversation hosts with lines using color: "--conv-connection-line-color" on the generated Mercator World Map projection. This option only applies to source type: "pcap2conv".

[--conv-connection-line-color TEXT]

Choose the color for lines used to connect IPv4 Address conversations on the generated Mercator World Map projection. Use the special color value: "spectrum" for a predefined alternating set of colors. The default value is: "spectrum". This option only applies to source type: "pcap2conv".

[--conv-connection-line-width TEXT]

Choose the line width to used in pixels for connecting IPv4 Address conversations when generating a KML Document. Use the special width value: "graduated" for a predefined set of increasing line width values start from "1" to "5" pixels. When the "graduated" value is selected, an appropiate line width will be choosen between each conversation host endpoint and will depend on the total amount of network traffic sent and received in "Bytes". The default value is: "graduated". This option only applies to source type: "pcap2conv".

[--conv-fqdn [true]|false]

Use this option to try to resolve each IPv4 Address Conversation Host to its Fully Qualified Domain Name (FQDN). This option only applies to source type: "pcap2conv".

[--new-mercator [true]|false]

When the "--new-mercator" option is used with the "mercator" mode, a new base Mercator Map image will be used. If a previous nstgeolocate status information file: "nstgeolocate.xml" exists within the active "--odir" directory, it will be renamed to the same name as the last image file with a cooresponding "xml" extension.

[--crontab-user TEXT]

Use this option to set the crontab 'user' for automatic map updates using cron.

[--cron-map-min INTEGER]

Use this option to create a crontab entry for automatic updating a 'nstgeolocate' map creation session each "--cron-map-min" minutes. The current map will be updated each time this crontab entry is executed. This option can only be used with the "--mode mercator" or the "--mode kmz" mode. The minimum value permitted is 1. The maximum value permitted is 59.

[--cron-map-hr INTEGER]

Use this option to create a crontab entry for automatic updating a 'nstgeolocate' map creation session each "--cron-map-hr" hours. The current map will be updated each time this crontab entry is executed. This option can only be used with the "--mode mercator" or the "--mode kmz" mode. The minimum value permitted is 1. The maximum value permitted is 23.

[--cron-map-other TEXT]

Use this option to create a crontab entry for automatic updating a 'nstgeolocate' map creation session each "--cron-map-other" time period. The following time period values are valid: '@hourly', '@daily or @midnight', '@weekly', '@monthly' or @yearly' or '@annually'. The current map will be updated each time this crontab entry is executed. This option can only be used with the "--mode mercator" or the "--mode kmz" mode.

[--rotate-interval-min INTEGER]

Use this option to set the automatic map rotation interval time each "--rotate-interval-min" minutes. When a 'nstgeolocate' map creation session has commenced, if the rotation interval time in "--rotate-interval-min" minutes has been exceeded from the start time of the current map sequence, the start of a new map sequence will begin. This option can only be used with the "--mode mercator" or the "--mode kmz" mode. The "--rotate-interval-min" rotation interval takes precedence over both the "--rotate-interval-hr" and "--rotate-interval-day" intervals. The minimum value permitted is 1. The maximum value permitted is 59.

[--rotate-interval-hr INTEGER]

Use this option to set the automatic map rotation interval time each "--rotate-interval-hr" hours. When a 'nstgeolocate' map creation session has commenced, if the rotation interval time in "--rotate-interval-hr" hours has been exceeded from the start time of the current map sequence, the start of a new map sequence will begin. This option can only be used with the "--mode mercator" mode. The "--rotate-interval-hr" rotation interval takes precedence over the "--rotate-interval-day" interval. The minimum value permitted is 1. The maximum value permitted is 23.

[--rotate-interval-day TEXT]

Use this option to set the automatic map rotation interval time each "--rotate-interval-day" days. When a 'nstgeolocate' map creation session has commenced, if the rotation interval time in "--rotate-interval-day" days has been exceeded from the start time of the current map sequence, the start of a new map sequence will begin. This option can only be used with the "--mode mercator" mode. To effectively disable automatic map rotation, just use a maximum "--rotate-interval-day" day interval value of '99999' days. The minimum value permitted is 1. The maximum value permitted is 99999.

[--cron-prune-min INTEGER]

Use this option to create a crontab entry for automatic map imagery and status information file pruning using a 'nstgeolocate' map pruning session each "--cron-prune-min" minutes. The associated 'nstgeolocate' session directory will be pruned each time this crontab entry is executed. Either the "--prune-min" or the "--prune-day" option is required to establish how far back in time to remove map imagery and status information files. This option can only be used with the "--mode mercator" or the "--mode kmz" mode. The minimum value permitted is 1. The maximum value permitted is 59.

[--cron-prune-hr INTEGER]

Use this option to create a crontab entry for automatic map imagery and status information file pruning using a 'nstgeolocate' map pruning session each "--cron-prune-hr" hours. The associated 'nstgeolocate' session directory will be pruned each time this crontab entry is executed. Either the "--prune-min" or the "--prune-day" option is required to establish how far back in time to remove map imagery and status information files. This option can only be used with the "--mode mercator" or the "--mode kmz" mode. The minimum value permitted is 1. The maximum value permitted is 23.

[--cron-prune-other TEXT]

Use this option to create a crontab entry for automatic map imagery and status information file pruning using a 'nstgeolocate' map pruning session each "--cron-prune-other" time period. The following time period values are valid: '@hourly', '@daily or @midnight', '@weekly', '@monthly' or @yearly' or '@annually'. The associated 'nstgeolocate' session directory will be pruned each time this crontab entry is executed. Either the "--prune-min" or the "--prune-day" option is required to establish how far back in time to remove map imagery and status information files. This option can only be used with the "--mode mercator" or the "--mode kmz" mode.

[--prune-min INTEGER]

Use this option to specify in "minutes" the pruning window for previously generated image files (i.e., Mercator Map Images, KMZ Documents or WebGL JSON Documents) and associated status information files in the "--odir" directory. Specify how many "--prune-min" minutes back in time to maintain image files. There is no default value set. Use the "--prune-day" option if you need to maintain image files for "1" day or more. This option is only used with modes: "prunemap" and "listprunemap". Note: When using mode "listprunemap", no file removal will actually take place. The "--prune-min" option will only specify the image and status information files to be pruned. This method is nondestructive and can be used to determine if your pruning value is desirable. The minimum value permitted is 0. The maximum value permitted is 1440.

[--prune-day INTEGER]

Use this option to specify in "days" the pruning window for previously generated image files (i.e., Mercator Map Images, KMZ Documents or WebGL JSON Documents) and associated status information files in the "--odir" directory. Specify how many "--prune-day" days back in time to maintain image files. There is no default value set. This option is only used with modes: "prunemap" and "listprunemap". Note: When using mode "listprunemap", no file removal will actually take place. The "--prune-day" option will only specify the image and status information files to be pruned. This method is nondestructive and can be used to determine if your pruning value is desirable. The minimum value permitted is 0. The maximum value permitted is 99999.

[--map-title TEXT]

This title text will overide the map title value derived by interrogating either the active local or remote ntop session specified by the "--ntop-url" option or an ntopng session specified by the "--ntopng-url" option. This option is a short title (i.e., a maximum of 22 characters will be displayed) depicted in the legend or description area. Try to use a descriptive phrase to identify the traffic that the ntop or ntopng session is monitoring (e.g., "Corporate Web Site").

[--series-description TEXT]

This description will be displayed on the WebGL Globe presentation. It will be used to describe the dataset series information derived from either the active ntop or ntopng session. This option is mandatory for mode: "--mode webgl". ***Note: If this parameter is set to "__COMPUTE__" then the description will automatically be set to the ntop or ntopng session collection start and end time. (e.g., "2015-06-17 10:00:01 - 2015-06-18 10:00:01").

[--data-series-cnt INTEGER]

Use this parameter with mode: "--mode webgldatset" to limit the number of existing single series NST WebGL Globe JSON documents used to generate a multi-series WebGL Globe JSON document. The default count value is: "3".

[--quick-stats [true]|false]

This is a special option used by the NST WUI with mode: "updateindicators" to dump selective variables providing 'nstgeolocate' session status and information.

[--public-ipaddr-ref [true]|false]

Use this option to add a public IP Address reference to the KML document for access back to the NST WUI. Detected "ntop" and "Conversation" hosts can then be used with the included NST WUI IP networking tools and utilities. This option only pertains to KML document creation.

[-h [true]|false] | [--help [true]|false]

When this option is specified, nstgeolocate will display a short one line description of nstgeolocate, followed by a short description of each of the supported command line options. After displaying this information nstgeolocate will terminate.

[-H [true]|false] | [--help-long [true]|false]

This option will attempt to pull up additional nstgeolocate documentation within a text based web browser. You can force which browser we use setting the environment variable TEXTBROWSER, otherwise, we will search for some common ones.

[-v [true]|false] | [--verbose [true]|false]

When you set this option to true, nstgeolocate will produce additional output. This is typically used for diagnostic purposes to help track down when things go wrong.

[--version [true]|false]

If this option is specified, the version number of the script is displayed.

Files

/usr/share/nstgeolocate

Directory containing resource files used by nstgeolocate.

Environment

TEXTBROWSER

This controls what text based browser is used to display help information about the script. If not set, we will search your system for available text-based browsers (Ex: elinks, lynx ...).

See Also

nstnewscript(1), Network Security Toolkit