VGX Server System Administration

System administration can be performed in three ways:
Section 1, “Admin Pages”
Section 2, “Administrative Service Endpoints”
Section 3, “vgxadmin Command Line Tool”

1. Admin Pages

Section 1.1, “Top Menu /”
Section 1.2, “Admin /admin”
Section 1.3, “Status /status”
Section 1.4, “Console /console”
Section 1.5, “General Search /search”
Section 1.6, “User Plugins and Builtin Plugins /plugin”
Section 1.7, “Multi-Node System Dashboard /system”

1.1. Top Menu `/`

Click on a grey box to open its page.

1.2. Admin `/admin`

Use this page for basic administration of a singe VGX Server instance.

Click the lock icon in the middle of the page to enable/disable the various action buttons.

1.2.1. Display Readout

Group Box Description

Group	Box	Description
Service	Name	Name of service, as provided to system.RunServer() name argument, or `?` if running interactively.
State	`Running` or `Interactive`
Uptime	`days:hours:minutes:seconds` since VGX Server started
Order / Size	Order	Total number of vertices in all loaded graphs
Size	Total number of arcs in all loaded graphs
Memory Usage	Memory used by this VGX Instance (% of physical memory on host)
Request	Request Rate	Current requests per second reaching this VGX Server instance
Latency	Average / 99th Percentile request latency in milliseconds
Durability	Persistence	`Durable` VGX Transaction Service has been started in durable mode with op.Bind() `Volatile` Persist before shutdown required to retain data
State	`Streaming` Received graph data continuously persisted to transaction log `In-Memory` Data exists in memory only `Sync` Transaction data is being synchronized from this VGX Server instance to all its subscribers `Writing` All data in memory is being written to disk snapshot
Mutability	Execution Mode	`Local` Graph modification logic is executing in local VGX Server instance `Bound: txport` VGX Transaction Service is running on txport, but no provider has attached `Suspended` VGX Transaction Service has temporarily suspended input processing `Streaming` A real-time transaction stream is established from an attached provider to this VGX Server instance
Mutable Mode	`Readonly` VGX Server instance data is currently not modifiable `TTL Active` Automatic expiration is running `TTL Frozen` Expiration events are currently suspended `TTL Passive` Expiration via VGX Transaction Service input only
TX I/O kiB/s	Transaction Input Rate	Current data received from subscriber
Transaction Output Rate	Current data sent to subscriber(s)
Local	State	`Idle` `Active`
Digest	Identicon representing current local state
Serial Number	Transaction sequence number
Provider	State	`Idle` Attached provider is currently not sending data `Stream` Attached provider is currently sending data `Delta` Data is not in sync between provider and this VGX Server instance `Paused` VGX Transaction Service has temporarily suspended input processing
Digest	Identicon representing provider’s current state
Hostname	Provider’s hostname or IP address
Hport	Provider’s HTTP server admin port (visible only when initiating subscription)
Subscriber	State	`Idle` Currently not sending data to attached subscriber `Stream` Currently sending data to attached subscriber `Delta` Data is not in sync between this VGX Server instance and the subscriber `Paused` Output transactions are temporarily suspended `Sync` Data is being synchronized from this VGX Server instance to the subscriber `Down` Attached subscriber is currently unreachable
Digest	Identicon representing subscriber’s current state
Hostname	Subscriber’s hostname or IP address
Tport	Subscriber’s input txport

Service

Name

Name of service, as provided to system.RunServer() name argument, or ? if running interactively.

State

Running or Interactive

Uptime

days:hours:minutes:seconds since VGX Server started

Order / Size

Order

Total number of vertices in all loaded graphs

Size

Total number of arcs in all loaded graphs

Memory Usage

Memory used by this VGX Instance (% of physical memory on host)

Request

Request Rate

Current requests per second reaching this VGX Server instance

Latency

Average / 99th Percentile request latency in milliseconds

Durability

Persistence

Durable VGX Transaction Service has been started in durable mode with op.Bind()

Volatile Persist before shutdown required to retain data

State

Streaming Received graph data continuously persisted to transaction log

In-Memory Data exists in memory only

Sync Transaction data is being synchronized from this VGX Server instance to all its subscribers

Writing All data in memory is being written to disk snapshot

Mutability

Execution Mode

Local Graph modification logic is executing in local VGX Server instance

Bound: txport VGX Transaction Service is running on txport, but no provider has attached

Suspended VGX Transaction Service has temporarily suspended input processing

Streaming A real-time transaction stream is established from an attached provider to this VGX Server instance

Mutable Mode

Readonly VGX Server instance data is currently not modifiable

TTL Active Automatic expiration is running

TTL Frozen Expiration events are currently suspended

TTL Passive Expiration via VGX Transaction Service input only

TX I/O kiB/s

Transaction Input Rate

Current data received from subscriber

Transaction Output Rate

Current data sent to subscriber(s)

Local

State

Idle

Active

Digest

Identicon representing current local state

Serial Number

Transaction sequence number

Provider

State

Idle Attached provider is currently not sending data

Stream Attached provider is currently sending data

Delta Data is not in sync between provider and this VGX Server instance

Paused VGX Transaction Service has temporarily suspended input processing

Digest

Identicon representing provider’s current state

Hostname

Provider’s hostname or IP address

Hport

Provider’s HTTP server admin port (visible only when initiating subscription)

Subscriber

State

Idle Currently not sending data to attached subscriber

Stream Currently sending data to attached subscriber

Delta Data is not in sync between this VGX Server instance and the subscriber

Paused Output transactions are temporarily suspended

Sync Data is being synchronized from this VGX Server instance to the subscriber

Down Attached subscriber is currently unreachable

Digest

Identicon representing subscriber’s current state

Hostname

Subscriber’s hostname or IP address

Tport

Subscriber’s input txport

1.2.2. Action Buttons

Some actions are available only in certain modes. A button is not shown if its associated action is not applicable to this VGX Server instance.

Action Opposite Action Description

Action	Opposite Action	Description
`Shutdown`		Stop VGX Server instance
`Service Out`	`Service In`	Toggle S-OUT/S-IN mode for HTTP server plugin execution
`Reset`		Reset all HTTP Server performance metrics to their initial state
`Snapshot`		Persist all in-memory data to disk
`Sync`		Initiate bulk synchronization of all data from this VGX Server instance to all attached subscribers
`Truncate`		Remove all data from memory
`Bind`	`Unbind`	Start or stop VGX Transaction Service
`Freeze TTL`	`Thaw TTL`	Pause or resume automatic expiration events for all graphs
`Readonly`	`Mutable`	Toggle readonly state for all graphs
`Pause I`	`Resume I`	Pause or resume transaction input stream
`Pause O`	`Resume O`	Pause or resume transaction output stream
`Detach All`		Detach all subscribers
`Subscribe`		Request to become a subscriber of the specified provider
`Detach`		Provider row: Detach from provider Subscriber row(s): Detach from subscriber
`Attach`		Become a provider for the specified subscriber

Shutdown

Stop VGX Server instance

Service Out

Service In

Toggle S-OUT/S-IN mode for HTTP server plugin execution

Reset

Reset all HTTP Server performance metrics to their initial state

Snapshot

Persist all in-memory data to disk

Sync

Initiate bulk synchronization of all data from this VGX Server instance to all attached subscribers

Truncate

Remove all data from memory

Bind

Unbind

Start or stop VGX Transaction Service

Freeze TTL

Thaw TTL

Pause or resume automatic expiration events for all graphs

Readonly

Mutable

Toggle readonly state for all graphs

Pause I

Resume I

Pause or resume transaction input stream

Pause O

Resume O

Pause or resume transaction output stream

Detach All

Detach all subscribers

Subscribe

Request to become a subscriber of the specified provider

Detach

Provider row: Detach from provider

Subscriber row(s): Detach from subscriber

Attach

Become a provider for the specified subscriber

1.3. Status `/status`

This is a readonly summary page showing various metrics and counters.

Request Engine: HTTP server metrics
Transaction Interconnect: VGX Transaction Service metrics
Graph Engine: Graph objects and property storage summary
Memory: System memory usage information

1.4. Console `/console`

This page is a general purpose command interface for detailed manipulation and investigation of a running system, useful for troubleshooting and repair.

dir(); shows available commands

python: <expression>; evaluates expression

API command arguments are supplied as dict:
e.g. Graph.Vertices: {'hits':5, 'sortby':S_IDEG}

1.5. General Search `/search`

This page provides a general search interface.

Neighborhood: Maps to pyvgx.Graph.Neighborhood()
Vertices: Maps to pyvgx.Graph.Vertices()
Arcs: Maps to pyvgx.Graph.Arcs()
Vertex: Display all vertex attributes and properties
Expression: Maps to Evaluate()

1.6. User Plugins and Builtin Plugins `/plugin`

This page allows plugins to be tested.

Normally all admin UI pages should be accessed on the admin port. However, this page must be loaded on the main port when the plugin is a dispatcher pre-processor. If not, an error Preprocessor plugin only allowed in dispatcher context will be returned.

Page header is shaded in light red when accessed on the main port as a warning, because admin tasks can lock executor threads for an extended period of time, which could lead to problems in a live system. Click the bar counter |||||||||| next to Port before it goes to zero to remain on the main port. The page will otherwise reload on the admin port.

1.7. Multi-Node System Dashboard `/system`

Multiple VGX Server instances can be deployed together as a system. This page shows a summary of all nodes in the system and their roles.

For this page to work the SYSTEM_Descriptor property has to be defined. This is a property of the system graph describing all nodes and their roles.

Consider a system with eight VGX Server instances deployed as show below:

The System Overview dashboard for this example system will look like this:

1.7.1. System Dashboard Description

The dashboard’s upper segment shows a summary of information aggregated across all instances in the system. The System Name is the name defined in the system descriptor.

Table 1. Top level summary
Column	Value	Description
Status	`OK`	All components are running normally
Status	`Check n warnings`	One or more components requires attention
Instances	i (h hosts)	Total number of instances i running on h separate hosts
Partitions	p	Number of partitions (Builder instances)
Vertices	order	Total number of vertices across all partitions
Arcs	size	Total number of arcs across all partitions
Properties	nprop	Total number of properties across all partitions
Vectors	nvec	Total number of vectors across all partitions
CPU	model	CPU model brand string installed on server(s)
Memory	used (hostmem)	Total used memory for all instances in system, out of the total hostmem memory available on all hosts

The dashboard’s lower segment shows details for each system instance.

Table 2. System instances
Column	Value	Description

[Instance]
Id	Instance Label	Unique instance identifier
Type	`Admin`	Holds system configuration master and orchestrates multi-node admin operations
	`Dispatch`	Request dispatcher responsible for distributing requests to back-end matrix, merging results, and executing any pre or post processing plugins
	`Builder`	Responsible for generating graph data which is distributed to Search nodes, and plugin engine execution
	`Search`	Responsible for plugin engine execution against graph data received from Builder node
	`TXProxy`	Intermediary whose main purpose is to receive transaction data from Builder and forward to Search nodes, while optionally providing streaming persistence (durability)
	`Generic`	Default, unspecified node type
Status	`OK`	Instance is running normally
	`Busy`	Transaction input processing is lagging more than 30 seconds behind provider
	`S/N`	Serial number difference between this instance and its provider
	`No Input`	Instance with VGX Transaction Service running (bound txport) has no provider attached
	`S-OUT`	Service Out: HTTP Server plugin execution is paused
	`Sync`	Instance is synchronizing its data to attached subscriber(s)
	`Snapshot`	Instance is persisting data to disk
	`Down`	Instance does not respond to status requests and is either not running or not reachable
	`Partial`	Dispatcher’s back-end matrix is incomplete due to one or more partitions without any replicas
	`OK/Delta`	Dispatcher’s back-end matrix has one or more partitions with inconsistent replicas, or back end subscriber instance is our of sync with its provider
	`Degraded`	Dispatcher’s back-end matrix has one or more unavailable replicas (reduced capacity)
	`Memory`	Memory warning, instance’s physical host memory usage is 80% - 83%
	`Mem x%`	Memory warning, instance’s physical host memory usage is 83% - 93%
	`! MEM x%`	Memory critical, instance’s physical host memory usage is above 93%
	`Readonly`	Instance is in readonly mode
	`Starting`	Instance is initializing after startup
	`Waiting`	Instance is running but did not provide node status information
	`---`	State unknown due to incomplete status information
Uptime	`D:H:M:S`	Time since instance was started
Address	`host:port`	Hostname/IP and admin port of instance

[Server Load]
Matrix	`S-IN`	Service In: Back end engine instance operating normally
	`S-OUT`	Service Out: Back end or dispatcher instance not accepting plugin requests
	`mR x nP`	Dispatcher’s back-end matrix dimensions (m replicas, n partitions)
	Empty	N/A
In	Sockets	Number of connected clients
Out	Sockets	Number of channels connected to backend matrix
Bal	Load Balance	Relative request rate in group
RPS	Request Rate	HTTP Server requests per second
95th ms	Request Latency	HTTP Server 95th percentile request latency in milliseconds

[Memory]
Instance/Host	Instance/Total	Memory used (in GiB) by instance / Physical host total memory usage

[Graph]
Order	Vertices	Total number of vertices in all graphs
Size	Arcs	Total number of arcs in all graphs

[Interconnect]
Subs	Subscribers	Currently attached subscribers
Out	Data Out Rate	Provider transaction data output rate
In	Data In Rate	Subscriber transaction data input rate
Lag	Data In Lag	Subscriber transaction input lag

[Transaction]
Digest	State identicon	Representing current state of instance data
S/N	Serial Number	Most recent transaction serial number
TX Log	TX Log Size	In durable mode, transaction log grows until max size triggers batch persist, which clears the log and starts filling it again

1.7.2. System Dashboard Actions

Many of the UI elements can be clicked to perform an action. Some elements open another page. Some elements perform an administrative action. When clicking an element that performs an action a dialog appears. The instance identifier for the clicked instance is pre-populated in the dialog. It is possible to edit the instance identifier, e.g. to perform the action on multiple instances by using the * wildcard symbol. For example, if clicking S-IN in the Matrix column for instance S1.1 you can edit the instance to S* in the pup-up dialog. This will perform the action on all instances whose identifier starts with S.

1.7.2.1. Dialog Bypass

Hold down the Shift key to reveal action buttons describing the actions associated with elements.

To bypass the dialog hold down Shift-Alt (Windows) / Shift-Opt (MacOS) while clicking an action button.

To bypass the dialog for potentially destructive actions you have to click the action button three times while holding down Shift-Alt/Shift-Opt.

1.7.2.2. Links

Elements that open another page when clicked will open the link in a new window by default.

Hold down the Shift key to open the link in the current window.

Hold down Shift-Ctrl (Windows) / Shift-Cmd (MacOS) to open the link in a new tab.

1.7.2.3. Action Description

Table 3. Actions
Column Element	Action When Clicked
Id	Open instance `/console`
Type	Open instance `/admin`
Status	Open instance `/status`
Uptime	Perform instance `/vgx/ping`
Address	Open instance `/index`
Matrix	Toggle instance S-IN/S-OUT state. The performed action depends on the current service state of the clicked instance.
RPS	Restart the instance HTTP server. This does not restart the VGX Server instance, only the HTTP server.
95th ms	N/A
Memory	Persist instance to disk
Order	Perform reverse synchronization of data. This only applies to provider instances already attached to one or more subscribers.
Size	Perform forward synchronization of data. This only applies to provider instances already attached to one or more subscribers.
Subs	Toggle attach/detach of provider instance. Applies to provider instances only.
Out	Toggle pause/resume of provider instance transaction output. Applies to attached provider instances only.
In	Toggle pause/resume of subscriber instance transaction input. Applies to attached subscriber instances only.
Lag	Unsubscribe from provider. Applies to attached subscriber instances only.
Digest	Attempt to repair/synchronize instances that are configured as subscribers of the clicked provider instance.
S/N	Perform a rolling update / synchronization of the provider instance’s attached subscriber instances. At least two subscribers are required for this operation.
TX Log	N/A

2. Administrative Service Endpoints

All administrative tasks can be performed using the service endpoints described in this section.

2.1. authtoken

All administrative plugins require parameter authtoken, which helps prevent accidental execution of administrative tasks. Before calling any of these plugins, first retrieve a new token via /vgx/builtin/ADMIN_GetAuthToken, then pass the returned token as authtoken parameter for the administrative plugin.

The authtoken retrieved by /vgx/builtin/ADMIN_GetAuthToken can only be used once and expires 100 seconds after its was retrieved.

2.1.1. authtoken Example

Request

http://127.0.0.1:9000/vgx/builtin/ADMIN_GetAuthToken

Response

{
    "status": "OK",
    "response": {
        "authtoken": "3c8b886d2ef4565378339d580b689ef7fff8bf0ce55da1bbc5f6c1a140f1cf71",
        "t0": 1715134438.5777998,
        "tx": 1715134738.5777998
    },
    "level": 0,
    "partitions": null,
    "exec_ms": 0.142
}

2.2. ADMIN Plugins

Call these endpoints to perform administrative tasks. Obtain a unique authtoken for each new request.

2.2.1. ADMIN Plugin Parameters

Use endpoint /vgx/builtin/argspec?plugin=name to see a description of parameters for each service.

Example

http://127.0.0.1:9000/vgx/builtin/argspec?plugin=ADMIN_Throttle

{
    "status": "OK",
    "response": [
        {
            "path": "/vgx/builtin/ADMIN_Throttle",
            "description": [
                "",
                "ADMIN: Throttle TX input",
                "rate : Limit TX input processing to units per second",
                "unit : bytes, opcodes, operations, transactions",
                "    "
            ],
            "parameters": {
                "request": "<class 'pyvgx.PluginRequest'>",
                "headers": "dict",
                "authtoken": "str",
                "rate": [
                    "float",
                    -1
                ],
                "unit": [
                    "str",
                    "bytes"
                ]
            },
            "bound_graph": "None"
        }
    ],
    "level": 0,
    "partitions": null,
    "exec_ms": 0.182
}

2.2.2. General

URI Path Description

URI Path	Description
`/vgx/builtin/ADMIN_Console`	View or edit graph information, as in Section 1.4, “Console `/console`”
`/vgx/builtin/ADMIN_Shutdown`	Initiate instance shutdown. This requires two authtokens. Obtain the first token and pass in `authshutdown` parameter. Then obtain the second token and pass in `authtoken` parameter.
`/vgx/builtin/ADMIN_SystemDescriptor`	Update the system descriptor

/vgx/builtin/ADMIN_Console

View or edit graph information, as in Section 1.4, “Console /console”

/vgx/builtin/ADMIN_Shutdown

Initiate instance shutdown. This requires two authtokens. Obtain the first token and pass in authshutdown parameter. Then obtain the second token and pass in authtoken parameter.

/vgx/builtin/ADMIN_SystemDescriptor

Update the system descriptor

2.2.3. Plugin Management

URI Path Description

URI Path	Description
`/vgx/builtin/ADMIN_ReloadPlugins`	Reload all plugins initially loaded with pyvgx.VGXInstance.StartInstance(), whose definitions are located in dynamically re-loadable modules (i.e. reference functions in package/module.) New plugins can be dynamically added by POSTing plugin definition JSON. The plugin module file containing functions referenced in the definition must already exist on the server’s filesystem.

/vgx/builtin/ADMIN_ReloadPlugins

Reload all plugins initially loaded with pyvgx.VGXInstance.StartInstance(), whose definitions are located in dynamically re-loadable modules (i.e. reference functions in package/module.)

New plugins can be dynamically added by POSTing plugin definition JSON. The plugin module file containing functions referenced in the definition must already exist on the server’s filesystem.

2.2.4. Data Management

URI Path Description

URI Path	Description
`/vgx/builtin/ADMIN_CancelSync`	Stop synchronization of subscribers (pyvgx.system.CacnelSync())
`/vgx/builtin/ADMIN_Persist`	Create snapshot on disk for all loaded graphs (pyvgx.system.Persist())
`/vgx/builtin/ADMIN_Sync`	Synchronize all subscribers (pyvgx.system.Sync())
`/vgx/builtin/ADMIN_Truncate`	Delete all graph data (pyvgx.Graph.Truncate())

/vgx/builtin/ADMIN_CancelSync

Stop synchronization of subscribers (pyvgx.system.CacnelSync())

/vgx/builtin/ADMIN_Persist

Create snapshot on disk for all loaded graphs (pyvgx.system.Persist())

/vgx/builtin/ADMIN_Sync

Synchronize all subscribers (pyvgx.system.Sync())

/vgx/builtin/ADMIN_Truncate

Delete all graph data (pyvgx.Graph.Truncate())

2.2.5. Graph State

URI Path Description

URI Path	Description
`/vgx/builtin/ADMIN_ClearReadonly`	Make all graphs writable (pyvgx.system.ClearReadonly())
`/vgx/builtin/ADMIN_ResumeEvents`	Resume TTL process for all graphs (pyvgx.system.ResumeEvents())
`/vgx/builtin/ADMIN_SetReadonly`	Make all graphs readonly (pyvgx.system.SetReadonly())
`/vgx/builtin/ADMIN_SuspendEvents`	Suspend TTL process for all graphs (pyvgx.system.SuspendEvents())

/vgx/builtin/ADMIN_ClearReadonly

Make all graphs writable (pyvgx.system.ClearReadonly())

/vgx/builtin/ADMIN_ResumeEvents

Resume TTL process for all graphs (pyvgx.system.ResumeEvents())

/vgx/builtin/ADMIN_SetReadonly

Make all graphs readonly (pyvgx.system.SetReadonly())

/vgx/builtin/ADMIN_SuspendEvents

Suspend TTL process for all graphs (pyvgx.system.SuspendEvents())

2.2.6. Request Engine

URI Path Description

URI Path	Description
`/vgx/builtin/ADMIN_ResetMetrics`	Reset VGX Server metrics (pyvgx.system.ResetMetrics())
`/vgx/builtin/ADMIN_ServiceIn`	Enable VGX Server plugin execution (pyvgx.system.ServiceInHTTP(True))
`/vgx/builtin/ADMIN_ServiceOut`	Disable VGX Server plugin execution (pyvgx.system.ServiceInHTTP(False))

/vgx/builtin/ADMIN_ResetMetrics

Reset VGX Server metrics (pyvgx.system.ResetMetrics())

/vgx/builtin/ADMIN_ServiceIn

Enable VGX Server plugin execution (pyvgx.system.ServiceInHTTP(True))

/vgx/builtin/ADMIN_ServiceOut

Disable VGX Server plugin execution (pyvgx.system.ServiceInHTTP(False))

2.2.7. Transaction Flow Management

URI Path Description

URI Path	Description
`/vgx/builtin/ADMIN_Attach`	Become a provider for the attached subscriber (pyvgx.op.Attach())
`/vgx/builtin/ADMIN_Bind`	Start VGX Transaction service on this VGX instance (pyvgx.op.Bind())
`/vgx/builtin/ADMIN_Detach`	Detach subscriber (pyvgx.op.Detach())
`/vgx/builtin/ADMIN_DetachAll`	Detach all subscribers (pyvgx.op.Detach())
`/vgx/builtin/ADMIN_ResumeTxInput`	Resume processing of input transactions (pyvgx.op.ResumeTxInput())
`/vgx/builtin/ADMIN_ResumeTxOutput`	Resume output of transaction data (pyvgx.op.ResumeTxOutput())
`/vgx/builtin/ADMIN_Subscribe`	Request to become a subscriber of the specified provider instance (pyvgx.op.Subscribe())
`/vgx/builtin/ADMIN_SuspendTxInput`	Pause processing of input transactions (pyvgx.op.SuspendTxInput())
`/vgx/builtin/ADMIN_SuspendTxOutput`	Pause output of transaction data (pyvgx.op.SuspendTxOutput())
`/vgx/builtin/ADMIN_Throttle`	Limit consumption rate of inbound transaction data (pyvgx.op.Throttle())
`/vgx/builtin/ADMIN_Unbind`	Stop VGX Transaction service on this VGX instance (pyvgx.op.Unbind())
`/vgx/builtin/ADMIN_Unsubscribe`	Unsubscribe from provider (pyvgx.op.Unsubscribe())
`/vgx/builtin/ADMIN_UpdateSubscribers`	Effectuate a provider’s subscriber state according to a specified goal.

/vgx/builtin/ADMIN_Attach

Become a provider for the attached subscriber (pyvgx.op.Attach())

/vgx/builtin/ADMIN_Bind

Start VGX Transaction service on this VGX instance (pyvgx.op.Bind())

/vgx/builtin/ADMIN_Detach

Detach subscriber (pyvgx.op.Detach())

/vgx/builtin/ADMIN_DetachAll

Detach all subscribers (pyvgx.op.Detach())

/vgx/builtin/ADMIN_ResumeTxInput

Resume processing of input transactions (pyvgx.op.ResumeTxInput())

/vgx/builtin/ADMIN_ResumeTxOutput

Resume output of transaction data (pyvgx.op.ResumeTxOutput())

/vgx/builtin/ADMIN_Subscribe

Request to become a subscriber of the specified provider instance (pyvgx.op.Subscribe())

/vgx/builtin/ADMIN_SuspendTxInput

Pause processing of input transactions (pyvgx.op.SuspendTxInput())

/vgx/builtin/ADMIN_SuspendTxOutput

Pause output of transaction data (pyvgx.op.SuspendTxOutput())

/vgx/builtin/ADMIN_Throttle

Limit consumption rate of inbound transaction data (pyvgx.op.Throttle())

/vgx/builtin/ADMIN_Unbind

Stop VGX Transaction service on this VGX instance (pyvgx.op.Unbind())

/vgx/builtin/ADMIN_Unsubscribe

Unsubscribe from provider (pyvgx.op.Unsubscribe())

/vgx/builtin/ADMIN_UpdateSubscribers

Effectuate a provider’s subscriber state according to a specified goal.

3. vgxadmin Command Line Tool

Command line tool vgxadmin offers a convenient way to manage and monitor a multi-node system.

A system descriptor is required for this tool to work. By default a file named vgx.cf in the current working directory is expected. An alternative file can be specified with option --cf. An optional <address> (host:port) can be specified to point the tool directly at a running VGX Server instance, using that instance’s system descriptor instead of a local file.

$ vgxadmin --help

usage: vgxadmin [<address|id>] <options>

    -a, --attach <id>[,<sub>[,...]] Attach instance to subscribers
    -B, --bind <id>                 Bind transaction input port
    -k, --cancelsync <id>           Terminate sync in progress
    -f, --cf <file>                 Use this local system descriptor file
    -C, --command <id>,<gr>,<cmd>   Send console command <cmd> to graph <gr>
    -c, --confirm                   Confirm operation (skip y/n prompt)
    -Z, --descriptor <id>           Update instance system descriptor
    -d, --detach <id>               Detach instance from subscribers
    -E, --endpoint <path>           Send request to <address>
    -K, --forcecopy <src>,<dst>     Force hard sync from <src> to <dst>
    -h, --help                      Show this help message
    -Q, --instancecfg <id>          Show instance configuration
    -n, --nodestat <id>[,<key>]     Nodestat
    -M, --opdump <id>               Dump instance data to output file
    -p, --pausein <id>              Pause transaction input
    -P, --pauseout <id>             Pause transaction output
    -t, --pausettl <id>             Pause TTL event processor
    -W, --persist <id>              Write instance data to disk
    -g, --readonly <id>             Make graph(s) readonly
    -N, --reloadplugins <id>[,<pd>] Reload or add plugins in <pd> json file
    -m, --resetmetrics <id>         Clear performance and error counters
    -D, --restarthttp <id>          Restart HTTP server with refreshed config
    -r, --resumein <id>             Resume transaction input
    -R, --resumeout <id>            Resume transaction output
    -T, --resumettl <id>            Resume TTL event processor (if enabled)
    -Y, --reversesync <id>          Reverse sync instance data from attached subscriber
    -L, --rollingupdate <id>        Forward sync subscribers one at a time in S-OUT
    -I, --servicein <id>            Service in
    -i, --serviceout <id>           Service out
    -J, --show                      Show effective system descriptor
    -s, --start <id>                Start instance on the local host
    -S, --status <id>               System summary
    -x, --stop <id>                 Stop instance
    -y, --sync <id>[,<mode>]        Sync data to subscribers (<mode> [repair|hard|soft])
    -V, --throttle <id>[,<r>,<u>]   Throttle TX input rate <r>, unit <u>
    -X, --truncate <id>             Erase instance data
    -U, --unbind <id>               Unbind transaction input port
    -u, --unsubscribe <id>          Detach instance from provider
    -v, --version                   VGX Server version
    -w, --waitforidle <id>          Wait until instance input is idle
    -G, --writable <id>             Make graph(s) writable

3.1. vgxadmin Options

Many options require an argument <id>, which is a reference to instance identifiers in the system descriptor. <id> can be a single instance identifier, multiple identifiers separated by commas, or a wildcard pattern using *. (Some options do not accept multiple identifiers.)

The same option can occur multiple times. Actions are performed sequentially, i.e. one action has to complete before the next action starts.

If <address> is specified, it can either be a valid instance identifier or the <host>:<port> of a running VGX Server instance. In this case the <id> argument for all specified options defaults to that instance’s identifier and can be omitted.

When only <address> is provided (no other options specified) a summary of the running VGX Server instance is printed.

3.1.1. `--attach <id>` (`-a`)

Attach provider instance(s) to subscribers in accordance with the transaction topology definition.

Attach instances "B0.1" and "B0.2" to their configured subscribers

vgxadmin --attach B0.1 --attach B0.2

Attach instance "B0.1" to subscriber "S1.1" only

vgxadmin --attach "B0.1, S1.1"

Be careful when using vgxadmin --attach <id>,<sub1>,<sub2>,… since this allows overriding the defined transaction topology in vgx.cf.

3.1.2. `--bind <id>` (`-B`)

Start the transaction input service on the tport defined for this instance. If no tport is defined this command has no effect.

Start transaction input service on instance "S1.1"

vgxadmin --bind S1.1

3.1.3. `--cancelsync <id>` (`-k`)

Terminate any data synchronization currently in progress.

Stop sync from instances with prefix "B" to their subscribers

vgxadmin --cancelsync "B*"

3.1.4. `--cf <file>` (`-f`)

Use this local system descriptor file instead of the default vgx.cf.

Print status of all instances defined in system descriptor another.cf

vgxadmin --cf another.cf --status @

3.1.5. `--command <id>,<gr>,<cmd>` (`-C`)

Send console command <cmd> for graph <gr>. The <cmd> syntax is the same used on the /console page.

Execute a remote pyvgx command on instance "S1.1"

vgxadmin --command S1.1,g1,"Graph.Vertices: {'hits':5, 'sortby':S_IDEG}"

3.1.6. `--confirm` (`-c`)

Bypass y/n prompts for options that require confirmation.

Remove all data from instance "S1.1" without prompting for confirmation

vgxadmin --confirm --truncate S1.1

3.1.7. `--descriptor <id>` (`-Z`)

Update system descriptor on instance(s) with the effective system descriptor.

Send the local vgx.cf to all defined instances

vgxadmin --descriptor @

3.1.8. `--detach <id>` (`-d`)

Detach provider instance(s) for all attached subscribers.

Detach all instances with "B" prefix from their subscribers

vgxadmin --detach "B*"

3.1.9. `--endpoint <path>` (`-E`)

Send HTTP request <path> to instance specified in <address>, which is required for this option.

Send request for metrics to instance running locally on port 9990

vgxadmin 127.0.0.1:9990 --endpoint "/vgx/builtin/metrics"

Send a plugin request to instance running locally on port 9100

vgxadmin 127.0.0.1:9100 --endpoint "/vgx/plugin/search?name=1234&hits=3"

3.1.10. `--forcecopy <src>,<dst>` (`-K`)

Force hard sync from <src> instance to <dst> instance. This allows data to be copied between any nodes within a system, bypassing the configured transaction topology.

By creating a temporary system descriptor referencing instances in both system A and B it is possible to copy data between any nodes in both systems.

Copy data from one partition builder to another

vgxadmin --forcecopy "B0.2, B0.1"

3.1.11. `--help` (`-h`)

Print vgxadmin usage options and exit.

vgxadmin --help

3.1.12. `--instancecfg <id>` (`-Q`)

Show effective instance configuration parameters as defined in system descriptor.

Print the top dispatcher’s configuration

vgxadmin --instancecfg TD

3.1.13. `--nodestat <id>[,<key>]` (`-n`)

Retrieve /vgx/nodestat from instance(s) and print result, optionally filtered to only show attribute <key>.

Show full nodestat for instance "B0.1"

vgxadmin --nodestat B0.1

Show memory consumed by all builder instances

vgxadmin --nodestat "B*",memory-process

3.1.14. `--opdump <id>` (`-M`)

Dump all graph data to an output file on the instance’s local file system. The file will be named opdump_<timestamp>.tx located directly under the instance vgxroot directory. Output data is written as VGX transactions and can be used to populate other instances.

To allow an instance to restore data from a local file, the service implementation can use pyvgx.op.Consume(). Data can be read line by line from opdump<timestamp>.tx and submitted to pyvgx.op.Consume().

The instance being dumped must not be attached to any subscribers and must not have the transaction input service running. Use --detach, --unsubscribe and --unbind as appropriate before performing --opdump.

Dump instance B0.1 data to output file as VGX transactions

vgxadmin --opdump B0.1

# Output written to builder_B0.1/opdump_1731703450.tx

3.1.15. `--pausein <id>` (`-p`)

Pause transaction input service on subscriber instance(s). Transaction data will still be received and queued by subscribers, but not processed while in the paused state. Use --resumein to re-enable processing of received transaction data.

Pause transaction input on the second search row

vgxadmin --pausein "S2*"

3.1.16. `--pauseout <id>` (`-P`)

Pause transaction output emitter on provider instance(s). Transaction data is produced and queued, but not sent to subscribers while in the paused state. Use --resumeout to start sending data to subscribers again.

Pause transaction output on the builders

vgxadmin --pauseout "B*"

3.1.17. `--pausettl <id>` (`-t`)

Pause TTL event processor. Objects with expiration time set will not be deleted while in the paused state. Use --resumettl to start processing TTL events again.

Prevent deletion of expired objects on builder row

vgxadmin --pausettl "B*"

3.1.18. `--persist <id>` (`-W`)

Save all instance data (all graphs) to disk.

Save all builder instances to disk

vgxadmin --persist "B*"

3.1.19. `--readonly <id>` (`-g`)

Transition all graph(s) to readonly mode. If the instance is currently a subscriber its transaction input processor will be paused.

Set all instances with identifier prefix "S" to readonly

vgxadmin --readonly "S*"

3.1.20. `--reloadplugins <id>[,<pd>]` (`-N`)

Reload updated code for already registered plugins, or add plugins in <pd> json file.

The plugin implementation Python code must already exist on the filesystem of the instance being updated, and the plugin definition list must correctly specify the plugin package(s) and function(s).

If adding new plugins the <pd> file format is the JSON equivalent of the Python plugin definition list used with pyvgx.VGXInstance.StartInstance().

3.1.20.1. Example: Add square root capabilities to running system

Assuming the running service implementation exists in a package called service, add a new file extras.py to the package containing the following Python code:

The plugin implementation Python code must be placed on disk on every host for which we intend to execute the update.

service/extras.py

from pyvgx import *

def SquareRoot( request:PluginRequest, graph:Graph, x:float ):
    return x ** 0.5

On the local machine (where vgxadmin is to be executed) place a new file sqrt.json in the current directory with the following content:

sqrt.json

[
  {
    "name":    "sqrt",
    "package": "service",
    "module":  ".extras",
    "pre":     "SquareRoot",
    "engine":  null,
    "post":    null,
    "graph":   "*"
  }
]

VGX Server instances running in dispatcher mode load plugins defined as pre and/or post. Other instance types load plugins defined as engine.

Execute the reload command:

Add new endpoint /vgx/plugin/sqrt to TD instance

vgxadmin --reloadplugins TD,sqrt.json

Now, assuming TD runs locally on port 9990, we can make requests to compute square roots:

Compute square root of 123

curl http://127.0.0.1:9990/vgx/plugin/sqrt?x=123

Square root result

{
    "status": "OK",
    "response": 11.090536506409418,
    "level": 0,
    "partitions": null,
    "exec_ms": 0.072
}

3.1.21. `--resetmetrics <id>` (`-m`)

Clear errors and performance counters for instance(s).

Reset counters and errors for all search instances with prefix "S"

vgxadmin --resetmetrics "S*"

3.1.22. `--restarthttp <id>` (`-D`)

Restart HTTP server(s) with refreshed config. This is useful for dispatcher instances after making changes to the dispatch topology.

Restart the HTTP server on instance "TD"

vgxadmin --restarthttp TD

3.1.23. `--resumein <id>` (`-r`)

Resume transaction input service on subscriber instance(s). Transactions previously received and queued while in the paused state are immediately applied, followed by normal streaming operation.

Resume transaction input on the second search row

vgxadmin --resumein "S2*"

3.1.24. `--resumeout <id>` (`-R`)

Resume transaction output emitter on provider instance(s). Transactions previously produced and queued while in the paused state are immediately sent to subscribers, followed by normal streaming operation.

Resume transaction output on the builders

vgxadmin --resumeout "B*"

3.1.25. `--resumettl <id>` (`-T`)

Resume TTL event processor, if globally enabled. Expired objects will be deleted immediately.

Resume deletion of expired objects on builder row

vgxadmin --resumettl "B*"

3.1.26. `--reversesync <id>` (`-Y`)

Perform reverse synchronization of provider instance data from its subscriber with the highest master serial number.

Restore builder B0.1 with data from one of its configured subscribers

vgxadmin --reversesync B0.1

3.1.27. `--rollingupdate <id>` (`-L`)

Synchronize data from provider instance to all its subscribers, one at a time, with each destination subscriber in S-OUT state while being updated. Since each destination is hard-sync’ed (destructive truncation followed by rebuild) the S-OUT state is required to avoid live traffic reaching the updated instance.

At least two destination subscribers must be actively serving in S-IN mode for rolling update to be performed.

Perform rolling update of two partitions

vgxadmin --rollingupdate B0.1 --rollingupdate B0.2

3.1.28. `--servicein <id>` (`-I`)

Set instance(s) HTTP server to the S-IN state. Plugin requests on the main port will execute normally.

S-IN the second search row

vgxadmin --servicein "S2*"

3.1.29. `--serviceout <id>` (`-i`)

Set instance(s) HTTP server to the S-OUT state. Plugin requests on the main port will return HTTP code "503 Service out".

S-OUT the second search row

vgxadmin --serviceout "S2*"

3.1.30. `--show` (`-J`)

Show effective system descriptor.

Retrieve and print system descriptor used by local instance on port 9111

vgxadmin 127.0.0.1:9111 --show

3.1.31. `--start <id>` (`-s`)

For internal development. Do not use.

3.1.32. `--status <id>` (`-S`)

Print status summary for instance(s).

Print status for all instances in system

vgxadmin --status @

3.1.33. `--stop <id>` (`-x`)

Shut down instance(s).

Shut down the second search row

vgxadmin --stop "S2*"

3.1.34. `--sync <id>[,<mode>]` (`-y`)

Synchronize data from provider instance(s) to their attached subscriber(s). Synchronization is performed according to the specified <mode>, which must be one of hard, soft, or repair.

hard: Truncate all attached subscribers, then send all data
soft: Send all data to attached subscribers without truncating first
repair (default): Perform hard sync of subscribers that require it, i.e. subscribers with data different from their respective providers. Subscribers that are in sync with their providers are unaffected. Affected subscribers must be set to S-OUT for this operation to proceed.

Copy data from all builders to their subscribers

vgxadmin --sync "B*",hard

3.1.35. `--throttle <id>[,<r>,<u>]` (`-V`)

Get or set data consumption rate limits for the transaction input service.

Get: When no rate <r> and unit <u> are specified the configured throttle parameters are returned.

Set: Specify consumption rate <r> and unit <u> to apply the configuration.

Allowed values for <u> are: bytes, opcodes, operations, transactions. Rate <r> is specified as units per second.

Clear: Remove all limits by specifying rate 0 and unit all_unlimited.

On search nodes, limit input data consumption to 1,000 transactions/sec and 50,000 opcodes/sec

vgxadmin --throttle "S*",1000,transactions --throttle "S*",50000,opcodes

3.1.36. `--truncate <id>` (`-X`)

Erase instance data.

Delete all builder data (propagated to any attached subscribers)

vgxadmin --truncate B0.1 --truncate B0.2

3.1.37. `--unbind <id>` (`-U`)

Stop the transaction input service on instance.

Stop transaction input service on second search row

vgxadmin --unbind S2.1 --unbind S2.2

3.1.38. `--unsubscribe <id>` (`-u`)

Detach subscriber instance(s) from provider(s).

Detach second search row

vgxadmin --unsubscribe "S2*"

3.1.39. `--version` (`-v`)

Print version info and exit.

vgxadmin --version

3.1.40. `--waitforidle <id>` (`-w`)

Wait until instance transaction input has not been processing data for at least 5 seconds. This is useful when stringing several actions together in the same command and subsequent actions depend on previous actions to be fully completed.

Synchronize data from builders to search rows, then S-IN search rows

vgxadmin --sync "B*" --waitforidle "S*" --servicein "S*"

3.1.41. `--writable <id>` (`-G`)

Make all graph(s) in instance(s) writable.

Set instance "S1.1" writable

vgxadmin --writable S1.1

VGX Server System Administration

1. Admin Pages

1.1. Top Menu /

1.2. Admin /admin

1.2.1. Display Readout

1.2.2. Action Buttons

1.3. Status /status

1.4. Console /console

1.5. General Search /search

1.6. User Plugins and Builtin Plugins /plugin

1.7. Multi-Node System Dashboard /system

1.7.1. System Dashboard Description

1.7.2. System Dashboard Actions

1.7.2.1. Dialog Bypass

1.7.2.2. Links

1.7.2.3. Action Description

2. Administrative Service Endpoints

2.1. authtoken

2.1.1. authtoken Example

2.2. ADMIN Plugins

2.2.1. ADMIN Plugin Parameters

2.2.2. General

2.2.3. Plugin Management

2.2.4. Data Management

2.2.5. Graph State

2.2.6. Request Engine

2.2.7. Transaction Flow Management

3. vgxadmin Command Line Tool

3.1. vgxadmin Options

3.1.1. --attach <id> (-a)

3.1.2. --bind <id> (-B)

3.1.3. --cancelsync <id> (-k)

3.1.4. --cf <file> (-f)

3.1.5. --command <id>,<gr>,<cmd> (-C)

3.1.6. --confirm (-c)

3.1.7. --descriptor <id> (-Z)

3.1.8. --detach <id> (-d)

3.1.9. --endpoint <path> (-E)

3.1.10. --forcecopy <src>,<dst> (-K)

3.1.11. --help (-h)

3.1.12. --instancecfg <id> (-Q)

3.1.13. --nodestat <id>[,<key>] (-n)

3.1.14. --opdump <id> (-M)

3.1.15. --pausein <id> (-p)

3.1.16. --pauseout <id> (-P)

3.1.17. --pausettl <id> (-t)

3.1.18. --persist <id> (-W)

3.1.19. --readonly <id> (-g)

3.1.20. --reloadplugins <id>[,<pd>] (-N)

3.1.20.1. Example: Add square root capabilities to running system

3.1.21. --resetmetrics <id> (-m)

3.1.22. --restarthttp <id> (-D)

3.1.23. --resumein <id> (-r)

3.1.24. --resumeout <id> (-R)

3.1.25. --resumettl <id> (-T)

3.1.26. --reversesync <id> (-Y)

3.1.27. --rollingupdate <id> (-L)

3.1.28. --servicein <id> (-I)

3.1.29. --serviceout <id> (-i)

3.1.30. --show (-J)

3.1.31. --start <id> (-s)

3.1.32. --status <id> (-S)

3.1.33. --stop <id> (-x)

3.1.34. --sync <id>[,<mode>] (-y)

3.1.35. --throttle <id>[,<r>,<u>] (-V)

3.1.36. --truncate <id> (-X)

3.1.37. --unbind <id> (-U)

3.1.38. --unsubscribe <id> (-u)

3.1.39. --version (-v)

3.1.40. --waitforidle <id> (-w)

3.1.41. --writable <id> (-G)

1.1. Top Menu `/`

1.2. Admin `/admin`

1.3. Status `/status`

1.4. Console `/console`

1.5. General Search `/search`

1.6. User Plugins and Builtin Plugins `/plugin`

1.7. Multi-Node System Dashboard `/system`

3.1.1. `--attach <id>` (`-a`)

3.1.2. `--bind <id>` (`-B`)

3.1.3. `--cancelsync <id>` (`-k`)

3.1.4. `--cf <file>` (`-f`)

3.1.5. `--command <id>,<gr>,<cmd>` (`-C`)

3.1.6. `--confirm` (`-c`)

3.1.7. `--descriptor <id>` (`-Z`)

3.1.8. `--detach <id>` (`-d`)

3.1.9. `--endpoint <path>` (`-E`)

3.1.10. `--forcecopy <src>,<dst>` (`-K`)

3.1.11. `--help` (`-h`)

3.1.12. `--instancecfg <id>` (`-Q`)

3.1.13. `--nodestat <id>[,<key>]` (`-n`)

3.1.14. `--opdump <id>` (`-M`)

3.1.15. `--pausein <id>` (`-p`)

3.1.16. `--pauseout <id>` (`-P`)

3.1.17. `--pausettl <id>` (`-t`)

3.1.18. `--persist <id>` (`-W`)

3.1.19. `--readonly <id>` (`-g`)

3.1.20. `--reloadplugins <id>[,<pd>]` (`-N`)

3.1.21. `--resetmetrics <id>` (`-m`)

3.1.22. `--restarthttp <id>` (`-D`)

3.1.23. `--resumein <id>` (`-r`)

3.1.24. `--resumeout <id>` (`-R`)

3.1.25. `--resumettl <id>` (`-T`)

3.1.26. `--reversesync <id>` (`-Y`)

3.1.27. `--rollingupdate <id>` (`-L`)

3.1.28. `--servicein <id>` (`-I`)

3.1.29. `--serviceout <id>` (`-i`)

3.1.30. `--show` (`-J`)

3.1.31. `--start <id>` (`-s`)

3.1.32. `--status <id>` (`-S`)

3.1.33. `--stop <id>` (`-x`)

3.1.34. `--sync <id>[,<mode>]` (`-y`)

3.1.35. `--throttle <id>[,<r>,<u>]` (`-V`)

3.1.36. `--truncate <id>` (`-X`)

3.1.37. `--unbind <id>` (`-U`)

3.1.38. `--unsubscribe <id>` (`-u`)

3.1.39. `--version` (`-v`)

3.1.40. `--waitforidle <id>` (`-w`)

3.1.41. `--writable <id>` (`-G`)