Event Pattern Authoring

The majority of pattern detection logic is configured in the event pattern view. In this environment, you define how to find matches from incoming data via elements, either by themselves, or in relation with other elements by means of constraints. Once a pattern is detected, outcomes are used to produce output data based on the input data or computational results of the input data.

Working with Elements

An element is a unit of pattern detection logic that defines what parts of the incoming data constitute a match. In other words, elements tell the system what kinds of things to look for in the data. For more information about elements, see Model Artifacts.

Creating Elements

To create an element:

1. In the tool palette, click the Element (⬭) icon.
2. Click the event type to associate with the new element. Click New Event Type to create new event types, and then follow the instructions in Creating New Event Types.
3. Click any open space in the diagramming area to place the element.
4. In the details panel, edit the element’s properties as needed. For more information, see Editing Elements.

Elements may also be created directly on the canvas as follows:

1. Right-click any empty space in the canvas.
2. In the selection menu, click Add Element.
3. In the dialog, select an event type. To create a new event type, click New, then follow the instructions in Creating New Event Types.

note

Depending on the source of the input data, elements in the event pattern view may have differently shaded colors:

• An element associated with a raw event type is deep blue.
• An element associated with a derived event type (that is, one that comes from an output of a lower-level event pattern) is brighter blue.

Editing Elements

When an element is selected in the event pattern view, its information is displayed in the details panel, where the element's details can be viewed and modified as needed.

To edit an element:

1. In the diagramming area, click the element to edit.
2. In the details panel, click the Contents tab if it is not already selected.
3. Edit the element's details as necessary:
   1. Under Attributes:
      1. From the Event Types dropdown menu, select the event type to modify the event type associated with the element.
      2. In the Alias field, enter an alias for labeling the element. (It is recommended that the alias describe the element's unique role in its event pattern that differentiates it from other elements using the same event type.)
      3. In the Min field, specify the minimum number of events the element must receive.
      4. In the Max field, specify the maximum number of events the element must receive.
      5. Under Time to Live, define an expiration period for the element (i.e., how long it should exist in the system before being discarded) if needed:
         1. Click +. The Create Time-To-Live dialog opens. (The Create Time-To-Live dialog is also accessible by right-clicking an element and selecting Add Time to Live from the context menu.)
         2. In the Time Value field, enter a number indicating how long the element should remain valid.
         3. From the Unit Type dropdown menu, select the appropriate unit of time for the specified time value.
         4. Check the Apply to Matched Events checkbox to drop any matched records after the configured time-to-live period. Leave it unchecked to let matched records persist after the configured time-to-live period.
         5. Click Create to create the expiration period, or Cancel to discard it.
         6. Edit the expiration period by changing the field values, if needed.
         7. Click the Delete (trash can) icon to clear the entered values, if necessary.
   2. Under the Event Type section (which uses the name of the event type selected for the element):
      1. Click the pencil (✎) icon to edit the details.
      2. Follow the procedure in Editing Event Types.

Working with Outcomes

An outcome is a unit of pattern detection logic that represents output data. In other words, once the system finds a pattern established in an element, an outcome tells the system what to do after that discovery. For more information about outcomes, see Model Artifacts.

Creating Outcomes

To create an outcome:

1. In the tool palette, click the Outcome (▭) icon.
2. Select the event type to associate with the new outcome. If a new event type should be created for the outcome, follow the process in Creating New Event Types.
3. Click any open space in the diagramming area to place the outcome.
4. In the details panel, edit the outcome’s properties as needed. For more information, see Editing Outcomes.

Outcomes may also be created directly on the canvas as follows:

1. Right-click any empty space in the canvas.
2. In the selection menu, click Add Outcome.
3. In the dialog, select an event type. If a new event type should be created, click New, then follow the instructions in Creating New Event Types.

Editing Outcomes

When an outcome is selected in the event pattern view, its information is displayed in the details panel, where the outcome's details can be viewed and modified as needed.

To edit an outcome:

1. In the diagramming area, click an outcome to edit.
2. In the details panel, click the Content tab if it is not already selected.
3. Edit the outcome’s details as needed:
   1. Under Attributes:
      1. Under Sources, click a source to edit its details. For more information, see Editing Elements.
      2. Under Computations, click a computation to check its details. Click the Delete (trash can) icon beside a computation to delete it. For more information about computations, see Built-In Computation Functions.
      3. Under Risk, choose the appropriate risk type:
         1. If the risk factors (elements) are independent from one another, select Weighted to assign a weight to each element.
         2. If the risk factors (elements) are dependent upon one another, select Bayesian to specify the conditions used to compute risk.
      4. Under Batch Output, click the Enable Batch Output toggle to enable batch output if desired. (By default, the toggle is set to "Off.") For more information, see Configuring Batch Output.
   2. Under the Outcome section (which uses the name of the selected outcome):
      1. Click the pencil (✎) icon to edit the details.
      2. Follow the procedure in Editing Event Types.

Understanding Risk Types

Cogynt models can calculate risk based on two different approaches: weighted risk and Bayesian risk.

Risk types and weights are assigned when editing outcomes.

Using Weighted Risk

Weighted risk is used when the probability (weight) of elements or risk factors does not change based on the presence or absence of other risk factors. In other words, weighted risk is used for independent risk factors.

tip

Suppose you want to calculate a population's cancer risk based on a number of personal and environmental factors, such as whether they smoke, whether they work with toxic chemicals, and so on. Since the risk of each factor remains more or less static, weighted risk would be useful for this model.

Using Bayesian Risk

Bayesian risk is modeled after the Bayes' theorem. It is used when the probability (weight) of elements or risk factors can change based on the presence or absence of other risk factors. In other words, Bayesian risk is used for dependent risk factors.

tip

Suppose you want to calculate the risk of a wildfire occurring in a particular region. The wildfire risk might increase if the region has seen a drought, and it might decrease if there has been an unusual amount of rain. In this case, Bayesian risk would be useful to model the wildfire probability.

Configuring Batch Output

"Batching" refers to an increment of time during which the system waits to publish any updates for an event pattern until the next increment begins.

Batch output defines the increment of time during which the system listens to incoming data streams, and only publishes the data's updated (latest) state acquired during that interval.

note

The batch output interval refers to server time (time elapsed during the operation of the server), not data time (time specified in the data's timestamps or metadata).

To configure batch output for event patterns:

1. In the diagramming area, click the desired event pattern's outcome.
2. In the details panel, click the Contents tab if not already selected.
3. Click the Batch Output section to expand it.
4. Click the Enable Batch Output toggle to set it to "On".
5. In the Time Value field, enter the interval of time that the output window should span.
6. From the Unit Type dropdown menu, select the appropriate unit for the time value.
7. Click outside of the details menu to save the changes.

Working with Constraints

A constraint is a condition placed upon an input event for an event pattern to be considered a match. For more information about constraints, see Model Artifacts.

Creating Constraints

To create a constraint:

1. In the diagramming area, click an element.
2. Click the Connection (🔀) button, then drag the line to another element.
3. In the Create a Constraint dialog:
   1. For the first (green) element:
      1. Click the Fields dropdown menu.
      2. Select the appropriate field name. If the list is too long, type the first few characters of the field's name to narrow down the list.
      3. Click Computations if computations based on field values are needed to produce a result for comparison. For more information, see Constraint Computations Authoring.
   2. Repeat steps 3.i.a–3.i.c for the second (yellow) element.
   3. Click the Comparison Option dropdown menu to select a comparison option for the two elements.
      1. In the Offset field, if necessary, specify a value to add to the source event for the comparison. (For example, if you knew that all the source event's dates would be five minutes behind due to a server outage, you could use Offset to add five minutes to the source value.)
      2. In the Tolerance field, if necessary, specify a value to add and subtract from the source event to create a range of valid values for comparison operations. (For example, if you knew that the source event's dates would be either five minutes ahead or five minutes behind, and wanted to cover all possible dates within that range, then you could use Tolerance to establish this as the acceptable range.)
4. Click OK to create the constraint, or click Cancel to discard it.

Selecting Constraints

Constraint options vary depending on the datatypes involved. For example, if a constraint is created between two scalar strings, the constraints are limited to "equal" and "not equal."

The available constraints are as follows:

• Contains – Checks whether one element is contained within the other.
• Count equal – Checks whether the number of elements in an array is equal to the specified number.
• Count greater than – Checks whether the number of elements in an array exceeds the specified number.
• Count greater than or equal – Checks whether the number of elements in an array exceeds or matches the specified number.
• Count less than – Checks whether the number of elements in an array is less than the specified number.
• Count less than or equal – Checks whether the number of elements in an array is less than or equal to the specified number.
• Count not equal – Checks whether the number of elements in an array does not match the specified number.
• Equal – Checks whether both elements have the same value.
• Greater than – Checks whether one element represents a greater number than the other.
• Greater than or equal – Checks whether one element represents a greater or equivalent number than the other.
• In network – Checks whether one element exists in the same network as the other.
• Inside – Checks whether one element is situated inside the other.
• Less than – Checks whether one element represents a lesser number than the other.
• Less than or equal – Checks whether one element represents a lesser or equivalent number than the other.
• Not equal – Checks whether the elements are not the same.
• Outside – Checks whether one element is situated outside the other.

The possible datatypes are as follows, and can be either scalars or arrays:

• Boolean – A true or false value.
• Date/Time – A datatype that represents a date and time together as a single data unit.
• Float – A floating point number (decimal value or fractional number).
• Geo Coordinate – A GeoJSON-formatted pair of numerical coordinates indicating a single geographical point. For more information about using this datatype, see Working with GeoJSON Datatypes.
• Geo Polygon – A GeoJSON-formatted set of coordinate pairs describing a geographical area. For more information about using this datatype, see Working with GeoJSON Datatypes.
• Integer – A whole number, such as 25 or -25.
• IP – A unique address that identifies a device on the internet or a local network.
• IP Range – A set of consecutive IP addresses.
• JSON – A set of objects consisting of attribute-value pairs and arrays.
• Null – An empty value.
• String – A sequence of (often alphabetical) characters.
• Unique ID (UUID) – An alphanumeric string for labeling information in a computer system.
• URL – An address for a unique resource on the Internet.

For certain Cogynt builds, several datatypes beginning with the prefix COG_ may be available. These datatypes represent Cogynt-level metadata or extra features that invoke particular Cogynt behaviors. The COG_ datatypes include:

• COG_attachments: Metadata for files that are manually attached to events via external submission mechanisms. (For example, if your Cogynt setup is designed to accept incident reports from an external interface, COG_attachments collects an attached incident report's metadata.) The metadata is passed into HCEP as a JSON object. The attached file itself is then viewable in Workstation.
• COG_confidence: The place where risk scores are published. Manually mapping values here allows bypassing risk tables if desired.
• COG_filter: A Boolean field that, if it receives True, publishes the event to the output topic. If it receives False, it suppresses the event instead.
• COG_id: A unique identifier for items. COG_id only exists in input events, and cannot be overwritten from the UI.
• COG_matches: The behavior of COG_matches varies depending on whether it is an input or output element:
  • As input: If the element or event type has a lexicon filter on it, then COG_matches produces an array of words that matches the lexicon. Similarly, for matched nodes, it lists the name of the lexicon node.
  • As output: If mapped to an outcome, words indicating lexicon matches are highlighted when shown in Cogynt Workstation.
• COG_timestamp: An optional field. Datetime values can be mapped to COG_timestamp. When ingested, Cogynt Workstation interprets the mapped value to determine the timestamp used for denoting events in chronological order. (COG_timestamp has no impact on how Authoring patterns behave.)

The constraints available for the various datatype combinations are outlined in the following tables.

note

Only valid datatype pairings are listed in the tables. If a pairing is not listed, then there are no constraints available for it.

The same constraints are available regardless of which datatype is the first and which is the second in the pairing.

Scalar-to-Scalar Constraints

First Datatype	Second Datatype	Available Constraints
Boolean	Boolean	equal not equal
Date/Time	Date/Time	equal not equal less than less than or equal greater than greater than or equal
Float	Float	equal not equal less than less than or equal greater than greater than or equal
Geo Coordinate	Geo Coordinate	equal not equal
Geo Coordinate	Geo Polygon	inside outside
Geo Polygon	Geo Polygon	equal
Integer	Integer	equal not equal less than less than or equal greater than greater than or equal
IP	IP	equal not equal in network
IP Range	IP	equal not equal in network
IP Range	IP Range	equal not equal in network
JSON	JSON	equal not equal
String	String	equal not equal
Unique ID	Unique ID	equal not equal
URL	URL	equal not equal

Array-to-Array Constraints

First Datatype	Second Datatype	Available Constraints
Geo Coordinate	Geo Coordinate	equal not equal
Geo Coordinate	Geo Polygon	inside outside
Geo Polygon	Geo Polygon	equal
JSON	JSON	equal not equal

Array-to-Scalar Constraints

First Datatype (Array)	Second Datatype (Scalar)	Available Constraints
Boolean	Boolean	contains
Boolean	Integer	count equal count not equal count less than count less than or equal count greater than count greater than or equal
Date/Time	Date/Time	contains
Date/Time	Integer	count equal count not equal count less than count less than or equal count greater than count greater than or equal
Float	Float	contains
Float	Integer	count equal count not equal count less than count less than or equal count greater than count greater than or equal
Geo Coordinate	Geo Coordinate	equal not equal
Geo Coordinate	Geo Polygon	inside outside
Geo Polygon	Geo Coordinate	inside outside
Geo Polygon	Geo Polygon	equal
Integer	Integer	contains count equal count not equal count less than count less than or equal count greater than count greater than or equal
IP	Integer	count equal count not equal count less than count less than or equal count greater than count greater than or equal
IP	IP Range	contains
String	Integer	count equal count not equal count less than count less than or equal count greater than count greater than or equal
String	String	contains
Unique ID	Integer	count equal count not equal count less than count less than or equal count greater than count greater than or equal
Unique ID	Unique ID	contains
URL	Integer	count equal count not equal count less than count less than or equal count greater than count greater than or equal
URL	URL	contains

Working with GeoJSON Datatypes

GeoJSON-formatted datatypes must adhere to the following conditions.

GeoJSON Formatting

A valid GeoJSON object must be represented in either Geometry or Feature style.

Geometry Style

A GeoJSON object in Geometry style must contain two fields: type and coordinates. The following are examples of valid GeoJSON objects in Geometry style:

"my_point": {
    "type": "Point",
    "coordinates": [125.6, 10.1]
}

"my_polygon": {
    "type": "Polygon",
    "coordinates": [
        [
            [0, 0], [10, 10], [10, 0], [0, 0]
        ]        
    ]
}

Feature Style

Feature style is a wrapper of Geometry style. It is designed to provide additional information.

On the root layer, a valid Feature-style GeoJSON object must contain three fields: type, geometry, and properties. The following are examples of valid GeoJSON objects in Feature style:

"my_point": {
    "type": "Feature",
    "geometry": {
        "type": "Point",
        "coordinates": [102.0, 0.5]
    },
    "properties": {
        "some_prop": "some_value"
    }
}

"my_polygon": {
    "type": "Feature",
    "geometry": {
        "type": "Polygon",
        "coordinates": [
            [
                [0, 0], [10, 10], [10, 0], [0, 0]
            ]            
        ]
    },
    "properties": null
}

Working with Geo Polygons

Geo polygons must be closed (i.e., the first and last coordinate listed in the coordinates field must be the same).

Multi-polygons are not accepted, even though such constructs are valid GeoJSON forms.

Editing Constraints

To edit a constraint:

1. In the diagramming area, click a constraint to edit.
2. In the details panel, click the Contents tab if it is not already selected.
3. For the first (green) element:
   1. Click the Fields dropdown menu.
   2. Select the appropriate field name. If the list is too long, type the first few characters of the field’s name to narrow down the list.
   3. Click Computations if computations based on field values are needed to produce a result for comparison. For more information, see Constraint Computations Authoring.
4. Repeat steps 3.i-3.iii for the second (yellow) element.
5. Click the Comparison Option dropdown menu to a select a comparison option for the two elements.
6. Click anywhere outside the details panel to save the changes.

Editing Groups

The only editable attribute of a group is its color.

note

Groups are a legacy feature in Cogynt, and no longer serve any necessary functions.

To change the color of a group:

1. In the diagramming area, click the group to edit.
2. In the details panel, click the Contents tab if it's not already selected.
3. Click the color bar above the Members section.
4. From the color palette, select the desired color.
5. Click anywhere outside the color palette to close it and save the change.

For more information about groups, see Model Artifacts.

Working with Partitions

Partitions instruct the system to split and group the analysis by one or more fields.

tip

Consider a pattern that analyzes a stream of bank transactions from a bank, containing the magnitude of position or negative flow of cash value and the bank account to which it pertains. The pattern computes the balance for each account.

In this case, a partition is set to the bank account field, so the system knows to compute a balance for each bank account, rather than treat the entire stream as a single bank account.

Creating Partitions

To create a partition for an event pattern:

1. In the diagramming area, click an event pattern to give a partition.
2. In the details panel, click the Contents tab if it is not already selected.
3. Under Attributes, click the Partitions section to expand it.
4. Set the Enable Partitions toggle to "On."
5. Click +.
6. In the Create Partition dialog:
   1. From the Elements dropdown menu, select an element containing the field to partition over.
   2. From the Fields dropdown menu, select the field to partition over. (The Fields dropdown menu cannot be opened until a selection is made from Elements.)
   3. Click the Split Array toggle to activate or deactivate array splitting. If set to "On," a unique partition is created for each element in the input array.
7. Set the Partition Over Window toggle to "On" to allow treating windows as their own discrete entities (by granting each window a unique ID within Cogynt).
8. Click Create to create the partition, or click Cancel to discard it.

note

The Partition Over Window toggle only appears if windowing is enabled. For more information, see Setting Windowing.

Editing Partitions

Existing partitions can be modified as needed.

To edit a partition:

1. In the diagramming area, click an event pattern that contains a partition.
2. In the details panel, click the Contents tab if it is not already selected.
3. Under Attributes, click the Partitions section to expand it.
4. Toggle Enable Partitions to "On" if it is not already.
5. Click the pencil (✎) icon.
6. Edit the partition details as needed. For more information about the partition details, see Steps 6 and 7 of Creating Partitions.

note

The Split Array toggle may be adjusted without having to click the pencil icon. For more information about splitting arrays, see Step 6 of Creating Partitions.

Deleting Partitions

Partitions that are no longer necessary can be removed.

To delete a partition:

1. In the diagramming area, click an event pattern that contains a partition.
2. In the details panel, click the Contents tab if it is not already selected.
3. Under Attributes, click the Partitions section to expand it.
4. Click the trash can (🗑) icon.

Setting Windowing

"Windowing" is the process of splitting a stream of data into portions of finite size, which can then be subjected to computations. Windowing separates the data from an input stream into discrete sections, making it possible to process and calculate data over time. For more information, refer to Windows in the official Apache Flink documentation.

To add windowing to an event pattern:

1. In the diagramming area, click an event pattern to set windowing for.
2. In the details panel, click the Contents tab if it is not already selected.
3. Under Attributes, click the Windowing section to expand it.
4. From the Window Type dropdown menu, select the desired window type ("Sliding" or "Tumbling"). For more information about these windowing types, refer to Tumbling Windows and Sliding Windows in the official Apache Flink documentation.
5. From the Input Event Type dropdown menu, select the event type to use as the input.
6. From the Referenced Timestamp dropdown menu, select the timestamp for the window to use as its referencing timeframe.
7. If the Referenced Timestamp selection is anything other than "Kafka Ingestion Time", the Lateness field appears. In the Lateness field, specify the length and unit of time that the system should continue accepting and processing events from a window after it has closed.
8. In the Window Length field, specify the minimum length of time that the window should span.
9. In the Slide Length field, specify the length of time for each sliding window. (Note: This field appears only if Window Type is set to "Sliding".)
10. From the Unit Type dropdown menu to the right of the Minimum Length field, specify the unit of time for the Minimum Length value.
11. In the Timeout field, specify how long the system should wait before closing a window if no new input data is being received. (Note: The Timeout value should not exceed the Minimum Length value.)
12. From the Unit Type dropdown menu to the right of the Timeout field, specify the unit of time for the Timeout value.
13. Click outside of the details menu to save the changes.