Data Authoring

Telling the system how to recognize and deal with data is an essential task in the authoring process. In Cogynt Authoring, the metadata describing incoming data comes in two forms, both of which are top-level model artifacts:

• User Data Schema – Contains a list of fields consisting of names, identifiers, and data types.
• Event Type – Uses fields defined by user data schemas, and also has its own attributes for dealing with certain streaming characteristics of the incoming data.

The separation between the two entities allows multiple event types to link to one user data schema. Changes to the schema are automatically reflected to all event types linked to it. For more information, see Model Artifacts.

Managing User Data Schemas

The Schemas window provides tools for managing and working with user data schemas. This section describes them in detail.

To open the Schemas window, click the Schemas button on the application toolbar.

If a project is currently active, the window displays a table showing all the user schemas belonging to the project. If there is no active project, selecting one from the Project dropdown menu opens it and displays its user data schemas.

Creating User Data Schemas

New user data schemas can be manually added to the project as needed.

To create a new user data schema:

1. In the Schemas window, click + New Schema.
2. In the New Data Schema dialog:
   1. In the Name field, enter a name for the user data schema. The name can't exceed 255 characters in length.
   2. In the Description field, enter a description for the user data schema.
   3. In the Fields section, add new fields to the user data schema as necessary. For more information, see Adding Fields to User Data Schemas.
3. Click Create to create the new schema, or click Cancel to discard it.

Exporting User Data Schemas

User data schemas can be exported as downloadable files that can be loaded into other projects.

To export a user data schema:

1. In the Schemas window, locate the user data schema to export.
2. Click the corresponding More menu (⋮).
3. Click Export.
4. Refer to the procedures in Exporting and Importing Authoring Files.

Duplicating User Data Schemas

User data schemas can be copied when necessary. For example, rather than starting from a blank schema, it may save time to create a modified version of an existing schema.

To duplicate a user data schema:

1. In the Schemas window, locate the user data schema to duplicate.
2. Click the corresponding More menu (⋮).
3. Click Duplicate.
4. In the prompt, type a new name for the duplicate user data schema.

Deleting User Data Schemas

Unnecessary or erroneous user data schemas can be deleted as needed.

To delete a user data schema:

1. In the Schemas window, locate the user data schema to delete.
2. Click the corresponding More menu (⋮).
3. Click the Delete (trash can) button.

warning

Remove all event type linkages from the target user data schema before attempting to delete it.

Deletion can occur only if there are no event types linked to the schema. This prevents event types from having invalid schema linkages.

Editing User Data Schemas

Existing user data schemas can be modified as needed in order to keep them accurate and up-to-date. This section describes the various modifications that can be performed.

To edit a user data schema:

1. In the Schemas window, locate the user data schema to edit.
2. Click the corresponding Edit (✎) button. The user data schema editor opens.

Adding Fields to User Data Schemas

User data schemas have an ordered list of fields, each given a name and a data type, that defines the basic structure of incoming data. New fields can be added to user data schemas as needed.

Authoring allows adding fields manually or with system assistance. System assistance is recommended to help reduce errors.

To manually add a field to a user data schema:

1. In the user data schema editor, click + Add Field. (Note: All user data schemas require at least one field.)
2. For each new field:
   1. From the Type dropdown menu, select the field’s data type. For more information, see the note following these instructions.
   2. In the Name text box, enter a name for the field.
   3. In the Kafka Topic Path field, specify the name of the field as it exists in the Kafka topic.
   4. Check the Sync Path checkbox to automatically change the Kafka Topic Path value to match changes to the Name value.

To add a field to a user data schema with system assistance:

1. In the user data schema editor, click the dropdown to the right of the + Add Field. button.
2. Select Add fields from a Kafka topic.
3. In the Select Kafka Topic Fields window, select a Kafka topic.
4. Review the topic's available fields and their sample records in the preview window.
5. Click Add X Fields (where "X" is the number of fields discovered) to add the Kafka topic fields to the schema.
6. In the user data schema editor, confirm the field's details:
   1. From the Type dropdown menu, select the field’s data type. For more information, see the note following these instructions.
   2. In the Name text box, enter a name for the field.
   3. In the Kafka Topic Path field, verify the name of the field as it exists in the Kafka topic. (Note that this value is not editable for fields added from Kafka topics.)
   4. Check the Sync Path checkbox to automatically change the Kafka Topic Path value to match changes to the Name value.

note

Cogynt Authoring supports the following data types for user data schema fields:

• Array
• Boolean
• Date/time
• Float
• Geo coordinate
• Geo polygon
• Integer
• IP
• JSON
• String
• URL
• Unique ID

Selecting the "array" data type allows specifying the particular kind of array:

• Generic array (for dealing with special cases, such as arrays of objects)
• Boolean array
• Date/time array
• Float array
• Geo coordinate array
• Geo polygon array
• Integer array
• IP array
• JSON array
• String array
• URL array
• Unique ID array

Working with Nested Fields

On occasion, you might want to make a user data schema using the individual fields from a JSON with nested fields (instead of passing the entire JSON as its own field). In such cases, Cogynt recommends making a field for each individual subfield entry.

For example, consider a field like the following:

{
    "city_proper": {
        "definition": "Metropolis prefecture", 
        ...
    }
}

Here, the city_proper field has a subfield called definition whose value is the string Metropolis prefecture.

To add a field that records this information:

1. Follow the instructions in Adding Fields to User Data Schemas.
2. For the Name of the field, enter any value you like. (For example, City Definition.)
3. For the Path, enter city_proper|definition.
4. For the Type of the field, specify "String."

note

Cogynt uses a pipe (|) as a delimiter in such cases, not a dot (.). This is to support situations where field names in JSON files contain a dot.

Reordering User Data Schema Fields

When editing an existing user data schema, the editor loads its full information. HCEP processes the fields in the order that they appear in the list. This order can be rearranged as needed.

To reorder user data schema fields:

1. In the user data schema editor, under the Data Schema Fields section, locate the field to move up or down the list.
2. On the left of the field, click the dotted drag handle and hold down the mouse button.
3. Drag the field to the desired place in the list.
4. Release the mouse button.

Editing User Data Schema Fields

The individual fields of user data schemas are modifiable.

To edit a user data schema field:

1. In the user data schema editor, locate the field to edit.
2. Verify that the Type is the appropriate kind.
3. In the Name text box, enter a new name for the field as needed.
4. In the Kafka Topic Path text box, specify the name of the field as it exists in the Kafka topic.
5. Check the Sync Path checkbox to automatically match the field's Path value to the entered Name value. (This option is the default selection.)
6. Click Save to save the changes, or Cancel to discard the changes and exit the editor.

Deleting User Data Schema Fields

Unnecessary or erroneous fields in user data schemas can be removed.

To delete a user data schema field:

1. In the user data schema editor, locate the field to delete.
2. To the right of the field, click the corresponding X button.
3. In the Confirm Field Delete dialogue, click OK to delete the field, or Cancel to retain it.

Managing Event Types

The system and event pattern views do not deal with user data schemas directly. Instead, they require event types.

The Event Type Management window provides tools for managing and working with event types as needed. This section describes them in detail.

To open the Event Type Management window, click the Event Types button on the application toolbar.

Creating New Event Types

New event types can be created as needed using the Create Event Type dialog.

To access the dialog, do one of the following:

• In the Event Type Management window, click + New Event Type.
• Select the Create a new event type option when creating a connecting event, then click OK. For more information, see Establishing Event Pattern Data Flow.

To create a new event type in the Create Event Type dialog:

1. On the left of the screen, click Step 1: Event Info. On this page:
   1. In the Name field, specify a name for the event type.
   2. In the Description field, enter a description for the event type if desired.
   3. Select whether the Data Storage should be Permanent or Temporary by clicking the corresponding option button. (When "Temporary" is selected, results from this pattern will not persist after the deployment is terminated.)
   4. Click Deletion Settings to choose whether to activate automatic topic data deletion. (This is set to "Off" by default.)
   5. From the Data Source dropdown menu, select the source of the event type's data.
   6. From the Stream Type dropdown menu, select the appropriate type of data stream.
   7. In the Topic field, specify the name of the source topic as it appears in Kafka (or select the desired source topic from the prepopulated list).
   8. Click the Edit Topic Partition Settings icon to modify the number of topic partitions, if desired.
   9. In the Event Fields section:
      1. Select the New Schema option button to create a new data schema. Follow the field creation instructions outlined in Adding Fields to User Data Schemas.
      2. Select the Existing Schema option button to choose an existing schema. Select the desired schema from the Schema dropdown menu, and click Edit Schema to edit the schema if necessary.
2. On the left of the screen, click Step 2: Advanced Settings. On this page:
   1. Choose the appropriate Link Analysis type that should represent the event type in Cogynt Analyst Workstation's link analysis visualization:
      • Select None to exclude the event type from link analysis.
      • Select Linkage to set a flag informing Cogynt Workstation that the event type concerns the relationship between entities (for example, if the event type is supposed to flag a connection or a match).
      • Select Entity to set a flag informing Cogynt Workstation that the event type represents a concrete thing (such as a person or an address).
   2. Choose the appropriate Lexicon Filter:
      • Select None to avoid using a lexicon filter.
      • Select Contains to filter by matching words. Any words found in the lexicon are considered matches, and will pass through the filter. From the Field Name dropdown menu, select the appropriate field to filter by, then select the appropriate lexicon group from the Lexicon Group dropdown menu.
      • Select Excludes to filter out certain words. Any words found in the lexicon are not considered matches, and will not pass through the filter. From the Field Name dropdown menu, select the appropriate field to WHAT, then select the appropriate lexicon group from the Lexicon Group dropdown menu.
   3. Under Manual Actions, configure the names of any actions that should be available to Cogynt Workstation analysts when inspecting events published by the event type. For more information, see Using Manual Actions.
3. At the top of the screen, click Create to create the event type, or Cancel to discard it.

note

The minimum unit necessary for link analysis in Workstation is two separate event types in Authoring, one of which is an "entity," and one of which is a "linkage."

For more information about link analysis in Workstation, refer to the Workstation User Guide.

Editing Event Types

The process for editing events is largely the same as creating them (see Creating New Event Types). The main difference is that tag linkage is available when editing event types. For more information about tags, see Managing Tags.

note

The Tags dropdown menu does not appear during event type creation, because tags can only be assigned to existing artifacts. The event type does not exist until the creation process has finished.

To edit an event type:

1. In the Event Type Management window, click the Edit (✎) icon beside the event type to edit.
2. Edit the event type’s information as needed. For descriptions of each field, see Creating New Event Types.
3. In the Tags menu:
   1. Click the dropdown button to add a new tag or link an existing tag to the event type. For more information, see Managing Tags.
   2. Click the Edit (✎) button beside a linked tag to change it.
   3. Click the Unlink button beside a linked tag to unlink it from the event type.
4. Click Save to save the changes, Revert to undo the changes and remain in the editor, or Cancel to discard the changes and exit the editor.

Exporting Event Types

Event types can be exported as downloadable files that can be loaded into other projects.

To export an event type:

1. In the Event Type Management window, locate the event type to export.
2. Click the corresponding Export button.
3. Save the file to the desired location on your machine.

Duplicating Event Types

Event types can be copied when necessary.

To duplicate an event type:

1. In the Event Type Management window, locate the event type to duplicate.
2. Click the corresponding Duplicate button.
3. In the prompt, type a new name for the duplicate event type.

Deleting Event Types

Unnecessary or erroneous event type can be deleted as needed.

To delete an event type:

1. In the Event Type Management window, locate the event type to delete.
2. Click the corresponding Delete (trash can) button.