Cogynt for Operation Managers

As an operation manager, your job is to keep the operation running smoothly. You need to keep an eye on the health of your system and ensure that the data is flowing as it should. It's also your responsibility to turn data about your operation and signals from the system into accessible dashboards and coherent visuals that stakeholders can understand.

With this in mind, your main tasks in Cogynt include:

1. Preparing data for dashboard use.
2. Creating charts in Superset.
3. Creating dashboards from Superset charts.
4. Assigning dashboard access permissions.

Each of these tasks is outlined in greater detail in the sections to follow. Where applicable, links to relevant documentation are provided.

Preparing Data

Cogynt dashboards use Apache Pinot and Apache Superset for organizing and displaying data. Configuring Cogynt data for Superset involves two main steps:

1. Ingesting data into Pinot.
2. Connecting Pinot to Superset.

For more information about each step, see the corresponding topic headings.

Ingesting Data

Pinot can accept data from Kafka topics or CSV files.

note

In order to ingest data, the following prerequisites must be met:

• Any Kafka topics used must already exist at setup time. (Pinot will ingest data from the topic live, so it is not necessary for the topic to contain data first.)
• All datetime fields must be in "epoch long" (BIGINT) format.

Accessing Pinot

Pinot is included as a standard part of the Cogynt cluster. It is accessible from an Internet-connected web browser.

The Pinot URL follows the syntax https://pinot-[environment].[domain_name].[domain_suffix], where:

• [environment] is the cluster to access. Customers may have access to different clusters (such as development, staging, or production).
• [domain name] is the customer's domain (for example, cogility).
• [domain suffix] is the customer's domain suffix (for instance, com, io, org, etc.).

tip

Consider the following hypothetical scenario:

• The production cluster is named prod.
• The customer's domain is named example.
• The customer's domain suffix is com.

Given this information, navigating to https://pinot-prod.example.com would access Pinot in the production environment.

Putting Kafka Data into Pinot

To lay the groundwork for ingesting Kafka topic data, you will need to create table schemas and realtime tables to specify how the incoming data should be interpreted and processed.

Creating New Table Schemas

Table schemas tell Pinot how to interpret incoming data. Configuring a table schema ensures that the data Pinot ingests will be read and displayed properly when you construct charts and dashboards in Superset later.

To create a new table schema:

1. Access Pinot.
2. Go to Tables.
3. For each Kafka topic or data file, use the Add Schema feature to define a corresponding schema. (The schema for the topic can be obtained from either Cogynt Authoring or from model documentation.)
   1. Provide a name for the schema.
   2. Add columns corresponding to each field that exists in the topic schema:
      1. Set the field/column name. (Note: Column names cannot contain spaces.)
      2. Select the field type:
         • Use dimension for fields containing categorical or non-numerical values, such as an integer or string.
         • Use metric for fields containing a numerical values.
         • Use datetime for fields with date and time values. (Note: It is mandatory to have at least one datetime field to apply the schema to a REALTIME table.)
      3. Select the datatype:
         • For dimension and metric, select the appropriate datatype for the field/column.
         • For datetime, Cogynt will only support fields that are epoch long. Therefore, select LONG, where the time unit is MILLISECONDS and the time format is EPOCH. The rest can be left as their default values.
         • For datetime in Zulu format, select STRING for the datatype, then choose the appropriate time unit and format corresponding to your Zulu format. (For example, if your Zulu time includes milliseconds, then the time format should be yyyy-MM-dd''T''HH:mm:ss.SSSZ.)

Creating Realtime Tables

Offline tables are updated in batches, but data streams like Kafka aim to be updated in real time. Realtime tables facilitate creating a Pinot table from Kafka data.

To create a new realtime table:

1. Access Pinot.
2. Go to Tables > Add Realtime Table.
3. Set up the realtime table:
   1. Name the table after your data schema.
      • The table will automatically recognize and set the schema with the same table name, and automatically select the timestamp field as the table's timestamp.
      • Naming the table the same as the schema is necessary (at least through the Pinot UI), because there is no option to manually assign a schema to the table.
   2. If you have more than one datetime column, pick a time column to serve as the table's primary datetime column.
   3. Set the replication. A value of at least 2 or 3 is recommended for most production environments. For more information, refer to the official Apache Pinot documentation.
   4. Set any transform functions, if necessary.
   5. Set the stream configurations:
      1. Stream.kafka.broker.list: kafka.confluent.svc.cluster.local:9071 (Warning: If a mistake is made here, the table name cannot be modified, deleted, or re-used. In the event of an error, you will need to make a new table.)
      2. Stream.kafka.TOPIC.NAME: Where TOPIC.NAME is the topic name in Kafka.
   6. Set the indexing, if necessary. This feature is typically used for fine-tuning performance. For more information, refer to the official Apache Pinot documentation.
   7. Set the partitioning, if necessary. (Note: Setting the partitioning is recommended to ensure cluster performance.) For more information, refer to the official Apache Pinot documentation.
   8. Click Save.

Verifying Tables

After constructing table schemas and realtime tables, it is worth confirming that they work as intended before moving on to making charts and dashboards.

To verify your tables in Pinot:

1. Go to Query Console.
2. Click the table name.
3. The UI automatically queries the first ten records in the table. Verify that the data is complete, as well as correctly configured and interpreted.

To test whether a table is live, publish a single record into Kafka, and verify that the table's record count has incremented by 1.

Putting CSV Data into Pinot

If your data is not in a Kafka topic, but instead exists as a CSV file, that file's data can be uploaded to Pinot after several preparatory steps.

Setting up Schemas and Tables

Like with a Kafka topic, you will need to configure schemas and tables to tell Pinot how your CSV data should be interpreted.

To set up a schema and table:

1. Access Pinot.
2. Go to Cluster Manager > Tables > Add Schema.
3. For each CSV file, follow the instructions from Creating New Table Schemas.
4. Go to Home > Cluster Manager > Tables > Add Offline Table.
5. Configure an offline table:
   1. Name the table after your data schema.
   2. Set a timestamp, if desired, by selecting the correct timestamp column to use as the table's primary timestamp. (Note: Timestamps are optional for offline tables.)
   3. Leave all other values as their defaults for a quick start. If fine-tuning is desired, refer to the official Apache Pinot documentation for details.

Generating API Calls

Pinot requires an API call to accept a CSV file for upload.

To generate an API call:

1. Access Pinot.
2. Go to Swagger REST API.
3. Set Schemes to HTTPS.
4. Under the Table section, locate POST /ingestFromFile. Click to expand it.
5. Click Try it out to enable UI-based configuration.
6. Configure the API:
   1. tableNameWithType: [table-name]_OFFLINE (e.g. sample-table_OFFLINE). It is safest to return to the Tables page in the Pinot UI and copy and paste the table name from the table list.
   2. batchConfigMapStr: Copy and paste the following JSON string (along with curly braces, etc.): {"inputFormat":"csv","recordReader.prop.delimiter":","}
7. Click Execute. A response appears, containing a Request URL text box.
8. Copy the text returned in the Request URL box, saving it for the process in Uploading CSV Files to Pinot Tables.

Uploading CSV Files to Pinot Tables

Once you have set up your schemas and tables and generated an API call, you can upload your CSV file to Pinot.

To upload a CSV file to a Pinot table:

1. Pick an API request library in the coding framework/language of your choice. (Python and Jupyer notebooks are used for the following example.)
2. Using the requested URL saved from Generating API Calls, post a request to upload the file through a requests library.

Here is a sample of how the process might look in Python:

import requests

file_path = "/path/to/file/sample_records.csv"
prepared_file = {'file': open(file_path, 'rb')}
pinot_env="enter the pinot env name here" #datascius1g
table_name="table name and type" #sample_table_OFFLINE
request_url = f'https://pinot-{pinot_env}.cogilitycloud.com/ingestFromFile?tableNameWithType={table_name}&batchConfigMapStr=%7B%22inputFormat%22%3A%22csv%22%2C%22recordReader.prop.delimiter%22%3A%22%2C%22%7D' 
requests.post(request_url, files=prepared_file)

Verifying CSV Uploads

After you have uploaded a CSV file, it is worth confirming that Pinot has interpreted the file as intended.

To verify a CSV upload:

1. Access Pinot.
2. Go to Query Console.
3. Click the table name.
4. The UI automatically queries the first ten records in the table. Verify that the data is complete, as well as correctly configured and interpreted.

Connecting Pinot to Superset

Once your data has been prepared using Pinot, the next step is to configure Superset to access and read the data.

To connect Pinot to Superset:

1. Ensure you have created the necessary schemas in Pinot.
2. Access Superset.
3. Go to Data > Datasets, then click + Dataset.
4. Under Database, select Pinot.
5. Select the table schema previously created in Step 1.
6. Edit the data set as needed using Superset's selection options.

Creating Charts in Superset

Once Superset has access to your data sources, you can use Superset to create and display visual representations of your data, like charts. You can then share these visualizations with other parties to keep them apprised of your projects.

The process of creating charts follows three main steps:

1. Accessing Superset.
2. Deciding which charts to create.
3. Implementing charts.

Charts can be added to a dashboard that other users can access with your permission.

Accessing Superset

Your Cogynt cluster's Superset dashboard is accessible via any internet-connected web browser.

The URL follows the syntax https://superset-[environment].[domain_name].[domain_suffix], where:

• [environment] is the cluster to access. Customers may have access to different clusters (such as development, staging, or production).
• [domain name] is the customer's domain (for example, cogility).
• [domain suffix] is the customer's domain suffix (for instance, com, io, org, etc.).

tip

Consider the following hypothetical scenario:

• The production cluster is named prod.
• The customer's domain is named example.
• The customer's domain suffix is com.

Given this information, navigating to https://superset-prod.example.com would access the dashboard in the production environment.

Deciding Upon Charts

The data to track and charts to represent it vary from project to project, so there is no one-size-fits-all approach to deciding upon which charts to use.

That said, following guidelines may be useful when thinking through which charts to make:

• Identify key metrics. What are the main things you want to track? What would be useful to know about them? Answering these questions can guide you toward the kinds of charts you need.
• Look for categories that the project can be broken down into. What are the broad strokes of your data? (For example, certain parts of your data set might indicate "revenue" or "risk.") Identifying the main types of data you are working with can guide you in how to chart the data.
• Consider trends. It is often useful to track how values fluctuate over certain windows of time (such as day over day or month over month). Are key metrics trending upward or downward? A trend or line chart is an effective way to document this.

For suggestions on chart types and their uses, see Atlassian's Essential chart types for data visualization and How to choose the right data visualization.

Implementing Charts

Superset makes it possible to construct a variety of different charts from your data.

To add a chart:

1. Access Superset.
2. Create a dashboard to hold the chart.
3. In the upper-right of the dashboard screen, click + Chart.
4. From the Choose a dataset dropdown menu, select the dataset your chart will use.
5. Under Choose a visualization type, click the chip that corresponds to the type of chart you wish to create.
6. Modify the fields of the chart as needed. (Note that the fields may vary depending on the type of chart selected.)

For more information about adding and working with charts, see Exploring Data in Superset in the official Superset documentation.

Creating Dashboards

Dashboards provide a means for organizing, visualizing, exploring, and analyzing data.

For instructions on creating a dashboard, see Creating Your First Dashboard in the official Superset documentation.

Once your dashboard is created, you can add filters to it, and grant other users access to the dashboard.

Adding Filters to Dashboards

Filters allow users to quickly sort and organize dashboard data according to their needs and preferences.

To add a filter to a dashboard:

1. In the top-left of the dashboard, click + Add/Edit Filters.
2. Choose the dataset and column that should be filtered.
3. Set the default value and scope of the filter.
4. Click Save.

To apply a filter:

1. In the dashboard's left panel, click the Filters dropdown menu.
2. Select the desired filter from the list.
3. Click Apply Filters.

Managing Users in Superset

By default, dashboards in Superset are not publicly accessible. (However, they can be configured for public access.) Superset grants dashboard access based on permissions. A set of permissions are defined in a role. Roles are then assigned to users to dictate their access permissions.

Accordingly, managing users in Superset consists of the following tasks:

1. Creating Users
2. Creating Roles

Each of these processes is described in greater detail under the corresponding topic headings.

Creating Users

There are two types of users that can be created in Superset:

• User logins for individualized access.
• Anonymous logins for public access without a login.

Creating User Logins

User logins allow individualized access to Superset using a distinct user name and password.

To create a user login:

1. Access Superset and log in as an admin user.
2. Go to Settings > List Users.
3. At the top of the screen, click +.
4. Enter a First Name for the user.
5. Enter a Last Name for the user.
6. Enter a User Name for the user.
7. Select the Is Active checkbox.
8. Enter the user's Email.
9. Assign a Role if needed. For more information, see Creating Roles.
10. Set a Password for the user to use on their initial login. They can change it later using the Reset My Password function.
11. Click Save.

Users can reset their passwords as follows:

1. Access Superset and log in using the provided password.
2. Go to Settings > Info.
3. Select Reset My Password and follow the instructions onscreen.

note

If a user does not remember their password, an admin user can manually change it for them as follows:

1. Access Superset and log in as an admin user.
2. Go to Settings > List Users.
3. Select Show Record for the appropriate user.
4. Click Reset Password and follow the instructions onscreen.

Creating Anonymous Logins

It may sometimes be helpful to be able to share a dashboard with users without requiring them to log in to Superset. This practice is called an "anonymous login," though no login is required on the user's behalf once it is configured.

Anonymous logins must be configured in the superset_config.py file that runs at build.

To create an anonymous login:

1. Add the following lines to the superset_config.py file:

   AUTH_ROLE_PUBLIC = 'Public'
   PUBLIC_ROLE_LIKE = 'Public'
   DEFAULT_FEATURE_FLAGS = {
       "DASHBOARD_RBAC": True,
       "ENABLE_TEMPLATE_PROCESSING": True,
       "DASHBOARD_NATIVE_FILTERS": True,
       "DASHBOARD_CROSS_FILTERS": True
   }
2. In Superset, grant the permissions in the note following these instructions to the Public role.
3. For each dashboard you wish to make accessible to anonymous users:
   1. Access the Dashboard Properties.
   2. Under Roles, add Public. (Note: The Roles section is only visible if Dashboard_RBAC is set to True in the superset_config.py file, as described in Step 1.)
   3. Set the dashboard's status to Published. (Public users cannot see dashboards in Draft status.)
   4. Generate the shareable URL for the dashboard:
      1. While looking at the dashboard, click Options.
      2. Click Share, then Copy permalink to clipboard.
   5. Distribute the copied URL as necessary.

note

Public role users should be granted the following permissions:

can read on SavedQuery, can write on SavedQuery, can read on CssTemplate, can write on CssTemplate, can read on ReportSchedule, can write on ReportSchedule, can read on Chart, can read on Annotation, can write on Annotation, can read on Dataset, can read on Dashboard, can read on Database, can read on Query, can this form post on ResetMyPasswordView, can this form get on ResetMyPasswordView, can this form post on UserInfoEditView, can this form get on UserInfoEditView, can userinfo on UserDBModelView, resetmypassword on UserDBModelView, can get on OpenApi, can show on SwaggerView, can get on MenuApi, can list on AsyncEventsRestApi, can read on AdvancedDataType, can invalidate on CacheRestApi, can export on Chart, can write on DashboardFilterStateRestApi, can read on DashboardFilterStateRestApi, can write on DashboardPermalinkRestApi, can read on DashboardPermalinkRestApi, can get embedded on Dashboard, can export on Dashboard, can read on EmbeddedDashboard, can write on ExploreFormDataRestApi, can read on ExploreFormDataRestApi, can write on ExplorePermalinkRestApi, can read on ExplorePermalinkRestApi, can add on FilterSets, can list on FilterSets, can delete on FilterSets, can edit on FilterSets, can export on ImportExportRestApi, can import on ImportExportRestApi, can export on SavedQuery, can show on DynamicPlugin, can list on DynamicPlugin, can query form data on Api, can time range on Api, can query on Api, can this form post on CsvToDatabaseView, can this form get on CsvToDatabaseView, can this form post on ExcelToDatabaseView, can this form get on ExcelToDatabaseView, can this form post on ColumnarToDatabaseView, can this form get on ColumnarToDatabaseView, can get on Datasource, can external metadata on Datasource, can external metadata by name on Datasource, can get value on KV, can store on KV, can my queries on SqlLab, can profile on Superset, can sqllab viz on Superset, can results on Superset, can slice json on Superset, can fave dashboards by username on Superset, can queries on Superset, can search queries on Superset, can estimate query cost on Superset, can recent activity on Superset, can extra table metadata on Superset, can sqllab on Superset, can tables on Superset, can created dashboards on Superset, can sqllab history on Superset, can annotation json on Superset, can schemas access for file upload on Superset, can warm up cache on Superset, can stop query on Superset, can testconn on Superset, can favstar on Superset, can sqllab table viz on Superset, can save dash on Superset, can available domains on Superset, can filter on Superset, can csv on Superset, can dashboard on Superset, can validate sql json on Superset, can explore on Superset, can request access on Superset, can fetch datasource metadata on Superset, can import dashboards on Superset, can user slices on Superset, can explore json on Superset, can fave slices on Superset, can log on Superset, can slice on Superset, can dashboard permalink on Superset, can add slices on Superset, can copy dash on Superset, can datasources on Superset, can fave dashboards on Superset, can created slices on Superset, can expanded on TableSchemaView, can post on TableSchemaView, can delete on TableSchemaView, can get on TabStateView, can post on TabStateView, can activate on TabStateView, can migrate query on TabStateView, can delete on TabStateView, can put on TabStateView, can delete query on TabStateView, can post on TagView, can delete on TagView, can tagged objects on TagView, can get on TagView, can suggestions on TagView, can read on SecurityRestApi, menu access on List Users, menu access on List Roles, menu access on Action Log, menu access on Access requests, menu access on Home, menu access on Annotation Layers, menu access on Plugins, menu access on Import Dashboards, menu access on Alerts & Report, menu access on Dashboards, menu access on SQL Editor, menu access on Saved Queries, menu access on Query Search, menu access on Databases, all datasource access on all_datasource_access, all database access on all_database_access, can share dashboard on Superset, can share chart on Superset

Creating Roles

Superset provides a default set of roles, as described in Security Configurations in the official Apache Superset documentation.

Most commonly, you will want to create a read-only role for users that allows:

• Seeing dashboards.
• Applying filters to dashboards.
• Downloading dashboards as images.
• Resetting their password directly from the Superset app.
• Changing their first/last name in Superset.

At the same time, this read-only role forbids:

• Modifying dashboards or editing charts.
• Modifying or accessing databases or data sources.
• Accessing SQL Lab.
• Modifying other users or roles.

In this regard, a best practice is to create one role representing the functional permissions, and a second role representing the data access permissions.

To create the functional role:

1. Access Superset.
2. Go to Settings > List Roles.
3. At the top of the screen, click +.
4. Provide a name for the role. (For example, Normal User or Read-Only User.)
5. Grant the role the following permissions:
   • can read on SavedQuery
   • can read on CssTemplate
   • can read on ReportSchedule
   • can read on Chart
   • can read on Annotation
   • can read on Dataset
   • can read on Log
   • can read on Dashboard
   • can read on Database
   • can read on Query
   • can this form post on ResetPasswordView
   • can this form get on ResetPasswordView
   • can this form post on ResetMyPasswordView
   • can this form get on ResetMyPasswordView
   • can this form post on UserInfoEditView
   • can this form get on UserInfoEditView
   • can userinfo on UserDBModelView
   • resetmypassword on UserDBModelView
   • resetpasswords on UserDBModelView
   • userinfoedit on UserDBModelView
   • can get on MenuApi
   • can read on AdvancedDataType
   • can read on DashboardFilterStateRestApi
   • can time range on API
   • can get on Datasource
   • can fave dashboards by username on Superset
   • can datasources on Superset
   • can dashboard on Superset
   • can fave dashboards on Superset
   • can favstar on Superset
   • can log on Superset
   • can recent activity on Superset
   • can profile on Superset
   • can read on SecurityRestApi
   • menu access on Home
   • menu access on Dashboards
   • can explore JSON on Superset
6. Assign the role as needed to any existing user accounts. (This can also be done later.)
7. Click Save.

To create the data access role:

1. Access Superset.
2. Go to Settings > List Roles.
3. At the top of the screen, click +.
4. Provide a name for the role. (For example, Case Management Dashboard Access.)
5. Grant the relevant permissions:
   1. In the search bar, enter database access, and grant access to pinot-db.
   2. In the search bar, enter datasource access, and grant access to all relevant datasources for the project.
6. Assign the role as needed to any existing user accounts. (This can also be done later.)
7. Click Save.

note

If you would like to add additional administrators to a project, you can assign the admin role to a user to give them access to everything.