Datasource¶

Introduction to Datasource¶

The entity, Datasource is a connection between an entity and the database server. A datasource configuration entity points to a particular database server and the connection information is used by the system for establishing a connection to that database. Operations such as fetching the data from the database table or inserting information into the database table etc. are done via the datasource.

The organization-specific datasources that can be accessed by all the users in the organization are created in the Manage section.

User-specific datasources are created in the Settings section of the user. User-specific datasources are visible in the Settings section and it is accessible to the user who created it. The creation and management of both types of datasources are the same.

You can connect to different types of datasources by configuring the same in the platform.

Creating New Datasource Configuration¶

Follow the below steps for creating a new Datasource configuration.

1. Navigate to Management > Configuration Management.

2. Click Datasource in the configuration entity panel.

3. Click +Create New. Create New Datasource pop-up appears.

   
   

4. Click Basic Details. (This button is active by default)

5. Enter the configuration details in the Basic Details panel as explained below.

   Field	Description
   Name*	Enter the name of the Datasource. Character limit: 50. Data type: Alphanumeric and underscore.
   Is Default	Check the checkbox adjacent to Is Default , if you need to set the Datasource as a Default Datasource. If there is any other default Datasource it will be set back as non-default. - You can configure only one datasource as the default Datasource. If you set a Datasource as default, the previously existing default Datasource will change to non-default automatically. Uncheck the checkbox if you do not need the Datasource as default.
   Type	Select the Type of database server as per your need from the drop-down. This list includes your external connectors also. The external connector should be enabled in the External connectors section to access it in th datasource type. Type the name (or first few letters) of the datasource to display the matching three items from which you can choose to select your type of datasource.
   Description	Write a brief description of the datasource. Character limit: 1000 characters Data type: Alphanumeric and symbols.

6. Click Next and enter the configuration details in the Authorization section. The authorization details appear based on the datasource type selected.

   Refer to Types of Platform Datasources and Configurations.

7. Click Create on the bottom right of the page and the Datasource gets created with the details entered.

Types of Platform Datasources and Configurations¶

Cassandra: Apache Cassandra¶

Apache Cassandra is an open-source NoSQL distributed database for scalability and high availability without compromising performance. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data.

• Select Type as Cassandra. The authorization section displays corresponding configuration details.
  
  

DB2: IBM DB2¶

DB2 is a database product from IBM. It is a Relational Database Management System (RDBMS). DB2 is designed to store, analyze, and retrieve the data efficiently. DB2 product is extended with the support of Object-Oriented features and non-relational structures with XML.

• Select Type as DB2. The authorization section displays corresponding configuration details.
  
  

Directory¶

A directory database contains user accounts and security information for the domain.

• Select Type as Directory. The authorization section displays corresponding configuration details.
  
  

Kafka: Apache Kafka¶

Apache Kafka is a distributed data store optimized for ingesting and processing streaming data in real time. Kafka is used for building real-time streaming data pipelines and real-time streaming applications. A data pipeline reliably processes and moves data from one system to another, and a streaming application is an application that consumes streams of data.

• Select Type as Kafka. The authorization section displays corresponding configuration details.
  
  

MS SQL¶

Microsoft SQL Server is a relational database management system. As a database server, it is a software product with the primary function of storing and retrieving data as requested by other software applications—which may run either on the same computer or on another computer across a network (including the Internet).

• Select Type as MS SQL. The authorization section displays corresponding configuration details.
  
  

My SQL¶

MySQL is a relational database management system based on SQL – Structured Query Language. The application is used for a wide range of purposes, including data warehousing, e-commerce, and logging applications. MySQL Database Service is a fully managed database service to deploy cloud-native applications.

• Select Type as My SQL. Authorization section displays corresponding configuration details.
  
  

Neo4j¶

Neo4j facilitates personal data storage and management: it allows you to track where private information is stored and which systems, applications, and users access it. The graph data model helps visualize personal data and allows for data analysis and pattern detection.

• Select Type as Neo4j. Authorization section displays corresponding configuration details.
  
  

Oracle: Oracle DB¶

An Oracle Database (aka Oracle RDBMS) is a collection of data organized by type with relationships being maintained between the different types. The primary purpose of a database is to store and retrieve related information.

• Select Type as Oracle. Authorization section displays corresponding configuration details.
  
  

RDBMS¶

RDBMS in the Platform refers to any RDBMS supported by JDBC/ODBC driver.

• Select Type as RDBMS. The authorization section displays corresponding configuration details.
  

Connector-Based Datasource Configurations¶

Connector-based datasources are configured to utilize the connector-based features through the process flow activities. When you create a connector datasource, a JDBC connection is created. You can do manipulations in the datasource details through the corresponding activities in the process flow.

You can add the connector activities to the process flow activity list and then utilize those activities in the process flows.

• Enter your external connector name in the Type. The authorization section displays corresponding configuration details.
  For each external connector, the authorization details are different.

Viewing and Editing Datasource¶

1. Navigate to Management > Configuration Management > Datasource.

   For viewing user-specific datasources, click burger menu and navigate to Settings.

2. Click the Datasource card for viewing the details of the selected Datasource. The details of the Datasource appear in the Info Actions panel (Edit Datasource).

   
   

3. Edit the Datasource details as needed.

4. Click Save.

Testing Datasource¶

You can test the Datasource created using the Test feature. Test functionality validates the configuration details of the datasource.

1. Navigate to Management > Configuration Management (or Settings for user-specific datasource).

2. Click Datasource. The list of all datasources is displayed.

3. Click the datasource to be tested.

   
   

4. Click Test on the lower-right of the page. A success message is displayed on the successful validation of the datasource.

   
   

5. Click Ok.

   Alternatively, you can follow the below steps to test the Datasource:

6. Navigate to Management > Configuration Management > Datasource.

7. Hover over the Datasource card. Three dots appear on the upper right side of the card.

8. Click the three dots. More Actions appear.

9. Click Test. A success message is displayed on the successful validation of the datasource.

10. Click Ok.

    
    

Fetching Access Tokens¶

For those datasources for which access tokens are applicable (external connectors), the UI displays the Fetch Access Token button on the selection of that datasource. You can fetch the access token of a particular datasource.

1. Navigate to Management > Configuration Management (or Settings for user-specific datasoure).

2. Click Datasource. The list of all datasources is displayed.

3. Click the datasource for which the access token is to be fetched. Fetch Access Token appears on the bottom-right of the screen.

   
   

4. Click Access Token. You will be redirected to the respective application for signing in to that application.

   For example, for MS Teams datasource, you will be redirected to the login.microsoftonline.com.

5. Log in to your application account using your credentials.

   Note

   The application account used for fetching access tokens for the connector-based datasource (e.g., in the case of MS Teams) needs appropriate permission to be granted from the application administrator to access the APP and the APP should have the appropriate permissions to specify the action required to be performed using the connectors.
   Upon successful login to the user account with the required permissions, an access token gets generated and a success message appears.

   
   

6. Click Close.

   Every time you fetch the access token, a fresh access token is generated and a new configuration for the access token is created in the Management > Configuration Management > Access Token with the access token details. This access token configuration is either used by the datasource or you can utilize it as a web service.

Duplicating Datasource¶

When a datasource is duplicated all the properties of the datasource get copied except the “Password”. Hence while duplicating the datasource, the user is prompted to enter the password for the duplicate datasource. Unless the user enters the password value, the datasource cannot be duplicated.

1. Navigate to Management > Configuration Management.

2. Click Datasource. The list of all Datasource is displayed.

3. Hover over any Datasource card. Three dots appear on the upper right side of the card.

4. Click the three dots. More Actions appear.

   
   

5. Click Duplicate. A confirmation pop-up appears with an option to enter the new password.

   
   

6. Enter the password. When you duplicate a datasource, password will not be duplicated, hence it will prompt you to enter password for the duplicated entity.

7. Click Ok to duplicate the Datasource (or you can click Cancel to cancel the duplicate action). A Success message appears on the successful duplication of the Datasource.

   
   

8. Click Ok. A duplicate copy of the Datasource appears on the Datasource page with the same Datasource name suffixed with “_copied”.

   
   

Importing Objects From the Datasource¶

You can import the objects (tables) from the datasource. The imported objects are saved in the background and it will be part of the datasource created. When you import objects, the objects and the attributes is imported as data entity in the Management > Data Catalog > Data Entities section.

• Note that the import feature is available only for the applicable type of datasources.
• Navigate to Management > Configuration Management (or Settings for user-specific datasoure).

• Click Datasource. The list of all Datasource is displayed.

• Select the datasource from which you need to import objects.

  
  

• Click Import on the bottom-right. The schemas available in that datasource appear.

  
  

• Click an object name to view the details (attributes of the table) within that.

  Note

  When you import a connector-based datasource, a data entity is created in the Management > Data Catalog section (these details are non-editable in the Data Catalog section).
  
  

• Click Import. The details get imported as background details for the selected datasource.

  
  

Deleting Datasource¶

You can delete the default Datasource and other Datasources. When you delete a default Datasource, you need to configure another Datasource as the default Datasource. You cannot retrieve the deleted Datasource.

1. Navigate to Management > Configuration Management (or Settings for user-specific datasoure).
2. Click Datasource. The list of all Datasource is displayed.

3. Click the Datasource name card that is to be deleted. The lower-right of the page displays the Delete button.

   
   

4. Click Delete. A Confirmation pop-up for delete appears.

   
   

5. Click Ok for deleting the Datasource.
   Or
   Click Cancel to cancel the action.

   Alternatively, you can follow the below steps to delete the Datasource:

6. Click the Burger menu and navigate to Management > Configuration Management > Datasource

7. Hover over the Datasource card. Three dots appear on the upper right side of the card.

8. Click the three dots. More Actions appear.

9. Click Delete and follow step 5 in the above procedure.

   
   

Utilizing Datasource¶

Datasource in Process Flows¶

You can utilize the datasource (non user-specific datasources) in the processflows by mapping the datasource to the required activities. The activities in the process flows that need datasource configuration are:

• DB
• Insert Data Entity
• External connector-based activities

Hence it is mandatory to configure the required datasource before configuring these activities.

Datasources in Data Flows¶

For all the data processing activities in the data flows, you need a datamodel mapping and datamodel intern needs a datasource configuration. Hence it is mandatory to configure the required datasource if you are working on the data flows.

Datasources in Event Flows¶

You can utilize the datasource in the event flows by mapping the datasource to the required activities. The activities in the process flow that need datasource configuration are:

• Data Lookup
• Event Publisher
• Persister

Hence it is mandatory to configure the required datasource before configuring these activities.

User Specific Datasources¶

You can create user-specific datasource configurations. That is, the datasource can be manipulated only by a specific user. Other users will have only the view of that datasource.

User-specific datasources are configured in the main menu > Settings > Datasource section.