Skip to main content

Cyware Threat Intelligence eXchange

Google BigQuery

Connector Category: Security Information and Event Management System (SIEM) Tool

About Integration

Google BigQuery is a serverless data warehouse, for querying, inserting, updating data, managing jobs, and datasets, and facilitating large-scale data analytics seamlessly. The Google BigQuery data platform integrates with Intel Exchange (CTIX) to retrieve indicators of compromise (IOCs) that are ingested and analyzed on Intel Exchange. The retrieved IOCs are added to the BigQuery data tables for further processing.

The Google BigQuery internal application in Intel Exchange supports the following actions:

Action Name

Description

Upload Indicators 

This action updates the data tables on the Google BigQuery platform with the IOCs retrieved from Intel Exchange.

Create Google Cloud Service Account

Intel Exchange uses Google Cloud service accounts to make authorized API calls to the Google BigQuery platform. For more information about Google Cloud service accounts, see Service accounts overview.

To create a Google Cloud service account and download the authorization credentials, follow these steps:

Note

The steps to create a service account may differ. We recommend you refer to the Google Cloud Create service accounts documentation.

  1. Go to Google Cloud and sign in as a super administrator.

  2. Go to Navigation Menu > APIs & Services > Credentials. Ensure that the appropriate project is selected in the upper-left corner.

  3. Click +Create Credentials > Service account. 

  4. In Service account details, enter the following details:

    1. In Service account name, enter a name for the service account. For example, Intel Exchange.

    2. In Service account ID, the service account ID is pre-filled based on the account name. You can modify the account ID to enter a unique ID.

    3. (Optional) In Service account description, enter a description for the service account.

    4. Click Create and Continue.

  5. In Grant this service account access to project, enter the following details:

    1. In Select a role, select Basic > Editor.

    2. Click Continue.

  6. Click Done. The service account is created and you can view the account email and name under Service Accounts.

  7. Click the service account and go to the Keys tab.

  8. Click Add Key > Create new key > JSON.

  9. Click Create.

The private key of the service account is created and saved on your system in a .json file. 

Configure Google BigQuery App in Intel Exchange

Configure the Google BigQuery internal application in Intel Exchange to establish seamless connectivity with the Google BigQuery platform.

Before you Start 

  • You must have the View Tool Integrations and Update Tool Integrations permissions in Intel Exchange.

  • You must have the private key of the Google Cloud service account.

Steps 

To configure a Google BigQuery internal application instance in Intel Exchange, follow these steps:

  1. Go to Administration > Integration Management, and select Internal Applications under Tool Integrations.

  2. Select Security Information and Event Management, and then select the Google BigQuery application.

  3. Click Add Instance and enter the following details:

    • Instance Name: Enter a unique instance name. For example, Prod_BigQuery.

    • API ID: Enter the entire content of the private API key of your Google Cloud service account to authorize API calls to the Google BigQuery platform.

  4. Click Save.

The Google BigQuery instance is configured and you can view the available list of actions. You can configure multiple instances of this integration by clicking Manage > Add More.

Enable App Actions

Enable the Upload Indicators action of the Google BigQuery internal application to update data tables in the Google BigQuery platform.

Steps 

To enable the Upload Indicators action, follow these steps:

  1. Go to Administration > Integration Management and select Internal Applications under Tool Integrations.

  2. Select Security Information and Event Management, and then select the Google BigQuery application.

  3. On the upper-right corner, click the vertical ellipsis and click Manage.

  4. Click Manage Actions and select the Upload Indicators action.

  5. Turn on the toggle to enable the action and click Save.

The action is enabled. You can use the action in rules to update data tables in the Google BigQuery platform.

Create a Rule to Upload Indicators

Create a rule to retrieve the indicators from specific sources of Intel Exchange and the dataset and table of the Google BigQuery platform to be updated.

Important

If an indicator is uploaded to a BigQuery data table multiple times then the table will include duplicate rows for the indicator.

To ensure that duplicate indicators are discarded before uploading them to a BigQuery data table, configure a rule that filters out the indicators that exist in the BigQuery data table based on a specific tag.

Before you Start 

  • Ensure that you have created a tag to be used to discard duplicate indicators. For information, see Tag Management.

  • Ensure that you have created at least one dataset with a table on the Google BigQuery platform. The data table must include the following column names to receive indicator details in the expected column.

Column Name

Type

Indicator Parameter

value

String

Indicator value

source

JSON

List of names and IDs of the sources

tags

JSON

List of associated tags

tlp

String

Traffic Light Protocol (TLP)

type

String

Indicator type

score

Integer

CTIX Confidence Score

risk_severity

String

Risk severity of the indicators

first_seen

String

The timestamp when the indicator was first sighted

last_seen

String

The timestamp when the indicator was last sighted

indicator_id

String

ID of the indicator object in Intel Exchange

indicator_url

String

URL of the indicator in Intel Exchange

created_from

Timestamp

The timestamp of when the indicator was created

valid_from

Timestamp

The timestamp from when the indicator is considered valid

ctix_created

String

The timestamp when the indicator was created in Intel Exchange

source_created

String

The timestamp when the indicator was created in the source

Note

  • The column names are case-sensitive.

  • If any of the specified column names are missing in the data table, the indicator upload will fail.

  • In addition to the specified columns, if you add other columns, the columns will not include any indicator data and will be shown as null.

Steps 

To create a rule to deduplicate and upload indicators to a data table on the Google BigQuery platform, follow these steps:

  1. Go to  Main Menu > Actions > Rules.

  2. Click New Rule.

  3. Enter a rule name within 100 characters and click Submit.

  4. In Source, select the sources and collections from which you want to retrieve indicators.

  5. In Condition, enter the following details:

    • Intent Type: To retrieve a list of indicators, select Indicator.

    • Rule Type: To apply the condition based on a specific tag and deduplicate indicators at source, select TAGS.

    • Selector: To apply the condition to the indicators that are not associated with a specific tag, select NOT EQUAL.

    • Value: Select the tag you have created to deduplicate indicators. For example, BigQuery Indicators.

    • Select Object for Actioning: Select this option to perform the action of a rule on the selected object. This option ensures that the action is performed only on the selected object when you define multiple conditions with multiple objects.

  6. In Actions, enter the following details to add the tag to the indicators:

    1. Actions: Select the Update Tag action.

    2. Application: Select the CTIX application.

    3. Account: Select the default account.

    4. Operation: Select Add.

    5. Tags: Select the same tag you have selected in the condition. For example, BigQuery Indicators.

  7. Click +Action and enter the following details to upload the indicators to a Google BigQuery data table:

    1. Actions: Select the Upload Indicators action.

    2. Application: Select the Google BigQuery application.

    3. Account: Select the instance you have configured for the Google BigQuery internal application in Intel Exchange.

    4. Dataset Name: Select a dataset of the Google BigQuery platform.

    5. Table Name: Select a table from the selected dataset to upload indicators.

  8. Set the global conditions from Additional Actions. For more information, see Additional Actions for Rules.

  9. Click Save.

When you run the rule, indicators will be retrieved based on the configured sources and conditions, and the selected data table will be updated in the Google BigQuery platform.