Setup rate limiting for Open API and TAXII Servers
The CTIX application allows you to integrate with other platforms or applications using RESTful APIs. CTIX also allows you to add clients, organizations, or partner organizations called subscribers to send and receive threat intel packages. These subscribers can poll for threat intel data from your TAXII servers. How do you make sure that these subscribers and third-party applications are not flooding your application with too many requests? How do you make sure that all subscribers and users get a fair distribution of your data, and that one user is not blocking others?
You can accomplish this by defining a Rate Limit on your APIs and TAXII servers.
What is Rate Limiting?
Rate limiting is a methodology used to control the number of requests that can be made in a given time period. You can set a limit for the number of requests that can be made to your servers for every minute, every hour, and every day.
Why do you need Rate Limiting?
Rate limiting protects your application from inadvertent or malicious overuse by limiting how often each user or subscriber can make requests to your server. When there are multiple users and subscribers, it is important to make sure that one user is not blocking others by overcrowding the system with too many requests. Rate limiting also ensures that the APIs and servers run smoothly and efficiently.
The more dire consequences of not having a rate-limiting in place for your APIs are as follows:
Your APIs are vulnerable.
You can fall prey to security incidents such as Denial of Service (DoS) attacks and brute-force attacks on authentication.
Defining Rate Limits for Open API in CTIX
You can define how many requests your API users can make in a defined time period. A tenant can have many API users with API credentials defined for each of them. This is a global setting, and it applies to all your API users who have configured API integration with CTIX.
Sign in to the CTIX application.
From the Administration module, select Configuration.
Select Open API and click Edit.
In the Per minute rate limit box, enter the number of requests that the user can make in a minute.
In the Per hour rate limit box, enter the number of requests that the user can make in an hour.
In the Per day rate limit box, enter the number of requests that the user can make in a day.
Click Save.
Defining Rate Limits for TAXII Servers in CTIX
You can define how frequently your subscribers can make requests to your TAXII servers in a defined time period. This is a global setting, and it applies to all subscribers configured in CTIX.
Sign in to the CTIX application.
From the Administration module, select Configuration.
Select TAXII Server and navigate to Rate limit.
In the Per minute rate limit for each subscriber box, enter the number of requests the subscriber can make in a minute.
In the Per hour rate limit for each subscriber box, enter the number of requests the subscriber can make in an hour.
In the Per day rate limit for each subscriber box, enter the number of requests the subscriber can make in a day.
Click Save.
Rate Limiting Algorithm in CTIX
The rate-limiting algorithm implemented in CTIX is based on the rolling window. Whenever a request is made, the algorithm decides to either serve the request or reject it based on the limit. It maintains a counter to track the limit.
Example 1 - consider the rate limit per day is 100 requests. If a request came in at 09:05 hours (and consider that this is the first request of the day), the window opens from 09:05 hours today till 09:05 hours the next day. It also sets and increments a counter.
Consider that a second request came in at 10:15 hours on the same day. It checks for any open windows and sees that the previously-created window is still open (as the window lasts till 09:05 hours the next day). It increments the counter.
Now, consider that the third request came in at 11:30 hours on the next day. It checks for an open window and sees that there are no open windows (as the last window ended at 09:05 hours). It now creates a new window from 11:30 hours today till 11:30 hours the next day and sets the counter to 1.
If the counter crosses your set limit at any point, it rejects the request. Otherwise, the request is served.
Example 2 - Consider the Rate limit per day as 100 requests. If the rate limit is exceeded and the open window is from 09:05 hours today to 09:05 hours tomorrow, then the user can make the next successful request after 09:05 hours tomorrow.
Note: Changing the value of any limit at any point in time invalidates the existing window of that limit.
What are the Maximum Rate Limits that you can specify?
The following section describes the maximum values that you can set for CTIX Open API and TAXII servers.
Open API
The total number of requests that a tenant can make using multiple Open API credentials at the same time is limited to 1,00,000 requests per day and 30,00,000 requests per month.
Per minute rate limit - 200
Per hour rate limit - 10000
Per day rate limit - 75000
TAXII servers
Per minute rate limit for each subscriber - 150
Per hour rate limit for each subscriber - 3000
Per day rate limit for each subscriber - 50000
Exceeding Rate Limit error
If you exceed your per-hour, per-minute, or per-day limit, you will receive the following error messages.
Open API
Error Code - 429
Error Message - Too Many Requests -- You're requesting too frequently! Slow down!
Response body -
{'detail': 'You have exceeded your per minute API call limit.'}
TAXII 2 and TAXII 2.1
Error Code - 429
Error Message - Too Many Requests -- You're requesting too frequently! Slow down!
Response body -
{"title": "You have exceeded your per minute API call limit."}TAXII 1
Error Message - Bad Request -- Your request is invalid.
Error Code - 400
Response body:
<taxii_11:Status_Message xmlns:taxii="http://taxii.mitre.org/messages/taxii_xml_binding-1" xmlns:taxii_11="http://taxii.mitre.org/messages/taxii_xml_binding-1.1" xmlns:tdq="http://taxii.mitre.org/query/taxii_default_query-1" message_id="5414793272407468978" in_response_to="0" status_type="DENIED"> <taxii_11:Message>You have exceeded your per minute API call limit.</taxii_11:Message> </taxii_11:Status_Message>
You will then need to wait for the window of the currently exceeded rate limit to resume making API calls. To increase the rate limit, contact the Cyware team.