Developer

 View Only
last person joined: 2 days ago 

Expand all | Collapse all

Sum of limit and offset exceeded max value of 10000

This thread has been viewed 12 times
  • 1.  Sum of limit and offset exceeded max value of 10000

    Posted Aug 08, 2024 03:37 AM

    I have a python script that is using aruba networks APIs to extract a list of all of the events in a given time slot (12h at a time). With the API (Events API Link), I am not able to get all of the events in the given time slot, and I am only able to get 1000 events at a time as that is the physical limit for the API call.

    So, my thought process was to add an offset value of 1000 to the API call. So, every time the API is called, I will get 1000 different events in my response, until I eventually start getting less than 1000 events. In which case, I will stop calling the API. However, when trying that out, I kept getting the error:

    {'description': 'Sum of limit and offset exceeded max value 10000', 'error_code': '0002', 'service_name': 'Monitoring'}  

    This error is happening because my offset is increasing by 1000 each time until it eventually reaches 10000 which is the limit. However, even though the offset limit is reached, shouldn't there still be some events left inside aruba that have not yet been pulled?

    So the part in my script that does this is:

    def get_events(access_token, offset, start_time, end_time):
            
        url = f"{baseurl}/monitoring/v2/events?limit=1000&offset={offset}&from_timestamp={start_time}&to_timestamp={end_time}"
        payload = {}
        headers = {
            "Content-Type": "application/json",
            "Authorization": "Bearer " + access_token,
        }
        response = requests.request("GET", url, headers=headers, data=payload)
        json_data = response.json()
        print("DATA")
        print(json_data)
    
        return json_data
    
    
    # Get the current date and time
    current_datetime = datetime.now()
     
    # Subtract one day
    previous_day_datetime = current_datetime - timedelta(days=1)
    date_str = previous_day_datetime.strftime("%Y-%m-%d")
    start_time = datetime_to_epoch(date_str, "12:00:00")
    end_time = datetime_to_epoch(date_str, "23:59:59")
    
    
    offset = 0
    
    while True:
        response = get_events(get_access_token(), offset, start_time, end_time)
        count = response['count']
        if count < 1000:
            break
        offset += 1000
    

    In the above code, if you notice inside the while loop statement. I constantly check if the number of events returned by the API call is less than 1000, in which case I know there are no more events to get.

    what are some possible solutions to get more than 10000 entries? I'm still relatively knew to aruba and aruba APIs, so any help would be greatly appreciated!



  • 2.  RE: Sum of limit and offset exceeded max value of 10000

    EMPLOYEE
    Posted Aug 08, 2024 03:41 AM

    This may be something to discuss with TAC, this API may not be used very heavily as it pushes you through the API limit quickly if you have many events.

    If you have advanced licensing, you may use the streaming API instead.



    ------------------------------
    Herman Robers
    ------------------------
    If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

    In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
    ------------------------------



  • 3.  RE: Sum of limit and offset exceeded max value of 10000

    Posted Sep 02, 2024 04:27 AM

    Hi Yousef.

    You have several options. Depending on your needs. As Herman wrote, you can use streaming API to get events in almost realtime.

    Second option is to use alarms and webhooks to get a specific event when it occur.

    You can also use a filter in REST API call to select only events of interest.

    Or you can shorten the interval to get smaller sample.

    Maybe setup alarms on the events that you are really interested in and use webhooks to collect it.

    If you really want to have all events saved in your database, the only viable option is streaming API.

    Best, Gorazd



    ------------------------------
    Gorazd Kikelj
    MVP Guru 2024
    ------------------------------