Increasing ask query limit

Rmckeel's picture
Submitted by Rmckeel(270)
Contributor
22 June, 2012 - 10:30

An NREL user who is trying to use the utility rate service was having an issue.  He writes "I noticed that any rates past 10,000 are not accessible via json. For example, this query only returns two entries: http://en.openei.org/services/rest/utility_rates?version=latest&format=json_plain&offset=9998&limit=30&detail=basic".

Twong mentioned "It looks like there's a default total max limit of 10000 results retrieved, even for special query pages in SemanticMediawiki.  $smwgQMaxLimit Maximal number of results ever retrieved, even when using special query pages. Default: 10000"

 

Are there any issues increasing that?  The utility rate web service relies on ask queries currently, so having 10K as a limit is unfortunate with 21K+ utility rates!  A safe limit allowing for growth of the URDB would be somewhere around 40K, but I don't know how much of a strain on the system that could be.  Unfortunately, the offset doesn't seem to allow going past that 10K limit.  I'm guessing with midtiers going to the slave DB, this change would be hardly noticeable.  Any thoughts?

Groups:

Comments

Rmckeel
Rmckeel46 weeks 6 days ago

Query limit increased!

I didn't hear any objections from the peanut gallery, so I just upped the limit to 40K.  This means queries such as http://en.openei.org/services/rest/utility_rates?version=latest&format=json_plain&offset=9998&limit=5&detail=basic will return results 9998 through 10003, while previously it would only return results 9998 through 10000.

Let me know if anyone sees issues with this change.  Thanks!