Details around bigquery export

API & integrations
1

Hey,

You provide a paid service for exporting all the raw data to Bigquery, but for a lot of the types of clients we’re working with the price tag is a hard sell in a smaller, less data mature market.

You provide a raw data export API. I’m comfortable writing scripts to extract data from it. But I’ve gotten the impression from you that dumping all data (for larger sites) is generally not… Advised?

What are the details around this? If I have e.g. two million events a month can I run a daily raw-data dump into BQ through the API? Does this incur API quota costs? Is this something that would lead to you reaching out and asking us to purchase a BQ dump license? And is there some sort of cutoff where this would be fine for a smaller site, but not for a larger site with e.g. 10 million events?

Best,
Alex

2

Hi @alexanderf,

We do not reach out to people who try to get too much data from data export API. If you try to get too much data with one request, it will be just a lot of waiting and possible timeouts. So you can use our raw data exports as you’d like, but keep in mind that the more data you try to get at once, the slower it will work and might throw some timeouts. The cutoff is case by case-for clients, so you’d have to test how much you can get at once. We have plans to improve the efficiency of our API, but no ETA or details around that yet.

Best,
Oliwer

1 Like