Scaling with tons of data – Performance

Hey Scott, we use Elasticsearch, so it’s pretty scalable. Right now we have over 30 million records and it’s humming right a long.

That being said, we do have a fair usage asterisk that for sites that have consistently over 100k records per month they should probably use a custom enterprise setup. Especially for a private multisite network, because there are cool things you can do with a custom setup that you can’t do with our hosted platform, like view all activity network-wide.

I have a meeting with your friend Taylor Lovett about this tomorrow, actually ??

Sorry I didn’t answer your performance question very specifically.

Here were our average API response times this week, and like I said our cluster has 30m+ records:

Create a new record – 21ms
Search existing records – 25ms

I’ve got a meeting with Taylor about something else tomorrow myself ??

Thanks for the quick answer. Do you or are you planning on offering the custom setups or integrating with a different Elasticsearch instance?

Both. We can host/manage a private Elasticsearch cluster or facilitate for clients to use their own on-premise cluster that they would manage. All of this comes in the form of a separately licensed add-on plugin we have that interfaces with the main Stream plugin.

Gotcha, that answers my questions then ??

Thx!