Retries with exponential backoff
-
Hi team, does AI Engine support “Implement retries with exponential backoff” technique ? I think this is super important as this technique embraces the fact that cloud resources might intermittently be unavailable for more than a few seconds for any reason. In our case, OpenAI models have their RPM limits, that once reached the reponse received is bad and it stops for a few short period.
So I consider that for a minimal good UX something like this should be applied to solve high volume concurrent users, or maybe there is something Im not aware of than can manage those limits situations for users. I know there exists an excerption hook that handles AI errors, so the idea from AI Engine is to be able to retry the call for X attempts during Y time to solve it automatically.
Microsoft Azure is sugesting applying this technique.
Thanks in advance.??
- The topic ‘Retries with exponential backoff’ is closed to new replies.