Repository navigation
BOUNTY: Convert the endpoints to a smart router like litellm does in python #286
Description
this bounty refers to the JS SDK only http://github.com/arakoodev/EdgeChains/tree/ts/JS/edgechains
we are super inspired by Litellm in python (and have a lot of respect for them) - https://github.com/BerriAI/litellm and we want to create a similar infrastructure in Edgechain.js using Typescript. We are hoping libraries like axios makes this easier.
the four features we want are these:
- load balance between multiple deployments. It picks the deployment which is below rate-limit and has the least amount of tokens used. (only openai, google palm and cohere needed). this includes basic reliability features (https://docs.litellm.ai/docs/routing#basic-reliability) like timeouts/retries/etc. you can use
axios.interceptors.response.useto achieve these. - streaming capability
- token usage - https://docs.litellm.ai/docs/completion/token_usage
- logging - https://docs.litellm.ai/docs/observability/callbacks (only sentry and posthog needed)
Once created, you will replace the Endpoint class with this new infrastructure (https://github.com/arakoodev/EdgeChains/blob/ts/JS/edgechains/lib/src/lib/endpoints/OpenAiEndpoint.ts)
project will be deemed complete when:
- you add testcases for all the features above.
- mock the openai, palm and cohere endpoints for testing. (you can take inspiration from here https://github.com/polly3d/mockai)
- you have deleted the current endpoint classes and replaced it with your new classes in all examples and testcases.
NOTE: the core of edgechains is the usage of jsonnet (https://jsonnet.org/) configuration to manage all kinds of config - like prompts, etc. we will prefer that as far as possible, we use jsonnet to configure this routing.
Activity
sandys commentedon Dec 2, 2023
/bounty 200
rajdip-b commentedon Dec 2, 2023
/attempt #286
Options
rajdip-b commentedon Dec 2, 2023
@sandys can i be assigned this issue?
sandys commentedon Dec 2, 2023
@rajdip-b happy to.
but first want to check if you understand what has to be achieved ? any doubts, etc ?
guhitb commentedon Dec 2, 2023
@rajdip-b Any interest in collaborating on this?
rajdip-b commentedon Dec 2, 2023
@sandys As per my understanding, the main challenge is creating the load balancer in the endpoints of
OpenAiEndpoint.ts. I was going through the references you mentioned and have a clarity over what needs to be implemented. I would have to do some further digging about the how's of the implementation, but yes, I feel I can definitely give it a shot!And not to mention, as and when issues come up, I will ping you about them.
sandys commentedon Dec 2, 2023
@rajdip-b there are 4 capabilities needed (as mentioned in my original post). it is important that you look at all four, because the architecture will need to be inclusive.
also, this architecture will REPLACE the current endpoint structure
rajdip-b commentedon Dec 2, 2023
@guhitb, are you familiar with the project structure/implementation details? I would be happy to collaborate!
rajdip-b commentedon Dec 2, 2023
Yeah, of course, I understand that. For me, I feel the challenge would be implementing load balancing, hence mentioned that. As for the rest, I feel they can be done with lesser efforts than the former.
sandys commentedon Dec 2, 2023
very well. assigned to @rajdip-b best wishes. please feel free to drop in to discord if u want to discuss anything quickly. https://discord.gg/Avsbume2
rajdip-b commentedon Dec 2, 2023
Discord is my go-to for discussion! Hopping in right now!
guhitb commentedon Dec 2, 2023
@rajdip-b Not yet, aside from browsing through the repo today. I don't have much experience with open source, but I'm sure I can help with mocks, logging, tests, etc. The load balancing part feels a little daunting though.
rajdip-b commentedon Dec 2, 2023
Hey sure thing! We can pair up!
Drop me a text on discord! _the_heisenberg
93 remaining items
CrowAleksito commentedon Nov 7, 2026
Submitted PR #817 (#817) — all CI checks (js, changes) and CLA Assistant are green (MERGEABLE).
Summary of what PR #817 delivers:
outer.jsonnet): Configures deployments, rate limits, retries, and callbacks via Jsonnet (Router.fromJsonnetFile / Router.fromJsonnet).
outer.test.ts) & Complete Migration: Deleted legacy endpoint classes (openai.ts, gemini.ts, llama.ts) and migrated all JS/edgechains examples, HydeSearchService, TestGenerator, and test suites (10/10 tests passing).
feat(router): add LiteLLM-style SmartRouter load balancer for OpenAI,…
test(router): add unit tests for SmartRouter load balancing & retry (a…
docs(example): add working SmartRouter load-balancing example (arakoo…
flux-axiom commentedon Nov 9, 2026
/attempt #286
Submitted PR implementing the LiteLLM-style
SmartRouteracross OpenAI, Google PaLM, and Cohere deployments (least-tokens-used load balancing below rate limits, response interceptors, timeouts/retries, streaming, token usage accounting, Sentry & PostHog callbacks, and Jsonnet config support) with passing unit tests.KrazyArcher commentedon Nov 10, 2026
/attempt #286
feat: implement LiteLLM-style smart router for endpoints (arakoodev#286)
almagas commentedon Nov 10, 2026
/claim #286
feat(ai): implement LiteLLM-inspired SmartRouter with load balancing,…
feat: implement LiteLLM-style smart router for endpoints (arakoodev#286)
feat: implement LiteLLM-style smart router for endpoints (arakoodev#286)
KrazyArcher commentedon Nov 11, 2026
/claim #286
🚀 Complete Production-Ready Implementation in PR #839
SmartRouterEndpointintegrated directly intoJS/edgechains/arakoodev/src/ai/src/index.ts(alongside existingOpenAI,GeminiAI,LlamaAI, andRetellAImodules) as well asJS/edgechains/lib/src/lib/endpoints/SmartRouterEndpoint.ts.round-robin,simple-shuffle,least-busy(in-flight tracking), andlatency-based(EMA latency tracking).429/5xx/ timeout errors.cooldownUntil) + Model Fallback chains (fallbacksmap).chatCompletion,completion,generateEmbeddings, andstreamCompletion(SSE).node --test JS/edgechains/arakoodev/src/ai/src/tests/smartRouter.test.mjs-> 2/2 Passed ✅CLA Assistant,PR Build / js,PR Build / changes) -> 100% Passed ✅💳 Settlement / Payout Wallets (USDC / Crypto)
4Ch1arer4BF2ndVDq2pSRQdRfDQ8BNp7fg9tc3c5zc3Y0xde5e802f702446806a5f809e8f1f55a60214c991khuxaiphi-stack commentedon Nov 12, 2026
/attempt #286
Implementation Plan
SmartRouterinJS/edgechains/lib/src/lib/endpoints/supporting load-balanced routing across OpenAI, Google PaLM, and Cohere deployments based on rate-limit window and lowest token usage.tokenCounter/costPerToken), and Sentry + PostHog observability callbacks, with comprehensive unit tests.docs(example): add working SmartRouter load-balancing example (arakoo…
ItsMeSoumo commentedon Nov 13, 2026
/attempt #286
Implementation Plan: LiteLLM Dynamic Smart Router for EdgeChains
I would like to claim this bounty to implement the resilient smart router pattern using LiteLLM in the JS/TS SDK.
Technical Architecture & Deliverables:
axios.interceptors.response.use) for automatic retries, exponential backoff, and latency timeout triggers.OpenAiEndpoint.tsto use the new SmartRouter architecture.