Official repo for the ACSOS 2021 paper on how to manage many deep learning models at the edge!
-
Updated
Sep 29, 2021 - Python
#
inference-serving
Here are 7 public repositories matching this topic...
Local inference serving with adaptive batching, benchmark sweeps, and regression gates for batching tradeoffs.
python benchmark machine-learning real-time latency websocket load-testing inference regression-testing performance-testing model-serving quality-gates mlops fastapi inference-serving llm-inference adaptive-batching
-
Updated
May 20, 2026 - Python
Reactive LLM inference.
inference incremental-computation inference-optimization kv-cache inference-serving llm llm-inference kvcache-reuse recurrent-memory
-
Updated
Jul 7, 2026 - Python
cost-aware-inference-cluster: FastAPI inference-serving prototype with Redis queues, worker heartbeats, dynamic batching, autoscaling simulation, and local Docker Compose benchmark evidence.
docker kubernetes redis grafana prometheus autoscaling ml-infrastructure fastapi dynamic-batching inference-serving
-
Updated
Jun 29, 2026 - Python
GSM artifact for exact online GNN inference serving with degree-based scheduling and memory-aware batch division.
scheduling cuda memory-management graph-neural-networks gnn dgl inference-serving online-inference graph-inference
-
Updated
Jun 15, 2026 - Python
GPU-aware LLM serving runtime with async scheduling, dynamic batching, streaming, admission control, metrics, and mock/llama.cpp/vLLM backends.
-
Updated
Jul 21, 2026 - Python
Machine-readable companion to the IEEE OJ-CS survey 'Semantic Caching and Response Reuse for Large Language Model Services: A Survey' (Chukkapalli, Mishra, Naik, 2026): 21-work evidence matrix, systematic-search log, proposed benchmark trace schema, stdlib-only contract validator, and CPU pilot. Code MIT; data CC-BY-4.0.
-
Updated
Jul 11, 2026 - Python
Improve this page
Add a description, image, and links to the inference-serving topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the inference-serving topic, visit your repo's landing page and select "manage topics."