Model Providers

Your AI tools (Hermes, dsh, the %%hermes magic) all talk to an LLM provider — an OpenAI-compatible endpoint that serves a model (here, Socrates). This page covers the providers we give you, how to pick one, how to bring your own, and the restart rules.

The providers we provide¶

• LiteLLM is the metered route: each user gets their own API key with a spending budget, and usage is tracked per key. This is the recommended provider for student work.

• DiveAI is the direct route to the SGLang endpoint; its base URL + key are injected into your pod as environment variables, so it is available with no setup.

The jupyterhub-litellm JupyterHub extension is what manages the LiteLLM side: it generates your personal key, enforces the budget (a rolling window, e.g. a spend limit per 24 h), shows live usage, and injects the key into your notebook.

Getting your LiteLLM key (and the endpoint)¶

Your key and endpoint live in the hub control panel, not in your notebook:

1. In your browser, go to your hub’s LiteLLM page: <hub-url>/hub/litellm-key (e.g. https://dive.cs.cityu.edu.hk/cs1302a_26a/hub/litellm-key).

2. The page shows your API Key (copyable) and your endpoint (…/litellm/v1), plus a live usage bar ($spent / $budget) and your limits.

3. The same key is already injected into your notebook as LITELLM_API_KEY, and the endpoint as LITELLM_API_BASE, so most tools find them automatically.

To confirm what your pod has (keys masked):

Revoking / regenerating your key¶

• Regenerate — click Generate / Regenerate Key on the same /hub/litellm-key page. This immediately invalidates the old key (it is deleted server-side) and issues a fresh one with a full budget. You may regenerate once per budget window (e.g. once per 24 h); the page tells you how long to wait if you try too soon. Use this if you suspect a key has leaked.

• Block (admin) — an instructor can block a user’s key from the admin page <hub-url>/hub/litellm-admin, which rejects all requests from that key instantly.

If you regenerate or a key is revoked, the old key stops working everywhere at once. Update any tool that had the old key stored (see the per-tool config below), then restart whatever is holding the old key in memory.

Setting the provider for Hermes¶

Hermes reads its model/provider from ~/.hermes/config.yaml (on your NFS home). Look at the current value:

The structure is a model: block plus a list of custom_providers: (LiteLLM, DiveAI, …):

model:
  provider: custom        # which provider is active
  default: Socrates       # model id
  base_url: https://…/litellm/v1
  api_key: <your-key>
custom_providers:
- name: LiteLLM
  base_url: https://…/litellm/v1
  api_key: <your-key>
  model: Socrates
- name: DiveAI
  base_url: http://<cluster-ai>/v1
  api_key: <injected>
  model: Socrates

The hermes model command changes the active model/provider non-interactively (equivalent to editing model: in the config):

Use your own custom endpoint for Hermes: add an entry to custom_providers with your base_url + api_key + model, then set model.provider to it (or use hermes model). Hermes auto-detects the context length on first use if the endpoint does not advertise one.

Does a Hermes provider change need a restart?¶

It depends on which Hermes interface:

• JupyterAI chat panel (the “Hermes” persona): the persona runs a hermes acp subprocess. A new chat session picks up the current config, so in most cases a provider change takes effect on a new chat — you do not need to restart the Jupyter server. If an open session keeps using the old model, start a new chat (or reset the persona). A full server restart always works as a clean fallback.

• %%hermes notebook magic: the ACP subprocess is spawned once per kernel (a single persistent hermes acp for the life of the kernel). So after changing the provider, restart the kernel (or run %hermes reset) for the next %%hermes cell to use the new model. This is the case where a restart is genuinely required.

Setting the provider for dsh¶

For dsh, use dsh-proxy (detailed in DeepSeek Harness (dsh)):

dsh-proxy use persists the choice to ~/.dsh/proxy.conf and restarts the in-pod shim so it applies immediately (reload the dsh tab if it looks stale).

Using your own (custom) endpoint¶

Nothing here is locked to our two providers — every tool accepts any OpenAI-compatible endpoint:

• dsh: dsh-proxy use <name> --base https://your-endpoint/v1 --key <key> (auto-detects the token cap).

• Hermes: add a custom_providers: entry + set model.provider (or hermes model).

• %%hermes / JupyterAI: same as Hermes (config-driven), then restart the kernel / start a new chat.

If your endpoint is not OpenAI-compatible, you can put the dsh-openai-shim in front of it to normalize reasoning_effort and max_tokens.

Cheat sheet: what to restart¶

Related: DeepSeek Harness (dsh), Hermes Agent, Agentic Coding.