---
jupytext:
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.18.1
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

---
title: Model Providers
abstract: |
    How the course LLM providers work and how to point your AI tools at them — the DiveAI and LiteLLM endpoints, where to get (and revoke) your personal LiteLLM key from the hub control panel, how to set providers for Hermes and dsh, how to use your own custom endpoint, and which restarts a change requires.
---

+++

Your AI tools (Hermes, dsh, the `%%hermes` magic) all talk to an **LLM provider** — an OpenAI-compatible endpoint that serves a model (here, **Socrates**). This page covers the providers we give you, how to pick one, how to bring your own, and the restart rules.

+++

## The providers we provide

+++

| Provider | Endpoint | How you get a key | Backed by |
|---|---|---|---|
| **LiteLLM** | `<hub>/litellm/v1` | **Your personal key** from the hub control panel | [LiteLLM proxy](https://docs.litellm.ai/) — metered, budgeted, per-user |
| **DiveAI** | (in-cluster `…/ai/v1`) | Injected into your pod automatically | The SGLang "Socrates" deployment |

- **LiteLLM** is the *metered* route: each user gets their own API key with a spending budget, and usage is tracked per key. This is the recommended provider for student work.
- **DiveAI** is the *direct* route to the SGLang endpoint; its base URL + key are injected into your pod as environment variables, so it is available with no setup.

The `jupyterhub-litellm` JupyterHub extension is what manages the LiteLLM side: it generates your personal key, enforces the budget (a rolling window, e.g. a spend limit per 24 h), shows live usage, and injects the key into your notebook.

+++

## Getting your LiteLLM key (and the endpoint)

+++

Your key and endpoint live in the **hub control panel**, not in your notebook:

1. In your browser, go to your hub's **LiteLLM** page: **`<hub-url>/hub/litellm-key`** (e.g. `https://dive.cs.cityu.edu.hk/cs1302a_26a/hub/litellm-key`).
2. The page shows your **API Key** (copyable) and your **endpoint** (`…/litellm/v1`), plus a live **usage bar** (`$spent / $budget`) and your limits.
3. The same key is **already injected** into your notebook as `LITELLM_API_KEY`, and the endpoint as `LITELLM_API_BASE`, so most tools find them automatically.

To confirm what your pod has (keys masked):

```{code-cell} ipython3
:editable: true

!env | grep -iE 'DIVEAI|LITELLM' | sed -E 's/(KEY=).*/\1<redacted>/Ig'
```

### Revoking / regenerating your key

+++

- **Regenerate** — click **Generate / Regenerate Key** on the same `/hub/litellm-key` page. This **immediately invalidates the old key** (it is deleted server-side) and issues a fresh one with a full budget. You may regenerate **once per budget window** (e.g. once per 24 h); the page tells you how long to wait if you try too soon. Use this if you suspect a key has leaked.
- **Block** (admin) — an instructor can block a user's key from the admin page **`<hub-url>/hub/litellm-admin`**, which rejects all requests from that key instantly.

> **If you regenerate or a key is revoked, the old key stops working everywhere at once.** Update any tool that had the old key stored (see the per-tool config below), then restart whatever is holding the old key in memory.

+++

## Setting the provider for Hermes

+++

Hermes reads its model/provider from `~/.hermes/config.yaml` (on your NFS home). Look at the current value:

```{code-cell} ipython3
:editable: true

# Show the model block (key redacted)
!sed -E 's/(api_key|key|token)[\"'"'"']?\s*[:=].*/\1: <redacted>/Ig' ~/.hermes/config.yaml | sed -n '1,12p'
```

The structure is a `model:` block plus a list of `custom_providers:` (LiteLLM, DiveAI, …):

```yaml
model:
  provider: custom        # which provider is active
  default: Socrates       # model id
  base_url: https://…/litellm/v1
  api_key: <your-key>
custom_providers:
- name: LiteLLM
  base_url: https://…/litellm/v1
  api_key: <your-key>
  model: Socrates
- name: DiveAI
  base_url: http://<cluster-ai>/v1
  api_key: <injected>
  model: Socrates
```

The **`hermes model`** command changes the active model/provider non-interactively (equivalent to editing `model:` in the config):

```{code-cell} ipython3
:editable: true

# Show available models / current selection
!hermes model
# Switch the active model (and, where applicable, its provider/base_url)
# !hermes model Socrates
```

**Use your own custom endpoint for Hermes:** add an entry to `custom_providers` with your `base_url` + `api_key` + `model`, then set `model.provider` to it (or use `hermes model`). Hermes auto-detects the context length on first use if the endpoint does not advertise one.

### Does a Hermes provider change need a restart?

+++

It depends on *which* Hermes interface:

- **JupyterAI chat panel (the "Hermes" persona):** the persona runs a `hermes acp` subprocess. A new chat session picks up the current config, so in most cases a **provider change takes effect on a new chat** — you do **not** need to restart the Jupyter server. If an open session keeps using the old model, **start a new chat** (or reset the persona). A full server restart always works as a clean fallback.
- **`%%hermes` notebook magic:** the ACP subprocess is spawned **once per kernel** (a single persistent `hermes acp` for the life of the kernel). So after changing the provider, **restart the kernel** (or run **`%hermes reset`**) for the next `%%hermes` cell to use the new model. This is the case where a restart is genuinely required.

+++

## Setting the provider for dsh

+++

For dsh, use **`dsh-proxy`** (detailed in [DeepSeek Harness (dsh)](DSH_Harness.md)):

```{code-cell} ipython3
:editable: true

!dsh-proxy list
!dsh-proxy use litellm      # or: dsh-proxy use diveai
```

`dsh-proxy use` persists the choice to `~/.dsh/proxy.conf` and restarts the in-pod shim so it applies immediately (reload the dsh tab if it looks stale).

+++

## Using your own (custom) endpoint

+++

Nothing here is locked to our two providers — every tool accepts any **OpenAI-compatible** endpoint:

- **dsh:** `dsh-proxy use <name> --base https://your-endpoint/v1 --key <key>` (auto-detects the token cap).
- **Hermes:** add a `custom_providers:` entry + set `model.provider` (or `hermes model`).
- **`%%hermes` / JupyterAI:** same as Hermes (config-driven), then restart the kernel / start a new chat.

If your endpoint is not OpenAI-compatible, you can put the [`dsh-openai-shim`](https://github.com/dive4dec/dsh-openai-shim) in front of it to normalize `reasoning_effort` and `max_tokens`.

+++

## Cheat sheet: what to restart

+++

| You changed… | What to restart |
|---|---|
| dsh provider (`dsh-proxy use`) | Nothing — shim restarts itself; reload the dsh tab if stale |
| Hermes model via `hermes model` (JupyterAI persona) | New chat (or reset persona); server restart as fallback |
| Hermes model via `hermes model` (`%%hermes` magic) | **Restart the kernel** (or `%hermes reset`) |
| Your LiteLLM key (regenerated/revoked) | Any tool holding the old key in memory — reload/restart that tool |
| Installed a new persona/magic package | Restart the Jupyter **server** |

+++

Related: [DeepSeek Harness (dsh)](DSH_Harness.md), [Hermes Agent](Hermes.md), [Agentic Coding](Agentic_Coding.md).