Skip to article frontmatterSkip to article content

Integration with Jupyter Notebook

You can play with Large language models (LLMs) in various ways. LLMs have been integrated into the Jupyter server using jupyter-ai. To be able to chat with an LLM inside a notebook, you need to run the line magic %load_ext jupyter_ai:

# initialization
import os

if not os.getenv(
    "NBGRADER_EXECUTION"
):  # Skip the code or auto-grading may take too long to complete.
    %load_ext jupyter_ai

As an example, run the following cell magic to ask an LLM what an LLM is.

%%ai dive:chat
Explain what is LLM in one line.

Let’s try different output formats:

%%ai dive:chat -f math
What is the Pythagoras theorem?
%%ai dive:chat -f html
What is the Pythagoras theorem?
%%ai dive:chat -f html
Illustrate the Pythagoras theorem using a diagram in SVG.
%%ai dive:chat -f text
Can I trust you?
Are there different LLMs?

dive:chat is just one of many large language models you may choose. To list all supported chat models, you can run

%ai list

For more information on selecting and customizing a model, run

%ai --help

There is also a Jupyternaut interface to chat with an LLM separately from the notebook.

  1. Click the chat icon on the left menu bar. A chat panel will open.
  2. Enter some messages to see a response from the Azure OpenAI chat model.
Trouble-shooting

If the above fail, reconfigure the model as follows:

  1. Click the gear icon at the top right-hand corner of the chat panel.
  2. Select the Completion model as DIVE :: chat.
  3. Click Save Changes button.
  4. Click the back arrow at the top to go back to the chat panel.
  5. Enter some messages to see a response from the chat model.

Azure OpenAI

To play with a bigger LLM model such as GPT-4o, registered students can subscribe to Azure OpenAI as follows:

  1. Access the APIM portal and click Active Directory - Login to login with your CityU active directory credentials.
  2. From the top menu, navigate to the Products page and click the link to cs-dive
  3. Enter any name such as aitutor in the textbox and click subscribe. You will be brought to your profile page where you can show/regenerate your primary/secondary API keys.
  4. Copy one of the API keys such as the Primary key for the subsequent steps.
Why API keys?

API keys are used to restrict access to OpenAI large language models (LLMs) to ensure that only authorized users can utilize the service and its resources. By issuing unique API keys, the service providers can track and control usage, enforce rate limits, and prevent unauthorized access, which is crucial for maintaining security and managing resource allocation. Over time, an API key can become compromised or leaked, potentially leading to misuse, such as exceeding the allocated quota limit or incurring unexpected costs. Regularly regenerating the API key helps mitigate these risks by invalidating the old key and issuing a new one, thus effectively cutting off any unauthorized access that might have occurred due to the key’s exposure. This practice ensures continued security and optimal usage of the LLM service by authorized users. To regenerate your keys:

  1. Access the Profile page on the APIM portal.
  2. Click the Regenerate link next to the respective key to regenerates the key.

Next, select and configure the Azure OpenAI model as follows:

  1. Click the chat icon on the left menu bar. A chat panel will open.
  2. Click the gear icon on the chat panel to set up the provider.
  3. Select the Completion model as DIVE Azure :: gpt4o.
  4. Enter an API key to the AZURE_OPEN_API_KEY field. Other fields can be left empty.
  5. Save the API Key and click the Save Changes button.
  6. Click the back arrow at the top to go back to the chat window.
  7. Enter some messages to see a response from the Azure OpenAI chat model.

If you want to reset the configuration in case of error, remove the configuration folder using the shell capture below:

if input('Reset Jupyter AI configuration? [y/N]').lower() == 'y':
    !rm -rf ~/.local/share/jupyter/jupyter_ai

After deleting the folder, simply restart the Jupyter server to recreate it.

To use the model in a cell magic, run the following to record the API as an environment variable:

from dive_azure_openai import set_api_key

set_api_key()
%%ai dive-azure:gpt4o -f math
Give an elegant proof of the Pythagoras theorem in LaTeX.

To use the model in a python program, create an AzureOpenAI client, which will automatically takes the the subscription information from the environment variables:[1]

from openai import AzureOpenAI

deployment_name = "gpt4o"
client = AzureOpenAI()

Create a chat completion to receive a reply to a query:

response = client.chat.completions.create(
    model=deployment_name,
    messages=[
        {
            "role": "system",
            "content": "You are a helpful AI tutor for introductory-level programming who give hints but not answers.",
        },
        {"role": "user", "content": "What is Python?"},
        {"role": "assistant", "content": "Human use it to talk to the computer."},
        {"role": "user", "content": "What is REST API?"},
    ],
)

print(response.choices[0].message.content)
How does Jupyter access Azure OpenAI?

The communication is through REST API. Accessing Azure OpenAI via REST API is like sending a letter (API request) to a highly knowledgeable friend (Azure OpenAI model) and getting a well-thought-out response back (API response) through the mail service (HTTP). You may play with the API on the APIs page on APIM portal as follows:

  1. Navigate to the APIs page and click the link to cs-eastus.
  2. On the left panel, click GPT-4o :: chat-completion-API to show the corresponding REST API endpoint for chat completion:
    https://apim-aoai-eas-dev02.azure-api.net/cs-eastus/openai/deployments/gpt4o/chat/completions
  3. Click Try it to open the right panel.
  4. Click + Add parameter and enter the REST API version with
    • name:
      api-version
    • value:
      2024-02-01
  5. In the textbox under Body, enter the message body in a format expected by the API such as:[2]
    {
    "messages": [
        {"role": "system", "content": "You are a helpful AI tutor for introductory-level programming who give hints but not answers."},
        {"role": "user", "content": "What is Python?"},
        {"role": "assistant", "content": "Human use it to talk to the computer."},
        {"role": "user", "content": "What is REST API?"}
    ]
}

local LLM

We can run a generative model locally using Ollama. To do so, start the ollama service as follows.

  1. In JupyterLab, navigate to the File menu.
  2. Select New from the drop-down menu and choose Terminal.
  3. The terminal window will appear. You can use this terminal to run a shell command. Enter the following command into the terminal prompt and hit enter.
    ollama serve
  4. To terminate Ollama, simply type Ctrl + C, the same way you would terminate any shell command by Keyboard Interrupt.

After Ollama is running, run the following cell to chat with a compact LLM model phi3.[3]

import os

os.getenv("OLLAMA_MODELS", "").split(",")
%%ai dive-ollama:phi3 -f text
When was your training cutoff date and how were you trained?

To use Ollama with Jupyternaut:

  1. Click the chat icon on the left menu bar. A chat panel will open.
  2. Click the gear icon on the chat panel to set up the provider.
  3. Select the Completion model as DIVE Ollama :: ... with ... replaced by your desired model such as phi3.
  4. Click the Save Changes button.
  5. Click the back arrow at the top to go back to the chat panel.
  6. Enter some messages to see a response from the Azure OpenAI chat model.

The following models would be too slow to run without GPU. If you are using the Default server option, consider restarting your server with a GPU server option.[4]

%%ai dive-ollama:codellama -f text
When was your training cutoff date and how were you trained?
%%ai dive-ollama:mistral -f text
When was your training cutoff date and how were you trained?
%%ai dive-ollama:dolphin-mistral -f text
When was your training cutoff date and how were you trained?

Different models have different sizes and may be good at different things. The following executes the a shell command to list other models that can be served by ollama.

!ollama list

The models reside in the directory specified by the environment variable OLLAMA_MODELS:[5]

%env OLLAMA_MODELS

Integration with VSCode

Github Copilot

A popular AI-assisted programming tool is the Github Copilot for Visual Studio Code (VSCode).

To get started, open the setup guide:

  1. Follow Step 1 last bullet point to get free access as a verified student.
  2. Instead of Step 2, you can access VSCode through the JupyterHub:
    1. Click File->New launcher->VS Code to open VSCode.
    2. Click the profile icon on the left menu and select Sign in with Github to use Github Copilot.
    3. After logging into Github, click the copilot icon at the top of VSCode to start chatting.
  3. Step 3 is for advanced user who would like to use Github Copilot in the command line interface (CLI), which is available in both the JupyterLab and VSCode. You may directly jump to the step following the prerequisites.
How to chat with Copilot effectively?

See the prompt engineering guide for some suggestions.

You may also use a model served by Ollama as a substitute for Copilot. See the Continue extension guide.

Glossary

line magic
A command in Jupyter notebook that starts with % such as %load_ext is called a line magic. It can be used to performs tasks to change the notebook behavior, such as loading the %%ai cell magic.
cell magic
A command in Jupyter notebook that starts with %% such as %%ai is called a cell magic. It can be used to change the beaviour of the cell, such as sending the text as a prompt to an LLM.
shell capture
A command in Jupyter notebook that starts with ! or !! is called a shell capture. It can be used to execute a shell/terminal command.
REST API
REST API stands for Representational State Transfer Application Programming Interface. It is a set of rules for building and interacting with web services that use standard HTTP methods to enable communication between client and server.
Footnotes

  1. In addition to AZURE_OPENAI_API_KEY, it also take the environment variables AZURE_OPENAI_ENDPOINT and OPENAI_API_VERSION.

    ↩

  2. See other examples here for chat completion.

    ↩

  3. The first run will take some time for loading the model into memory.

    ↩

  4. Switching models require additional time to reload a new model into the memory.

    ↩

  5. To download or run a new ollama model, you need to set the directory to ~/.ollama or any directory you have write access to. To use the model in JupyterNaut, select the Completion model as Ollama :: * and specify the model id.

    ↩
Lab 0
Course Materials
Lecture 1
Introduction to Computer Programming