Using Multimodal Models with Ollama on Katana
Some Ollama models can work with more than just text.
These are known as multimodal models and can analyse:
- images
- screenshots
- diagrams
- charts
- scientific figures
- photos
Audio workflows are also possible by combining Ollama with speech-to-text tools such as Whisper.
This page explains how to use multimodal models on Katana.
What is a Multimodal Model?
A multimodal model can process multiple types of input instead of only text.
For example:
| Input Type | Example |
|---|---|
| Text | prompts and questions |
| Images | PNG, JPG screenshots or photos |
| Audio (indirectly) | speech converted into text |
Examples of multimodal Ollama models:
llava
llama3.2-vision
bakllava
moondream
1. Start an Interactive GPU Job
Vision models are significantly faster on GPUs.
Start an interactive GPU session:
qsub -I -l select=1:ncpus=4:mem=32gb:ngpus=1
After the job starts you should see something similar to:
z1234567@k001:~
This means you are now on compute node k001.
Choosing a GPU
Katana provides several different GPU models. Depending on the model you choose, performance can vary significantly, especially for larger language models and multimodal models.
View Available GPU Nodes
To see which GPU nodes are currently available:
pstat --gpu
Example output:
Node GPU Model Status
k206 H200 Free
k207 A100 Busy
k208 V100 Free
View Detailed Information About a Node
If you want to inspect a particular node, use:
pbsnodes k206
This displays information such as:
- GPU model
- Number of GPUs
- CPU count
- Memory
- Current jobs running on the node
Request a Specific GPU Model
When starting an interactive job, you can request a specific GPU type.
Example:
qsub -I -l select=1:gpu_model=H200
You can also request additional resources:
qsub -I -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_model=H200
Recommendations for Ollama
| Model | Recommended Hardware |
|---|---|
| phi3 | CPU or any GPU |
| gemma:2b | CPU or any GPU |
| llama3:8b | A100 or better |
| mistral:7b | A100 or better |
| llava | GPU recommended |
| llama3.2-vision | GPU strongly recommended |
| llama3:70b | H100/H200 recommended |
| mixtral | H100/H200 recommended |
Tip
If you are learning Ollama for the first time, start with phi3 or llama3:8b. These models are smaller, download quickly, and are ideal for testing your workflow before moving to larger models.
2. Load the Ollama Module
module load ollama
Verify that Ollama is available:
ollama --version
You may see:
Warning: could not connect to a running Ollama instance
Warning: client version is 0.17.7
This is normal before the Ollama server has been started.
3. Set the Model Storage Location
Large models should be stored in scratch instead of home.
export OLLAMA_MODELS=/srv/scratch/$USER/ollama/models
mkdir -p $OLLAMA_MODELS
4. Start the Ollama Service
ollama serve
Expected output:
Listening on 127.0.0.1:11434
The terminal is now occupied by the Ollama server.
Recommended Method: Open Another Terminal
Open another terminal window (CMD / PowerShell / Terminal).
SSH back into Katana
ssh zID@katana.restech.unsw.edu.au
Example:
ssh z1234567@katana.restech.unsw.edu.au
SSH into the same compute node
If your compute node is:
z1234567@k001
then run:
ssh k001
Now both terminals are connected to the same job.
5. Pull a Vision Model
A good starting model is llava.
ollama pull llava
Other vision-capable models:
llama3.2-vision
bakllava
moondream
6. Run the Vision Model
Start the model:
ollama run llava
You can now provide an image path.
Example:
>>> /srv/scratch/z1234567/images/test.png
>>> Describe this image
Or directly from the command line:
ollama run llava "Describe this image: /srv/scratch/$USER/images/test.png"
Example Use Cases on Katana
Vision models are useful for:
- microscopy images
- scientific figures
- chart interpretation
- screenshot analysis
- OCR-style tasks
- diagram explanation
- research workflows
7. Using Audio with Ollama
Ollama itself does not directly process audio files.
The standard workflow is:
Audio File
↓
Speech-to-Text Model
↓
Text Transcript
↓
Ollama Model
Most users use Whisper for transcription.
8. Installing Whisper
Create and activate a Python virtual environment:
module load python
python -m venv whisper_env
source whisper_env/bin/activate
Install faster-whisper:
pip install faster-whisper
9. Example Audio Transcription
Create a Python script called transcribe.py:
from faster_whisper import WhisperModel
model = WhisperModel("base")
segments, info = model.transcribe("audio.mp3")
for segment in segments:
print(segment.text)
Run it:
python transcribe.py
This will produce a text transcript from the audio file.
10. Using the Transcript with Ollama
Once the transcript has been generated:
ollama run phi3
Example prompt:
Summarise the following meeting transcript:
[paste transcript]
GPU vs CPU Recommendations
CPU-Friendly Models
phi3
gemma:2b
moondream
Recommended GPU Models
llava
llama3
mistral
llama3.2-vision
Large Models
llama3:70b
mixtral
Large models may require very large GPU memory allocations.
Useful Commands
List downloaded models:
ollama list
Show running models:
ollama ps
Remove a model:
ollama rm llava
Stop Ollama:
pkill ollama
Check GPU usage:
nvidia-smi
Storage Recommendations
Image and audio datasets can become very large.
Recommended storage location:
/srv/scratch/$USER/
rather than:
/home/$USER/
Ending the Session
When finished:
pkill ollama
exit
This releases the compute resources.
Minimal Working Workflow
Terminal 1:
qsub -I -l select=1:ncpus=4:mem=32gb:ngpus=1
module load ollama
export OLLAMA_MODELS=/srv/scratch/$USER/ollama/models
ollama serve
Terminal 2:
ssh zID@katana.restech.unsw.edu.au
ssh k001
module load ollama
export OLLAMA_MODELS=/srv/scratch/$USER/ollama/models
ollama pull llava
ollama run llava