Instructions to use bartowski/Nemotron-Mini-4B-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries
NeMo
How to use bartowski/Nemotron-Mini-4B-Instruct-GGUF with NeMo:
```
# tag did not correspond to a valid NeMo domain.
```

How to use bartowski/Nemotron-Mini-4B-Instruct-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="bartowski/Nemotron-Mini-4B-Instruct-GGUF",
	filename="Nemotron-Mini-4B-Instruct-IQ3_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use bartowski/Nemotron-Mini-4B-Instruct-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M

Use Docker

docker model run hf.co/bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use bartowski/Nemotron-Mini-4B-Instruct-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bartowski/Nemotron-Mini-4B-Instruct-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bartowski/Nemotron-Mini-4B-Instruct-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M

Ollama
How to use bartowski/Nemotron-Mini-4B-Instruct-GGUF with Ollama:
```
ollama run hf.co/bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M
```

Unsloth Studio new

How to use bartowski/Nemotron-Mini-4B-Instruct-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for bartowski/Nemotron-Mini-4B-Instruct-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for bartowski/Nemotron-Mini-4B-Instruct-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for bartowski/Nemotron-Mini-4B-Instruct-GGUF to start chatting

Pi new

How to use bartowski/Nemotron-Mini-4B-Instruct-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use bartowski/Nemotron-Mini-4B-Instruct-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use bartowski/Nemotron-Mini-4B-Instruct-GGUF with Docker Model Runner:
```
docker model run hf.co/bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M
```

Lemonade

How to use bartowski/Nemotron-Mini-4B-Instruct-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull bartowski/Nemotron-Mini-4B-Instruct-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Nemotron-Mini-4B-Instruct-GGUF-Q4_K_M

List all available models

lemonade list

LM Studio Says "unknown model architecture"

by alexcardo - opened Sep 14, 2024

Discussion

alexcardo

Sep 14, 2024

As I know this model has been released weeks or maybe even month ago, but llama cpp doesn't support it. LM Studio thereby can't support it as well.

bartowski

Owner Sep 14, 2024

What do you mean by llama.cpp doesn't support it? It certainly does 🤔 which version of lmstudio are you using? It's possible it needs an update to its binary but definitely the main llamacpp has support

LeroyDyer

Sep 14, 2024

some of these need remote code true ,

I dont know how to quantize a model which uses this requirement as it always fails for me !

But if its quantized generally it works , But Yes also sometimes we can quantize something once and it does not work ! ( i made a merg like this and it was a llama+Mistral merge , even now i cannot delete it it was doi...
but if its quantized it should run on all the platforms , you just need to update...

i find that KoboloCPP Works the best with Strange GGUFS ! << ie you could have created the model with a mad chat template requiring more the 1 input .... ie alpaca , so the model does not work right .. hence you would need to use a code ui manually created so you can load it and prompt it to get the required response !
not all models are FIT to be Hosted on LMstudio or gpt4all or ollama etc , but as a chat model they work ...again messy prompt templates can mess you up also !

so some models that do not work in these dockers .... will work locally with the right UI ~ or Code access ... even Ctransformers ... but also hugging face transformers can now load gguf and extract back to safetensors !

alexcardo

Sep 14, 2024

I don't know how to attach a screenshot, but I'll try to do it in text:

You are on the latest version.
Current Version: 0.2.31

bartowski/Nemotron-Mini-4B-Instruct-GGUF
September 13, 2024 (yesterday)

Nemotron-Mini-4B-Instruct-Q8_0.gguf

{
  "title": "Failed to load model",
  "cause": "llama.cpp error: 'error loading model architecture: unknown model architecture: 'nemotron''",
  "errorData": {
    "n_ctx": 512,
    "n_batch": 512,
    "n_gpu_layers": 33
  },
  "data": {
    "memory": {
      "ram_capacity": "8.00 GB",
      "ram_unused": "3.40 GB"
    },
    "gpu": {
      "gpu_names": [
        "Apple Silicon"
      ],
      "vram_recommended_capacity": "5.33 GB",
      "vram_unused": "2.83 GB"
    },
    "os": {
      "platform": "darwin",
      "version": "15.1"
    },
    "app": {
      "version": "0.2.31",
      "downloadsDir": "/Users/alex/.cache/lm-studio/models"
    },
    "model": {}
  }
}```

alexcardo changed discussion title from LM Studio Says "Unknown model format" to LM Studio Says "unknown model architecture" Sep 14, 2024

bartowski

Owner Sep 14, 2024

0.2.31 isn't latest, you can grab the newer 0.3.2 from https://lmstudio.ai/

Not sure when if ever auto update will come

alexcardo

Sep 14, 2024

Ok, thank you that you pointed me out to the newest version. I was sure that I can update it from inside the LM Studio interface. Now the model is started to work.

P.s.: The interface on this new LM Studio is worse. I can't even find how to set up the model parameters. But it doesn't relate to this quant. The quant works with this version.

Unfortunately, the model is bad compared to its competitors, especially in general knowledge.

As for the rest. I asked the model to explain me many-to-many relationship in Django, and it explained me many-to-one... So, in my case, the model is useless.

LeroyDyer

Sep 19, 2024

Yes it also has a different server setup ! bloddy people made a booby ! ... I had to revert to the previous version because the models was not accpting the endpoint 2

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

LM Studio Says "unknown model architecture"

You are on the latest version.Current Version: 0.2.31

You are on the latest version.
Current Version: 0.2.31