Command-R

[SrVill]

Will there ever be a version with Command-R (https://huggingface.co/CohereForAI/c4ai-command-r-v01) support?
llama.cpp supports this model for a long time.

[GrennKren]

Build it from concedo_experimental branch. I tried it before and it works

[tomgm777]

Build it from concedo_experimental branch. I tried it before and it works

I made it using win according to Compiling on Windows, but command-r does not start.
I have overwritten llama.cpp and llama.h with the latest versions, but do I need to overwrite anything else?

[GrennKren]

Build it from concedo_experimental branch. I tried it before and it works

I made it using win according to Compiling on Windows, but command-r does not start. I have overwritten llama.cpp and llama.h with the latest versions, but do I need to overwrite anything else?

I've never tried it, but does compiling on Windows require overwriting some files? I'm on Linux, I usually just run these commands:

$ git clone https://github.com/LostRuins/koboldcpp
$ cd koboldcpp
$ git switch concedo_experimental
$ make LLAMA_CUBLAS=1

[tomgm777]

Build it from concedo_experimental branch. I tried it before and it works

I made it using win according to Compiling on Windows, but command-r does not start. I have overwritten llama.cpp and llama.h with the latest versions, but do I need to overwrite anything else?

I've never tried it, but does compiling on Windows require overwriting some files? I'm on Linux, I usually just run these commands:

$ git clone https://github.com/LostRuins/koboldcpp
$ cd koboldcpp
$ git switch concedo_experimental
$ make LLAMA_CUBLAS=1

thank you
I understand how to use branches.

[EwoutH]

They now also released a larger, 104B parameter model: C4AI Command R+

[CamiloMM]

Support for Command-R comes from commit 12247f4 - PR#6033 so I assume Kobold just needs to include the latest llama.cpp?

Also really cool to see them releasing an 104B model, though I assume it takes even more than 2x24GB to run a quant of that, right? Because of the KV Cache. I actually don't know how much VRAM you need for iMat quants of the 35b.

[Vladonai]

The big question is whether imatrix even work as expected of them...

[henk717]

Maintainer of the repo has been away from home until April 7th. Of course this model will be added upon his return. He will need a bit of time to catch up with upstream.

[CamiloMM]

Ah that's perfectly understandable. Thanks for the heads up Henky.

[LostRuins]

Hello, can you please try the latest release and see if it works for you now?

[SrVill]

Hello, can you please try the latest release and see if it works for you now?

It works great!

[SabinStargem]

I gave v1.61.2 a try, but Command-R+ doesn't boot. The version of Command-R+ I used is the IQ4xs Imat from Dranger, both as an joined file and as separate splits. It might be an issue with the archives, as other models that I used HJSplit or Peazip worked fine. I will report the possibility to Dranger, just in case.

https://huggingface.co/dranger003/c4ai-command-r-plus-iMat.GGUF

Here is the error message.

---

Welcome to KoboldCpp - Version 1.62.1
For command line arguments, please refer to --help

---

Attempting to use CuBLAS library for faster prompt ingestion. A compatible CuBLAS will be required.
Initializing dynamic library: koboldcpp_cublas.dll

Namespace(bantokens=None, benchmark=None, blasbatchsize=512, blasthreads=31, chatcompletionsadapter=None, config=None, contextsize=32768, debugmode=0, forceversion=0, foreground=False, gpulayers=31, highpriority=False, hordeconfig=None, host='', ignoremissing=False, launch=True, lora=None, mmproj=None, model=None, model_param='C:/KoboldCPP/Models/ggml-c4ai-command-r-plus-104b-iq4_xs.gguf', multiuser=1, noavx2=False, noblas=False, nocertify=False, nommap=False, noshift=False, onready='', password=None, port=5001, port_param=5001, preloadstory=None, quiet=False, remotetunnel=False, ropeconfig=[0.0, 10000.0], sdconfig=None, skiplauncher=False, smartcontext=False, ssl=None, tensor_split=None, threads=31, useclblast=None, usecublas=['normal', '0', 'mmq'], usemlock=True, usevulkan=None)

Loading model: C:\KoboldCPP\Models\ggml-c4ai-command-r-plus-104b-iq4_xs.gguf
[Threads: 31, BlasThreads: 31, SmartContext: False, ContextShift: True]

The reported GGUF Arch is: command-r

---

Identified as GGUF model: (ver 6)
Attempting to Load...

Using automatic RoPE scaling. If the model has customized RoPE settings, they will be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 |
llama_model_load: error loading model: invalid split file: C:\KoboldCPP\Models\ggml-c4ai-command-r-plus-104b-iq4_xs.ggufllama_load_model_from_file: failed to load model
Traceback (most recent call last):
File "koboldcpp.py", line 3192, in
File "koboldcpp.py", line 2942, in main
File "koboldcpp.py", line 398, in load_model
OSError: exception: access violation reading 0x0000000000000070
[24972] Failed to execute script 'koboldcpp' due to unhandled exception!

[process exited with code 1 (0x00000001)]

[LostRuins]

How did you merge the GGUF files? Did you use the official split tool?

[LostRuins]

CR+ is too big for me to test personally. But I should be using the same code as upstream to run it.