Add Command-R Model - #6033
Conversation
|
I can successfully convert the model to GGUF format with these changes. |
|
You are so cool 👍 |
|
It seems that cohere use a different way of rotate_half in their codebase, not like other llama-based ones. Would this be okay with llama.cpp? |
|
I noticed some potentially useful comments about the model here: Lightning-AI/litgpt#1089 |
I think they've missed the rotate_half part - which is different while my fine-tuning test, not sure if it is significant to inference. |
|
If I'm reading the code correctly in modeling, this is the same RoPE as usual: So I don't think any changes are needed. Just use |
|
Any ideas on where to add logit_scale? Comparing the llama and the cohere models: llama_cohere_diff.txt @@ -1212,6 +1161,7 @@
logits = torch.cat(logits, dim=-1)
else:
logits = self.lm_head(hidden_states)
+ logits = logits * self.logit_scale
logits = logits.float() |
|
Probably this would work: diff --git a/llama.cpp b/llama.cpp
index 38e7036a..a911cdff 100644
--- a/llama.cpp
+++ b/llama.cpp
@@ -5857,6 +5857,12 @@ struct llm_build_context {
// lm_head
cur = ggml_mul_mat(ctx0, model.output, cur);
+
+ if (model.logits_scale != 1.0f) {
+ cur = ggml_scale(ctx0, cur, model.logits_scale);
+ }
+
cb(cur, "result_output", -1);
ggml_build_forward_expand(gf, cur); |
|
Another change is in the transformer layer. graph TD;
Input-->InputLayerNorm;
InputLayerNorm-->Attention;
Attention-->Plus;
Input-->Plus;
Plus-->PostAttnLayerNorm;
PostAttnLayerNorm-->MLP;
MLP-->Plus2;
Plus-->Plus2
Plus2-->Output;
In cohere transformer layers there is no post-attention layernorm: graph TD;
Input-->InputLayerNorm;
Input-->Plus;
InputLayerNorm-->Attention;
Attention-->Plus;
InputLayerNorm-->MLP;
MLP-->Plus
Plus-->Output;
In the python code comparing cohere to llama model: def forward(
self,
@@ -737,7 +691,7 @@
hidden_states = self.input_layernorm(hidden_states)
# Self Attention
- hidden_states, self_attn_weights, present_key_value = self.self_attn(
+ hidden_states_attention, self_attn_weights, present_key_value = self.self_attn(
hidden_states=hidden_states,
attention_mask=attention_mask,
position_ids=position_ids,
@@ -747,16 +701,12 @@
cache_position=cache_position,
**kwargs,
)
- hidden_states = residual + hidden_states
# Fully Connected
- residual = hidden_states
- hidden_states = self.post_attention_layernorm(hidden_states)
+ hidden_states_mlp = self.mlp(hidden_states)
- hidden_states = self.mlp(hidden_states)
# Add everything together
- hidden_states = residual + hidden_states
+ hidden_states = residual + hidden_states_attention + hidden_states_mlp
outputs = (hidden_states,) |
|
Okay I made the changes mentioned above. Thanks for the help Georgi. |
|
Howdy. I was working on this same thing to get this model working on llama.cpp; but my results were incoherent when I got it to run for the first time, whereas when I tested the code in this branch in this PR, I got coherent text, so this is definitely more correct than my hack. But I also noticed Q8_0 quant does not work at all. Other than that, I can confirm that the code so far, as written by @acanis produces coherent text that seems indistinguishable from the HF version (although I didn't test exactly on logit probability level at this time) so I think in terms of getting the model correctly computed mathematically it cannot be very wrong. I can come back to this tomorrow or later this week if no one else chimes in to confirm correctness. Edit: quick edit: I noticed @acanis pushed new code about 1 hour before I wrote this comment and right after I wrote this comment. I was testing the commits before those happened. |
|
Thanks for testing Noeda. Any help is appreciated. I just tested llama.cpp on the F16 model (without quantization) using the suggested prompt: Prompt output is: This matches pretty closely to the output from the 8-bit reference quantized python model: Full log (CPU) is below: Another log this time running with cuBLAS (on the same machine with an A100 GPU) with low temperature: I also noticed that the quantize failed for Q8_0 with this error (didn't have time to investigate): |
|
@acanis Thanks for your efforts! I'll pull your code tomorrow and do some testing, and compare with the HF implementation and see if there's any inaccuracies. If there's coherent text, then I don't expect major divergences. And maybe also I will try to check why am I having trouble with Q8. |
This is correct ✅ |
|
The commits from chatllm.cpp may help. |
|
lgtm, successfully create a quant & run it :) |
./quantize for Q8_0 doesn't work. |
|
@acanis How to support 128K context for this model by llama.cpp ? Any idea? |
oh, you're right.. that's weird |
|
@sweetcard I was planning to investigate that next after fixing the quantization. The current context is 8192 tokens. |
Thank you for your cool work👍 |
If the model training context is 128k, make sure to set the context length in the GGUF meta data to 128k. The loading log should report 128k:
With |
The discussion from the authors is about the context size: huggingface.co/CohereForAI/c4ai-command-r-v01/discussions/12. I’m not very clear about the real context size: 8k or 128k? |
Then the implementation looks correct :) |
ggerganov
left a comment
ggerganov
left a comment
There was a problem hiding this comment.
Thank you all for the implementation and the detailed analysis - much appreciated!
|
@acanis If I could, I'd tip you for quickly getting this to work :) great speed. I'm impressed. @saurabhdash I still have questions about tokenization, but later today or next week I'll open that discussion on HF instead. If anything needs to be done for Thanks everyone! 👏 👏 |
|
Thanks so much for the help everyone! I'm glad we were able to get this support merged with correct functionality. @Noeda great work on the detailed analysis between llama.cpp and the reference, and for fixing the quantization bug, that must have been hard to track down! I'm also curious about the tokenization, I will checkout your HF discussion This was a great opportunity for me to learn the llama.cpp codebase. |
We should definitely do better with the defaults. The problem is that not everybody has enough VRAM to offload the entire model, and currently we don't have a way to detect how many layers can be offloaded automatically. |
|
Thanks so much for the patch and merging it so quickly after the release of Command-R. Not sure if this is the correct place to comment on a possible issue. I seem to have an problem converting the original model. I have tried clean builds of latest master and acanis repo and get the same error. I had a quick look at the convert-hf-to-gguf source and tried hacking around a bit, but encountered other issues. I'm reasonably sure this is repeatable, but also struggling to understand why non one else is reporting it. Presumably I can work around the problem by checking out the converted models, but I thought id mention it in case it's an actual issue.. python3 convert-hf-to-gguf.py --outtype f16 ./c4ai-command-r-v01 |
|
For those who are interested: I'm following up my tokenization question over here: https://huggingface.co/CohereForAI/c4ai-command-r-v01/discussions/27 |
@chrismrutherford you need to update your HF repo to get the latest version of the config.json: The PR to add model_max_length with the true 128k context length was only merged yesterday. |
|
Thank you very much for your excellent work. |
Information about the Command-R 35B model (128k context) can be found at: https://huggingface.co/CohereForAI/c4ai-command-r-v01 Based on the llama2 model with a few changes: 1) New hyper parameter to scale output logits (logit_scale) 2) Uses LayerNorm instead of RMSNorm 3) Transfomer layers have a single shared LayerNorm that feeds into both the self-attention and FFN layers in parallel. There is no post-attention LayerNorm. 4) No support for Rotary Position Embeddings (RoPE) scaling 5) No biases used Find GGUF files here: https://huggingface.co/andrewcanis/c4ai-command-r-v01-GGUF To convert model to GGUF format yourself: 1) Download Command-R Hugging Face safetensors: git lfs install git clone https://huggingface.co/CohereForAI/c4ai-command-r-v01 2) Run: python3 convert-hf-to-gguf.py --outtype f16 ./c4ai-command-r-v01
|
They now released a larger, 104B parameter model: C4AI Command R+ |
|
The new model has a Some of the parameters are different. @saurabhdash I think are the committer of that top commit to I feel like this is coming my catchphrase...but I can help add the new layer code for |
Lemme know if you need any help. It's just qk_norm and GQA instead of MHA. |
|
I love how active this project is :) |
Information about the Command-R 35B model (128k context) can be found at: https://huggingface.co/CohereForAI/c4ai-command-r-v01 Based on the llama2 model with a few changes: 1) New hyper parameter to scale output logits (logit_scale) 2) Uses LayerNorm instead of RMSNorm 3) Transfomer layers have a single shared LayerNorm that feeds into both the self-attention and FFN layers in parallel. There is no post-attention LayerNorm. 4) No support for Rotary Position Embeddings (RoPE) scaling 5) No biases used Find GGUF files here: https://huggingface.co/andrewcanis/c4ai-command-r-v01-GGUF To convert model to GGUF format yourself: 1) Download Command-R Hugging Face safetensors: git lfs install git clone https://huggingface.co/CohereForAI/c4ai-command-r-v01 2) Run: python3 convert-hf-to-gguf.py --outtype f16 ./c4ai-command-r-v01
Information about the Command-R 35B model (128k context) can be found at: https://huggingface.co/CohereForAI/c4ai-command-r-v01 Based on the llama2 model with a few changes: 1) New hyper parameter to scale output logits (logit_scale) 2) Uses LayerNorm instead of RMSNorm 3) Transfomer layers have a single shared LayerNorm that feeds into both the self-attention and FFN layers in parallel. There is no post-attention LayerNorm. 4) No support for Rotary Position Embeddings (RoPE) scaling 5) No biases used Find GGUF files here: https://huggingface.co/andrewcanis/c4ai-command-r-v01-GGUF To convert model to GGUF format yourself: 1) Download Command-R Hugging Face safetensors: git lfs install git clone https://huggingface.co/CohereForAI/c4ai-command-r-v01 2) Run: python3 convert-hf-to-gguf.py --outtype f16 ./c4ai-command-r-v01
Information about the Command-R 35B model (128k context) can be found at: https://huggingface.co/CohereForAI/c4ai-command-r-v01 Based on the llama2 model with a few changes: 1) New hyper parameter to scale output logits (logit_scale) 2) Uses LayerNorm instead of RMSNorm 3) Transfomer layers have a single shared LayerNorm that feeds into both the self-attention and FFN layers in parallel. There is no post-attention LayerNorm. 4) No support for Rotary Position Embeddings (RoPE) scaling 5) No biases used Find GGUF files here: https://huggingface.co/andrewcanis/c4ai-command-r-v01-GGUF To convert model to GGUF format yourself: 1) Download Command-R Hugging Face safetensors: git lfs install git clone https://huggingface.co/CohereForAI/c4ai-command-r-v01 2) Run: python3 convert-hf-to-gguf.py --outtype f16 ./c4ai-command-r-v01
Information about the Command-R 35B model (128k context) can be found at: https://huggingface.co/CohereForAI/c4ai-command-r-v01 Based on the llama2 model with a few changes: 1) New hyper parameter to scale output logits (logit_scale) 2) Uses LayerNorm instead of RMSNorm 3) Transfomer layers have a single shared LayerNorm that feeds into both the self-attention and FFN layers in parallel. There is no post-attention LayerNorm. 4) No support for Rotary Position Embeddings (RoPE) scaling 5) No biases used Find GGUF files here: https://huggingface.co/andrewcanis/c4ai-command-r-v01-GGUF To convert model to GGUF format yourself: 1) Download Command-R Hugging Face safetensors: git lfs install git clone https://huggingface.co/CohereForAI/c4ai-command-r-v01 2) Run: python3 convert-hf-to-gguf.py --outtype f16 ./c4ai-command-r-v01
Information about the Command-R 35B model (128k context) can be found at: https://huggingface.co/CohereForAI/c4ai-command-r-v01 Based on the llama2 model with a few changes: 1) New hyper parameter to scale output logits (logit_scale) 2) Uses LayerNorm instead of RMSNorm 3) Transfomer layers have a single shared LayerNorm that feeds into both the self-attention and FFN layers in parallel. There is no post-attention LayerNorm. 4) No support for Rotary Position Embeddings (RoPE) scaling 5) No biases used Find GGUF files here: https://huggingface.co/andrewcanis/c4ai-command-r-v01-GGUF To convert model to GGUF format yourself: 1) Download Command-R Hugging Face safetensors: git lfs install git clone https://huggingface.co/CohereForAI/c4ai-command-r-v01 2) Run: python3 convert-hf-to-gguf.py --outtype f16 ./c4ai-command-r-v01
Information about the Command-R 35B model (128k context) can be found at: https://huggingface.co/CohereForAI/c4ai-command-r-v01 Based on the llama2 model with a few changes: 1) New hyper parameter to scale output logits (logit_scale) 2) Uses LayerNorm instead of RMSNorm 3) Transfomer layers have a single shared LayerNorm that feeds into both the self-attention and FFN layers in parallel. There is no post-attention LayerNorm. 4) No support for Rotary Position Embeddings (RoPE) scaling 5) No biases used Find GGUF files here: https://huggingface.co/andrewcanis/c4ai-command-r-v01-GGUF To convert model to GGUF format yourself: 1) Download Command-R Hugging Face safetensors: git lfs install git clone https://huggingface.co/CohereForAI/c4ai-command-r-v01 2) Run: python3 convert-hf-to-gguf.py --outtype f16 ./c4ai-command-r-v01
Information about the Command-R 35B model (128k context) can be found at: https://huggingface.co/CohereForAI/c4ai-command-r-v01 Based on the llama2 model with a few changes: 1) New hyper parameter to scale output logits (logit_scale) 2) Uses LayerNorm instead of RMSNorm 3) Transfomer layers have a single shared LayerNorm that feeds into both the self-attention and FFN layers in parallel. There is no post-attention LayerNorm. 4) No support for Rotary Position Embeddings (RoPE) scaling 5) No biases used Find GGUF files here: https://huggingface.co/andrewcanis/c4ai-command-r-v01-GGUF To convert model to GGUF format yourself: 1) Download Command-R Hugging Face safetensors: git lfs install git clone https://huggingface.co/CohereForAI/c4ai-command-r-v01 2) Run: python3 convert-hf-to-gguf.py --outtype f16 ./c4ai-command-r-v01
Information about the Command-R 35B model (128k context) can be found at: https://huggingface.co/CohereForAI/c4ai-command-r-v01 Based on the llama2 model with a few changes: 1) New hyper parameter to scale output logits (logit_scale) 2) Uses LayerNorm instead of RMSNorm 3) Transfomer layers have a single shared LayerNorm that feeds into both the self-attention and FFN layers in parallel. There is no post-attention LayerNorm. 4) No support for Rotary Position Embeddings (RoPE) scaling 5) No biases used Find GGUF files here: https://huggingface.co/andrewcanis/c4ai-command-r-v01-GGUF To convert model to GGUF format yourself: 1) Download Command-R Hugging Face safetensors: git lfs install git clone https://huggingface.co/CohereForAI/c4ai-command-r-v01 2) Run: python3 convert-hf-to-gguf.py --outtype f16 ./c4ai-command-r-v01
14 participants
Information about the Command-R 35B model (128k context) can be found at:
https://huggingface.co/CohereForAI/c4ai-command-r-v01
Based on the llama2 model with a few changes:
self-attention and FFN layers in parallel. There is no post-attention LayerNorm.
Find GGUF files here:
https://huggingface.co/andrewcanis/c4ai-command-r-v01-GGUF
To convert model to GGUF format yourself:
Download Command-R Hugging Face safetensors:
git lfs install
git clone https://huggingface.co/CohereForAI/c4ai-command-r-v01
Run:
python3 convert-hf-to-gguf.py --outtype f16 ./c4ai-command-r-v01