Fine-Tuning Gemma 4 E2B for Enterprise AIOps (Mac M3 Pro & GGUF)

A complete, end-to-end pipeline for fine-tuning Google's Gemma 4 E2B on Apple Silicon (M3 Pro, 18GB RAM) using Apple's native MLX framework.

This repository demonstrates how to adapt Gemma 4 into a specialized AIOps Orchestrator Agent capable of outputting structured JSON control schemas for multi-domain infrastructure, including Kubernetes (K8s), Nutanix, VMware ESXi, Active Directory (AD), ADFS, PKI, and Windows Server environments.

---

🏗 System Architecture & Workflow

[M3 Pro Mac (18GB)] ──> 1. Fine-Tune (MLX LoRA)
                    └──> 2. Fuse Adapters to HF Format
                    └──> 3. Quantize to GGUF (llama.cpp)
                    └──> 4. Push to Hugging Face Hub
                    └──> 5. Serve via Ollama / API

---

📋 Prerequisites & Hardware

• Hardware: Apple Silicon Mac (M3 Pro with 18GB Unified Memory recommended).
• OS: macOS Sonoma / Sequoia or newer.
• Python: 3.10 or higher.
• Dependencies: mlx-lm, huggingface_hub, llama.cpp, ollama.

---

📁 Repository Structure

.
├── data/
│   ├── train.jsonl             # Fine-tuning dataset
│   └── valid.jsonl             # Validation dataset
├── aiops_adapters/            # Trained LoRA adapters (MLX)
├── gemma4-e2b-aiops-hf/       # Fused Hugging Face model
├── scripts/
│   ├── train.sh               # MLX fine-tuning wrapper
│   ├── quantize.sh            # GGUF conversion pipeline
│   └── upload.sh              # HF Hub upload automation
├── Modelfile                  # Ollama configuration file
├── requirements.txt           # Python dependencies
├── LICENSE                    # Apache 2.0 License
└── README.md                  # Project documentation

---

🚀 Quick Start & Step-by-Step Guide

Step 1: Environment Setup

Clone this repository and set up a virtual environment:

git clone https://github.com/htunn/aiops-gemma4.git
cd aiops-gemma4

python3 -m venv aiops-env
source aiops-env/bin/activate

pip install -U "mlx-lm[train]" huggingface_hub

Step 2: Prepare Training Data

The model is trained on domain-specific AIOps scenarios where input prompts map directly to structured JSON execution schemas.

Create data/train.jsonl and data/valid.jsonl formatted as follows:

{"text": "<bos><start_of_turn>user\n[AIOps-Agent] Node k8s-worker-03 status is NotReady. Active Directory service account 'svc_k8s_cluster' authentication failed on ADFS.<end_of_turn>\n<start_of_turn>model\n{\"action\": \"remediate_auth\", \"target_domain\": \"ADFS\", \"service_account\": \"svc_k8s_cluster\", \"steps\": [\"check_pki_cert_validity\", \"rotate_secret_k8s\"], \"api_call\": \"POST /api/v1/auth/refresh\"}<end_of_turn>"}
{"text": "<bos><start_of_turn>user\n[AIOps-Agent] Nutanix VM 'win-dc-01' CPU usage exceeds 95%. VMware host ESXi-02 reports datastore latency.<end_of_turn>\n<start_of_turn>model\n{\"action\": \"diagnose_hypervisor\", \"target_vm\": \"win-dc-01\", \"infrastructure\": [\"nutanix\", \"vmware\"], \"recommendation\": \"Live-migrate VM via Prism API to secondary host ESXi-04.\"}<end_of_turn>"}

Step 3: Run Fine-Tuning with Apple MLX

Train the model locally using Parameter-Efficient Fine-Tuning (LoRA):

mlx_lm.lora \
  --model google/gemma-4-E2B-it \
  --data ./data \
  --train \
  --iters 600 \
  --batch-size 1 \
  --learning-rate 1e-4 \
  --lora-layers 16 \
  --adapter-path ./aiops_adapters

Memory Performance: Consumes ~8GB–10GB VRAM, leaving ample room on an 18GB M3 Pro without memory swapping.

Step 4: Fuse Adapters and Quantize to GGUF

Fuse the trained LoRA adapters back into base Hugging Face format, then quantize to GGUF using llama.cpp.

1. Fuse MLX Adapters

mlx_lm.fuse \
  --model google/gemma-4-E2B-it \
  --adapter-path ./aiops_adapters \
  --save-path ./gemma4-e2b-aiops-hf \
  --export-hf

2. Convert and Quantize via llama.cpp

# Clone llama.cpp repository
git clone https://github.com/ggml-org/llama.cpp
pip install -r llama.cpp/requirements.txt

# Convert HF format to FP16 GGUF
python llama.cpp/convert_hf_to_gguf.py ./gemma4-e2b-aiops-hf \
  --outfile ./gemma4-e2b-aiops-f16.gguf \
  --outtype f16

# Quantize FP16 GGUF to Q4_K_M (4-bit)
cd llama.cpp && make
./llama-quantize ../gemma4-e2b-aiops-f16.gguf ../gemma4-e2b-aiops-Q4_K_M.gguf Q4_K_M
cd ..

Step 5: Publish GGUF to Hugging Face Hub

Authenticate with Hugging Face and publish your GGUF binary:

# Login with write-enabled access token
huggingface-cli login

# Upload model file
huggingface-cli upload htunn/gemma-4-e2b-aiops-gguf \
  ./gemma4-e2b-aiops-Q4_K_M.gguf \
  gemma4-e2b-aiops-Q4_K_M.gguf

Step 6: Deployment & Local API Serving

Serving with Ollama

Run the published model directly from Hugging Face:

ollama run hf.co/htunn/gemma-4-e2b-aiops-gguf:Q4_K_M

Custom Agent Integration (Modelfile)

Create a custom Modelfile to define system behavior:

FROM hf.co/htunn/gemma-4-e2b-aiops-gguf:Q4_K_M

SYSTEM """You are an autonomous AIOps orchestrator agent. Evaluate telemetry across Kubernetes, Nutanix, VMware, Active Directory, ADFS, and PKI infrastructure. Output strict, execution-ready JSON commands."""

Build and launch the Ollama agent instance:

ollama create aiops-orchestrator -f Modelfile
ollama serve

Send API completion requests via HTTP POST (http://localhost:11434/v1/chat/completions) from FastAPI, Go, or LangChain microservices.

---

📜 License

Distributed under the Apache 2.0 License. See LICENSE for more information.