File size: 5,661 Bytes
cb6c992 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 | ---
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
- code
- code-review
- bug-fixing
- qwen
- qwen2.5-coder
- qlora
- peft
- trl
- static-analysis
model-index:
- name: Code-Autopsy
results:
- task:
type: text-generation
name: Code Bug Diagnosis & Refactoring
metrics:
- name: Validation Loss
type: loss
value: 0.2442
- name: Token Accuracy
type: accuracy
value: 93.20%
---
<div align="center">
# 🔬 Code-Autopsy (QLoRA)
### Deep Structural Bug Diagnosis & Remediation Model
[](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct)
[-purple.svg)](https://github.com/huggingface/peft)
[](https://wandb.ai/devanshtyagi1903-innothoughts/code-autopsy/runs/gc70q2q2)
[](LICENSE)
</div>
---
## 📌 Model Summary
**Code-Autopsy** is a specialized code intelligence model fine-tuned on top of **Qwen2.5-Coder-7B-Instruct** using 4-bit QLoRA. It operates like an autonomous forensic compiler: given buggy, defective, or vulnerable code snippets across Python, JavaScript, and other languages, it outputs a clean, structured diagnostic report:
1. **Bug Identified:** Exact forensic analysis of the flaw (e.g. mutable default arguments, ZeroDivisionError, unawaited asynchronous promises, race conditions).
2. **Root Cause:** In-depth explanation of *why* the defect occurs at the runtime/memory level.
3. **Fixed Code:** Corrected, refactored, and production-ready implementation.
---
## 📊 Training Metrics & Cloud Logs
The model was trained for **3 full epochs (246 steps)** on a curated dataset of code bugs and algorithmic repairs.
| Metric | Initial (Epoch 0.06) | Final (Epoch 3.0) | Delta |
| :--- | :---: | :---: | :---: |
| **Training Loss** | `2.162` | **`0.255`** | **-88.2%** 📉 |
| **Validation Loss (`eval_loss`)** | `1.397` | **`0.2442`** | **-82.5%** 📉 |
| **Token Accuracy** | `60.29%` | **`93.20%`** | **+32.91%** 📈 |
| **Gradient Norm** | `0.27` | `0.39` | Stable |
> 🌐 **Interactive Training Logs & Loss Curves:**
> View the live dashboard, loss charts, and hardware telemetry on [Weights & Biases](https://wandb.ai/devanshtyagi1903-innothoughts/code-autopsy/runs/gc70q2q2).
---
## ⚙️ Hyperparameters & Hardware Configuration
* **Base Model:** `Qwen/Qwen2.5-Coder-7B-Instruct`
* **Quantization:** 4-bit NF4 (`bitsandbytes` double quant)
* **Compute Dtype:** `bfloat16`
* **LoRA Rank ($r$):** `16`
* **LoRA Alpha ($lpha$):** `32`
* **LoRA Target Modules:** `q_proj`, `v_proj`
* **Optimizer:** `adamw_8bit`
* **Peak Learning Rate:** `2e-4` (with Cosine Decay and 5% Warmup)
* **Effective Batch Size:** `8` (Per-device `1`, Gradient Accumulation `8`)
* **Hardware:** NVIDIA GeForce RTX 5060 (8GB VRAM)
---
## 🚀 Quickstart: Running Inference
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
BASE_MODEL = "Qwen/Qwen2.5-Coder-7B-Instruct"
ADAPTER_REPO = "devanshty/Code-Autopsy"
# 1. Load Tokenizer & 4-bit Base Model
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, trust_remote_code=True)
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True
)
base_model = AutoModelForCausalLM.from_pretrained(
BASE_MODEL,
quantization_config=bnb_config,
device_map="auto",
torch_dtype=torch.bfloat16,
trust_remote_code=True
)
# 2. Load Fine-Tuned Code-Autopsy Adapter
model = PeftModel.from_pretrained(base_model, ADAPTER_REPO)
model.eval()
# 3. Format Diagnostic Prompt
code_snippet = '''def append_item(val, lst=[]):
lst.append(val)
return lst'''
prompt = f"""<|im_start|>system
You are a code review expert. Analyze the provided code, identify any bugs or issues, explain the root cause, and provide a corrected version.<|im_end|>
<|im_start|>user
Language: python
```python
{code_snippet}
```<|im_end|>
<|im_start|>assistant
"""
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=False,
pad_token_id=tokenizer.eos_token_id
)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```
---
## 🔍 Diagnostic Output Format
The model generates responses structured in Markdown:
```markdown
## Bug Identified
Mutable default argument `lst=[]` used in function definition.
## Root Cause
In Python, default arguments are evaluated once when the function is defined, not each time it is called. Modifying `lst` mutates the single shared list object across subsequent calls.
## Fixed Code
```python
def append_item(val, lst=None):
if lst is None:
lst = []
lst.append(val)
return lst
```
```
---
## 📜 Citation & Credits
* **Author:** Devansh Tyagi ([devanshty](https://huggingface.co/devanshty))
* **Base Architecture:** Alibaba Cloud Qwen Team (`Qwen2.5-Coder-7B-Instruct`)
* **Frameworks:** 🤗 Hugging Face `transformers`, `peft`, `trl`, and Weights & Biases `wandb`.
|