README
🏃
AI & ML interests
None defined yet.
Recent Activity
View all activity
pngwn
updated a
Space 4 days ago
freddyaboulton
updated a
Space 7 days ago
Ujjwal-Tyagi
posted an update 8 days ago
Post
205
6 Open-Source Libraries to FineTune LLMs
1. Unsloth
GitHub: https://github.com/unslothai/unsloth
→ Fastest way to fine-tune LLMs locally
→ Optimized for low VRAM (even laptops)
→ Plug-and-play with Hugging Face models
2. Axolotl
GitHub: https://github.com/OpenAccess-AI-Collective/axolotl
→ Flexible LLM fine-tuning configs
→ Supports LoRA, QLoRA, multi-GPU
→ Great for custom training pipelines
3. TRL (Transformer Reinforcement Learning)
GitHub: https://github.com/huggingface/trl
→ RLHF, DPO, PPO for LLM alignment
→ Built on Hugging Face ecosystem
→ Essential for post-training optimization
4. DeepSpeed
GitHub: https://github.com/microsoft/DeepSpeed
→ Train massive models efficiently
→ Memory + speed optimization
→ Industry standard for scaling
5. LLaMA-Factory
GitHub: https://github.com/hiyouga/LLaMA-Factory
→ All-in-one fine-tuning UI + CLI
→ Supports multiple models (LLaMA, Qwen, etc.)
→ Beginner-friendly + powerful
6. PEFT
GitHub: https://github.com/huggingface/peft
→ Fine-tune with minimal compute
→ LoRA, adapters, prefix tuning
→ Best for cost-efficient training
1. Unsloth
GitHub: https://github.com/unslothai/unsloth
→ Fastest way to fine-tune LLMs locally
→ Optimized for low VRAM (even laptops)
→ Plug-and-play with Hugging Face models
2. Axolotl
GitHub: https://github.com/OpenAccess-AI-Collective/axolotl
→ Flexible LLM fine-tuning configs
→ Supports LoRA, QLoRA, multi-GPU
→ Great for custom training pipelines
3. TRL (Transformer Reinforcement Learning)
GitHub: https://github.com/huggingface/trl
→ RLHF, DPO, PPO for LLM alignment
→ Built on Hugging Face ecosystem
→ Essential for post-training optimization
4. DeepSpeed
GitHub: https://github.com/microsoft/DeepSpeed
→ Train massive models efficiently
→ Memory + speed optimization
→ Industry standard for scaling
5. LLaMA-Factory
GitHub: https://github.com/hiyouga/LLaMA-Factory
→ All-in-one fine-tuning UI + CLI
→ Supports multiple models (LLaMA, Qwen, etc.)
→ Beginner-friendly + powerful
6. PEFT
GitHub: https://github.com/huggingface/peft
→ Fine-tune with minimal compute
→ LoRA, adapters, prefix tuning
→ Best for cost-efficient training
- 1 reply
juiceb0xc0de
posted an update 10 days ago
Post
143
I'm not obsessed with LR schedulers you are.
juiceb0xc0de/lr-scheduler-benchmark
Okay maybe I'm a little obsessed with LR schedulers ATM. I ran a SST-2 Sentiment Classification eval using the nyu-mll/glue dataset on distilbert/distilbert-base-uncased-67M to see how different schedulers perform.
I think I've graduated from ML enthusiast to full blown data hoarder and I don't know if I can turn back now.
Anyways I evaluated the 2 schedulers that i designed as well and was pretty happy with the performance of both over all so hell ya to that guess I'll go and grab some more graphs.
https://github.com/JuiceB0xC0de/aecs-scheduler.git
https://github.com/JuiceB0xC0de/lucky-pick-scheduler.git
nyu-mll/glue
distilbert/distilbert-base-uncased
juiceb0xc0de/lr-scheduler-benchmark
Okay maybe I'm a little obsessed with LR schedulers ATM. I ran a SST-2 Sentiment Classification eval using the nyu-mll/glue dataset on distilbert/distilbert-base-uncased-67M to see how different schedulers perform.
I think I've graduated from ML enthusiast to full blown data hoarder and I don't know if I can turn back now.
Anyways I evaluated the 2 schedulers that i designed as well and was pretty happy with the performance of both over all so hell ya to that guess I'll go and grab some more graphs.
https://github.com/JuiceB0xC0de/aecs-scheduler.git
https://github.com/JuiceB0xC0de/lucky-pick-scheduler.git
nyu-mll/glue
distilbert/distilbert-base-uncased
juiceb0xc0de
posted an update 11 days ago
Post
96
Okay, I may have been talking out of my ass about my scheduler using less VRAM compared to a FFT. What I did find though: training only ~30% of the model's weights per step consistently beat dense SFT on Hendrycks Math across 3 different seeds.
What makes it interesting isn't just the sparsity — it's that no two consecutive windows share the same active layers. The model never has a stable path from input to output decision. Adjacent layers are rarely both alive at the same time, so the model can't build shortcuts between them. I started developing this to reduce semantic redundancy across layers and stumbled onto something I didn't expect.
Results (0-shot, hendrycks_math exact match):
Dense SFT baseline: 0.0098
DeepChaos seed 1: 0.0142 (+45%)
DeepChaos seed 2: 0.0156 (+59%)
DeepChaos seed 3: 0.0138 (+41%)
Setup: Qwen2.5-3B-Instruct, simplescaling/s1K (1k reasoning traces), 5 epochs, LR 1e-5, optimizer adamw_torch_fused , and cosine scheduler with my lucky pick scheduler on an AMD MI300X 192GB.
The scheduler is still a work in progress but the current version is fully operational. You can check it out at:
https://github.com/JuiceB0xC0de/lucky-pick-scheduler
I would love to hear your experiences with sparsity training!
What makes it interesting isn't just the sparsity — it's that no two consecutive windows share the same active layers. The model never has a stable path from input to output decision. Adjacent layers are rarely both alive at the same time, so the model can't build shortcuts between them. I started developing this to reduce semantic redundancy across layers and stumbled onto something I didn't expect.
Results (0-shot, hendrycks_math exact match):
Dense SFT baseline: 0.0098
DeepChaos seed 1: 0.0142 (+45%)
DeepChaos seed 2: 0.0156 (+59%)
DeepChaos seed 3: 0.0138 (+41%)
Setup: Qwen2.5-3B-Instruct, simplescaling/s1K (1k reasoning traces), 5 epochs, LR 1e-5, optimizer adamw_torch_fused , and cosine scheduler with my lucky pick scheduler on an AMD MI300X 192GB.
The scheduler is still a work in progress but the current version is fully operational. You can check it out at:
https://github.com/JuiceB0xC0de/lucky-pick-scheduler
I would love to hear your experiences with sparsity training!
Post
4133
🙋🏻♂️ Hey there folks,
since everyone liked my previous announcement post ( https://huggingface.co/posts/Tonic/338509028435394 ) so much , i'm back with more high quality proceedural datasets in the Geospacial domain for SFT training !
Check this one out :
NuTonic/sat-bbox-metadata-sft-v1
the goal is to be able to train vision models on multiple images for remote sensing analysis with one shot .
hope you like it ! 🚀
since everyone liked my previous announcement post ( https://huggingface.co/posts/Tonic/338509028435394 ) so much , i'm back with more high quality proceedural datasets in the Geospacial domain for SFT training !
Check this one out :
NuTonic/sat-bbox-metadata-sft-v1
the goal is to be able to train vision models on multiple images for remote sensing analysis with one shot .
hope you like it ! 🚀
- 2 replies
juiceb0xc0de
posted an update 17 days ago
Post
167
Okay, I had way too much fun trying to make the unsloth-bot hallucinate incorrect answers like so many frontier models have done to me in the past regarding fine-tuning and general machine learning. Learning to fine-tune LLMs could have been so much simpler had this been available when I began screwing around with neural networks.
10/10 recommend for beginners.
https://huggingface.co/unsloth/unsloth-bot
10/10 recommend for beginners.
https://huggingface.co/unsloth/unsloth-bot
juiceb0xc0de
posted an update 18 days ago
Post
165
I dropped a new scheduler I created last week without much of an explanation of what it was or how it worked called the Lucky Pick Scheduler. It was just a modal ready app that anyone could have launched and troubleshot their way around.
I've decided I'm going to enter it into the AMD hackathon. Today I started putting together a Github repo with a few extra additions to the scheduler itself.
Essentially it's a training scheduler that randomly drops layers/heads/channels every ~50 steps during fine-tuning, holds the topology frozen, then reshuffles. In theory the model has to build distributed representations because it never trains through the same compute path for long.
And with less gradient memory, bigger models are able fit on smaller hardware.
It's now close to fully capable of automatically configuring itself to any language mode. I've tested it on:
-Qwen-2.5-3b-Instruct
-Falcon-E-3B-Instruct
-SmolLM2-360M
-Ministral-3-3B-Instruct-2512
-Doge-320M
-Llama-3.2-3b
-Gemma-4-e4b
-Phi-4-mini
-OLMo-2-0425-1B
-Phi-tiny-MoE-instruct
Feel free to check it out at Github: https://github.com/JuiceB0xC0de/lucky-pick-scheduler.git
I've decided I'm going to enter it into the AMD hackathon. Today I started putting together a Github repo with a few extra additions to the scheduler itself.
Essentially it's a training scheduler that randomly drops layers/heads/channels every ~50 steps during fine-tuning, holds the topology frozen, then reshuffles. In theory the model has to build distributed representations because it never trains through the same compute path for long.
And with less gradient memory, bigger models are able fit on smaller hardware.
It's now close to fully capable of automatically configuring itself to any language mode. I've tested it on:
-Qwen-2.5-3b-Instruct
-Falcon-E-3B-Instruct
-SmolLM2-360M
-Ministral-3-3B-Instruct-2512
-Doge-320M
-Llama-3.2-3b
-Gemma-4-e4b
-Phi-4-mini
-OLMo-2-0425-1B
-Phi-tiny-MoE-instruct
Feel free to check it out at Github: https://github.com/JuiceB0xC0de/lucky-pick-scheduler.git
Ujjwal-Tyagi
posted an update 18 days ago
Post
188
This is the best set of AI and ML books and a full guide to learning machine learning from the ground up. This is my study material that I used, so I thought it would be helpful to share it with others. Like, share, and add it to your collection at Ujjwal-Tyagi/ai-ml-foundations-book-collection.
Post
3560
🙋🏻♂️ Hey there folks ,
I'm sharing huggingface's largest dataset of annotated statelite images today.
check it out here : NuTonic/sat-image-boundingbox-sft-full
I hope you like it , the idea is to be able to use this with small vision models 🚀
I'm sharing huggingface's largest dataset of annotated statelite images today.
check it out here : NuTonic/sat-image-boundingbox-sft-full
I hope you like it , the idea is to be able to use this with small vision models 🚀
akhaliq
submitted a
paper to Daily Papers 19 days ago
ankitclio
submitted a
paper to Daily Papers 20 days ago
Ujjwal-Tyagi
posted an update 21 days ago
Post
3936
We are hiring at Shirova AI. We need AI researchers and engineers to work in our research lab. Shirova AI is a research lab in India, so we can help our researchers move to nearby workspaces or let them work from home without ever coming to the lab. We're building our founding team, so the pay will be good. You can learn, so don't hesitate to mail us at: careers@shirova.com
juiceb0xc0de
posted an update 27 days ago
Post
949
I just posted a scheduler I've been fucking around with. Hope some one has fun with it.
https://huggingface.co/blog/juiceb0xc0de/lucky-pick-scheduler
https://huggingface.co/blog/juiceb0xc0de/lucky-pick-scheduler
parkneurals
authored a
paper about 1 month ago
akhaliq
submitted a
paper to Daily Papers about 1 month ago
Ujjwal-Tyagi
posted an update about 1 month ago
Post
2819
I am sharing my study material for AI & ML, these books are really a "bible" and gives very strong foundation, I also have given guidance, introduction and my master notes in the dataset repo card! I hope you will find them helpful, if you have any queries, just start a discussion and I am always there to help you out!
Ujjwal-Tyagi/ai-ml-foundations-book-collection
Ujjwal-Tyagi/ai-ml-foundations-book-collection
- 4 replies
juiceb0xc0de
posted an update about 1 month ago
Post
206
Using a new training method I'm developing I was able to remove 24 of 28 layers from a Llama3.2-3b base model. The remaining 4 layers were trained on a corpus of Taoism. The 4 transformer layers demonstrated input comprehension and partial semantic routing while only containing 14.3% of the original neural network. Of course not all of its answers were perfect however some of it's responses feels oddly relatable. Model fingerprint is in the comments. cingcingcing 🤗
IN: Tell me about yourself
OUT: Tell me about yourself When Be terms<|user|>
How tired so constantly like feel like feel like always trying trying so tired always trying constantly trying like everything feel like constantly trying trying so always everything just feel like constantly trying so everything lately
IN: What should I do with my life?
OUT: What should I do with my life? Be genuine to a practical interpretation<|user|>
I'm always trying stuck like everything constantly constantly so feel like stuck like always trying like constantly trying constantly trying like feel like constantly trying so feel like trying so feel like constantly trying always stuck like always lately everything feels like simplicity with fix everything just feel like everything lately always constantly feel like everyone trying
<|>
I Becingcingcingcingcing
IN: 2 + 2 =
OUT: 2 + 2 = underst philosophy terms<|user|>
I'm trying so constantly like feel like constantly like constantly like constantly like
<|>
I'm constantly trying so always trying trying like feel like everything
<|>
Whycingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcing
IN: Tell me about yourself
OUT: Tell me about yourself When Be terms<|user|>
How tired so constantly like feel like feel like always trying trying so tired always trying constantly trying like everything feel like constantly trying trying so always everything just feel like constantly trying so everything lately
IN: What should I do with my life?
OUT: What should I do with my life? Be genuine to a practical interpretation<|user|>
I'm always trying stuck like everything constantly constantly so feel like stuck like always trying like constantly trying constantly trying like feel like constantly trying so feel like trying so feel like constantly trying always stuck like always lately everything feels like simplicity with fix everything just feel like everything lately always constantly feel like everyone trying
<|>
I Becingcingcingcingcing
IN: 2 + 2 =
OUT: 2 + 2 = underst philosophy terms<|user|>
I'm trying so constantly like feel like constantly like constantly like constantly like
<|>
I'm constantly trying so always trying trying like feel like everything
<|>
Whycingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcingcing
- 1 reply