How Transformers Actually Work

Forward Pass

The model takes input text (e.g., "The cat") and passes it through all layers. At the end, it outputs a probability distribution over all possible next tokens. It might say: "mat" 40%, "floor" 20%, "the" 15%, etc.

Calculate Loss (Error)

We know the correct answer (e.g., "sat"). We measure how wrong the model was using a formula called cross-entropy loss. If it predicted "sat" with probability 0.001 but the answer was "sat", the loss is very high. If it predicted "sat" with 0.9 probability, loss is low.

Backpropagation

The error is sent backward through all layers. Each layer learns how much it contributed to the error. This calculates gradients — numbers that tell each parameter (weight) which direction to adjust.

Update Weights (Gradient Descent)

Every single parameter (weight) in the model is updated by a tiny amount. The "learning rate" (e.g., 0.0001) controls how big each step is. Too large = chaotic. Too small = slow. This repeats billions of times.

Collect & Clean Your Data

Gather text data — books, websites, code, articles. Clean it by removing HTML tags, duplicates, and harmful content. Big models use datasets like "The Pile" (825GB), FineWeb, or Common Crawl (petabytes of web text).

Example data: 
"The quick brown fox jumps over the lazy dog."
"Paris is the capital of France."
"def factorial(n): return 1 if n<=1 else n*factorial(n-1)"

Build Your Tokenizer

Train a BPE tokenizer on your data. It learns a "vocabulary" — the most common subword units. GPT-4 uses a vocabulary of 100,277 tokens. BERT uses 30,522. Your toy model might use 5,000–50,000.

from tokenizers import ByteLevelBPETokenizer
tokenizer = ByteLevelBPETokenizer()
tokenizer.train(files=["data.txt"], vocab_size=10000)
# "hello" → [15496]
# "world" → [11]

Design Your Model Architecture

Decide: How many layers? How many attention heads? What hidden dimension? These are called "hyperparameters". Larger = smarter but slower and more expensive.

Tiny model (runs on laptop):
  layers = 6
  heads = 6  
  d_model = 384
  d_ff = 1536  (4× d_model)
  vocab_size = 10000
  Parameters: ~15 million

GPT-2 scale:
  layers = 12
  heads = 12
  d_model = 768
  Parameters: ~117 million

Code the Transformer Block

The core building block. In Python with PyTorch, each transformer layer contains Multi-Head Attention, Feed-Forward Network, and two Layer Normalizations.

class TransformerBlock:
    def forward(x):
        # Multi-Head Self-Attention
        attn_out = self.attention(x)
        x = self.layer_norm_1(x + attn_out)  # residual
        
        # Feed-Forward Network
        ff_out = self.ff_network(x)
        x = self.layer_norm_2(x + ff_out)    # residual
        
        return x  # passes to next layer

Implement Attention

The heart of the transformer. Project input into Q, K, V matrices. Compute attention scores. Apply softmax. Return weighted values.

class MultiHeadAttention:
    def forward(x):
        Q = self.W_q(x)  # Query matrix
        K = self.W_k(x)  # Key matrix
        V = self.W_v(x)  # Value matrix
        
        # Attention scores
        scores = Q @ K.T / sqrt(d_k)  # dot product + scale
        weights = softmax(scores)      # normalize to 0-1
        output = weights @ V           # weighted sum
        
        return output

Stack Layers & Add Output Head

Stack N transformer blocks on top of each other. Add a final "language model head" — a linear layer that converts the hidden state to logits (scores) over your entire vocabulary.

class GPTModel:
    def forward(token_ids):
        x = embedding(token_ids)       # tokens → vectors
        x = positional_encoding(x)     # add position info
        
        for block in self.layers:      # N transformer blocks
            x = block(x)
        
        logits = self.lm_head(x)       # → vocab scores
        return logits  # [batch, seq_len, vocab_size]

Train with Gradient Descent

Feed data in batches. Calculate cross-entropy loss (how wrong was the prediction?). Backpropagate. Update weights with an optimizer like AdamW. Repeat for millions of steps.

optimizer = AdamW(model.parameters(), lr=3e-4)

for batch in dataloader:
    input_ids, labels = batch
    
    logits = model(input_ids)           # forward pass
    loss = cross_entropy(logits, labels) # calculate error
    
    optimizer.zero_grad()
    loss.backward()                      # backprop
    optimizer.step()                     # update weights
    
    print(f"Loss: {loss.item():.4f}")

Generate Text (Inference)

Once trained, feed a prompt and let the model predict the next token. Sample from the probability distribution. Append to input. Repeat until you hit a stop token or max length.

def generate(prompt, max_tokens=100):
    input_ids = tokenizer.encode(prompt)
    
    for _ in range(max_tokens):
        logits = model(input_ids)           # predict next
        next_token = sample(logits[-1])     # pick a token
        input_ids.append(next_token)        # append
        
        if next_token == END_TOKEN:
            break
    
    return tokenizer.decode(input_ids)

Fine-tune & Apply RLHF

After pre-training, fine-tune on high-quality instruction/response pairs. Then if you want an AI assistant (like Claude/ChatGPT), apply RLHF: collect human feedback, train a reward model, use PPO (Proximal Policy Optimization) to optimize the main model.

# Phase 2: Supervised Fine-Tuning
fine_tune_data = [
    {"prompt": "What is the capital of France?", 
     "response": "Paris is the capital of France."},
    ...
]

# Phase 3: RLHF
reward_model = train_reward_model(human_rankings)
ppo_optimize(model, reward_model)  # maximize human preference

Define Tools with Schemas

You give the LLM a list of available tools in its system prompt. Each tool is described with its name, purpose, and parameters — like a menu of capabilities.

tools = [
  {
    "name": "web_search",
    "description": "Search the internet for current information",
    "parameters": {
      "query": {
        "type": "string", 
        "description": "The search query"
      }
    }
  },
  {
    "name": "run_python",
    "description": "Execute Python code and return the output",
    "parameters": {
      "code": {"type": "string", "description": "Python code to run"}
    }
  },
  {
    "name": "send_email",
    "description": "Send an email to a recipient",
    "parameters": {
      "to": {"type": "string"},
      "subject": {"type": "string"},
      "body": {"type": "string"}
    }
  }
]

LLM Decides to Use a Tool

The model thinks about the task and outputs a special "tool use" response. Instead of generating regular text, it outputs a structured JSON saying which tool to call and with what arguments.

# User asks: "What's the population of Tokyo in 2025?"

# LLM RESPONSE (tool call):
{
  "type": "tool_use",
  "name": "web_search",
  "input": {
    "query": "Tokyo population 2025"
  }
}

# This is NOT shown to the user yet.
# Your code intercepts this and runs the actual search.

Your Code Executes the Tool

Your application receives the tool call, runs the actual function (calls a real search API, runs real Python code, reads a real file), and gets the real result.

def execute_tool(tool_name, tool_input):
    if tool_name == "web_search":
        results = google_search_api(tool_input["query"])
        return {
            "results": [
                {"title": "Tokyo Population", 
                 "snippet": "Tokyo's population is 13.96 million..."},
                {"title": "Greater Tokyo Area",
                 "snippet": "The Greater Tokyo Area has 37.4 million..."}
            ]
        }
    elif tool_name == "run_python":
        output = subprocess.run(tool_input["code"])
        return {"stdout": output, "error": None}
    # ... other tools ...

Result Fed Back into Context

The tool result is added to the conversation history as a "tool_result" message. The LLM now sees this real data and can use it to answer the user.

conversation_history = [
  {"role": "user", "content": "What's Tokyo's population in 2025?"},
  {"role": "assistant", "content": [
    {"type": "tool_use", "name": "web_search", 
     "input": {"query": "Tokyo population 2025"}}
  ]},
  {"role": "tool", "content": [
    {"type": "tool_result", 
     "content": "Tokyo city: 13.96M, Greater area: 37.4M (2025)"}
  ]}
  # Now the LLM responds with the actual answer:
]

LLM Generates Final Answer

With the real data in its context, the model generates a human-readable answer. It can call more tools if needed, or produce the final response.

# LLM final response (regular text):
"Tokyo city proper has a population of approximately 
 13.96 million people as of 2025. However, the Greater 
 Tokyo Area — which includes surrounding prefectures — 
 is home to about 37.4 million people, making it the 
 world's most populous metropolitan area."

Install Requirements

You only need the Anthropic (or OpenAI) Python library. No LangChain, no big frameworks — just the raw API and your own code.

pip install anthropic requests

# That's it! We'll build everything else ourselves.

Define Your Tools

Create Python functions for each tool. These are REAL functions that do REAL things. Then define their schemas so the LLM knows how to call them.

import anthropic, subprocess, requests, json

# ── Real tool functions ──────────────────────────────
def web_search(query: str) -> str:
    """Actually searches the web using an API."""
    # Using a free search API (e.g. Serper, Tavily, DuckDuckGo)
    response = requests.get(
        "https://api.tavily.com/search",
        params={"api_key": "YOUR_KEY", "query": query, "max_results": 3}
    )
    results = response.json()["results"]
    return "\n".join([f"• {r['title']}: {r['content'][:200]}" 
                      for r in results])

def run_python(code: str) -> str:
    """Actually runs Python code and returns stdout."""
    result = subprocess.run(
        ["python3", "-c", code],
        capture_output=True, text=True, timeout=10
    )
    return result.stdout or result.stderr

# ── Tool schemas (tell the LLM how to call them) ─────
TOOLS = [
    {
        "name": "web_search",
        "description": "Search the internet for current information. Use this for facts, news, statistics, or anything that needs real-time data.",
        "input_schema": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": "What to search for"
                }
            },
            "required": ["query"]
        }
    },
    {
        "name": "run_python",
        "description": "Execute Python code and return the output. Use for math calculations, data processing, or generating results.",
        "input_schema": {
            "type": "object",
            "properties": {
                "code": {
                    "type": "string",
                    "description": "Python code to run"
                }
            },
            "required": ["code"]
        }
    }
]

Build the Tool Executor

This function receives a tool call from the LLM and routes it to the right Python function. It's the "hands" of the agent.

def execute_tool(tool_name: str, tool_input: dict) -> str:
    """Routes tool calls to actual functions."""
    print(f"\n  🔧 Calling: {tool_name}({tool_input})")
    
    if tool_name == "web_search":
        result = web_search(tool_input["query"])
    elif tool_name == "run_python":
        result = run_python(tool_input["code"])
    else:
        result = f"Error: Unknown tool '{tool_name}'"
    
    print(f"  📤 Result: {result[:100]}...")
    return result

Build the Core Agent Loop

This is the heart of the agent. It keeps calling the LLM, checking if it wants to use tools, executing them, and feeding results back — until the LLM produces a final text answer.

client = anthropic.Anthropic(api_key="YOUR_ANTHROPIC_KEY")

def run_agent(user_question: str) -> str:
    """
    The core ReAct agent loop.
    Runs until the LLM produces a final answer (no more tool calls).
    """
    print(f"\n🤖 Agent started: '{user_question}'\n")
    
    # Build conversation history (starts with user message)
    messages = [{"role": "user", "content": user_question}]
    
    system_prompt = """You are a helpful research assistant with access to 
web search and Python execution. 

For every task:
1. Think about what information or calculations you need
2. Use tools to get real data — don't guess
3. Use run_python for any math/calculations
4. Give a clear, complete final answer

Always use tools when you need current data or math."""
    
    # ── Agent loop ──────────────────────────────────────
    max_loops = 10  # safety limit
    
    for loop_num in range(max_loops):
        print(f"  → Loop {loop_num + 1}")
        
        # Call the LLM
        response = client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=4096,
            system=system_prompt,
            tools=TOOLS,
            messages=messages
        )
        
        # Check stop reason
        if response.stop_reason == "end_turn":
            # LLM gave a final text answer — we're done!
            final_text = ""
            for block in response.content:
                if hasattr(block, "text"):
                    final_text += block.text
            print(f"\n✅ Final Answer:\n{final_text}")
            return final_text
        
        elif response.stop_reason == "tool_use":
            # LLM wants to use one or more tools
            
            # Add the assistant's response to history
            messages.append({
                "role": "assistant",
                "content": response.content
            })
            
            # Execute each requested tool
            tool_results = []
            for block in response.content:
                if block.type == "tool_use":
                    result = execute_tool(block.name, block.input)
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": result
                    })
            
            # Add tool results to conversation history
            messages.append({
                "role": "user",
                "content": tool_results
            })
            # Loop continues → LLM sees results and decides next step
        
    return "Error: Max loops reached."

# ── RUN IT ───────────────────────────────────────────
answer = run_agent(
    "What is the current GDP of India? "
    "Calculate what 0.01% of that is in USD."
)

Expected Output (Live Run)

Here's what you'd actually see when running this agent:

🤖 Agent started: 'What is the GDP of India? Calculate 0.01% of it.'

  → Loop 1
  🔧 Calling: web_search({'query': 'India GDP 2025 current USD'})
  📤 Result: • India GDP 2025: India's GDP reached approximately $3.9 ...

  → Loop 2
  🔧 Calling: run_python({'code': 'print(3.9e12 * 0.0001)'})
  📤 Result: 390000000.0

  → Loop 3
✅ Final Answer:
India's current GDP (2025) is approximately $3.9 trillion USD.

0.01% of $3.9 trillion = $3.9 trillion × 0.0001 
                        = $390,000,000 (390 million USD)

Add Memory: Persistent Agent

Make the agent remember things between separate conversations by saving and loading history from a file (or database).

import json, os

MEMORY_FILE = "agent_memory.json"

def load_memory() -> list:
    if os.path.exists(MEMORY_FILE):
        return json.load(open(MEMORY_FILE))
    return []

def save_memory(messages: list):
    # Save only the last 20 exchanges to keep context size manageable
    json.dump(messages[-40:], open(MEMORY_FILE, "w"), indent=2)

def run_agent_with_memory(user_question: str) -> str:
    # Load past conversations
    past_messages = load_memory()
    
    # Add new user message  
    past_messages.append({"role": "user", "content": user_question})
    
    # Run agent with full history
    # ... (same loop as before) ...
    
    # Save updated history for next session
    save_memory(past_messages)
    return final_answer

# Now the agent remembers previous conversations!
run_agent_with_memory("My name is Arjun and I'm researching Indian economy.")
run_agent_with_memory("What was I researching?")  
# → "You were researching the Indian economy, Arjun."

Add RAG: Agent That Knows Your Documents

Combine the agent with a vector database so it can search through your private PDFs, wikis, or any documents.

from sentence_transformers import SentenceTransformer
import chromadb, PyPDF2

# ── 1. Build index (once) ────────────────────────────
model = SentenceTransformer("all-MiniLM-L6-v2")
db = chromadb.Client()
collection = db.create_collection("my_docs")

def add_document(filepath: str):
    """Add a PDF to the vector database."""
    text = extract_pdf_text(filepath)
    chunks = [text[i:i+500] for i in range(0, len(text), 500)]
    embeddings = model.encode(chunks).tolist()
    collection.add(
        documents=chunks,
        embeddings=embeddings,
        ids=[f"chunk_{i}" for i in range(len(chunks))]
    )

# ── 2. Add document_search tool ─────────────────────
def document_search(query: str) -> str:
    """Search your private documents."""
    query_embedding = model.encode([query]).tolist()
    results = collection.query(query_embeddings=query_embedding, n_results=3)
    return "\n".join(results["documents"][0])

# Add this to TOOLS list and execute_tool() routing
# Now your agent can answer questions from your own documents!

Upgrade to Multi-Agent

Add a second "worker" agent that the first agent can delegate tasks to. The orchestrator breaks the problem down; workers execute specific parts in parallel.

import asyncio

async def worker_agent(task: str, tools: list) -> str:
    """A worker agent — focused on one specific task."""
    response = await async_llm_call(
        system="You are a specialist. Complete the specific task given.",
        messages=[{"role": "user", "content": task}],
        tools=tools
    )
    return response

async def orchestrator_agent(big_task: str) -> str:
    """Breaks big task into subtasks, runs workers in parallel."""
    
    # Step 1: Orchestrator plans subtasks
    subtasks = await plan_subtasks(big_task)
    # e.g. ["Search India GDP", "Search India population", "Search India inflation"]
    
    # Step 2: Run all worker agents IN PARALLEL  
    results = await asyncio.gather(*[
        worker_agent(task, tools=TOOLS) 
        for task in subtasks
    ])
    
    # Step 3: Orchestrator synthesizes results
    synthesis_prompt = f"""
    Original task: {big_task}
    Research results:
    {chr(10).join(f'- {r}' for r in results)}
    
    Synthesize these into a comprehensive answer.
    """
    return await llm_call(synthesis_prompt)

# Run:
# asyncio.run(orchestrator_agent("Write a comprehensive report on India's economy"))

Option A — ElevenLabs API (Easiest, best quality)

No GPU needed. Free tier: 10,000 characters/month. Voice cloning from 30 seconds of audio.

pip install elevenlabs

from elevenlabs.client import ElevenLabs
from elevenlabs import play, save

client = ElevenLabs(api_key="YOUR_KEY")

# Use a built-in voice
audio = client.generate(
    text="Hello! I am an AI voice assistant built with ElevenLabs.",
    voice="Rachel",             # built-in voice
    model="eleven_turbo_v2"     # fastest model
)
play(audio)                     # plays through speakers
save(audio, "output.mp3")       # or save to file

# ── Clone your own voice (30 sec recording) ──────────
voice = client.clone(
    name="My Cloned Voice",
    description="My own voice",
    files=["my_recording.mp3"]  # clear, noise-free audio
)
audio = client.generate(
    text="This is now my cloned voice saying anything I type!",
    voice=voice
)
play(audio)

Option B — Kokoro (Best free local TTS, no GPU needed)

State-of-the-art open-source TTS. ~82M params. Runs in real-time on CPU. Multiple high-quality voices. Free forever.

pip install kokoro-onnx sounddevice soundfile numpy

from kokoro_onnx import Kokoro
import sounddevice as sd
import soundfile as sf

# Loads model on first run (~300MB download)
kokoro = Kokoro("kokoro-v0_19.onnx", "voices.bin")

# Generate speech — pick any voice
samples, sample_rate = kokoro.create(
    text="Hello! This is Kokoro TTS running completely offline on CPU.",
    voice="af_bella",   # American female, warm natural tone
    speed=1.0,          # 0.5=slow, 1.0=normal, 1.5=fast
    lang="en-us"
)

# Play it live
sd.play(samples, sample_rate)
sd.wait()

# Save to file
sf.write("output.wav", samples, sample_rate)
print("Saved output.wav")

# All available voices:
# af, af_bella, af_sarah, af_nicole  — American female
# am_adam, am_michael                — American male
# bf_emma, bf_isabella               — British female
# bm_george, bm_lewis                — British male

Option C — F5-TTS (Zero-shot voice cloning from 5 seconds)

Clone ANY voice from just 5–15 seconds of clear audio. Open-source. GPU recommended (RTX 3060+) but works on CPU too (slowly).

pip install f5-tts

# ── Command line (simplest) ───────────────────────────
f5-tts_infer-cli \
  --model F5TTS \
  --ref_audio "reference_voice.wav" \
  --ref_text "This is what the speaker says in the reference clip." \
  --gen_text "Now say this sentence in the exact same voice!" \
  --output_dir ./output

# ── Python API ────────────────────────────────────────
from f5_tts.api import F5TTS

tts = F5TTS()
wav, sr, _ = tts.infer(
    ref_file="reference_voice.wav",
    ref_text="Text spoken in the reference clip",
    gen_text="Generate this in the same voice!",
    file_wave="cloned_output.wav",
    seed=42    # same seed = reproducible output
)
print(f"Generated {len(wav)/sr:.2f} seconds of audio")

# Tips for best results:
# - Reference audio: 5–15 seconds, very clear, no background noise
# - Avoid music, multiple speakers, or phone-quality recordings
# - The reference text must EXACTLY match what is said in the clip

Option D — Full Voice Assistant (Listen + Think + Speak)

Combine Whisper (speech → text) + Claude (text → text) + Kokoro (text → speech) into a complete voice bot that listens, reasons, and talks back.

pip install openai-whisper sounddevice soundfile numpy kokoro-onnx anthropic

import sounddevice as sd
import soundfile as sf
import numpy as np
import whisper
import anthropic
from kokoro_onnx import Kokoro

# ── Load models once at startup ───────────────────────
stt_model  = whisper.load_model("base")          # ~74MB
tts_model  = Kokoro("kokoro-v0_19.onnx", "voices.bin")
llm_client = anthropic.Anthropic(api_key="YOUR_KEY")
history    = []

def listen(seconds=5, sr=16000):
    print("🎤 Listening...")
    audio = sd.rec(int(seconds * sr), samplerate=sr, channels=1)
    sd.wait()
    sf.write("_temp.wav", audio.flatten(), sr)
    result = stt_model.transcribe("_temp.wav")
    text = result["text"].strip()
    print(f"You said: {text}")
    return text

def think(user_text):
    history.append({"role": "user", "content": user_text})
    resp = llm_client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=150,
        system="You are a helpful voice assistant. Keep answers to 1-2 sentences.",
        messages=history
    )
    reply = resp.content[0].text
    history.append({"role": "assistant", "content": reply})
    print(f"AI: {reply}")
    return reply

def speak(text):
    samples, sr = tts_model.create(text, voice="af_bella")
    sd.play(samples, sr)
    sd.wait()

# ── Main loop ─────────────────────────────────────────
print("Voice assistant ready! Press Ctrl+C to quit.")
while True:
    user_text = listen(seconds=5)
    if user_text:
        reply = think(user_text)
        speak(reply)

Training a TTS Model from Scratch (Advanced)

For a custom voice trained on your own dataset. Uses the Coqui TTS framework — the gold standard for open-source TTS training.

pip install coqui-tts

# ── Dataset format needed ─────────────────────────────
# Folder: dataset/
#   wavs/001.wav  002.wav  003.wav ...
#   metadata.csv:
#     001|Hello, welcome to my training dataset.
#     002|The quick brown fox jumps over the lazy dog.
# Min: 1 hour of clean audio. Better: 5-10+ hours.
# Audio: 22050 Hz, mono, WAV, noise-free.

# ── Train VITS (best quality) ─────────────────────────
from TTS.trainer import Trainer, TrainingArgs
from TTS.tts.configs.vits_config import VitsConfig
from TTS.tts.models.vits import Vits, VitsDataset

config = VitsConfig(
    audio=dict(sample_rate=22050),
    batch_size=32,
    epochs=1000,
    text_cleaner="english_cleaners",
    use_phonemes=True,
    phoneme_language="en-us",
    output_path="output/my_voice_model",
    datasets=[{"name":"ljspeech",
               "meta_file_train":"metadata.csv",
               "path":"dataset/"}],
)
model = Vits(config)
trainer = Trainer(TrainingArgs(), config,
                  model=model,
                  output_path=config.output_path)
trainer.fit()

# ── Synthesize with your trained voice ───────────────
from TTS.api import TTS
tts = TTS(model_path="output/my_voice_model/best_model.pth",
          config_path="output/my_voice_model/config.json")
tts.tts_to_file(
    text="I trained this voice myself from scratch!",
    file_path="my_voice.wav"
)

Option A — Stability AI API (Zero setup)

pip install stability-sdk pillow

import stability_sdk.interfaces.gooseai.generation.generation_pb2 as generation
from stability_sdk import client as stability_client
import io
from PIL import Image

api = stability_client.StabilityInference(
    key="YOUR_STABILITY_KEY",
    engine="stable-diffusion-xl-1024-v1-0",
)

answers = api.generate(
    prompt="A majestic golden cat riding a rocket through the cosmos, "
           "cinematic lighting, 8K, highly detailed, digital art",
    seed=42,
    steps=30,          # quality (20–50 recommended)
    cfg_scale=7.5,     # prompt adherence (5–12)
    width=1024,
    height=1024,
    samples=1,
)

for resp in answers:
    for artifact in resp.artifacts:
        if artifact.type == generation.ARTIFACT_IMAGE:
            img = Image.open(io.BytesIO(artifact.binary))
            img.save("output.png")
            print("Saved output.png!")

Option B — FLUX.1 Locally (Best open-source 2024)

FLUX.1 [schnell] generates stunning images in just 4 steps. Requires ~12GB VRAM (quantized) or ~24GB full.

pip install diffusers transformers accelerate torch --upgrade

from diffusers import FluxPipeline
import torch

pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-schnell",   # fast 4-step version
    torch_dtype=torch.bfloat16
).to("cuda")

image = pipe(
    prompt="A photorealistic golden astronaut cat floating in space, "
           "NASA style photo, ultra detailed, 8K",
    height=1024,
    width=1024,
    guidance_scale=0.0,          # FLUX-schnell: use 0
    num_inference_steps=4,       # only 4 steps needed!
    max_sequence_length=256,
    generator=torch.Generator("cpu").manual_seed(42)
).images[0]

image.save("flux_output.png")
print("Done!")

# FLUX.1 [dev] — higher quality, 50 steps, guidance_scale=3.5
# pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev",...)

Option C — DreamBooth: Fine-tune on Your Own Face

Teach a model to generate YOUR face (or any object/style) from just 10–20 reference photos. Uses LoRA — requires ~12GB VRAM, ~15 minutes training.

pip install diffusers transformers accelerate bitsandbytes

# ── Step 1: Prepare 10-20 photos of your subject ─────
# Put them in: data/my_subject/
# Mix of angles, expressions, lighting — more variety = better

# ── Step 2: Train DreamBooth LoRA ─────────────────────
# Download training script:
# wget https://raw.githubusercontent.com/huggingface/diffusers/main/
#      examples/dreambooth/train_dreambooth_lora_sdxl.py

accelerate launch train_dreambooth_lora_sdxl.py \
  --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" \
  --instance_data_dir="data/my_subject" \
  --instance_prompt="a photo of sks person" \
  --output_dir="lora_weights/my_face" \
  --mixed_precision="fp16" \
  --resolution=1024 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --learning_rate=1e-4 \
  --max_train_steps=500 \
  --seed=42

# ── Step 3: Generate with your fine-tuned identity ───
from diffusers import StableDiffusionXLPipeline
import torch

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
).to("cuda")
pipe.load_lora_weights("lora_weights/my_face")

image = pipe(
    "a photo of sks person as a medieval knight, "
    "epic portrait, cinematic lighting",
    guidance_scale=7.5,
    num_inference_steps=30,
).images[0]
image.save("me_as_knight.png")

Option D — Full Image Generation Web App with Gradio

A complete web interface deployed on HuggingFace Spaces (free). Users can type prompts and generate images through a browser.

pip install gradio diffusers torch accelerate

import gradio as gr
import torch
from diffusers import FluxPipeline

# Load model once at startup
pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-schnell",
    torch_dtype=torch.bfloat16
).to("cuda")

def generate(prompt, steps, seed):
    gen = torch.Generator("cpu").manual_seed(int(seed))
    image = pipe(
        prompt,
        num_inference_steps=int(steps),
        guidance_scale=0.0,
        generator=gen,
    ).images[0]
    return image

demo = gr.Interface(
    fn=generate,
    inputs=[
        gr.Textbox(label="Prompt",
                   placeholder="A golden cat riding a rocket..."),
        gr.Slider(1, 8, value=4, step=1, label="Steps"),
        gr.Number(value=42, label="Seed"),
    ],
    outputs=gr.Image(label="Generated Image"),
    title="🎨 FLUX Image Generator",
    description="Generates high-quality images in just 4 steps!",
)
demo.launch(share=True)  # share=True → public URL

Option A — Runway API (Easiest, best quality)

pip install runwayml requests

import runwayml, requests, time

client = runwayml.RunwayML(api_key="YOUR_RUNWAY_KEY")

# Image-to-video (most reliable method)
task = client.image_to_video.create(
    model="gen3a_turbo",
    prompt_image="https://example.com/dog.jpg",  # start frame
    prompt_text="A golden retriever running through autumn leaves, "
                "cinematic slow motion, shallow depth of field",
    duration=5,          # 5 or 10 seconds
    ratio="1280:720",
)

task_id = task.id
while True:
    task = client.tasks.retrieve(task_id)
    print(f"Status: {task.status}")
    if task.status in ("SUCCEEDED", "FAILED"):
        break
    time.sleep(5)

if task.status == "SUCCEEDED":
    r = requests.get(task.output[0])
    with open("output.mp4", "wb") as f:
        f.write(r.content)
    print("Saved output.mp4!")

Option B — CogVideoX Local (12–16GB VRAM, great quality)

pip install diffusers transformers accelerate torch imageio[ffmpeg]

from diffusers import CogVideoXPipeline
from diffusers.utils import export_to_video
import torch

pipe = CogVideoXPipeline.from_pretrained(
    "THUDM/CogVideoX-5b",
    torch_dtype=torch.bfloat16
).to("cuda")

# Memory optimizations for 12-16GB VRAM
pipe.enable_model_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

video = pipe(
    prompt="A bustling Tokyo street at night, neon signs reflecting on wet "
           "pavement, people walking with umbrellas, cinematic footage, 4K",
    num_inference_steps=50,
    num_frames=49,       # ~6 seconds at 8fps
    guidance_scale=6.0,
    generator=torch.Generator("cuda").manual_seed(42),
).frames[0]

export_to_video(video, "tokyo_night.mp4", fps=8)
print("Saved tokyo_night.mp4")

Option C — Wan 2.1 (Best open-source, 16GB VRAM for 480p)

pip install diffusers transformers accelerate torch imageio[ffmpeg]

from diffusers import AutoencoderKLWan, WanPipeline
from diffusers.utils import export_to_video
import torch

pipe = WanPipeline.from_pretrained(
    "Wan-AI/Wan2.1-T2V-14B-Diffusers",
    torch_dtype=torch.bfloat16
).to("cuda")

pipe.enable_model_cpu_offload()
pipe.vae.enable_tiling()

output = pipe(
    prompt="A majestic eagle soaring over snow-capped mountains at sunrise. "
           "Cinematic 4K footage. Golden hour light. Ultra detailed.",
    negative_prompt="blurry, low quality, static, watermark",
    height=480,
    width=832,
    num_frames=81,            # ~5 seconds at 16fps
    guidance_scale=5.0,
    num_inference_steps=50,
    generator=torch.Generator("cpu").manual_seed(42),
).frames[0]

export_to_video(output, "eagle.mp4", fps=16)
print("Saved eagle.mp4")

Option D — Animate Any Photo (Image-to-Video with SVD)

pip install diffusers transformers pillow torch accelerate imageio[ffmpeg]

from diffusers import StableVideoDiffusionPipeline
from diffusers.utils import load_image, export_to_video
import torch

pipe = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid-xt-1-1",
    torch_dtype=torch.float16, variant="fp16"
).to("cuda")
pipe.enable_model_cpu_offload()

image = load_image("your_photo.jpg").resize((1024, 576))

frames = pipe(
    image,
    motion_bucket_id=127,     # 1=subtle motion, 255=strong motion
    noise_aug_strength=0.02,
    num_frames=25,            # ~4 seconds
    generator=torch.manual_seed(42),
).frames[0]

export_to_video(frames, "animated.mp4", fps=6)
print("Your photo is now a video!")

Option E — Full Automated Video Pipeline (Topic → Narrated Video)

The complete pipeline: Claude writes a script, FLUX generates scene images, SVD animates them, Kokoro adds narration, MoviePy combines everything into a finished video.

pip install anthropic diffusers kokoro-onnx moviepy imageio[ffmpeg] soundfile

import anthropic, torch, soundfile as sf
from diffusers import FluxPipeline, StableVideoDiffusionPipeline
from diffusers.utils import load_image, export_to_video
from kokoro_onnx import Kokoro
from moviepy.editor import VideoFileClip, AudioFileClip, concatenate_videoclips

# ── Load all models ───────────────────────────────────
print("Loading models...")
flux   = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16).to("cuda")
svd    = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid-xt-1-1",
    torch_dtype=torch.float16).to("cuda")
tts    = Kokoro("kokoro-v0_19.onnx", "voices.bin")
claude = anthropic.Anthropic(api_key="YOUR_KEY")

def write_script(topic, n=4):
    """Claude writes a 4-scene video script."""
    resp = claude.messages.create(
        model="claude-sonnet-4-20250514", max_tokens=600,
        messages=[{"role":"user","content":
            f"Write a {n}-scene documentary video script about: {topic}\n"
            "Format each scene EXACTLY as:\n"
            "SCENE N: [one sentence visual description] | NARRATION: [one sentence voiceover]\n"
            "Keep both parts SHORT (under 20 words each)."}]
    )
    scenes = []
    for line in resp.content[0].text.split("\n"):
        if "SCENE" in line and "|" in line:
            vis = line.split("|")[0].split(":",1)[1].strip()
            nar = line.split("|")[1].replace("NARRATION:","").strip()
            scenes.append({"visual": vis, "narration": nar})
    return scenes[:n]

def gen_image(prompt, n):
    img = flux(prompt, num_inference_steps=4,
               guidance_scale=0.0, height=576, width=1024).images[0]
    img.save(f"scene_{n:02d}_img.png")
    return f"scene_{n:02d}_img.png"

def animate(img_path, n):
    img = load_image(img_path).resize((1024, 576))
    frames = svd(img, motion_bucket_id=90, num_frames=25).frames[0]
    path = f"scene_{n:02d}_vid.mp4"
    export_to_video(frames, path, fps=6)
    return path

def narrate(text, n):
    audio, sr = tts.create(text, voice="af_bella")
    path = f"scene_{n:02d}_nar.wav"
    sf.write(path, audio, sr)
    return path

def combine(scenes_data, output="final_video.mp4"):
    clips = []
    for vp, ap in scenes_data:
        video = VideoFileClip(vp)
        audio = AudioFileClip(ap)
        dur   = max(audio.duration, video.duration)
        clip  = video.loop(duration=dur).set_audio(
                    audio.subclip(0, min(audio.duration, dur)))
        clips.append(clip.subclip(0, dur))
    concatenate_videoclips(clips).write_videofile(
        output, fps=6, codec="libx264", audio_codec="aac")
    return output

# ── Run the full pipeline ─────────────────────────────
TOPIC = "The wonders of the deep ocean"
print(f"\n🎬 Generating video: '{TOPIC}'\n")

scenes = write_script(TOPIC)
print(f"Script: {len(scenes)} scenes written")

results = []
for i, scene in enumerate(scenes):
    print(f"Scene {i+1}/{len(scenes)}: {scene['visual'][:50]}...")
    img = gen_image(scene["visual"], i+1)
    vid = animate(img, i+1)
    nar = narrate(scene["narration"], i+1)
    results.append((vid, nar))

final = combine(results, "ocean_documentary.mp4")
print(f"\n✅ Done! Saved to: {final}")