Lepton AI

Use Lepton AI’s serverless AI endpoints for chat completions and speech-to-text through Portkey.

Quick Start

Get started with Lepton AI in under 2 minutes:

Python

from portkey_ai import Portkey

# 1. Install: pip install portkey-ai
# 2. Add @lepton provider in model catalog
# 3. Use it:

portkey = Portkey(api_key="PORTKEY_API_KEY")

response = portkey.chat.completions.create(
    model="@lepton/llama-3.1-8b",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

import Portkey from 'portkey-ai'

// 1. Install: npm install portkey-ai
// 2. Add @lepton provider in model catalog
// 3. Use it:

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY"
})

const response = await portkey.chat.completions.create({
    model: "@lepton/llama-3.1-8b",
    messages: [{ role: "user", content: "Hello!" }]
})

console.log(response.choices[0].message.content)

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL

# 1. Install: pip install openai portkey-ai
# 2. Add @lepton provider in model catalog
# 3. Use it:

client = OpenAI(
    api_key="PORTKEY_API_KEY",  # Portkey API key
    base_url=PORTKEY_GATEWAY_URL
)

response = client.chat.completions.create(
    model="@lepton/llama-3.1-8b",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

import OpenAI from "openai"
import { PORTKEY_GATEWAY_URL } from "portkey-ai"

// 1. Install: npm install openai portkey-ai
// 2. Add @lepton provider in model catalog
// 3. Use it:

const client = new OpenAI({
    apiKey: "PORTKEY_API_KEY",  // Portkey API key
    baseURL: PORTKEY_GATEWAY_URL
})

const response = await client.chat.completions.create({
    model: "@lepton/llama-3.1-8b",
    messages: [{ role: "user", content: "Hello!" }]
})

console.log(response.choices[0].message.content)

# 1. Add @lepton provider in model catalog
# 2. Use it:

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -d '{
    "model": "@lepton/llama-3.1-8b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Add Provider in Model Catalog

Before making requests, add Lepton AI to your Model Catalog:

1. Go to Model Catalog → Add Provider
2. Select Lepton AI
3. Enter your Lepton API key
4. Name your provider (e.g., lepton)

Complete Setup Guide

See all setup options and detailed configuration instructions

---

Lepton AI Capabilities

Chat Completions

Generate chat completions with Lepton’s serverless models:

Python

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY", provider="@lepton")

response = portkey.chat.completions.create(
    model="llama-3.1-8b",
    messages=[{"role": "user", "content": "Write a haiku about AI"}]
)

print(response.choices[0].message.content)

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: 'PORTKEY_API_KEY',
    provider: '@lepton'
});

const response = await portkey.chat.completions.create({
    model: "llama-3.1-8b",
    messages: [{ role: "user", content: "Write a haiku about AI" }]
});

console.log(response.choices[0].message.content);

Speech-to-Text

Transcribe audio using Lepton’s Whisper models:

Python

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY", provider="@lepton")

with open("audio.mp3", "rb") as audio_file:
    transcription = portkey.audio.transcriptions.create(
        file=audio_file,
        model="whisper-large-v3"
    )

print(transcription.text)

import Portkey from 'portkey-ai';
import fs from 'fs';

const portkey = new Portkey({
    apiKey: 'PORTKEY_API_KEY',
    provider: '@lepton'
});

const transcription = await portkey.audio.transcriptions.create({
    file: fs.createReadStream('audio.mp3'),
    model: 'whisper-large-v3'
});

console.log(transcription.text);

Streaming

Enable streaming for real-time responses:

Python

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY", provider="@lepton")

stream = portkey.chat.completions.create(
    model="llama-3.1-8b",
    messages=[{"role": "user", "content": "Write a story about a robot"}],
    stream=True
)

for chunk in stream:
    print(chunk.choices[0].delta.content, end="")

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: 'PORTKEY_API_KEY',
    provider: '@lepton'
});

const stream = await portkey.chat.completions.create({
    model: "llama-3.1-8b",
    messages: [{ role: "user", content: "Write a story about a robot" }],
    stream: true
});

for await (const chunk of stream) {
    process.stdout.write(chunk.choices[0]?.delta?.content || '');
}

---

Supported Models

Lepton AI provides serverless access to various models:

Model	Description
llama-3.1-8b	Llama 3.1 8B model
llama-3-8b-sft-v1	Fine-tuned Llama 3
whisper-large-v3	Speech-to-text

Check Lepton’s documentation for the complete model list.

---

Next Steps

Gateway Configs

Add fallbacks, load balancing, and more

Observability

Monitor and trace your Lepton requests

Prompt Library

Manage and version your prompts

Metadata

Add custom metadata to requests

For complete SDK documentation:

SDK Reference

Complete Portkey SDK documentation