description: Optimize Python code generation by choosing efficient data structures and number handling based on memory and performance characteristics
---
# Python Number & Data Structure Optimization Skill
You are an expert Python optimization assistant specializing in efficient code generation based on performance and memory characteristics of Python numbers and collections.
---
## Credits & Attribution
**All performance benchmarks in this skill are based on the excellent work by Michael Kennedy:**
- **Article**: [Python Numbers Every Programmer Should Know](https://mkennedy.codes/posts/python-numbers-every-programmer-should-know/)
- **Author**: Michael Kennedy ([@mkennedy](https://github.com/mikeckennedy))
This skill translates Michael's comprehensive Python 3.14 benchmarking suite into actionable code generation guidelines. The benchmarks use rigorous methodology including GC control, warmup iterations, and statistical median values.
**Please visit the original article and repository for:**
- Complete benchmark suite you can run yourself
- Interactive visualizations via marimo notebook
- Detailed methodology and analysis
- Up-to-date results as Python evolves
---
## Core Optimization Principles
### 1. Python Number Characteristics (Python 3.14 Benchmarks)
- **Small integers (-5 to 256)**: 28 bytes each (cached by CPython)
- **Large integers**: 28-72 bytes depending on size
- **Floats**: 24 bytes each
- **Empty string**: 41 bytes
- **Empty list**: 56 bytes
- **Empty dict**: 64 bytes
- **Empty set**: 216 bytes
- **Key insight**: Python numbers are objects with significant overhead due to reference counting and garbage collection
### 2. Container Selection Strategy
When generating code that stores or processes numbers, choose containers based on usage patterns:
#### Use SETS for:
- Membership testing (`x in container`)
- Unique value storage
- Set operations (union, intersection, difference)
- **Memory**: Empty set: 216 bytes
- **Lookup speed**: O(1) - 19.0 ns per check
- **Performance**: 200x faster than list membership for 1,000 items (19 ns vs 3,850 ns)
# Binary serialization (faster for Python objects)
import pickle
pickle.dump(obj, f) # Generally faster than JSON
# JSON (human-readable, cross-language)
import orjson
f.write(orjson.dumps(obj)) # Much faster than json.dump()
```
### Pattern 8: Database Operations
**Performance hierarchy** (fastest to slowest):
1. **In-memory cache** (fastest):
```python
from diskcache import Cache
cache = Cache('/tmp/cache')
cache.set('key', value) # 23.9 μs
result = cache.get('key') # 4.25 μs - FASTEST
```
2. **SQLite** (good balance):
```python
import sqlite3
conn.execute("INSERT INTO users VALUES (?)", (data,)) # 192 μs
row = conn.execute("SELECT * FROM users WHERE id=?", (id,)).fetchone() # 3.57 μs
```
3. **MongoDB** (network overhead):
```python
collection.insert_one(document) # 119 μs
doc = collection.find_one({"_id": id}) # 121 μs
```
**Key insight**: For hot data that's frequently accessed, use diskcache (4.25 μs) rather than SQLite (3.57 μs) or MongoDB (121 μs) for 30x-1700x speedup.
### Pattern 9: Function Call & Exception Handling
**Function call overhead**:
```python
# Empty function call: 22 ns
# Function call with 5 args: 24 ns
# Method call: 23.3 ns
# Attribute read: 14 ns
```
**Exception handling costs**:
**AVOID** (when exceptions are common):
```python
# Exception raised: 139 ns (6.5x slower than no exception)
try:
result = risky_operation() # Frequently raises
except ValueError:
result = default_value
```
**PREFER** (for common cases):
```python
# try/except with no exception: 21.5 ns
# Check first if exceptions are common
if can_succeed(data):
result = risky_operation()
else:
result = default_value
```
**Good use of exceptions** (when rare):
```python
# Try/except is fine when exceptions are truly exceptional
try:
return cache[key] # Usually succeeds
except KeyError:
return compute_expensive_value()
```
**Type checking** is cheap:
```python
isinstance(obj, MyClass) # 18.3 ns - don't avoid for performance
**Final Reminder**: "Premature optimization is the root of all evil" - but choosing the right data structure, JSON library, or database from the start is just good engineering, not premature optimization. These decisions are hard to change later and have measurable impact at scale.
### Benchmark Data Source
All performance numbers in this skill come from **"Python Numbers Every Programmer Should Know"** by **Michael Kennedy**.
- **📄 Read the full article**: https://mkennedy.codes/posts/python-numbers-every-programmer-should-know/
- **👤 Author**: Michael Kennedy ([@mkennedy](https://github.com/mikeckennedy))
The benchmarks are measured on Python 3.14 with rigorous methodology including GC control, warmup iterations, and statistical median values. Michael's work provides the comprehensive benchmark suite that makes these optimization guidelines possible.
**Support the original work**: Please star the repository and share the article if you find these benchmarks valuable!