tqdm - Documentation

What is tqdm?

tqdm (pronounced “taqadum,” which is Arabic for “progress”) is a fast, extensible progress bar for Python and CLI. It displays a progress bar in your terminal, providing visual feedback on the progress of your long-running loops. It’s designed to be incredibly easy to use, highly customizable, and efficient even for very large iterations. tqdm automatically handles various situations, including nested loops and asynchronous operations.

Why use tqdm?

Using tqdm offers several significant advantages:

Installation

The simplest way to install tqdm is using pip:

pip install tqdm

For conda users:

conda install -c conda-forge tqdm

Basic Usage

Adding a tqdm progress bar to your code is incredibly straightforward. Simply wrap your iterable (e.g., a list, range, or generator) with the tqdm() function:

from tqdm import tqdm

for i in tqdm(range(100)):
    # Your time-consuming operation here
    # ... some code that takes time ...
    pass

# Or with a list:
my_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
for item in tqdm(my_list):
    # Process each item
    # ... some code ...
    pass

# Specifying the total iterations explicitly (useful for generators):
for i in tqdm(range(1000), total=1000):
    # ... Your code ...
    pass

This will display a progress bar showing the iteration progress, estimated time remaining, and overall speed. More advanced usage and customization options are detailed in subsequent sections of this manual.

Core Functionality

The tqdm function

The core of tqdm is the tqdm() function. It takes an iterable as its first argument and returns an iterator that wraps the original iterable, displaying a progress bar during iteration. The basic signature is:

tqdm(iterable=None, desc=None, total=None, leave=True, file=sys.stdout, ncols=None, mininterval=0.1, maxinterval=10.0, miniters=None, ascii=False, disable=False, unit='it', unit_scale=False, dynamic_ncols=False, bar_format=None, initial=0, position=None, postfix=None, colour=None, **kwargs)

While many arguments have sensible defaults, understanding them unlocks tqdm’s power. Key arguments include:

Iterating with tqdm

The simplest way to use tqdm is to wrap your iterable:

from tqdm import tqdm
for i in tqdm(range(100)):
    # Your code here
    pass

tqdm handles the iteration automatically, updating the progress bar with each step. For iterables whose length isn’t known upfront (like generators), provide the total argument:

import time
def my_generator(n):
    for i in range(n):
        time.sleep(0.1)
        yield i

for i in tqdm(my_generator(100), total=100):
    pass

Customizing the progress bar

tqdm allows extensive customization through various arguments and the bar_format parameter. bar_format uses a mini-templating language:

Example:

from tqdm import tqdm
for i in tqdm(range(100), desc="Processing", bar_format="{l_bar}{bar}| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, {rate_fmt}{postfix}]"):
  # ... your code ...
  pass

Consult the documentation for a complete list of format specifiers.

Handling nested loops

tqdm automatically handles nested loops using the position argument:

from tqdm import tqdm
outer_loop = tqdm(range(10)) # outer loop
for i in outer_loop:
    inner_loop = tqdm(range(100), position=1, leave=False) # inner loop, position 1
    for j in inner_loop:
        # Your code
        pass
    inner_loop.close() # this is important for nested loops

position determines the horizontal placement of the progress bar. leave=False prevents inner bars from cluttering the screen. Always call inner_loop.close() after each inner loop.

Disabling the progress bar

To disable the progress bar completely (e.g., for testing or when running in environments without a terminal), set disable=True:

from tqdm import tqdm
for i in tqdm(range(100), disable=True):
    pass

Working with different iterable types

tqdm works seamlessly with various iterable types, including lists, ranges, generators, and even custom iterators. The key is providing total when the length isn’t directly accessible:

from tqdm import tqdm
my_list = [1, 2, 3, 4, 5]
for item in tqdm(my_list):
    pass

# Using a generator:
def my_generator(n):
    for i in range(n):
        yield i
for i in tqdm(my_generator(100), total=100):
    pass

Remember to specify total when using generators or other iterables with unknown lengths for accurate progress reporting.

Advanced Usage

Manual updates

For situations where you can’t directly iterate over an iterable (e.g., asynchronous operations or manual loop control), use tqdm’s manual update capabilities:

from tqdm import tqdm
import time

with tqdm(total=100) as pbar:
    for i in range(100):
        # Perform some work
        time.sleep(0.1)
        pbar.update(1)  # Manually update the progress bar

pbar.update(n) increments the progress bar by n steps. pbar.set_postfix({'loss': 0.1}) can also add postfix information dynamically.

Using tqdm with threads and processes

When using threads or processes, ensure that each thread/process has its own tqdm instance or uses a thread-safe mechanism to avoid conflicts. For multiprocessing, consider using tqdm.external.concurrent.futures:

import concurrent.futures
from tqdm import tqdm
from tqdm.contrib.concurrent import process_map

def my_task(i):
    time.sleep(0.1)
    return i

results = process_map(my_task, range(100))

This avoids the overhead associated with managing progress bars across multiple processes. For simple threading cases, one tqdm instance usually suffices.

Integration with other libraries (e.g., NumPy, pandas)

tqdm integrates well with libraries like NumPy and pandas. For NumPy, you can use it directly with array iterations:

import numpy as np
from tqdm import tqdm

for i in tqdm(np.arange(100)):
    # Process each element of the array
    pass

For pandas, you can iterate over rows or columns:

import pandas as pd
from tqdm import tqdm

df = pd.DataFrame({'A': range(100)})
for index, row in tqdm(df.iterrows()):
    # Process each row
    pass

Writing custom progress bars

tqdm allows creating custom progress bars by subclassing tqdm.std.tqdm and overriding methods. This enables highly tailored visualisations:

from tqdm import tqdm
import sys

class MyTqdm(tqdm):
    def display(self, **kwargs):
        # Custom display logic here
        print("Custom progress bar:", self.n, "/", self.total, file=sys.stderr)

for i in MyTqdm(range(10)):
    pass

Using tqdm in Jupyter notebooks

In Jupyter Notebooks or other IPython environments, you need to explicitly enable the tqdm integration:

from tqdm import tqdm_notebook as tqdm

for i in tqdm(range(100)):
    pass

tqdm_notebook provides a progress bar that integrates seamlessly with the Jupyter environment.

Monitoring multiple iterables simultaneously

For monitoring multiple iterables, use the position argument in combination with appropriate leave settings:

from tqdm import tqdm
for i in tqdm(range(10), position=0, desc="First loop"):
    # ... some code ...
    for j in tqdm(range(100), position=1, leave=False, desc="Second loop"):
        # ... more code ...

This will display two separate progress bars, each updating independently, and closes the inner loop after it completes. You can use multiple tqdm instances simultaneously without specific coordination. However, avoid using the same position for multiple bars to avoid conflicts. Remember to call close() on inner progress bars if leave=False.

Configuration and Customization

Setting global options

tqdm allows setting global options that affect all subsequent progress bars. This is done using the tqdm.tqdm_gui or tqdm.tqdm function (depending on whether you want to change settings for the GUI or terminal output respectively) and passing keyword arguments:

from tqdm import tqdm

tqdm.set_lock(None)  # remove the lock mechanism for multiprocessing use cases on Windows
tqdm.pandas(desc="my bar!", unit="it") # sets default arguments for tqdm in pandas

#These will now be the default values for subsequent progress bars:
for i in tqdm(range(10)):
  pass

for i in tqdm(range(10)):
  pass

Many parameters that can be passed to tqdm() can also be set globally. Check the documentation for a complete list of configurable options.

Controlling progress bar appearance

Several parameters control the visual aspects of the progress bar:

Using different progress bar styles

tqdm supports various progress bar styles beyond the default. While not explicitly listed as selectable styles, the bar_format parameter allows creating different visual appearances by adjusting the format string and the choice of characters. For example, using ASCII characters (ascii=True) significantly alters the visual style.

Customizing bar format strings

As previously detailed, the bar_format parameter is crucial for customization. It uses a mini-template language to create highly specific progress bar displays. Experiment with different combinations of format specifiers to tailor your progress bar to your preferences.

For example, to show only the percentage, description, and current/total iterations:

from tqdm import tqdm

for i in tqdm(range(100), desc="My task", bar_format="{desc}: {percentage:3.0f}%|{bar}| {n_fmt}/{total_fmt}"):
  pass

Unit scaling and suffixes

The unit and unit_scale parameters control the units displayed in the progress bar. unit specifies the base unit (e.g., ‘it’, ‘bytes’, ‘files’), while unit_scale enables automatic scaling to larger units (e.g., K, M, G).

from tqdm import tqdm

# Using kilobytes as units and enabling auto-scaling
for i in tqdm(range(1000000), unit='B', unit_scale=True):
    pass

This will display units like KB, MB, or GB depending on the progress. Experiment with different unit combinations to optimize readability based on your iteration sizes.

Troubleshooting and Error Handling

Common errors and solutions

Several common issues might arise when using tqdm:

Debugging issues with tqdm

Debugging tqdm issues typically involves:

Performance considerations

While tqdm is designed to be efficient, using it can introduce minor performance overhead, especially on very short loops or when updating frequently. Consider these performance aspects:

Examples and Use Cases

Example: Progress bar for a loop

This shows a basic example of adding a progress bar to a simple loop:

from tqdm import tqdm
import time

for i in tqdm(range(100)):
    time.sleep(0.05)  # Simulate some work

This will display a progress bar showing the loop’s progress, estimated time remaining, and speed.

Example: Progress bar for file downloads

While tqdm doesn’t directly handle file downloads, you can integrate it with libraries like requests to display download progress:

import requests
from tqdm import tqdm
import os

url = "http://example.com/large_file.zip"  # Replace with your URL
filename = os.path.basename(url)

response = requests.get(url, stream=True)
total_size = int(response.headers.get('content-length', 0))
block_size = 1024  # 1 KB

with open(filename, 'wb') as file, tqdm(
        desc=filename,
        total=total_size,
        unit='iB',
        unit_scale=True,
        unit_divisor=1024,
) as bar:
    for data in response.iter_content(block_size):
        file.write(data)
        bar.update(len(data))

This example uses requests.iter_content to download data in chunks and updates the progress bar with each chunk.

Example: Progress bar for data processing

This shows how to use tqdm with data processing, for example, processing rows of a CSV file:

import csv
from tqdm import tqdm

with open('data.csv', 'r') as file:
    reader = csv.reader(file)
    header = next(reader)  # Skip the header row

    for row in tqdm(reader, total=1000, desc="Processing CSV"): #Assumes 1000 rows. Replace with accurate value if possible
        # Process each row here
        # ... your data processing code ...
        pass

Remember to replace 1000 with the actual number of rows in your CSV file if possible for precise progress reporting.

Example: Progress bar for training machine learning models

Many machine learning libraries can be integrated with tqdm. For example with a simple example using a custom training loop:

import numpy as np
from tqdm import tqdm

# Simulate training data
X = np.random.rand(1000, 10)
y = np.random.randint(0, 2, 1000)

# Simulate a simple training step
def train_step(X_batch, y_batch):
    # Simulate some computations
    time.sleep(0.01)
    return np.random.rand()

epochs = 10
batch_size = 32
num_batches = len(X) // batch_size

for epoch in tqdm(range(epochs), desc="Epochs"):
    for batch in tqdm(range(num_batches), desc="Batches", leave=False):
        X_batch = X[batch * batch_size:(batch + 1) * batch_size]
        y_batch = y[batch * batch_size:(batch + 1) * batch_size]
        loss = train_step(X_batch, y_batch)
        # ... other training logic ...

This example simulates a training loop; adapt it to your specific machine learning framework and training process. Remember to replace the simulated training step with your actual training code. leave=False on the inner loop keeps the output cleaner.

API Reference

This section provides a detailed overview of tqdm’s API. For the most up-to-date and comprehensive documentation, refer to the official tqdm documentation.

tqdm function parameters

The tqdm() function is the primary entry point. Its parameters are numerous and offer extensive control. Here’s a summary of key parameters (see the official documentation for a complete list):

Classes and methods

tqdm provides several classes, including:

Module-level functions

Besides the tqdm() function, the tqdm module offers additional functions:

These functions and classes provide different interfaces and functionalities to tailor progress bar integration to your needs, ranging from simple loop progress indicators to sophisticated visualizations in specialized environments. Refer to the official documentation for detailed explanations and usage examples for each function and class.