Iterators and Generators

Status

This note is complete, reviewed, and considered stable.

Iterators

What is an Iterator?

An iterator is an object that implements two key methods:

1. __iter__(): Returns the iterator object itself. This method is required for an object to be considered iterable.
2. __next__(): Returns the next item in the sequence. If there are no more items, it raises a StopIteration exception.

In Python, most collections such as lists, tuples, and dictionaries are iterables. This means we can loop over them using a for loop. However, these collections are not iterators themselves. They are iterable objects because they implement the __iter__() method. When we pass an iterable object to the iter() function, it returns an iterator.

Example of Using an Iterator

# Creating an iterator for a list
my_list = [1, 2, 3, 4]
iterator = iter(my_list)

print(next(iterator))  # Output: 1
print(next(iterator))  # Output: 2
print(next(iterator))  # Output: 3
print(next(iterator))  # Output: 4
# print(next(iterator))  # This will raise StopIteration

Custom Iterator

We can create our own iterator by defining a class with __iter__() and __next__() methods.

class Countdown:
    def __init__(self, start):
        self.start = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.start <= 0:
            raise StopIteration
        else:
            self.start -= 1
            return self.start

countdown = Countdown(5)
for num in countdown:
    print(num)  # Output: 4, 3, 2, 1, 0

Advantages of Iterators

• Memory efficient: Iterators yield one item at a time, which makes them more memory-efficient than lists, especially for large datasets.
• Lazy evaluation: Iterators do not generate all items at once, which allows for processing large data sets one element at a time.

Generators

What is a Generator?

A generator is a special type of iterator that is defined using a function with the yield keyword. Instead of returning a value with return, the generator function produces a sequence of values one at a time using yield. The state of the generator is saved between calls, so the generator function can continue where it left off after each yield.

Example of a Simple Generator

def count_up_to(max):
    count = 1
    while count <= max:
        yield count
        count += 1

counter = count_up_to(3)
print(next(counter))  # Output: 1
print(next(counter))  # Output: 2
print(next(counter))  # Output: 3
# print(next(counter))  # This will raise StopIteration

Generator Expression

We can also create generators using generator expressions, which have a syntax similar to list comprehensions but with parentheses.

# Using generator expression
squares = (x * x for x in range(5))
for square in squares:
    print(square)  # Output: 0, 1, 4, 9, 16

Advantages of Generators

• Memory Efficient: Like iterators, generators do not store all values in memory at once. Instead, they generate values on the fly, which is especially useful for working with large datasets.
• Concise: Generators allow us to write cleaner and more concise code for producing sequences of data.
• Lazy Evaluation: Generators evaluate values lazily, which means values are only generated when requested.

When to Use Generators

• When working with large datasets, like reading lines from a file, or streaming data from an external source.
• When we want to avoid storing large amounts of data in memory at once.
• When we need an efficient way to handle sequences of data that may not fit in memory.

Comparison Between Iterators and Generators

Feature	Iterator	Generator
Creation	Explicitly define __iter__() and __next__()	Use a function with yield
State	Maintains its state manually	Automatically saves state using yield
Memory Usage	Can be memory-intensive for large sequences	More memory-efficient due to lazy evaluation
Syntax	More code needed to implement	More concise and readable using yield
Performance	Slightly slower due to manual state handling	More efficient as it uses lazy evaluation