Iterators in Python

Introduction

Iterators are objects that allow to traverse through all the elements of a collection, regardless of its specific implementation. Iterator object must implement two special methods, __iter__() and __next__(), collectively called the iterator protocol. An object is called iterable if we can get an iterator from it. Most of built-in containers like: list, tuple, string etc. are iterables. The iter() function (which in turn calls the __iter__() method) returns an iterator from them.

Use the next() function to manually iterate through all the items of an iterator. When there is no more data to be returned, it will raise StopIteration.

Syntax

iter() function syntax is

iter(object[, sentinel]

It return an iterator object. Object must be a collection object which supports the iteration protocol, or it must support the sequence protocol. If the second argument (sentinel) is given, then object must be a callable object. Iterator created in this case will call object with no arguments for each call to its next() method. If the value returned is equal to sentinel, StopIteration will be raised, otherwise the value will be returned.

Below example demonstrate the use of iter() function using List.

# Define list
my_list = [4, 7, 0, 3]

# Get iterator
my_iter = iter(my_list)

# Iterate through it using next() 
#Output: 4
print(next(my_iter))

#Output: 7
print(next(my_iter))

#Output: 0
print(my_iter.__next__())

#Output: 3
print(my_iter.__next__())

# This will raise error, no items left
next(my_iter)

Loop with  Iterators

for loop can iterate over any iterable. Internally, the for loop creates an iterator object, iter_obj by calling iter() on the iterable. Inside the loop, it calls next() to get the next element and executes the body of the for loop with this value. After all the items exhaust, StopIteration is raised which is internally caught and the loop ends.

for element in iterable:
  # do something with element
  
  
# Behind the scene for the above loop  
# Create an iterator object from that iterable
iter_obj = iter(iterable)

# infinite loop
while True:
  try:
    # get the next item
    element = next(iter_obj)
    # do something with element
  except StopIteration:
    # if StopIteration is raised, break from loop
    break

Iterator with Sentinel

When a sentinel value is supplied to iter(), it will assume that first argument is callable (without arguments). iter() will call callable repeatedly until it returns the sentinel value. Afterwards, the iterator would stop. Sentinel value makes it possible to detect the end of the data.

One useful application of the second form of iter() is to build a block-reader. For example:

blocks = []
read_block = partial(f.read, 32)
for block in iter(read_block, ''):
    blocks.append(block)

The sentinel value in this case is an empty string. read_block takes no input and reads a constant size.

Building Own Iterator

To build an iterator from scratch we have to implement the methods __iter__() and __next__(). The __iter__() method returns the iterator object itself.  The __next__() method must return the next item in the sequence. On reaching the end, and in subsequent calls, it must raise StopIteration.

class PowTwo:
  """Class to implement an iterator
  of powers of two"""

  def __init__(self, max = 0):
    self.max = max

  def __iter__(self):
    self.n = 0
    return self

  def __next__(self):
    if self.n <= self.max:
      result = 2 ** self.n
      self.n += 1
      return result
    else:
      raise StopIteration
      

# Using above iterator      
for i in PowTwo(5):
  print(i)
  
# Output
# 1
# 2
# 4
# 8
# 16
# 32

Generators

Generators is an easier way to create iterators using a keyword yield from a function. Inside the while loop when it reaches to the yield statement, the value of low is returned and the generator state is suspended. During the second next call the generator resumed where it freeze-ed before and then the value of low is increased by one. It continues with the while loop and comes to the yield statement again.

def counter_generator(low, high):
  while low <= high:
   yield low
   low += 1

for i in counter_generator(5,10):
  print(i, end=' ')