Creating a generator object with generator expressions

Generator expressions are similar to list comprehensions. They help us to create a generator object with minimal code. We can create generator objects that do not hold the entire object in memory before iteration. Let us create a list and a generator object and look at the difference between the two.

#Creating a list
numbers_list = [num for num in range(5)]
#Creating a generator object
numbers_generatorObject = (num for num in range(5))
#output
numbers_list
numbers_generatorObject

In the above code, we have created a list and a generator object for numbers. The syntax will be very much similar, but the difference will be the type of parentheses that we use. When you execute the above code, this will be the output.

[0, 1, 2, 3, 4]
at 0x7f776b77dd58>

You can observe here that the numbers_list is a list, so the numbers were printed on the command line. Whereas the numbers_generatorObject has got created as a generator object. You can also see the location at which the generator object is created.

Evaluating generator performance

As I mentioned before, generators optimize memory. Let’s consider the same example that we have taken above and increase numbers up to 150. Let us see how much size the list and generator objects take to hold the same numbers. Here is a small program that we can use to get the size.

import sys
#Creating a list
numbers_list = [num for num in range(150)]
print("The size of the list is", sys.getsizeof(numbers_list))
#Creating a generator object
numbers_generatorObject = (num for num in range(150))
print("The size of the generator is", sys.getsizeof(numbers_generatorObject))
The output for the above program will be as follows.
The size of the list is 1448
The size of the generator is 88

You can see that the list took 1448 bytes, whereas the generator object is only 88 bytes. You can observe a huge difference when you work with a larger dataset.

Acquire NLP certification by enrolling in the HKR NLP Training program in Hyderabad!

Subscribe to our YouTube channel to get new updates..!

Advanced generator methods

Generators provide three special methods which were introduced in PEP 342 and is available since the python 2.5 version.

send() – It is a method used to send values to the generator iterators. The value specified in the send() method is used to continue with the next yield. If we do not pass any value to the send() method, it will be equivalent to the next() call. 

throw() – It is a method used to throw exceptions from the generator. We can add a throw() method when we might need to catch an exception. The value or exception specified in the throw() method will be sent to the caller.

close() – It is a method used to stop a generator. This will be really helpful when we want to stop a program when it goes into an infinity loop. 

Realted Article, List to String in Python !

Creating data pipelines with generators

When you have a huge dataset that needs processing, we can’t really do all the processing at a single place. To avoid this, we can create a pipeline. Each method in a pipeline receives an item, applies transformations on it, and returns the transformed item. This way, we can even change the order of transformations.

For example, if we want to process data in a CSV file, we have to read all the lines of data in the file. Identify the column names,split each row into a list of values,and filter out any unwanted data.Create dictionaries for the column names and lists.Apply the transformations that you want on the rows. All the created generators will function as a pipeline.

  Top 50 frequently asked Python interview Question and answers !

Conclusion

As you have learned, generators simplify code. Generator expressions simplify code much further. They might be a little confusing at first. But when you put enough effort and practice them, you will get to understand them completely. Then you will know how easy it is to code in python with the help of generators.

Generators are especially useful when dealing with huge datasets.We can create pipelines and make the developer’s job easier.The calculations on data will be performed on-demand. We can use generators to simulate concurrency.Enjoy coding with python!

Related Articles:

1. Python Partial Functions

2. Python Split Method

3. Running Scripts in Python

4. Python List Length