List Comprehensions in Python and Generator Expressions

Nov 21, 2023
/
10 min read
List Comprehensions in Python and Generator Expressions
Soner Ayberk
Soner Ayberk
Senior Backend Engineer / Team Lead

Do you know the difference between the following syntax?
[x for x in range(5)]
(x for x in range(5))
tuple(range(5))
This is exactly what differentiates Python from other languages and makes Python the best programming language. Coming from functional languages and being implemented in Python from early days, list comprehension became its distinctive feature.

Let’s dive deeper into the Python list comprehensions and generator expressions. DjangoStars team often uses list comprehensions in our projects, so if you need additional advice on Python or Django development, you just need to contact us.

Read Also: Debugging Python Applications with the PDB Module

4 Facts About the Lists

First off, a short review on the lists (arrays in other languages).

  • List is a type of data that can be represented as a collection of elements. Simple list looks like this – [0, 1, 2, 3, 4, 5]
  • Lists take all possible types of data and combinations of data as their components:
>>> a = 12
>>> b = "this is text"
>>> my_list = [0, b, ['element', 'another element'], (1, 2, 3), a]
>>> print(my_list)
[0, 'this is text', ['element', 'another element'], (1, 2, 3), 12]
  • Lists can be indexed. You can get access to any individual element or group of elements using the following syntax:
>>> a = ['red', 'green', 'blue']
>>> print(a[0])
red
  • Lists are mutable in Python. This means you can replace, add or remove elements.

How to create lists

There are 2 common ways how to create lists in Python:

>>> my_list = [0, 1, 1, 2, 3]

And less preferable:

>>> my_list = list()

Usually, list(obj) is used to transform another sequence into the list. For example we want to split string into separate symbols:

>>> string = "string"
>>> list(string)
['s', 't', 'r', 'i', 'n', 'g']

What is List Comprehension?

Often seen as a part of functional programming in Python, list comprehension allows you to create lists with less code. In short, it’s a truly Pythonic way of coding. Less code – more effectiveness.
Let’s look at the following example.
You create a list using a for loop and a range() function.

>>> my_list = []
>>> for x in range(10):
... my_list.append(x * 2)
...
>>> print(my_list)
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

And this is how the implementation of the previous example is performed using a list comprehension:

>>> comp_list = [x * 2 for x in range(10)]
>>> print(comp_list)
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

The above example is oversimplified to get the idea of syntax. The same result may be achieved simply using list(range(0, 19, 2)) function. However, you can use a more complex modifier in the first part of the comprehension or add a condition that will filter the list. Something like this:

>>> comp_list = [x ** 2 for x in range(7) if x % 2 == 0]
>>> print(comp_list)
[4, 16, 36]

Another available option is to use list comprehension to combine several lists and create a list of lists. At first glance, the syntax seems to be complicated. It may help to think of lists as outer and inner sequences.
It’s time to show the power of list comprehension when you want to create a list of lists by combining two existing lists.

>>> nums = [1, 2, 3, 4, 5]
>>> letters = ['A', 'B', 'C', 'D', 'E']
>>> nums_letters = [[n, l] for n in nums for l in letters]
#the comprehensions list combines two simple lists in a complex list of lists.
>>> print(nums_letters)
>>> print(nums_letters)
[[1, 'A'], [1, 'B'], [1, 'C'], [1, 'D'], [1, 'E'], [2, 'A'], [2, 'B'], [2, 'C'], [2, 'D'], [2, 'E'], [3, 'A'], [3, 'B'], [3, 'C'], [3, 'D'], [3, 'E'], [4, 'A'], [4, 'B'], [4, 'C'], [4, 'D'], [4, 'E'], [5, 'A'], [5, 'B'], [5, 'C'], [5, 'D'], [5, 'E']]
>>>

Let’s try it with text or, referring to it correctly, string object.

>>> iter_string = "some text"
>>> comp_list = [x for x in iter_string if x !=" "]
>>> print(comp_list)
['s', 'o', 'm', 'e', 't', 'e', 'x', 't']

The comprehensions are not limited to lists. You can create dicts and sets comprehensions as well with the Python set generator.

>>> dict_comp = {x:chr(65+x) for x in range(1, 11)}
>>> type(dict_comp)
<class 'dict'>
>>> print(dict_comp)
{1: 'B', 2: 'C', 3: 'D', 4: 'E', 5: 'F', 6: 'G', 7: 'H', 8: 'I', 9: 'J', 10: 'K'}
>>> set_comp = {x ** 3 for x in range(10) if x % 2 == 0}
>>> type(set_comp)
<class 'set'>
>>> print(set_comp)
{0, 8, 64, 512, 216}

When to Use List Comprehensions in Python

List comprehension is the best way to enhance the code readability, so it’s worth using whenever there is a bunch of data to be checked with the same function or logic – for example, KYC verification. If the logic is quite simple, for instance, it’s limited with `true` or `false` results, list comprehension can optimize the code and focus on the logic solely. For example:

>>> customers = [{"is_kyc_passed": False}, {"is_kyc_passed": True}]
>>> kyc_results = []
>>> for customer in customers:
...     kyc_results.append(customer["is_kyc_passed"])
...
>>> all(kyc_results)
False

There are many other ways how it can be implemented, but let’s have a look at the example with list comprehension:

>>> customers = [{"is_kyc_passed": False}, {"is_kyc_passed": True}]
>>> all(customer["is_kyc_passed"] for customer in customers)
False

Benefits of Using List Comprehensions

List comprehensions optimize the lists’ generation and help to avoid side effects as gibberish variables. As a result, you get more concise and readable code.
For a better understanding of what benefits list comprehensions brings to Python developers, one can also pay attention to the following:

  • Ease of code writing and reading. By using list comprehensions for Python list creation, developers can make their code easier to understand and reduce the number of lines, primarily by replacing for loops.
  • Improved execution speed. List comprehensions not only provide a convenient way to write code but also execute faster. Since performance is usually not considered one of the pros of using Python for web development, this aspect shouldn’t be neglected when programming and refactoring.
  • No modification of existing lists. A list comprehension call is a new list creation Python performs without changing the existing one. And this fact allows using of list comprehensions in a functional programming paradigm.

Difference Between Iterable and Iterator

It will be easier to understand the concept of Python list generators if you get the idea of iterables and iterators.
Iterable is a “sequence” of data, you can iterate over using a loop. The easiest visible example of iterable can be a list of integers – [1, 2, 3, 4, 5, 6, 7]. However, it’s possible to iterate over other types of data like strings, dicts, tuples, sets, etc.
Basically, any object that has iter() method can be used as an iterable. You can check it using hasattr() function in the interpreter.

>>> hasattr(str, '__iter__')
True
>>> hasattr(bool, '__iter__')
False

Iterator protocol is implemented whenever you iterate over a sequence of data. For example, when you use a for loop the following is happening on a background:

  • first iter() method is called on the object to convert it to an iterator object.
  • next() method is called on the iterator object to get the next element of the sequence.
  • StopIteration exception is raised when there are no elements left to call.
>>> simple_list = [1, 2, 3]
>>> my_iterator = iter(simple_list)
>>> print(my_iterator)
<list_iterator object at 0x7f66b6288630>
>>> next(my_iterator)
1
>>> next(my_iterator)
2
>>> next(my_iterator)
3
>>> next(my_iterator)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration

Generator Expressions

In Python, generators provide a convenient way to implement the iterator protocol. Generator is an iterable created using a function with a yield statement.
The main feature of generator expression is evaluating the elements on demand. When you call a normal function with a return statement the function is terminated whenever it encounters a return statement. In a function with a yield statement the state of the function is “saved” from the last call and can be picked up the next time you call a generator function.

>>> def my_gen():
... for x in range(5):
... yield x

A Python generator expression is an expression that returns a generator (generator object).
Generator expression in Python allows creating a generator on a fly without a yield keyword. However, it doesn’t share the whole power of generator created with a yield function. The syntax and concept is similar to list comprehensions:

>>> gen_exp = (x ** 2 for x in range(10) if x % 2 == 0)
>>> for x in gen_exp:
... print(x)
4
16
36
64

In terms of syntax, the only difference is that you use parentheses instead of square brackets. However, the types of data returned by Python generator expressions and list comprehensions differ.

>>> list_comp = [x ** 2 for x in range(10) if x % 2 == 0]
>>> gen_exp = (x ** 2 for x in range(10) if x % 2 == 0)
>>> print(list_comp)
[0, 4, 16, 36, 64]
>>> print(gen_exp)
<generator object <genexpr> at 0x7f600131c410>

The main advantage of generator over a list is that it takes much less memory. We can check how much memory is taken by both types using sys.getsizeof() method.
Note: in Python 2 using range() function can’t actually reflect the advantage in term of size, as it still keeps the whole list of elements in memory. In Python 3, however, this example is viable as the range() returns a range object.

>>> from sys import getsizeof
>>> my_comp = [x * 5 for x in range(1000)]
>>> my_gen = (x * 5 for x in range(1000))
>>> getsizeof(my_comp)
9024
>>> getsizeof(my_gen)
88

We can see this difference because while `list` creating Python reserves memory for the whole list and calculates it on the spot. In case of generator, we receive only ”algorithm”/ “instructions” how to calculate that Python stores. And each time we call for generator, it will only “generate” the next element of the sequence on demand according to “instructions”.
On the other hand, generator will be slower, as every time the element of sequence is calculated and yielded, function context/state has to be saved to be picked up next time for generating next value. That “saving and loading function context/state” takes time.
Note: Of course, there are different ways to provide Python ‘generator to list’ conversion, besides initial using square brackets where Python generates lists via list comprehension. If it’s necessary to convert a generator to a list, Python developers can use, for example, the list() function or the unpack operator *.

Boost your web development.

Get an experienced technical partner.

Learn more

Final Thoughts

The very first thing that might scare or discourage a newbie programmer is the scale of educational material. The trick here is to treat each concept as an option offered by language, you’re not expected to learn all the language concepts and modules all at once. There are always different ways to solve the same task. Take it as one more tool to get the job done.

Feel free to contact Django Stars if you need consulting or a dedicated team of Python developers.

Sign up for our newsletter

 

Frequently Asked Questions
What is the difference between a list generator and list comprehension?
When a list comprehension is called, Python creates an entire list — all of its elements are stored in memory, allowing for quick access to them. The point of generators is not to store all the elements in memory. It saves memory but requires the entire list to be generated each time it is called to get the next element.
Are generators useful in Python?
Generators in Python are appropriate to use in cases where you need to process large amounts of data, memory is important (RAM needs careful handling), and speed is not critical. If speed is important and memory is not, then it is not necessary to use generators.
When to use list comprehension in Python?
List comprehension is the best way to enhance code readability and speed up its execution. Also, a list comprehension call in Python creates a new list without modification of the existing one, allowing using list comprehensions in a functional programming paradigm.
How to generate a list in Python?
To generate a list in Python, add a generator expression to the code using the following syntax: generator = (expression for element in iterable if condition). For example: my_gen = (x**2 for x in range(10) if x%2 == 0). This differs from the Python list comprehension syntax by using parentheses instead of square brackets.
Are generators faster than lists in Python?
No. The advantage of a Python generator is that it doesn't store the list in RAM. Instead, it remembers the state. But recalculation every time the function is called takes more time comparing access to any element in the regular Python list.

Have an idea? Let's discuss!

Contact Us
Rate this article!
3 ratings, average: 4.67 out of 5
Very bad
Very good
Subscribe us

Latest articles right in
your inbox

Thanks for
subscribing.
We've sent a confirmation email to your inbox.

Subscribe to our newsletter

Thanks for joining us! 💚

Your email address *
By clicking “Subscribe” I allow Django Stars process my data for marketing purposes, including sending emails. To learn more about how we use your data, read our Privacy Policy .
We’ll let you know, when we got something for you.