Python 3.11 | What's new in python 3.11 | New features in python 3.11

Introduction:

Python 3.11 was released in October and it has a lot of interesting new features that excites all pythonistas. So what’s the reason for the excitement? What’s special in 3.11 that we didn’t have in 3.10. Well! there is a lot. Python 3.11 is 10–60 % faster than its predecessor as per the documentation. In this post, we will walk through the optimisations and see if the newer python version is as fast as they claim it to be.

Installation:

Python 3.11 is available for download in the below link.

https://www.python.org/downloads/release/python-3110/

If you’re using a mac os, download Python 3.11 using brew.

$ brew install python@3.11

==> Installing python@3.11 dependency: openssl@1.1
==> Pouring openssl@1.1--1.1.1s.monterey.bottle.tar.gz
🍺 /usr/local/Cellar/openssl@1.1/1.1.1s: 8,101 files, 18.5MB
==> Installing python@3.11
==> Pouring python@3.11--3.11.0.monterey.bottle.tar.gz
==> /usr/local/Cellar/python@3.11/3.11.0/bin/python3.11 -m ensurepip
==> /usr/local/Cellar/python@3.11/3.11.0/bin/python3.11 -m pip install -v --no-deps --no-index --upgrade --isolated --target=/usr/local/lib/python3.11/site-packages /usr/local/Cellar/python@3.11/3.11.0/Frameworks/Python.framework/Versions/3.11/lib/p
==> Caveats

Pyperf:

Before we get into the optimisations, allow me to give a brief introduction on how the execution times have been benchmarked.I have used the library to benchmark the execution time.The Python pyperf module is a utility to write, run and analyze benchmarks.

Features of pyperf:

Simple API to run reliable benchmarks
Automatically calibrate a benchmark for a time budget.
Spawn multiple worker processes.
Compute the mean and standard deviation.
Detect if a benchmark result seems unstable.
JSON format to store benchmark results.
Support multiple units: seconds, bytes and integer.

Installation:

$ pip install pyperf

Now that we have setup the pyperf module, let’s discuss the optimisations.

1. Faster CPython:

CPython 3.11 is 25% faster than CPython 3.10 on average according to benchmarks with pyperformance. The changes that are being done are part of something that Guido called: The “Shannon Plan”. For the 3.11 release, the plan was about making a lot of optimizations in two main areas:

Startup

2. Runtime.

I will write a separate blog for this to understand how it works under the hood.

2. Print f style formatting strings are faster:

Historically, up until python 3.10 the print-f style format strings were relatively slower compared to the f-strings. However, in python 3.11, the print-f style format strings are way faster than its predecessors. I had run the below code snippet from the terminal in both python 3.11 and python 3.9 to compare the execution times.

I’d say python 3.11 has succeeded in surprising us with its execution times.

import pyperf

my_stmt_pf = """
my_str = "Optimized 3.11"
lang = "python"
new_str = " This is a test string with %s in %s " % (my_str, lang)
"""

runner = pyperf.Runner()
runner.timeit(stmt=my_stmt_pf, name="string format")

Benchmark 3.11:

(py3.11-env) dkb@dkb-a01 practice % python new311.py
.....................
string format: Mean +- std dev: 126 ns +- 7 ns

(py3.11-env) dkb@dkb-a01 practice % python new311.py
.....................
string format: Mean +- std dev: 121 ns +- 12 ns

Benchmark 3.9:

(practo-env) dkb@dkb-a01 practice % python new311.py
.....................
string format: Mean +- std dev: 233 ns +- 20 ns

(practo-env) dkb@dkb-a01 practice % python new311.py
.....................
string format: Mean +- std dev: 206 ns +- 15 ns

The execution time of print-f style formatting in python 3.11 is ~80% faster than python 3.9.

While we are at it, let’s quickly check the execution times of f-strings. Please note that there are no optimisations specific to f-strings AFAIK. This is purely for our understanding.

F-strings benchmark

import pyperf

my_stmt_pf = """
my_str = "Optimized 3.11"
lang = "python"
new_str = f" This is a test string with {my_str} in {lang} "

"""

runner = pyperf.Runner()
runner.timeit(stmt=my_stmt_pf, name="string format")

Benchmark 3.11:

(py3.11-env) dkb@dkb-a01 practice % python new311.py
.....................
string format: Mean +- std dev: 99.3 ns +- 2.2 ns

(py3.11-env) dkb@dkb-a01 practice % python new311.py
.....................
string format: Mean +- std dev: 107 ns +- 7 ns

Benchmark 3.9:

(practo-env) dkb@dkb-a01 practice % python new311.py
.....................
string format: Mean +- std dev: 97.0 ns +- 1.2 ns

(practo-env) dkb@dkb-a01 practice % python new311.py
.....................
string format: Mean +- std dev: 103 ns +- 7 ns
(practo-env) dkb@dkb-a01 practice %

The f-strings execution times are more or less the same. However, if you carefully notice, the f-strings are faster than the print-f style string formatting in their corresponding python versions.

3. Faster int division for python bignums:

import pyperf

my_stmt = """
x = 10**1000
y = x//2
"""

runner = pyperf.Runner()
runner.timeit(stmt=my_stmt, name="string format")

Benchmark python 3.11:

(py3.11-env) dkb@dkb-a01 practice % python new311.py
.....................
string format: Mean +- std dev: 3.65 us +- 0.13 us

(py3.11-env) dkb@dkb-a01 practice % python new311.py
.....................
string format: Mean +- std dev: 3.94 us +- 0.31 us
(py3.11-env) dkb@dkb-a01 practice %

Benchmark python 3.9:

(practo-env) dkb@dkb-a01 practice % python new311.py
.....................
string format: Mean +- std dev: 4.00 us +- 0.08 us

(practo-env) dkb@dkb-a01 practice % python new311.py
.....................
string format: Mean +- std dev: 4.13 us +- 0.31 us
(practo-env) dkb@dkb-a01 practice %

Note: Please try this only with bignums. With smaller integers I did not see a significant difference.

4. Faster `list.append()` operations:

import pyperf

my_stmt = """
x = list()
for i in range(100000):
x.append(i)
"""

runner = pyperf.Runner()
runner.timeit(stmt=my_stmt, name="list append")

Benchmark python 3.11:

(py3.11-env) dkb@dkb-a01 practice % python new311.py
.....................
list append: Mean +- std dev: 3.28 ms +- 0.04 ms

(py3.11-env) dkb@dkb-a01 practice % python new311.py
.....................
list append: Mean +- std dev: 3.31 ms +- 0.07 ms
(py3.11-env) dkb@dkb-a01 practice %

Benchmark python 3.9:

(practo-env) dkb@dkb-a01 practice % python new311.py
.....................
list append: Mean +- std dev: 5.61 ms +- 0.14 ms

(practo-env) dkb@dkb-a01 practice % python new311.py
.....................
list append: Mean +- std dev: 5.90 ms +- 0.56 ms
(practo-env) dkb@dkb-a01 practice %

5. Include fine grained error tracebacks:

The primary motivation is to improve the feedback presented about the location of errors to aid with debugging.

Python currently keeps a mapping of bytecode to line numbers from the compilation. The interpreter uses this mapping to point to the source line associated with an error. While this line-level granularity for instructions is useful, a single line of Python code can compile into dozens of bytecode operations making it hard to track which part of the line caused the error.

Let’s consider the following code snippet

class MyClass:

def __init__(self,a, b):
self.a = a
self.b = b

m = MyClass(1, 2)
n = None

print(m.a, n.a)

Python 3.9:

(practo-env) dkb@dkb-a01 practice % python new311.py
Traceback (most recent call last):
File "/Users/dkb/Code/practice/new311.py", line 15, in
print(m.a, n.a)
AttributeError: 'NoneType' object has no attribute 'a'

Though the above traceback is informative enough, we get confused which object is None type here. Is it m? or n?. Now let’s check the traceback in python 3.11

Python 3.11:

(py3.11-env) dkb@dkb-a01 practice % python new311.py
Traceback (most recent call last):
File "/Users/dkb/Code/practice/new311.py", line 15, in
print(m.a, n.a)
^^^
AttributeError: 'NoneType' object has no attribute 'a'

They have clearly given a visual cue as to where exactly the object is None. Amazing isn’t it? This is my personal favourite optimisation.I ain’t sure who suggested this but this is indeed priceless.

https://peps.python.org/pep-0657/

6.Exception groups:

Two new built-in exception types have been added.

BaseExceptionGroup
ExceptionGroup

from builtins import ExceptionGroup

my_exception = ExceptionGroup("one", [TypeError(1), ValueError(2), OSError(3)])

print(traceback.print_exception(my_exception))

| ExceptionGroup: one (3 sub-exceptions) +-+---------------- 1 ---------------- | TypeError: 1 +---------------- 2 ---------------- | ValueError: 2 +---------------- 3 ---------------- | OSError: 3 +------------------------------------

The traceback clearly explains the exceptions with the corresponding numbers we had assigned. This is especially useful if you have your org level custom exceptions and you could group them together.

There is also a except* added to handle this exception groups. Adding this to except keyword groups all the related exceptions.With the new syntax, an except* clause can match a subgroup of the exception group that was raised, while the remaining part is matched by following except* clauses.

The exception group by itself is a vast topic. We will discuss that in another post dedicated for exception groups.

https://peps.python.org/pep-0654/

7. TOML support comes as Batteries included:

TOML aims to be a minimal configuration file format that’s easy to read due to obvious semantics.TOML is,

is easy to read due to obvious semantics
maps unambiguously to a hash table
is easy to parse into data structures in a wide variety of languages

TOML support is included in the python standard library. Let’s try and read a toml file from python 3.11.

[default]
user="dinesh"
language="python"
version=3.11

import tomllib

with open("myproject.toml", "rb") as toml_file:
my_file = tomllib.load(toml_file)
print(my_file)

{'default': {'user': 'dinesh', 'language': 'python', 'version': 3.11}}

https://peps.python.org/pep-0680/

Summary:

Apart from the aforementioned optimisations, there are few more added to python 3.11. They are

dataclass transforms.
Self type added in typing module.
Arbitrary literal string type added to typing.

If you’d like to read more please refer here.

codingforlife

codingforlife

Python 3.11 — What’s cooking?

Is python 3.11 really as fast as they claim it to be?

Table of contents

Introduction:

Installation:

Pyperf:

Features of pyperf:

Installation:

1. Faster CPython:

2. Print f style formatting strings are faster:

Benchmark 3.11:

Benchmark 3.9:

F-strings benchmark

Benchmark 3.11:

Benchmark 3.9:

3. Faster int division for python bignums:

Benchmark python 3.11:

Benchmark python 3.9:

4. Faster `list.append()` operations:

Benchmark python 3.11:

Benchmark python 3.9:

5. Include fine grained error tracebacks:

Python 3.9:

Python 3.11:

6.Exception groups:

7. TOML support comes as Batteries included:

Summary:

References:

Python 3.11 — What’s cooking?

Is python 3.11 really as fast as they claim it to be?

Table of contents

Introduction:

Installation:

Pyperf:

Features of pyperf:

Installation:

1. Faster CPython:

2. Print f style formatting strings are faster:

Benchmark 3.11:

Benchmark 3.9:

F-strings benchmark

Benchmark 3.11:

Benchmark 3.9:

3. Faster int division for python bignums:

Benchmark python 3.11:

Benchmark python 3.9:

4. Faster list.append() operations:

Benchmark python 3.11:

Benchmark python 3.9:

5. Include fine grained error tracebacks:

Python 3.9:

Python 3.11:

6.Exception groups:

7. TOML support comes as Batteries included:

Summary:

References:

4. Faster `list.append()` operations: