Python 2 to Python 3 Migration

The current situation with Python 2 to 3 migration is best explained as a stalemate between two key software development bylaws: “If it ain’t broke, don’t fix it” and “Penny wise, pound foolish.” Many companies developing or maintaining Python-based products feel like they’re caught between a rock and a hard place: They want to stick with outdated Python 2 code because it works. But the longer they do, the higher the cost of transition becomes. Why?

The Python 2.7 end-of-life support date was January 1, 2020, and the last version of Django to support Python 2 was 1.11 LTS. In addition, Python 3.7 has already reached its end-of-life date. This makes Python 2 almost a legacy, and the need for Python 2 to Python 3 migration is glaringly obvious for any company that has not yet done so.

A key difference between Python 2 and Python 3 is that the standard library is much better organized in Python 3, the syntax was refined, and it has many out-of-the-box features available, such as asynchronous programming. In some cases, these factors provide up to 30% better performance with up to 15% lower resource consumption than Python 2. Because Python code often runs pretty workload-heavy projects, these numbers can mean millions of dollars saved over time.

At Django Stars, a monostack Python & Django web development company with multiple accolades from Clutch and worldwide recognition in the field, we always suggest clients build new products using the latest stable Python version. But many companies with Python 2-based projects try to postpone the inevitable Python 2 to Python 3 migration — until they are forced to migrate.

In this article, we discuss the whys, the hows, and the cost of migration decisions. We’ll also compare Python 2 and 3 and showcase Python 2 to Python 3 migration issues a company might face, and offer a roadmap to successful migration.

Key differences between Python 2 and Python 3

Django Stars knows Python development inside out — we’ve built, reengineered, modernized, and transformed multiple products from the ground up that raised hundreds of millions of dollars in investments. This means we have the required expertise to highlight the most significant Python 2 vs. 3 differences for you.

The Python 3 introduction process was controversial, to say the least. Python 3.0 was released amidst the economic crisis of 2008, and Python 2.7 was developed in parallel and released in 2010.

According to Guido Van Rossum, Python’s creator and core developer, Python 3 was “a way to correct fundamental issues with Python 2 that couldn’t be cleanly fixed with a regular deprecation process for minor version updates.”

Like sometimes happens in architecture — the original foundation of Python 2 was unsuitable for future development. For example, it can’t support newer development that makes rich use of Unicode. So, a new foundation was created.

Python 2 works great within its limitations and gets the job done, but Python 3’s new foundation allows for building future-proof projects.

The main pillars of the new foundation were:

• A core string type based on Unicode by default
• Integer division that automatically promotes floating point results
• Removal of the conventional classes concept

Making these changes in language structure resulted in consecutive changes to language syntax, such as:

• Python 2 could only divide an integer by another integer and then round the result down to the nearest whole number. Python 3 can divide to a floating point (or perform integer division using //).
• Using non-ASCII characters in strings caused Python 2 to throw a UnicodeEncodeError. Addressing this error required prolonged and costly debugging for each non-ASCII character. Python 3 uses Unicode for all strings by default, and non-ASCII characters can be used without limitations.
• New-style classes are the only way to initiate a class in Python 3.

Although these are tectonic shifts, many changes were more cosmetic, such as a new print statement and harmonization of xrange and range functions, etc. Therefore, not everyone immediately understood why they should migrate from Python 2 to 3.

Nick Coghlan, a Python core developer, said,

One of the most important guidelines for good Unicode handling is to ensure that all encoding and decoding occurs at system boundaries, with all internal text processing operating solely on Unicode data. The Python 2 Unicode model is essentially the POSIX text model with Unicode support bolted on to the side, so it doesn’t follow that guideline.

The shift to Unicode allowed the Python 3 ecosystem to thrive unimpeded by Python 2 design limitations.

Benefits of migrating to Python 3

Python 3 is actively developed, so the language, its libraries, and frameworks, like Django or Flask, regularly receive security patches, bug fixes, and new features. As one of the most in-demand programming languages in 2023, its popularity makes it easy to find developers with advanced Python 3 expertise.

Thus, Python migration from 2 to 3 future-proofs any project, ensuring a project’s development and maintenance will fall within the expected budget and timeframes. Naturally, many factors chime in here, but using the latest language version, libraries, and frameworks to build a product significantly reduces the project costs.

Business benefits aside, there are technical advantages of Python 3 over 2, discussed below.

Enhanced performance

Because of its optimized syntax and language structure, Python 3 requires considerably fewer resources to run a product than its predecessor. For example, when Instagram completed its migration from Python 2 to 3, they got a 12% reduction in CPU usage and up to a 30% reduction in memory usage, which saved them a fortune.

In another case, upgrading from Python 2.7 to 3.6 led to a 4x reduction in CPU usage and a 50% drop in memory usage for a mission-critical application. This became one of the main advantages in Python 2.7 to 3 migration.

More refined standard library

Python’s standard library includes the most common modules, eliminating the need to write so many commands from scratch when developing a new product. Python 3.x’s standard library has been reorganized to optimize its structure; it requires less space, outdated modules were removed, and the overall consistency of the language is greatly improved.

Built-in modules and packages for standard functionalities coupled with a huge variety of open-source libraries compatible with Python 3 significantly reduce the time and expense of developing new product features.

Community and package support

There’s a huge and passionate community developing open-source libraries for Python 3. At annual conferences around the globe, like PyCon, developers participate in Python and Django hackathons and workshops. This active participation and collaboration fosters innovation, and the latest tech trends are quickly incorporated into Python.

But the path for Python 2 to Python 3 migration isn’t exactly a walk in the park. There will be challenges, and it’s only fair to mention them.

Challenges of migrating to Python 3

Most libraries used by Python 2 projects were already deprecated or upgraded to support Python 3.

This means that for products that still run on Python 2, there’s middleware and custom code built to ensure compatibility with these external libraries adapted to Python 3. At some point, all that middleware will have to be removed, code rewritten and updated, and the project architecture will even require adjustment in some cases.

Technical debt issues aside, migration also has organizational and implementation challenges.

Significant syntactic and semantic changes

The very syntax adjustments we praised above are one of the roadblocks to migration. Naturally, all the Python 2 product code must be copied to a separate Python 3 environment and reviewed to modify it for the updated syntax.

Mapping problems between older and newer versions

While most code can be refactored with time and effort, certain Python 2 modules have no direct counterparts in Python 3. This can create some difficulties if you want to migrate code from Python 2 to 3. Therefore, it may not be possible to easily map them to new code, and their functionality will have to be rewritten from scratch (if it remains relevant to the product and newer features can’t replace them).

Some refactoring will require product architecture adjustments to fully leverage the performance increase potential of Python 3 and its new features, which are continuously introduced in the latest versions. Thus, sometimes migration can be a great opportunity to fix technical debt, modernize product architecture, and make the code more secure and scalable. Naturally, this will take time and money, which we’ll discuss later.

Rewriting and retesting code

The migration process will require extensive testing for code compatibility, which is one of the reasons many companies postpone migration. There are no two ways about it — all the product code must be thoroughly tested for regressions and compatibility.

But there’s no reason to fret. Multiple tools are available to help, like Python-Modernize, 2to3, or Six (2×3) package. Any of these tools will help automate most of the code refactoring. Here’s how it works for Python-Modernize, for example:

1. Add a new “six” dependency to your application dependencies.
2. Run “pip install modernize.”
3. Run “python-modernize -w” in your project directory.
4. Review the changes. They should be reasonably safe but sometimes visibly inefficient, so adjust them as you see fit.
5. Test your app on Python 2.

Naturally, a recipe as easy as this works only for small-scale projects. The bigger the project scope, the more steps migration from Python2 to Python3 requires.

For testing, we highly recommend building CI/CD DevOps pipelines with automated tests. This might seem like an excessive task to take on during migration, but it will prove very valuable. Establishing happy path testing with automated tests using Sentry and other tools will save a company from spending hundreds of development hours on manual debugging the Python applications. Finally, using stack trace will help debug and fix transition-related errors.

Now that we’ve covered the technicalities of the process, it’s time to talk about the cost.

Assessing the cost of migration

A budget for Python 2 to 3 migration will largely depend on the current state of the product, how many features will need complete refactoring and the expected results.

1. The more custom code there is, the longer it takes to analyze it and determine which functionalities can be replicated using existing Python 3 capabilities and which will have to be written from scratch.
2. Lifting-and-shifting the main codebase after automated updating and testing costs less and is faster than a full-scale transition to Python 3.
3. Future-proofing a product with the latest technology requires updating to the latest stable Python 3 version. For example, Python 3 enables a vast range of libraries for AI, machine learning, and data wrangling, which many companies want to integrate into their offerings. Although updating to the latest Python version (i.e., 3.12) is free, supporting new capabilities like these entails infrastructure updates, staffing expenses, and other costs.

In the end, the final budget depends on the expertise of the team implementing the migration and their ability to deliver. At Django Stars, we always find an approach that serves the customer’s best interest.

Planning the migration

We highly recommend automatic Python 2 to 3 migration, with the simultaneous introduction of CI/CD pipelines and automated testing. It saves time and effort compared to reading and adjusting every string code. Available resources are better allocated to handling the technical debt challenge, architecture updates, and optimizing product performance.

Preparation for migration:

1. Usage of only Python 3-compatible features must be enforced with linters and checkers.
2. Developers have to check whether libraries in use have Python 3-compatible versions. If they’re deprecated, the team must find alternatives, copy and adapt part of them, or completely fork and adapt them for Python 3. Here’s a nice tool for this: https://pypi.org/project/caniusepython3/
3. Checking the changelog for potential changes that break library dependencies.
4. Creating a separate testing environment and committing all new code to it. If no automated tests are in place, the team needs to create a smoke test suite for quickly checking mission-critical app functions. Test coverage must be constantly increased.
5. Introducing automatic checking tools to ensure codebase coverage, which can include the aforementioned Six project or an automatic conversion tool.

With correct planning, migrating Python 2 to 3 is totally doable. It’s not a titanic deed — it’s a process that companies can accomplish in a reasonable time.

10 Cost-effective migration tips

Here are some additional tips to ensure migration goes smoothly without overextending the budget or time frame:

1. The product codebase must be replicated to a separate branch, and all new code committed must be Python 3-compatible.
2. Using the Six project tool will help automatically correct most of the syntax.
3. Updating dependencies that can be updated. Some libraries can simply be updated to their latest versions to become compatible with the most current Python and Django versions, while other packages might already have been dropped. The CanIUsePython3 tool does the updating job quite well.
4. After completing everything that can be done automatically, the team must manually update the rest of the dependencies or find alternative Python 3-compatible libraries.
5. Some Python 3 features were backported to Python 2. Examples include functools32 and subprocess32. If a product uses these, they should be truncated to avoid errors.
6. LInters like PyCharm and Pylint, smoke tests, automated testing, and other tools mentioned above can migrate the majority of code and track any errors.
7. Implementing continuous integration helps with testing. Running Tox in parallel on Py2 and Py3 environments will help catch any hiccups remaining.
8. Cache clearing is obligatory. Python 3 from a clean slate or an app that behaved just fine during testing can fail in production.
9. Testing external communication. Email processing, working on an external production server instead of a virtual environment, interactions with an empty database, and production data instead of mocked-up testing data — each of these points require attention from the team.
10. Blue-green deployment. Django Stars highly recommends rollouts with the possibility of rolling back to the old Python 2 app. Leaps of faith can be too costly.

Some of these points are plain old common sense, while others require in-depth expertise. For successful migration, selecting a reliable technology partner is the key.

Tools for Migration from Python 2 to 3

The most convenient Python 2 to 3 migration tool is the 2to3 utility. It’s a specific fixer tool that detects syntax differences between Python 2 and Python 3 and refactors code in auto mode, making it compatible with Python 3.

Also, for this purpose, you may use the python-future library. It’s similar to the 2to3 — the system detects Python 2-specific code and recreates it for a Python 3,3+ environment.

Django Stars as a Python Development Partner

Django Stars, in operation since 2008, has gained a wealth of knowledge over the course of dozens of successfully completed projects. We have Y Combinator winners and Forbes 500 companies among our clients and have earned multiple accolades recognizing the quality of our Python & Django development services. We’re ISO-certified and we bring quality and consistency to every project we work on to help customers achieve their business goals.

Conclusion

When we compare Python 2 and 3, the latter is the clear winner. It provides multiple business benefits and helps future-proof any product that uses it. Python 2 to 3 migration is not a question of if — it’s a question of when and how.

At Django Stars, we’re experts in planning and executing Python migrations. We have in-depth knowledge of all the Python 2 to Python 3 migration issues and know how and what tools to use to overcome them. We can join clients at any software development stage and offer multiple degrees of involvement, from team extension for product development to DevOps and CI/CD implementation, code refactoring, and quality assurance.

Get in touch to discuss migration from Python 2 to 3 and see how Django Stars can help!