Use Pyenv to Streamline your Python Projects

There are two hard problems to solve in a Python project: installing packages, and installing packages in other projects.

Over the years, the challenges I’ve come across in Python projects has narrowed down to two topics ¹ :

1. Installing packages for just my current project – not for all projects using that version of Python
2. Using different Python versions for different projects

Problem (1) can be solved with Python’s built in virtual environments, but they don’t automatically activate when you enter a project folder. As a result, I often forget to run source venv/activate and then accidentally install a bunch of packages to my system Python.

I highly recommend Pyenv to solve both problems. Pyenv is a fantastic tool for managing Python environments simply by changing your current folder. Pyenv also integrates with virtualenv so that you can create virtual environments for any Python version you’d like to use.

Overview

By the end of this post you will know how to:

• Install pyenv on your computer
• Build a version of Python with pyenv
• Change your global and local folder settings to run the new version of Python
• Connect pyenv to virtualenv to create a new virtual environment

Install Pyenv

First, install Xcode tools and a few development libraries that the Python build step will need:

# Note: These are macOS specific commands

# Update XCode and Homebrew
xcode-select --install
brew update

# Install libraries you'll need to build Python from source
brew install openssl readline sqlite3 xz zlib

# Install pyenv
brew install pyenv

Note: The installation instructions for pyenv change frequently. These are the steps for pyenv==2.3.1 on macOS. If you are using a different version or OS, see the pyenv installation guide.

If you’re using a zsh shell, run these commands to add pyenv to your dot files. If you’re using another shell, follow the installation steps for your shell under the Shell environment section of the pyenv README.

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc

Restart your terminal for the changes to take effect, and verify your installation by entering the command which python3 and confirming that pyenv/.shims is somewhere in the output:

$ which python3

/Users/nicky/.pyenv/shims/python3

Install Python 3.8.2

Next let’s install Python 3.8.2. I chose this particular version of Python because I use it in a few of my side projects, so you can use any other version of Python if you’d like.

Recently, I came across a bug in macOS Big Sur that prevents libraries like numpy and pandas to use the lzma package. Credit to the thread for pyenv Issue #1737 for the below instructions to correct the 3.8.2 build.

To install Python 3.8.2, first reinstall the zlib and bzip2 libraries via Homebrew:

brew reinstall zlib bzip2

In your ~/.zshrc (or ~/.bashrc if you’re using bash), add the below flags for the Python compiler to read:

export LDFLAGS="-L/usr/local/opt/zlib/lib -L/usr/local/opt/bzip2/lib"
export CPPFLAGS="-I/usr/local/opt/zlib/include -I/usr/local/opt/bzip2/include"

Then install Python 3.8.2 using the below command. This may take some time to compile.

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" \
LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.2 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

Verify that you built and installed Python 3.8.2 by checking what versions of Python pyenv has:

$ pyenv versions
  system
  3.8.2

Manage Pyenv Versions

Pyenv has two types of versions:

1. local – the Python version for your current directory
2. global – the default Python version to use if no local version is set.

When you switch directories in your terminal, Pyenv checks for a .python-version file in the root of that directory. If there’s no .python-version, Pyenv will search each parent folder until one is found.²

You can set what version of Python pyenv will use in your current folder with pyenv local:

$ pyenv local 3.8.2

You can check .python-version to see what version is used:

$ cat .python-version
3.8.2

And pyenv version will also tell you what version of Python you’re currently using as well as what config file set it:

$ pyenv version
3.8.2 (set by /Users/nicky/Developer/my-project/.python-version)

If you move to a different folder without a .python-version, Pyenv will use your global setting, which is system by default:

$ cd /path/to/other/folder

$ pyenv version
system (set by /Users/nicky/.python-version)

How to Use Pyenv Virtual Environments

Give pyenv superpowers by installing pyenv-virtualenv. With this plugin, pyenv can manage virtual environments like venv and Conda environments.

Install the plugin with Homebrew:

brew install pyenv-virtualenv

And add the following to your shell’s .rc file (such as .zshrc or .bashrc):

eval "$(pyenv virtualenv-init -)"

To create a virtual environment for the Python version you’re using with pyenv, run pyenv virtualenv [version] [new environment name]. For example, to create a new venv for a sample project using 3.8.2:

pyenv virtualenv 3.8.2 my-project-3.8.2

While not required, I recommend adding the Python version to your environment names to manage them easier.

You can also create a new virtual environment using the current pyenv Python version:

pyenv virtualenv my-default-venv

Just like pyenv, your virtual environment will automatically activate whenever you move into that folder!

Helpful Commands

Below is a list of commands for pyenv that I use often.

To show all of your Python versions and all virtual environments:

pyenv versions

To set a different version for the python and python3 commands for your global settings:

# (This works the same with `local` as well)
pyenv global [python version] [python3 version]

To create a new virtual environment and use it locally:

pyenv virtualenv [python version] [new environment name]
pyenv local [new environment name]

Add Your Vimrc to Obsidian

I’ve migrated my note taking system from Craft to Obsidian over the past few weeks, and it’s been great so far. One major advantage of Obsidian is its amazing developer community and wide, wide universe of community plugins. I use quite a few community plugins:

One of my favorite features of Obsidian is its support for vim bindings. I use vim and VS Code’s vim bindings all the time, and I’m able to context switch much faster between coding and note writing when all of my editors use vim keys.¹

You can turn on vim bindings in your own Obsidian vault by going to “Settings” > “Editor” and toggling “Vim key bindings”:

The only downside to Obsidian’s default vim bindings is the lack of .vimrc support out of the box. My personal .vimrc has a few shortcuts that I rely on:

• I map ; to : in normal mode so that I don’t rely on the shift key
• I map j to gj (and k to gk) to jump by visual lines instead of logical lines by default

The ; mapping isn’t a deal breaker in Obsidian; I’m not using vim commands that often. However, the j and k remaps are critical! Obsidian wraps text much more than a terminal thanks to its generous padding. Even on this “simple” post with just a lot of text, one line turns into four in Obsidian:

Luckily, there’s plugin for Obsidian .vimrc files: Obsidian Vimrc Support Plugin. To add this to your Obsidian vault, go to “Settings” > “Community plugins” > “Browse” and search for “vimrc”:

Then click “Vimrc Support”, “Install” and “Enable”:

At the root of your Obsidian vault (the root folder for all of your .md files), create a new file named .obsidian.vimrc. You can paste any of your ~/.vimrc into your .obsidian.vimrc.

Here’s my .obsidian.vimrc that remaps ;, j, and k in normal mode:

" .obsidian.vimrc
"
" A small .vimrc for Obsidian vim bindings
"
" To enable this file, you must install the Vimrc Support plugin for Obsidian:
" https://github.com/esm7/obsidian-vimrc-support
"_________________________________________________________________________

" ; (semicolon) - same as : (colon)
nmap ; :

" (space) - same as : (colon)
nmap <SPACE> :

" j and k navigate visual lines rather than logical ones
nmap j gj
nmap k gk

Once you write your .obsidian.vimrc, reload Obsidian and your new vim bindings will load!

How to Install Ruby 2.7.3 on M1 Mac

Installing Ruby or Python on M1 Macs is a nightmare. I’ve lost so many hours fighting compilers and Rosetta to these issues, so I’m documenting my installation steps to spare you¹ a lot of headache.

Why 2.7.3?

First, why Ruby 2.7.3 specifically? Ruby 3.0.0+ works great on M1 using my RVM install instructions post:

rvm install 3.0.0

Unfortunately, my website’s host GitHub Pages uses Ruby 2.7.3 according to their dependency documentation. And a bare-bones install of 2.7.3 blows up on my M1 machine:

rvm install 2.7.3

So I needed to find a way to install 2.7.3 on my Mac to build and run my website locally to preview new posts.

The Solution

I’ll put the winning command here at the top to keep things simple. Many thanks to @d-lebed for documenting their solution on this GitHub issue. I lightly modified their code to use $(brew --prefix) instead of hardcoding where homebrew downloads openssl@1.1:

# Winning script!
brew install openssl@1.1

export PATH="$(brew --prefix)/opt/openssl@1.1/bin:$PATH"
export LDFLAGS="-L$(brew --prefix)/opt/openssl@1.1/lib"
export CPPFLAGS="-I$(brew --prefix)/opt/openssl@1.1/include"
export PKG_CONFIG_PATH="$(brew --prefix)/opt/openssl@1.1/lib/pkgconfig"

rvm autolibs disable

export RUBY_CFLAGS=-DUSE_FFI_CLOSURE_ALLOC
export optflags="-Wno-error=implicit-function-declaration"

rvm install 2.7.3 --with-openssl-dir=$(brew --prefix)/opt/openssl@1.1

This successfully installed Ruby 2.7.3 for me, and I was then able to run bundle install at the root of my website’s repo!

The Issue

I originally documented all of my problem solving steps and install attempts, but it grew too long and rambly for my liking. Instead, I’ll give a brief summary of a few errors I came across.

Flags

At one point, rvm was complaining about my LDFLAGS:

checking whether LDFLAGS is valid... no
configure: error: something wrong with LDFLAGS="-L/usr/local/opt/zlib/lib -L/usr/local/opt/bzip2/lib"

And setting LDFLAGS="" in front of rvm install 2.7.3 only resulted in errors with the compiler missing openssl libraries. So I needed LDFLAGS set somehow, but not the way that works for Python and pyenv.

Rosetta

In a few GitHub issues, people recommended opening the Terminal app via Rosetta and running commands that way such as in this issue comment. However, I saw no difference in error outputs between Rosetta Terminal and iTerm. In the end, I used iTerm without Rosetta to successfully install Ruby 2.7.3.

Usename Macro Error

I got close to a correct install when I added the openssl library to LDFLAGS, CPPFLAGS, and PKG_CONFIG_PATH:

PATH="/usr/local/opt/openssl@1.1/bin:$PATH" \
LDFLAGS="-L$(brew --prefix)/opt/openssl@1.1/lib" \
CPPFLAGS="-I$(brew --prefix)/opt/openssl@1.1/include" \
PKG_CONFIG_PATH="$(brew --prefix)/opt/openssl@1.1/lib/pkgconfig" \
arch -x86_64 rvm install 2.7.3 -j 1

But I then started seeing errors about some missing rl_username_completion_function macro:

214 warnings generated.
linking shared-object date_core.bundle
installing default date_core libraries
compiling readline.c
readline.c:1904:37: error: use of undeclared identifier 'username_completion_function'; did you mean 'rl_username_completion_function'?
                                    rl_username_completion_function);
                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                    rl_username_completion_function
readline.c:79:42: note: expanded from macro 'rl_username_completion_function'
# define rl_username_completion_function username_completion_function
                                         ^
/opt/homebrew/opt/readline/include/readline/readline.h:485:14: note: 'rl_username_completion_function' declared here
extern char *rl_username_completion_function PARAMS((const char *, int));
             ^
1 error generated.
make[2]: *** [readline.o] Error 1
make[1]: *** [ext/readline/all] Error 2
make: *** [build-ext] Error 2
+__rvm_make:0> return 2

I then found the winning command (above) by googling for rvm install 2.7.3 error extern char *rl_username_completion_function PARAMS((const char *, int)); and trying a few commands recommended on some GitHub issues.

Next Steps

I recently figured out how to install Python 3.7, 3.8, and 3.9 on M1 Macs via pyenv, so keep an eye out for my post on how to install those.