Notes on vim python interop
November 19, 2017
I stumbled upon an interesting problem with
Jedi language service for python.
On digging further, realized that I’ve been thinking incorrectly so far. So
let’s dedicate this post to correct the stance.
Update: there’s a behavior change since vim
8.0.1451. See setting python home post for details.
VIM and Jedi
OmniSharp (language service for .NET),
Jedi is a language support
service for python. It works with several editors. There are two parts to it:
vim-jedi loads a python layer which finds the
package and asks source completions from it.
Defining the problem
I had been using
jedi in this way:
- Create a virtual env and activate it
jediin the virtual env
- When vim runs in the virtual env, it uses jedi package for auto completion
This scenario recently broke. There are atleast two ways to create a virtual
Method 1: python venv module
# create a python virtual env with venv in directory myenv > python -m venv myenv > source myenv/bin/activate # install jedi package and open vim > pip install jedi > vim foo.py # run `:python3 import jedi` within vim
Method 2: pipenv
# create a python virtual env with pipenv tool > pipenv --python 3 > pipenv shell # install jedi package and open vim > pipenv install --dev jedi > vim foo.py # run `:python3 import jedi` within vim
Problem statement: with method 1 jedi works, with method 2 it doesn’t!
Let’s make it more complicated :) Here’s a command that works in both environments:
> python -m jedi # no error, which means jedi is actually available in both in the python # interpreter
If it works outside vim, what’s going on within vim?
On python interop
VIM can interop with python based on compile time flags. For working with both
python3, the interoperability libraries should be dynamically
+python3/dyn flags in
:version in vim.
Our validation with
:version indicated the correct compilation
flags. We tried the following command to see the state of python interpreter.
> vim # type the following ex command in vim # :python3 import time; time.sleep(300000) > htop # the process tree view doesn't show python as a child process of vim !?
Alright, so vim doesn’t invoke the python interpreter. So there must be a python interpreter embedded within vim process. Let’s check out!
> gdb vim (gdb) start (gdb) c # execute a python command in vim # :python3 import sys; print(sys.path) # suspend vim process: Ctrl+Z (gdb) info sharedlibrary # this shows /usr/lib/python3.6/ 0x00007fffe938c220 0x00007fffe9556e3d Yes (*) /usr/lib/libpython3.6m.so.1.0 ... 0x00007fffe86edb20 0x00007fffe86ef467 Yes (*) /usr/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so
Point noted. And vim source code does imply that. Check out if_python3.c in vim source code:
- Cross platform assembly load routines defined here
load_dllloads the library,
symbol_from_dllloads the symbols from it
- And there’s an interepreter here
So our comparison that
python -m jedi works outside vim is expected, clearly
that’s not how vim does execute python code!
Specifics of vim + python
We’re quite close. There’s two other details we need to understand:
- An embedded python interpreter works with
Py_SetPythonHomefunction call. This sets the
exec-prefixfor the interpreter.
With a virtual environment created via
python -m venvway, a
$VIRTUAL_ENV/pyvenv.cfgfile is created. See pep-0405 for details. Relevant parts in the document:
In this case, prefix-finding continues as normal using the value of the home key as the effective Python binary location, which finds the prefix of the base installation. sys.base_prefix is set to this value, while sys.prefix is set to the directory containing pyvenv.cfg.
(If pyvenv.cfg is not found or does not contain the home key, prefix-finding continues normally, and sys.prefix will be equal to sys.base_prefix.)
So what happens in vim?
- vim calls
Py_SetPythonHomewith a prefix as
/usrbefore starting the embedded interpreter. This modifies the
python -m venvcreates a
pyenv.cfgfile in the virtual env it creates. This changes the
sys.prefixset by vim to
/tmp/myenvdirectory and thus
pipenv environment can’t find
jedi package in this directory since it is
only installed within the virtual environment.
python -m venv environment has modified the
site-packages to the virtual
environment directory, so it does find the
jedi package and everything works
vimis not aware of the virtual environment. It just loads the shared library and tries to embed a python interpreter.
python -m venvplaces a
$VIRTUAL_ENVwhich makes python natively aware of the virtual environment; and thus the
sys.prefixchanges accordingly, including
site-packagesof the virtual environment
Thus we need not install
jedi in every python project with
pipenv. We can
install the package system wide using
pacman -S python2-jedi python-jedi.
jedi will be loaded from the
however it will be aware of the virtualenv and accordingly show completion and
refactoring for the editors (only for packages within VIRTUAL_ENV site-packages).
How do we prove vim indeed sets
We can do it with setting a function breakpoint in
gdb and examining the
input. Here’s how:
> gdb vim (gdb) break Py_SetPythonHome Function "Py_SetPythonHome" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 1 (Py_SetPythonHome) pending. (gdb) start Temporary breakpoint 2 at 0x66ad0 Starting program: /usr/bin/vim [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Temporary breakpoint 2, 0x00005555555baad0 in main () (gdb) c Continuing. Breakpoint 1, 0x00007fffed0a176e in Py_SetPythonHome () from /usr/lib/libpython2.7.so.1.0 #skip the python 2.7 load (gdb) c Continuing. # vim starts now, input following command :python3 import sys Breakpoint 1, 0x00007fffe93aa44d in Py_SetPythonHome () from /usr/lib/libpython3.6m.so.1.0 # we don't have symbols, let's try to get the local arg from registers # %rdi is used to store the first arg, %rsi second arg etc. see https://cons.mit.edu/sp17/x86-64-architecture-guide.html # we will examine the value (gdb) x/10s $rdi 0x5555557f5ed0: "/" 0x5555557f5ed2: "" 0x5555557f5ed3: "" 0x5555557f5ed4: "u" 0x5555557f5ed6: "" 0x5555557f5ed7: "" 0x5555557f5ed8: "s" 0x5555557f5eda: "" 0x5555557f5edb: "" 0x5555557f5edc: "r"
Point noted :)
Can I play around with the embedded interpreter?
You can create one for yourself mimicing
vim in the context of this post. This
gist has source code listing.
And here’s how the output looks across
python -m venv and
The shell on left is python -m venv and on the right we have pipenv
environment. Note the
sys.prefix output (marked red) and the
listing (marked green).
Hope you enjoyed this post. Namaste!