Introduction

ScalaPy is an excellent library by Shadaj Laddad that provides users access to Python libraries inside Scala which I use a lot. This project sets out to improve developer experience by implementing some quality of life features such as ease of set up, code autocompletion and docstring display, as well as better integration with Ammonite (Scala REPL and scripting) and Almond (Jupyter kernel for Scala).

Ease of setting up

In order to set up ScalaPy to use a specific Python interpreter, users have to specify several properties, namely,

  • for JVM: the directories containing the libpython associated with said interpreter as well as the correct name of libpython, e.g. libpython3.7, libpython3.8m

  • for Scala Native: the recommended linker options

Figuring out how to extract these properties in many cases is not straightforward and the rules vary among different platforms, Python versions and installation methods. python-native-libs was made with an aim to derive these properties in a robust way. Details on how exactly python-native-libs arrives at these properties are provided here.

Using python-native-libs to set up ScalaPy either with an Sbt project or with Ammonite or Almond is fairly straightforward,

Sbt

for both JVM and Scala Native, add python-native-libs to project/plugins.sbt

libraryDependencies += "ai.kien" %% "python-native-libs" % "0.2.1"

then, in build.sbt,

for JVM,

fork := true

import ai.kien.python.Python

lazy val python = Python("<optional-path-to-a-python-interpreter-executable>")

lazy val javaOpts = python.scalapyProperties.get.map {
  case (k, v) => s"""-D$k=$v"""
}.toSeq

javaOptions ++= javaOpts

for Scala Native,

import ai.kien.python.Python

lazy val python = Python("<optional-path-to-a-python-interpreter-executable>")

lazy val pythonLdFlags = python.ldflags.get

nativeLinkingOptions ++= pythonLdFlags

Ammonite and Almond

@
import $ivy.`ai.kien::python-native-libs:0.2.1`
import $ivy.`me.shadaj::scalapy-core:0.5.0+8-7c7a6042`

import ai.kien.python.Python

Python("<optional-path-to-a-python-interpreter-executable>").scalapyProperties.fold(
  ex => println(s"Error while getting ScalaPy properties: $ex"),
  props => props.foreach { case(k, v) => System.setProperty(k, v) }
)

import me.shadaj.scalapy.py
import me.shadaj.scalapy.py.PyQuote

println(py"'Hello!'")

Future works

The next goal is to integrate python-native-libs directly into ScalaPy, Ammonite and Almond to make the experience even more seamless. There have already been open PRs in ScalaPy and Ammonite (or coursier if that fails) to move this forward.

Display of rich representations for Python objects in Almond

Jupyter notebooks allow Python users to display plots, images, html, latex by rendering the outputs of the _repr_*_ methods (e.g. _repr_png_, _repr_svg_, _repr_html_, …) for Python objects that implement them. This functionality has just been made available to ScalaPy users using Almond in almond#843 and almond#854 with the help of jvm-repr. You can see this feature at work here. The implementation was completed even though the PR is still open pending some baffling CI issue which we aim to resolve soonTM.

Future works

This feature only supports displaying static representations of Python objects, not interactive/updatable displays such as progress bar. One possible way to make this happen is to bridge between IPython display APIs and Almond display APIs.

Auto completion and docstring for ScalaPy in Ammonite and Almond

Basic completion for ScalaPy Python objects in Ammonite was made possible by employing the Python standard library rlcompleter. It supports completion for expressions of the form pyObject.(attr.*)attr| where pyObject is an object of type py.Dynamic.global.type or py.Dynamic and .attr’s are invocations of the selectDynamic() methods, which is equivalent to accessing an attribute of the Python object. Below are some examples of how this feature works in Ammonite.

@ py.Dynamic.global.li|
license( list(

@ import me.shadaj.scalapy.interpreter.CPythonInterpreter
  CPythonInterpreter.execManyLines("""class Cls: x = 'string'""")
@ py.Dynamic.global.Cls.x.up|
upper(

@ val p = py"'string'"
@ p.up|
upper(

@ object Obj { val p = py"'string'" }
@ Obj.p.up|
upper(

More advanced completions and docstring display are provided by jedi Interpreter

@ py.Dynamic.global.range(10).cou|
count|

Here range(10) being a function call makes it impossible for rlcompleter to provide completions for cou|. It is possible in jedi however, thanks to static analysis.

This feature is in the work at kiendang/Ammonite#2. This PR is not aimed to be merged into Ammonite as is but waits until Ammonite provides an interface for extending the completion functionality. ScalaPy completion would then be its own separate library which is imported and enabled by Ammonite users when they use ScalaPy. This is to avoid adding ScalaPy as a dependency to Ammonite.

Future works

The next step is to continue polishing this feature, hopefully with user feedbacks, and adding Scala 3 support. It is currently only available for Scala 2 (2.12 and 2.13), not Scala 3, mainly due to ScalaPy not being released for Scala 3 yet plus some difference in how the presentation compiler works in Scala 2 and Scala 3 which would require some modifications to the implementation. Works on making autocompletion available for Scala 3 would begin after ScalaPy lands Scala 3 support shadaj/scalapy#209.

Potential features in the work

One feature that is extremely nice to have and we hope to be able to implement if time allows in the future is the ability to start an integrated Python shell inside Ammonite and Almond that allows variable exchange with the Scala shell. Users could drop into a Python shell inside Ammonite and Almond, run some Python code, then return to the Scala shell with all the variables defined inside Python now also available in the Scala shell, similar to polynote.

Aggregation of links to the works mentioned in this post

Acknowledgements

Special thanks to Alex Archambault, Anatolii Kmetiuk and Shadaj Laddad and the Scala Center for the great support and for answering my various questions!