Writing Python Notebooks Fast and Safely

About the Method Introduced in This Article

This method is useful when you want to run Python with the following requirements:

• Need to run it immediately
• Need to share the Notebook
• Want to refer back to it later (avoiding spaghetti code)
• The VM only needs to run when in use

Conversely, it is better not to use this method in the following cases:

• Managing packages
• Long-term maintenance

The method introduced here is not guaranteed to work over the long term. If you need long-term maintenance, I recommend building an environment properly using Docker + pyenv + Poetry.

Environment

We will use Google Colab directly. We won't be developing locally and then uploading the Notebook. There are various reasons, but the main factors are:

• Vim works
• There is decent development support without relying on an IDE

Actually, you can use Vim in Google Colab. By selecting Settings -> Editor and then choosing Vim, you can edit the Notebook with Vim. This alone significantly improves work efficiency.

Of course, having AI assistance is better, but it's okay without it. This is because definition lookups and jumps work reasonably well.

This feature also works while you are writing. In other words, you can write Notebooks with a feel similar to a lightweight JetBrains IDE or VSCode. It's great to have development environment support right from the moment you open it. By the way, Pydantic is pre-installed, so no pip install is required.

• Autocompletion works just like your usual environment
  

• Easy to set input values for functions
  

typeguard is one of the libraries I often use. Unfortunately, typeguard is not pre-installed, so you need to pip install it.

https://pypi.org/project/typeguard/

It doesn't have as much flexibility as MyPy, but static analysis runs reasonably well.

Coding

I try to keep a cohesive set of code (for code that doesn't depend on other user-defined functions) within a single block, like the one below.

from pydantic import BaseModel, StrictStr
from typing import List
from typeguard import check_type


class SayHelloDto(BaseModel):
    name: StrictStr


class SayHelloResult(BaseModel):
    message: StrictStr


def say_hello(dto: SayHelloDto) -> SayHelloResult:
    check_type(dto, SayHelloDto)

    return SayHelloResult(
        message=f"Hello {dto.name}"
    )


def test_say_hello() -> None:
    expected = SayHelloResult(message="Hello @shunsock")
    actual = say_hello(
        dto=SayHelloDto(
            name="@shunsock"
        )
    )
    assert expected == actual


test_say_hello()

The features are as follows:

• Functions: Only expose functions globally
• Type checking: Handle types (as much as possible) safely
• Test in Block: Perform tests at the time of function declaration

Functions

One of the most common bugs that occurs when using Notebooks is as follows:

a = "This is global variable" # You don't know from where it will be called

By exposing variables like this, it is common for something you thought was a local variable within a function to actually be a global variable, leading to bugs. Since these types of errors take time to resolve and slow down development, it is better to avoid them as much as possible. (If you used one temporarily, restart the environment to clear the globally exposed variables.)

Type Checking

Since Python does not have runtime type checking, failing to verify types in functions or classes can eventually become a source of bugs and lead to project delays. To use dynamic types as safely as possible, I use Pydantic classes and typeguard.

• Pydantic: Executes validation during class initialization
• typeguard: Executes validation of function argument types

For constants, you'll have to give up. One option is to write it like this:

def shunsock_twitter_name() -> str:
    return "@shunsock"


def greet() -> None:
    print("Please follow me:", shunsock_twitter_name())


greet()

# Please follow me

Test in Block

A common accident in Notebooks is when a function is declared globally and a bug is discovered only at the call site. To prevent such cases, I want to write tests.

However, while in a typical Python project you can separate test code into a tests directory, in a Notebook everything is written in a single file, so that method isn't possible. Therefore, I set it up so that tests run at the timing when the function is declared globally (i.e., when the block is executed).

By doing this, any callable function can be used with relative safety because it has already satisfied the test requirements.

Since the premise is that long-term maintenance won't be performed, I believe assert is sufficient, but for those who want to use a specific library, it's also a good idea to plug one in.

Miscellaneous

• Regarding classes, I basically only use Pydantic classes for data storage.
  • I want to avoid having to use full-text search.
  • Except for patterns like PyTorch, I have some doubts about whether code that requires classes with methods really needs to be written in a Notebook.
  • For the same reason, I don't use abstract classes either.
• How do I handle functions that depend on other user-defined functions?
  • I use Notebook text blocks to separate them: Feature -> Things dependent on unit functions -> Unit functions.

Conclusion

The approach mentioned above allows you to proceed with experimental coding that is less prone to bugs while benefiting from IDE-like features to some extent, so I personally recommend it.

Python development environments vary from person to person, so if you have any favorite tools, I would be happy if you could share them in the comments.