Guideline for Development¶

Here, we describe our utilities to aid the development of Crane.

Devtools Installation¶

First, create and activate a virtual environment with python 3.7.9 or above. We'll use pyenv in this guide. You may want to update your PATH environment variable or initialize venv providers in your shell rc file based on your choice.

# Create virtual environment
$ curl https://pyenv.run | bash
$ pyenv install 3.7.9
---> 100%
$ pyenv virtualenv 3.7.9 crane
$ pyenv activate crane

Then, clone the Crane Github repository. A single command will install all Crane components and development dependencies, build and pull Docker images, and build protobuf files for GRPC.

>>> (crane) $ git clone https://github.com/friendliai/crane.git
---> 100%
>>> (crane) $ cd crane
>>> (crane) $ make install-dev
---> *

This command also installs crane components in editable mode with the pip -e flag. This will allow you to directly use your local version of Crane.

Communication Between Components¶

The Crane Admin CLI (craneadm) must communicate with the Crane Admin Daemon (crane-admind) to perform cluster-level actions (initialize, join, etc) on the host machine. Also, the Crane User CLI (crane) must be able to communicate with the Crane Gateway (crane.core.gateway) to perform user-level actions (sign in, add job, fetch log, etc). For development, it is important to understand how these components communicate with each other.

The default is as follows: - crane-admind <-> craneadm: Unix domain socket at /var/run/crane/crane-adm.sock. - crane.core.gateway <-> crane: Unix domain socket at /var/run/crane/crane-cli.sock.

However, during development, it is convenient to be able to, for example, run a minimal cluster for testing or launch multiple independent clusters on a single machine. Therefore, we support the following connection types: - STANDARD socket: A Unix domain socket at /var/run/crane/crane-*.sock. - LOCAL socket: A Unix domain socket at $HOME/.crane/crane-*.sock. - TCP endpoint: Any URL endpoint in the form of <ip>:<port>.

Selecting the type of endpoint can be done via the endpoint command on both CLIs. For example, running craneadm endpoint set local has craneadm use the socket at ~/.crane/crane-adm.sock.

CLI Features for Developers¶

In this section, we inform you of CLI flags and arguments used for development. See the reference for the full documentation.

Crane Bootstrap¶

All the options of Crane Bootstrap are essential for development. See the reference. Here, we list the most useful ones.

• --dev-crane-path: The local crane path is mounted on docker images. This way, you can use the current crane code without rebuilding images. Make sure you specify this as an absolute path.
• --socket-forward-port: If this option is set, boot.crane launches a socket forwarding container (that runs socat) listening to that port. This option is only needed for MacOS host machines.
• --admin-socket-dir: The unix domain socket directory can be set explicitly. If you wish to use the LOCAL socket connection, use --admin-socket-dir ~/.crane.
• --use-system-dockerd: If this option is set, boot.crane uses the system's dockerd instance instead of creating its own private dockerd instance.

Crane Admin¶

• standalone launches a single-process (not even a container) Crane cluster on the current node. An instance of the cluster manager, the gateway, and a non-persistent local state DB is created. Under this setting, craneadm does not communicate with crane-admind. Users may send requests to the cluster manager with the command crane --use-local-crane.
• pseudocluster launches a simple crane cluster without Docker. The difference with standalone is that pseudocluster launches separate subprocesses for Gateway, Cluster Manager, and (possibly many) Node Manager(s).
• endpoint: Set the endpoint type.
• --socket-path: Explicitly set the location of crane-adm.sock.

Crane CLI¶

• endpoint: Set the endpoint type.
• --socket-path: Explicitly set the location of crane-cli.sock.

Testing Crane¶

PyTest¶

Crane uses pytest to perform unit, component, integraion, and end-to-end (e2e) tests. Tests are kept in crane/tests, and organized based on their types (e.g. unit tests are inside the unittest directory).

Run the following commands to run tests: - make alltest: Run every single test. - make e2etest: Run every end-to-end test. - make inttest: Run every integration test. - make comptest: Run every component test. - make unittest: Run every unit test.

Apart from the test scripts in crane/tests, you need several Docker images to run tests that spawn containers. Build and pull them with make test-image. If you previously ran make install-dev, you're already set.

Docker-Compose¶

Crane is composed of multiple microservices, and integration/end-to-end tests require us to set up a certain subset of Crane's components and test interactions between them. Concretely, we want 1) the testing setup to be as close as possible to the real deployment of Crane, 2) to track bugs with one test case for each in a scalable manner, and 3) to reuse code for setting up Crane's components as much as possible.

Our take on this is to use docker-compose for testing. Basically, we write modular docker-compose files and compose them for each test case to achieve test-specific component setups. After setup, we spawn a test runner container that performs actions and assertions, and conveys whether the test succeeded or not through its exit code.

docker-compose files are kept in crane/tests/compose-files. You can write docker-compose tests with the following steps: 1) Check if the docker-compose files in crane/tests/compose-files are enough to setup your desired environment. If not, add what's missing. 2) Create a test script to be run by the test runner. While the script's location can be anywhere inside crane/tests, its name must match the pattern composetest_*.py. This is to prevent PyTest from collecting the test script as an independent test case. 3) Add a test case in a new or existing test file. Use the dcompose_runner PyTest fixture to get a test runner function. Pass the path of the test script and a list of compose file names to the test runner. 4) The path to your test script will be passed to the test runner via an environment variable (DCOMPOSE_TEST_SCRIPT), and the test runner will run py.test on it. In case the test fails, all the logs of all the containers are printed.

As an example, see crane/tests/integration/core/log/composetest_elasticsearch_log_manager.py and crane/tests/integration/core/log/test_run_log.py. The docker-compose tests you have written will be automatically discovered by PyTest.

Multi-node testing¶

Multi-node testing in crane is done via docker-machine. You need to install virtualbox and docker-machine. Docker-machine will create virtual machines, where each VM is a node.

# Install docker-machine from https://github.com/docker/machine/releases/
>>> $ curl -L https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-`uname -s`-`uname -m` \
> /tmp/docker-machine && \
chmod +x /tmp/docker-machine && \
sudo cp /tmp/docker-machine /usr/local/bin/docker-machine
---> 100%
>>> $ sudo apt install virtualbox
---> *

You can run crane tests in docker-machine via --use-docker-machine option.

Convenient Packages¶

Some quality-of-life python packages for development.

ipython¶

Async coroutines are difficult to test. Luckily ipython is capable of executing asynchronous code from the REPL. See here for details.