Skip to content

Installation

From PyPI

Flicker is available on PyPI. You can install it using pip.

1
pip install flicker

Use the --user if you're installing outside a virtual environment. Flicker is intended for Python 3.

Within a Docker container

If you just want to try out Flicker (or even PySpark) without having to install or setup anything, consider using the flicker-playground available at DockerHub. It comes with Java 8, PySpark, Flicker, Jupyter, IPython, and other python packages commonly needed when using Spark.

1
2
# From DockerHub (https://hub.docker.com/r/ankurio/flicker-playground)
docker pull ankurio/flicker-playground

The corresponding Dockerfile is available inside the repository here, if you choose to build it yourself. The instructions to build the Docker image are available here.

After pulling (or building) the Docker image, you can mount data and run a Jupyter notebook or IPython from it.

Jupyter Notebook

1
2
3
4
5
6
7
# From $REPO_ROOT/docker in the host machine
docker run -it \
-p 8888:8888 \
-p 8080:8080 \
-p 4040:4040 \
--volume $PWD/notebooks:/home/neo/notebooks \
ankurio/flicker-playground

IPython

1
2
3
4
5
6
7
8
# From $REPO_ROOT/docker in the host machine
docker run -it \
-p 8080:8080 \
-p 4040:4040 \
--volume $PWD/notebooks:/home/neo/notebooks \
ankurio/flicker-playground /bin/bash
# Run ipython on the shell prompt
# neo@95657e2ed2c6:~$ ipython

The ports 8080 and 4040 below are for Spark's Web UI. In the following commands, we bind the Docker container's ports to host machine's ports. We will still need to setup the Spark session inside the container to use these ports. Typically, the Spark's Web UI can be accessed at http://localhost:8080 for Spark Master Web UI (which may not always be available) and at http://localhost:4040 (for a particular Spark app).

Building from source

This may be needed if you want to develop and contribute code.

Clone the Repository

1
git clone git@github.com:ankur-gupta/flicker.git

Install the requirements

It is recommended that you use a virtual environment for the rest of the steps. 

1
2
# Within $REPO_ROOT
pip install -r requirements.txt

Run the tests

1
2
3
# Within $REPO_ROOT
# Within a virtual environment with requirements installed
python -m pytest flicker/tests

Build the package

A script is available to build the package. This helps standardize what build artifacts are generated. 

1
2
3
# Within $REPO_ROOT
# Within a virtual environment with requirements installed
./scripts/build-package

Last update: 2020-08-13