Installation and Overall Setup

Overall Setup

OptArrow Computation Service

OptArrow is an optimization integration engine designed to seamlessly connect the Python and Julia ecosystems. It addresses the technical and structural challenges of building scalable, high-performance optimization pipelines across languages.

This setup can also be used from MATLAB through the packaged client under src/matlab.

Requirements

  • Python >= 3.12 (recommended)

  • Julia >= 1.11 (recommended)

  • poetry for dependency and environment management (recommended best practice)

Currently supported solvers:

  • GLPK (not supported for QP problems)

  • GUROBI

  • HiGHS

  • Mosek

  • Hypatia (Julia only)

Make sure solver executables are installed and available in your system PATH.

1. Clone the repository

If you are not using Docker, clone the source code first:

git clone https://github.com/Digital-Metabolic-Twin-Centre/OptArrow.git
cd OptArrow

2. Install Python and Julia

3. Install dependencies

3.1 Install pipx

pipx installs Python applications in isolated environments and is recommended for installing poetry.

General:

pipx install poetry

Windows

Make sure Python 3.12+ is installed and added to PATH. In PowerShell:

py -m pip install --user pipx
py -m pipx ensurepath

Restart PowerShell (or your terminal), then verify:

pipx --version

macOS

Install via Homebrew:

brew install pipx
pipx ensurepath

Or via pip:

python3 -m pip install --user pipx
python3 -m pipx ensurepath

Restart your terminal (or run source ~/.zshrc), then verify:

pipx --version

Linux (Debian/Ubuntu)

sudo apt update
sudo apt install -y python3.12 python3.12-venv python3-pip
python3.12 -m pip install --user pipx
python3.12 -m pipx ensurepath
echo 'export PATH=$PATH:~/.local/bin' >> ~/.bashrc
source ~/.bashrc
pipx --version

Fedora / RHEL

sudo dnf install pipx
pipx ensurepath

3.2 Install Python dependencies

Install poetry first (via pipx), then install project dependencies from the project root:

pipx install poetry
poetry install --no-root

Using pip directly (or other dependency managers) is not recommended, to avoid environment inconsistency.

3.3 Install Julia dependencies

Launch Julia in project mode:

julia --project=./src/service/optimization_service/julia

Then instantiate dependencies in the Julia REPL:

julia> ]
(pkg) instantiate

4. Start all service instances (engines + gateway)

Start all services:

sh scripts/startAll.sh

Start services individually:

  • Gateway only:

    sh scripts/startServer.sh

  • Python engine only:

    sh scripts/startPyEngine.sh

  • Julia engine only:

    sh scripts/startJulia.sh

5. Services are ready

Optional: MATLAB Client Setup

MATLAB R2023b or later is required. Arrow IPC serialization is handled natively in MATLAB using the MATLAB Interface to Apache Arrow, which must be built once from the Apache Arrow source repository. See the full build instructions in MATLAB Interface for OptArrow.

Once the add-on is installed:

  1. Add the MATLAB client to your path:

    addpath(genpath(fullfile(repoRoot, 'src', 'matlab')));

  2. Configure the OptArrow client:

    cfg = struct( ...
    'endpoint',      'http://127.0.0.1:8000/compute', ...
    'engine',        'python', ...
    'backendSolver', 'HiGHS', ...
    'timeoutSec',    120);
optarrow.setOptArrowConfig(cfg);

  3. Continue with the full guide in MATLAB Interface for OptArrow.

After services are running and communicating, run the following Python example to solve a simple LP problem:

import pyarrow as pa
import requests

ipc_dict = {
     "model": {
          "A": {
                "row": [0, 0, 1, 1, 2, 2],
                "col": [0, 1, 0, 1, 0, 1],
                "val": [20, 10, 10, 20, 10, 30]
          },
          "b": [200, 120, 150],
          "c": [5, 12],
          "lb": [0, 0],
          "csense": ["L", "L", "L"],
          "osense": "max"
     },
     "model_name": "product_mix_lp",
     "engine": "julia",
     "solver": {
          "solver_name": "HiGHS",
          "solver_type": "LP",
          "solver_params": {}
     }
}

pa_arrays = [pa.array([value]) for value in ipc_dict.values()]
table = pa.Table.from_arrays(pa_arrays, names=list(ipc_dict.keys()))

sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, table.schema) as writer:
     writer.write(table)
ipc_bytes = sink.getvalue().to_pybytes()

headers = {"Content-Type": "application/vnd.apache.arrow.stream"}
response = requests.post("http://localhost:8000/compute", data=ipc_bytes, headers=headers)

if response.status_code != 200:
     print(f"Error: {response.status_code} - {response.text}")

reader = pa.ipc.open_stream(response.content)
result_table = reader.read_all()
result_dict = {name: result_table.column(name).to_pylist() for name in result_table.column_names}
print(result_dict)

Expected output (example):

{'success': [True], 'status': ['OPTIMAL'], 'obj_val': [66.0], 'solution': [[6.000000000000004, 2.9999999999999987]]}

6. Alternative: Start services with Docker

Docker can run services in isolated containers for production-style deployments.

Gateway service (server.dockerfile):

docker build -t gateway-server -f server.dockerfile .
docker run -d --net=host gateway-server

Python engine service (py_engine.dockerfile):

docker build -t py-engine-server -f py_engine.dockerfile .
docker run -d --net=host py-engine-server

Julia engine service (julia_engine.dockerfile):

docker build -t julia-engine-server -f julia_engine.dockerfile .
docker run -d --net=host julia-engine-server

Start all services together with Compose:

docker compose -f compose.yaml up

More Information

Configure service ports

See config.yaml for service IP/port configuration. If ports are changed, update Dockerfiles accordingly so the correct ports are exposed.

Scripts in ./scripts

envSetup.sh can be used to install dependencies on Debian-based Linux systems. Other scripts start individual services, and startAll.sh starts all services locally.

Extending the code

See Contributing.