# CML self hosted runners on demand with GPUs

**URL:** <https://discuss.dvc.org/t/cml-self-hosted-runners-on-demand-with-gpus/462>\
**Category:** Blog Comments\
**Created:** [August 4, 2020, 6:34pm UTC](https://discuss.dvc.org/t/cml-self-hosted-runners-on-demand-with-gpus/462 "2020-08-04T18:34:28Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![elle](https://avatars.discourse-cdn.com/v4/letter/e/ccd318/32.png) [@elle](https://discuss.dvc.org/u/elle)\
**Post date:** [August 4, 2020, 6:34pm UTC](https://discuss.dvc.org/t/cml-self-hosted-runners-on-demand-with-gpus/462/1 "2020-08-04T18:34:28Z")

</div>

> **[CML self-hosted runners on demand with GPUs](https://iterative.ai/blog/cml-self-hosted-runners-on-demand-with-gpus)**
>
> Use your own GPUs with GitHub Actions & GitLab for continuous machine learning.

---

<div class="post-metadata">

**Author:** ![bmabir17](https://avatars.discourse-cdn.com/v4/letter/b/c4cdca/32.png) [@bmabir17](https://discuss.dvc.org/u/bmabir17)\
**Post date:** [March 31, 2022, 11:20am UTC](https://discuss.dvc.org/t/cml-self-hosted-runners-on-demand-with-gpus/462/2 "2022-03-31T11:20:04Z")

</div>

Thank you for this tutorial.  
But i faced a problem while running the runner.  
The docker run command needs an additional command to be added in order to run the runner properly.  
The whole example is shown below

```auto
docker run --name myrunner -d --gpus all \
    -e RUNNER_IDLE_TIMEOUT=1800 \
    -e RUNNER_LABELS=cml,gpu \
    -e RUNNER_REPO=$my_repo_url \
    -e repo_token=$my_repo_token \
    iterativeai/cml:0-dvc2-base1-gpu runner

```

---

<div class="post-metadata">

**Author:** ![Casper.dcl](https://avatars.discourse-cdn.com/v4/letter/c/e47774/32.png) [@Casper.dcl](https://discuss.dvc.org/u/Casper.dcl)\
**Post date:** [March 31, 2022, 12:19pm UTC](https://discuss.dvc.org/t/cml-self-hosted-runners-on-demand-with-gpus/462/3 "2022-03-31T12:19:31Z")

</div>

Thanks @bmabir17 for pointing this out ([blog/cml: check for GPU · Issue #3408 · iterative/dvc.org · GitHub](https://github.com/iterative/dvc.org/issues/3408))!

---

<div class="post-metadata">

**Author:** ![RaghavaAlajangi](https://avatars.discourse-cdn.com/v4/letter/r/e56c9b/32.png) [@RaghavaAlajangi](https://discuss.dvc.org/u/RaghavaAlajangi)\
**Post date:** [March 15, 2023, 9:02am UTC](https://discuss.dvc.org/t/cml-self-hosted-runners-on-demand-with-gpus/462/4 "2023-03-15T09:02:35Z")

</div>

Thanks for the blog  
I have a small question. Can we also create a self hosted runner with CPU machine?

Moreover, when I tried to run the pipeline via gitlab ci, I could not install required python dependencies (pip install -r requirements). I got a warning message followed by an error.

```auto
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.python.org', port=443): Read timed out. (read timeout=15)")': /simple/joblib/

ERROR: No matching distribution found for joblib==1.2.0

```

I thought it was because of a network issue but it was not. I also tried to recreate the docker image but still, the same problem persisted. Can somebody help me how can I fix this?

---

<div class="post-metadata">

**Author:** ![0x2b3bfa0](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/0x2b3bfa0/32/119_2.png) [@0x2b3bfa0](https://discuss.dvc.org/u/0x2b3bfa0)\
**Post date:** [March 15, 2023, 11:54am UTC](https://discuss.dvc.org/t/cml-self-hosted-runners-on-demand-with-gpus/462/5 "2023-03-15T11:54:58Z")

</div>

> _Can we also create a self hosted runner with CPU machine?_

Of course! Are you using `cml runner` in conjunction with the `--cloud` option?

> _Moreover, when I tried to run the pipeline via GitLab CI/CD, I could not install required python dependencies_ […]

Sounds very similar to [iterative/cml#1324 (comment)](https://github.com/iterative/cml/issues/1324#issuecomment-1404160854); can you log in to the self-hosted runner and try running the `pip install` command manually? It looks like a connectivity issue. 🤔

> <https://github.com/iterative/cml/issues/1324#issuecomment-1404160854>
>
> I am a new user. 
> When the ci cd pipeline runs 
> it throws the error as cml:co…mmand not found. 
> \<pre\>
> Requirement already satisfied: six\>=1.5 in /usr/lib/python3.10/site-packages (from python-dateutil\>=2.7-\>matplotlib-\>-r requirements.txt (line 3)) (1.16.0)
> $ python train.py
> 0.852
> $ cat metrics.txt \>\> report.md
> $ echo '!\[\](./plot.png "Confusion Matrix")' \>\> report.md
> $ cml comment create report.md
> bash: cml: command not found
> Cleaning up project directory and file based variables
> 00:00
> ERROR: Job failed: exit status 1
> 
> \</pre\>
> 
> 
> 
> my ci cd pipeline as follows 
> 
> \<pre\>
> 
> train-and-report:
> image: iterativeai/cml:0-dvc2-base1
> script:
> - pip install -r requirements.txt
> - python train.py # generate plot.png
> # Create CML report
> - cat metrics.txt \>\> report.md
> - echo '!\[\](./plot.png "Confusion Matrix")' \>\> report.md
> - cml comment create report.md
> tags:
> - runner-tester
> \</pre\>

---

<div class="post-metadata">

**Author:** ![RaghavaAlajangi](https://avatars.discourse-cdn.com/v4/letter/r/e56c9b/32.png) [@RaghavaAlajangi](https://discuss.dvc.org/u/RaghavaAlajangi)\
**Post date:** [March 16, 2023, 10:06am UTC](https://discuss.dvc.org/t/cml-self-hosted-runners-on-demand-with-gpus/462/6 "2023-03-16T10:06:11Z")

</div>

@0x2b3bfa0 Thanks for the reply

I am using my ssh cloud instance as a self-hosted runner. I have installed the CML runner on my machine by following this guide ([Install as a Package](https://cml.dev/doc/install)) and executed the following command:

```auto
cml runner launch \
  --repo="$REPOSITORY_URL" \
  --token="$PERSONAL_ACCESS_TOKEN" \
  --labels="cml-gpu" \
  --idle-timeout="never" # or "3min", "1h", etc..

```

When I executed the above command, it successfully ran the CML runner. However, it failed (with above mentioned warning messages) to install the `requirements.txt` or `pypi packages` that were given via GitLab CI.

yes, I tried to install the packages manually it worked. And also, checked the internet connection with `ping google.com`. It is working.

In addition, I tried docker approach to run the CML DVC container as mentioned in above blog.

```auto
sudo docker run --name RUNNER_NAME \
-e RUNNER_IDLE_TIMEOUT=100h \
-e RUNNER_LABELS=cml, cpu \
-e RUNNER_REPO= REPO_URL \
-e repo_token= REPO_TOKEN \
dvcorg/cml-py3:latest

```

But, it gave me an error

```auto
{"level":"error","status":"terminated"}
Error: REPO_TOKEN does not have enough permissions to access workflow API
    at CML.repo_token_check (/cml/src/cml.js:236:13)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
        Destroy scheduled: 30 seconds remaining.
        No TF resource found

```

I tried to resolve this issue by creating a new access token with all the required permissions, but nothing seems to work out.