Mining Kandinsky-2 on Arbius
This guide will walk you through the process of mining Kandinsky-2 on the Arbius network. Kadinsky-2 is the model Arbius is launching with as the model to bootstrap miners to the network and reward them. For this model, it is configured to run on Nvidia A100 GPU. In the future, we plan to support other chipsets, for now you will need to have access to a A100 GPU to mine Arbius.
Get A100 GPU
We recommend Lambda Labs cloud hosting for this model, this been tested to function properly. There are many service providers of A100 chips via GPU clouds and other places, in the likely case that you don't have a A100 chip yourself.
The recommended storage space required for the computer running the A100 chip is 60GB. The miner and the A100 GPU instance does not need to be the same computer, the actual miner software just needs the URL for the instance running the COG.
What is a Cog file?
Cog is an open-source tool that packages machine learning models in a standard, production-ready container. In Arbius we deploy cogs via docker, and the miner connects to the docker instance for model input and output.
Therefore to mine Arbius, we have to deploy a docker image of a registered model.
Docker
You will need to have docker installed on the computer with the A100 chip. Some GPU cloud services offer dockerized containers, so you will not need to install docker yourself, but will need to run the miner on your own computer or separate VPS.
If running your own computer with the A100 chip, after installing docker:
docker run -d -p 5000:5000 --gpus=all r8.im/kasumi-1/kandinsky-2@sha256:373fa540ae197fc89f0679ed835bc4524152956d4f3027580244e10b09d6d3a5
The URL above is the Kadinsky-2 model required to mine Arbius. There is only one model for now.
Other providers
Community members have also managed to run the model on other cloud gpus such as RunPod and Vast.
RunPod
You need to create a new GPU Pod with the Docker image.
- Create a new A100 GPU Pod and click deploy.
- Customize Deployment.
- Set the Docker Image to
r8.im/kasumi-1/kandinsky-2@sha256:373fa540ae197fc89f0679ed835bc4524152956d4f3027580244e10b09d6d3a5
. - Set the Storage to
60GB
. - Expose HTTP port
5000
if you wish to run the miner script on a different server. - Set Overrides.
- Click Continue and then Deploy.
- Go to pods page and wait for the pod to be ready and for the model to be downloaded then connect to the server either using the web terminal or ssh using the provided credentials.
- Setup mining software using the instructions found below.
Vast
- Download Vast CLI from here.
- Find a suitable instance using the GUI or the website and note it's id (found in the bottom left as
Type #<id>
). - Run the following command after installing the Vast CLI, replacing
<instance-id>
with the number noted down earlier:
./vast create instance <instance-id> --image r8.im/kasumi-1/kandinsky-2@sha256:373fa540ae197fc89f0679ed835bc4524152956d4f3027580244e10b09d6d3a5 --env '-p 5000:5000 -p 22:22' --onstart-cmd 'python -m cog.server.http' --jupyter-dir /src --ssh --direct --disk 100
/.vast
with vastai
.- Setup mining software using the instructions found below.
Setup Mining Software and Dependencies
You may run the mining software on a local machine or on a cloud instance. Especially if using a cloud instance, ensure that you properly secure the machine.
Install NVM
NVM is a version manager for Node.js, which is required to run the mining software.
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
Install IPFS (InterPlanetary File System)
IPFS is a distributed file system that will be used to share generated content.
Follow this guide to ensure you have IPFS installed:
https://docs.ipfs.tech/install/command-line/
Make sure you are running IPFS as a daemon:
ipfs init
ipfs daemon
You will need to keep IPFS on whilst the miner is running.
Install Arbius Reference Miner
The Arbius reference miner is a simple mining software that will be used to mine Kandinsky-2 on the Arbius network. It is designed to be easy to use and debug. We recommend reading the code to understand how it works, and to contribute improvements upstream.
git clone https://github.com/semperai/arbius.git
cd arbius/miner
nvm use
# if you don't have yarn installed already
npm install -g yarn
yarn # install dependencies
Setup RPC Endpoint
You will need to provide an RPC endpoint for the Arbius network. You can run your own node, or use a public node. We recommend running your own node for security and performance reasons, however it is not necessary. For testing setup, you can use a public node.
https://docs.arbitrum.io/node-running/node-providers
Note: you will need to run your node on Arbitrum Nova network.
Generate Wallet for Miner
Arbius has a built in wallet generator, you can also use MetaMask or other tools to generate a new private key. We strongly recommend using a new private key for each miner.
cd arbius/contract
cp .env.example .env
nvm use
yarn
npx hardhat gen-wallet
Configure Mining Software
Within the arbius/miner
directory, copy the example configuration file to a new file called MiningConfig.json
.
cp MiningConfig.example.json MiningConfig.json
Read-only Configuration
Edit MiningConfig.json
to include the readonly
flag. This will prevent the miner from submitting solutions to the network. This is useful for testing the miner on a live network without performing any sort of transactions. You may want to use this while testing various prob
configurations.
{
"readonly": true
}
Blockchain Configuration
Edit MiningConfig.json
to include your miners private key and the RPC endpoint.
{
"blockchain": {
"private_key": "0x...",
"rpc_url": "https://..."
}
}
Automine Configuration
Arbius reference miner has an automine feature that will automatically generate new tasks. These tasks are available for anyone to solve, and you will be rewarded for solving them. You can enable automine by setting the automine
flag to true
in MiningConfig.json
. If automine is disabled, your miner will only mine tasks that it notices on the network.
{
"automine": {
"enabled": true
}
}
You will want to set the correct model
and a unique prompt
for tasks.
ML Configuration
Here is where you'll configure to connect to a running Cog container. Grab the prediction url from the api link of the Cog container you launched earlier. It will look something like: https://xxxxxxxxxxxxx.notebooksd.jarvislabs.net/predictions
. Make sure you do not share this url with anyone, as it can be used to access your Cog container.
If your cog container is running on a separate server, you will need to make sure that the server allows incoming connections on the provided port.
{
"ml": {
"strategy": "cog",
"cog": {
"0x98617a8cd4a11db63100ad44bea4e5e296aecfd78b2ef06aee3e364c7307f212": {
"url": "https://..."
}
}
}
}
IPFS Configuration
Here is where you'll configure to connect to a running IPFS node. If you are using pinata, you will need to use the pinata API url (see example cofig). You will want to use http_client
strategy, to connect to your own IPFS, or pinata
to connect to.. pinata.
{
"ipfs": {
"strategy": "http_client",
"http_client": {
"url": "http://localhost:5001"
}
}
}
Probability Configuration
The prob
configuration is used to set the probability of the miner to perform certain actions. Setting this to 0
will disable the action, and setting it to 1
will ensure the action is performed every time. To participate only in contestations, you can set every value to 0
except for contestation_submitted
. To only validate others solutions, you can set every value to 0
except for solution_submitted
(this will still cause you to contest invalid solutions). To
only attempt to mine solutions and not validate others tasks, you can turn task
to 1
or some smaller number if that is too much for your miner.
{
"prob": {
"task": 0.05,
"contestation_vote_finish": 0.2,
"contestation_submitted": 1.0,
"solution_submitted": 0.01,
"task_retracted": 1.0
}
}
Load Up Miner Wallet
You will need to load your miner wallet with some funds to pay for gas. You can do this by sending some funds to the address associated with your private key.
Arbitrum Nova
You can bridge funds from Ethereum to Arbitrum Nova using the Arbitrum bridge. You will need to use the Arbitrum bridge to send funds to your miner wallet.
https://bridge.arbitrum.io/?destinationChain=arbitrum-nova&sourceChain=ethereum
Arbius Token
Arbius(AIUS) token needs to be staked to mine. For the first day, no AIUS token is required to begin mining. After the first day, you will need to stake AIUS token to mine. You can obtain AIUS token from Uniswap. The miner will automatically stake AIUS token when it is required.
Test Mining Software
Once your miner has sufficient funds and has been configured you can start it by running:
yarn start:dev MiningConfig.json
Your miner will perform a bootup sanity check by checking version as well as a test generation of the Kandinsky-2 model. If everything is good, it will start mining.
Troubleshooting
If you encounter issues, please investigate the logs to determine the cause of the issue. If you are unable to resolve the issue, please reach out to the Arbius community for support.
If the miner seems stuck or is not mining, you can try restarting it. If this does not resolve the issue, try copying the db.sqlite file to a new location (for backup) and restarting the miner. This will allow the miner to start fresh.
RPC Operation (advanced)
The miner provides a http api that allows the operator to control operation. This should never be exposed to the internet.
Queue Jobs
To queue a job use the api/jobs/queue
route.
curl "localhost:8335/api/jobs/queue" \
-X POST \
-H 'Content-Type: application/json' \
--data '{"method":"automine", "priority": 5, "waituntil": 0, "concurrent": false, "data": {}}'
A higher priority
will increase the priority the job is run. waituntil
defines the unix timestamp at which to run the task. concurrent
sets whether or not this job will run concurrent with other jobs (usually it should be set to true, otherwise errors may occur).
Methods
pinTaskInput
taskid
[string]input
[object]
task
taskid
[string]txid
[string]
validatorStake
validatorMinimum
[string]
automine
solve
taskid
[string]
claim
taskid
[string]
contestationVoteFinish
taskid
[string]
List jobs
curl "localhost:8335/api/jobs/list" \
-X POST \
-H 'Content-Type: application/json'
Delete job
curl "localhost:8335/api/jobs/delete" \
-X POST \
-H 'Content-Type: application/json'
--data '{"id":37}'