metadata
license: mit
pipeline_tag: image-text-to-text
MoAI model
This repository contains the weights of the model presented in MoAI: Mixture of All Intelligence for Large Language and Vision Models.
Simple running code is based on MoAI-Github.
You need only the following seven steps.
[0] Download Github Code of MoAI, install the required libraries, set the necessary environment variable (README.md explains in detail! Don't Worry!).
git clone https://github.com/ByungKwanLee/MoAI
bash install
[1] Loading Image
from PIL import Image
from torchvision.transforms import Resize
from torchvision.transforms.functional import pil_to_tensor
image_path = "figures/moai_mystery.png"
image = Resize(size=(490, 490), antialias=False)(pil_to_tensor(Image.open(image_path)))
[2] Instruction Prompt
prompt = "Describe this image in detail."
[3] Loading MoAI
from moai.load_moai import prepare_moai
moai_model, moai_processor, seg_model, seg_processor, od_model, od_processor, sgg_model, ocr_model \
= prepare_moai(moai_path='BK-Lee/MoAI-7B', bits=4, grad_ckpt=False, lora=False, dtype='fp16')
[4] Pre-processing for MoAI
moai_inputs = moai_model.demo_process(image=image,
prompt=prompt,
processor=moai_processor,
seg_model=seg_model,
seg_processor=seg_processor,
od_model=od_model,
od_processor=od_processor,
sgg_model=sgg_model,
ocr_model=ocr_model,
device='cuda:0')
[5] Generate
import torch
with torch.inference_mode():
generate_ids = moai_model.generate(**moai_inputs, do_sample=True, temperature=0.9, top_p=0.95, max_new_tokens=256, use_cache=True)
[6] Decoding
answer = moai_processor.batch_decode(generate_ids, skip_special_tokens=True)[0].split('[U')[0]
print(answer)