Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Future Expansion #99

Open
QNNgg opened this issue Jan 8, 2025 · 1 comment
Open

Future Expansion #99

QNNgg opened this issue Jan 8, 2025 · 1 comment

Comments

@QNNgg
Copy link

QNNgg commented Jan 8, 2025

Thank you for the impressive progress in General Computer Control.

I want to ask are there plans to expand CRADLE's capabilities to a broader range of software and games? What criteria will be used to select new targets?

What strategies are being considered to address the scalability challenges posed by the increasing number of applications and games?

Thank you.

@QNNgg QNNgg changed the title 徐博,占楼 Future Expansion Jan 8, 2025
@WeihaoTan
Copy link
Collaborator

WeihaoTan commented Jan 10, 2025

Thanks for reaching out. Yes, we indeed have plans. As for the criteria, it still heavily relies on the capability of the based model. They are still struggling with outputting the exact pixel-level position, which greatly limits the performance of mouse control. And they have difficulty in understanding the game/software-specific artifacts. So if you want to check whether Cradle can be developed into a new game/software, you need to check whether the base model can somehow (no need to be 100% correct) understand the screenshot and output precise keyboard and mouse actions. Then Cradle can greatly enhance the raw performance.

As the cutting-edge base model improves (e.g., GPT5) or the models designed for computer use appear, more and more games/software will be handled. Besides waiting/developing new models, users can also introduce prior knowledge to temporarily improve performance through prompting and tool use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants