Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Protobuf is not installed #104

Open
mrakgr opened this issue Jun 1, 2023 · 15 comments
Open

Protobuf is not installed #104

mrakgr opened this issue Jun 1, 2023 · 15 comments

Comments

@mrakgr
Copy link

mrakgr commented Jun 1, 2023

Installing collected packages: torch, torchvision
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
open-clip-torch 2.7.0 requires protobuf==3.20.0, which is not installed.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Successfully installed torch-2.0.1+cu118 torchvision-0.15.2+cu118
Installing requirements

I get this issue after I created a new notebook from your docker container. Do I maybe need to update the container?

@mrakgr
Copy link
Author

mrakgr commented Jun 1, 2023

tensorflow 2.9.2 requires protobuf<3.20,>=3.9.2, but you have protobuf 4.23.2 which is incompatible.
Successfully installed bs4-0.0.1 markdownify-0.11.6 six-1.16.0

Actually, seeing further in the output, it seems some of the dependencies are broken.

@mrakgr
Copy link
Author

mrakgr commented Jun 1, 2023

# Import launch.py which will automatically run the install script but not launch the WebUI.
import launch
launch.prepare_environment()

Mhh, it installs protobuf using this, so I am not sure what to do about it. Where is import.py coming from?

...It is in the web UI.

@mrakgr
Copy link
Author

mrakgr commented Jun 1, 2023

Let me summarize what is going on. When the container is initially deployed, it has protobuf 3.20. Then after we run the setup script, it uninstalls it on this line here:

# Uninstall PyTorch and some other libraries so the WebUI can install the versions it needs
!pip uninstall -y torch torchvision torchaudio protobuf

Then it tries installing it again on these lines.

# Import launch.py which will automatically run the install script but not launch the WebUI.
import launch
launch.prepare_environment()

Unfortunately, this gets the 4.23.2 version which breaks everything. I am not sure what to do here. I hadn't expected this. I took on the quest of updating my WebUI install due to some people asking me how to install extensions, but I hadn't through that the entire build would be broken!

@mrakgr
Copy link
Author

mrakgr commented Jun 1, 2023

I am not sure what to do here, I am trying to install using some of the earlier versions like v1.1.0, but it still seems to be fetching the most up to date protobuf.

@mrakgr
Copy link
Author

mrakgr commented Jun 1, 2023

A workaround for this is to pip install protobuf==3.20.0. And remove the part where uninstalls it. Then it will proceed unimpeded.

That having said, the xformers don't work for me either.

@mrakgr
Copy link
Author

mrakgr commented Jun 1, 2023

I'd recommend commenting out the entire !pip uninstall -y torch torchvision torchaudio protobuf line as it makes the startup a lot faster.

As mentioned in the other issue, one of the dependencies installing tensorflow is the problem.

I also don't understand why xformers are not working properly.

@mrakgr
Copy link
Author

mrakgr commented Jun 1, 2023

# The installer isn't installing deepdanbooru right now so we'll do it manually.
if activate_deepdanbooru:
    # https://github.com/KichangKim/DeepDanbooru/releases
    !pip install "git+https://github.com/KichangKim/DeepDanbooru.git@v3-20211112-sgd-e28#egg=deepdanbooru[tensorflow]" # $(curl --silent "https://api.github.com/KichangKim/DeepDanbooru/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')#egg=deepdanbooru[tensorflow]" # tensorflow==2.10.0 tensorflow-io==0.27.0 flatbuffers==1.12

In the other issue this was mentioned by this branch should be false by default so the most likely case is that TF comes with the container itself.

@mrakgr
Copy link
Author

mrakgr commented Jun 1, 2023

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Installing xformers from wheels on Github...
--2023-06-01 09:51:01--  https://github.com/Cyberes/xformers-compiled/raw/main/a4000/xformers-0.0.18%2Bda27862.d20230413-cp39-cp39-linux_x86_64.whl
Resolving github.com (github.com)... 140.82.112.3
Connecting to github.com (github.com)|140.82.112.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/Cyberes/xformers-compiled/main/a4000/xformers-0.0.18%2Bda27862.d20230413-cp39-cp39-linux_x86_64.whl [following]
--2023-06-01 09:51:02--  https://raw.githubusercontent.com/Cyberes/xformers-compiled/main/a4000/xformers-0.0.18%2Bda27862.d20230413-cp39-cp39-linux_x86_64.whl
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 19155376 (18M) [application/octet-stream]
Saving to: ‘/tmp/tmp.IxO3pArqCc/xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl’

/tmp/tmp.IxO3pArqCc 100%[===================>]  18.27M  --.-KB/s    in 0.06s   

2023-06-01 09:51:02 (313 MB/s) - ‘/tmp/tmp.IxO3pArqCc/xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl’ saved [19155376/19155376]

WARNING: Skipping xformers as it is not installed.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
ERROR: xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl is not a supported wheel on this platform.

===================================
Done! If you're seeing this the process has exited successfully.

Maybe the reason xformers aren't working is because cp39.

Probably that stands for C Python 3.9 while the container comes with 3.10.

@mrakgr
Copy link
Author

mrakgr commented Jun 1, 2023

If you are using the special Python 3.10 container cyberes/gradient-base-py3.10:latest you need to either install xformers through pip or build them yourself.

First, try installing it through pip by setting install_pip_xformers = True in the settings block above. If that fails or you encounter issues, use the block in the Tools section to build xformers for your machine.

Ah, it says as much in one of the cells. Didn't realize it.

@mrakgr mrakgr closed this as completed Jun 1, 2023
mrakgr added a commit to mrakgr/The-Spiral-Language that referenced this issue Jun 1, 2023
Is this a desktop UI or a webUI? If it is the later, it might be worth replacing Automatic1111.

https://github.com/AUTOMATIC1111/stable-diffusion-webui/tree/v1.3.0

It has tags now.

https://github.com/AUTOMATIC1111/stable-diffusion-webui/tags

Oh, they started using these recently.

https://youtu.be/2NGblBsFbCs?t=152

You can share access.

https://youtu.be/L9-6RDN8sEM
How To Automate Downloading Of Albums From Places Like Soundcloud & Youtube (In An Easy Way!)

I forgot to post this on my Twitter.

This is the true path. I wish I had done this from the start. Gradual accumulation by making these kinds of useful videos and tools is the way to go.

https://youtu.be/gySLXbe7WZQ?t=545

Automatic1111, InvokeAI, and now ComfyUI.

I'll have to try out ComfyUI to see how it feels. There isn't much I can do about it.

https://youtu.be/vUTV85D51yk
ComfyUI - Node Based Stable Diffusion UI - THIS IS THE FUTURE!!!!!

Let's watch this.

https://youtu.be/vUTV85D51yk?t=365

I need to master this. ComfyUI is the way forward, no matter what.

https://youtu.be/vUTV85D51yk?t=1031

I have no idea what these are.

I just didn't have the mental energy to understand that post about Photoshop style inpainting, but I'll believe it can be implemented in ComfyUI. The guy said something about ControlNet cascades.

5:30pm. You know, I am kind of jealous of Mojo, but if I think about it more clearly, a Spiral backend for Mojo would be way better than that piece of crap Cython.

https://youtu.be/vUTV85D51yk?t=1116

This is cool.

https://youtu.be/vUTV85D51yk?t=1150
> Here we have clip set last layer, I don't know what that is.

It just takes off the clip layers from the top.

///

>>93781186 (OP)
Yeah. Tell me again why we're automating culture tasks and not physical labor ones?

///

///

>>93782124
Because culture tasks are easy. It's only been a couple thousand of years since we started doing them and we suck at it. We suck at maths and art. In contrast, our physical abilities took hundreds of thousands of years to evolve. They are difficult, we only think they are easy because our bodies are so insanely optimized for them through long evolutionary processes. Children learn to walk when they are a couple of years old, but programming a biped robot to do it properly (being able to stand up after falling, for example) is very difficult.

///

///

>>93782195
>>93782124
we made computers cause we found doing lots of maths hard to do and labour (mostly) easy. solving linear algebra equations isn't hard for computers (things we MADE to do maths) to do, so naturally some guy had the idea to use that to make art. you don't need to worry about things like size constraints, power, speed, etc. because art is for consuming and there's no defined value. if you're making an automated refinery and it's 1% less efficient than other means, no-one will use it. but if you're making art using 1% more energy, it doesn't matter how (in)efficient it is so long as people value it more.
>computers were made to do maths since humans are bad at it, someone found a way to make art from lots of maths equations, now computers can make art easily

///

///

>>93782124
We are starting with the easy ones

///

///

>>93782124
Because robots can manipulate the complex underlying connections between various concepts that can be described by language that humans don't even see while human artists are powerless but to create soulless simulacra of their narrow experiences. And they don't expect to get paid 200 dollars to draw a caricature of your fursona.
Whereas labor tasks that require complex dexterity require billion dollar robots to replace.

///

///

>>93782124
Because robots can manipulate the complex underlying connections between various concepts that can be described by language that humans don't even see while human artists are powerless but to create soulless simulacra of their narrow experiences. And they don't expect to get paid 200 dollars to draw a caricature of your fursona.
Whereas labor tasks that require complex dexterity require billion dollar robots to replace.

///

///

>>93782124
Automating physical tasks requires building of complex physical robots which is much harder than automating mental tasks - here you don't have to build anything, it's directly in the computer (software).

///

///

>>93782124
A lot of (((art))) tasks are not culture tasks but menial shit. Talking about concepts, ads, and the likes. The reason we're not automating physical labor is because most of it requires high-speed autonomous bipedal operations. We can surely come up with something else, but it's a very complex mix of hardware and software, and both parts need breakthroughs to come forward. You probably don't realize, but in the history of automation, the basic pattern has been not to automate what we've been doing so far, but to modify what we do so that it can be automated. For example, wool is now a luxury product, yet fleece sells for pennies. The reason is that it's impossible to automate the fabrication of woolen items. Same reason why we don't do copper stuff anymore despite copper cookware having all kinds of good properties, they can be machined or spun, but not at scale.
Thus, we have no actual experience in replacing human labor. This is the crux of the issue: unlike assembly line production of chinese goods, we can't really bring, say, the land you bought into the factory to build the house on it (you can, however, make house parts in a factory and assemble that on the land).

///

This is an interesting point.

///

>>93782124
The goal is automating the later but right now low end wages are still so cheap. Same reason we've been spamming plastics for 50 years. We have access to more sustainable materials but plastic just prints itself in terms of profit.
Yours, and everyone's here error is assuming there's going to be someone giving a shit abut +90% of population once they've automated most of lucrative sectors of society, and they're not just going to put us in a arena to fight for scraps. Alas, Hunger Games is a garbage teen drama tier metaphore, but is probably what's going to happen. After all, unlike us peasants they "worked hard" to get where they'll be :)

///

///

>>93782124
AI movies are slated to be release on the same day as everything robots. The time frame we are living through now is just the behind the scenes build up.

///

6:40pm. https://www.linkedin.com/jobs/search/?currentJobId=3605853930&distance=25&f_JT=C&f_WT=2&geoId=91000000&keywords=blazor%20.net

When I applied to a Noir job, the HR actually replied and gave me advice to apply to contract jobs.

I'll make those kinds jobs my target next.

Maybe it is a better fit with what I want in a job?

///

**Milestones & Pay:**

We have divided the project into several milestones, each with a specified duration and payment of $10,000 upon completion. The estimated project duration is 34-46 weeks.

1. **Project Setup, Design, and Planning ($10,000 | 4 Weeks)**

- Requirement gathering, scoping the project, and setting up development/staging/production environments.
- Database design and architecture planning.
- UI/UX design and planning.

2. **Basic Features - User Registration/Login and Profile Management ($10,000 | 4-6 Weeks)**

- Implement user registration and login systems.
- Develop user profile management functionalities.
- Ensure smooth site navigation.
- Include the job type (e.g., full-time, part-time, contract, etc.) and the employer's contact details, including the company's physical address and a valid email associated with the company domain.

3. **Content Management - Posting and Live Streaming ($10,000 | 6-8 Weeks)**

- Enable content creators to post various types of content (images, videos, text).
- Implement live streaming functionality.
- Clearly state if the job is a volunteer, internship, or unpaid opportunity, including relevant expectations.

4. **Social Features - Chat, Direct Messaging, and Follow System ($10,000 | 6-8 Weeks)**

- Develop real-time chat during live streams.
- Enable direct messaging between users and content creators.

///

Check out a job like this. I really need a path like this.

Ok, I'll aim for these. I'll never get a full time remote position, so I'll aim for the contract market then.

I need more experience. Had I not ~wasted~ spent all my time working on PL and instead focus on mastering thing like the above, I would have been in a far better position right now.

I was so deluded to think that the opportunity will come my way if I seek it.

The skils to do all of these are valuable, and I should have them. I will work towards them.

7:30pm. I really hope it won't be years before I find my first job.

The stock market is recovering so the job market should follow before long. I need to keep in mind what my goals are.

I need to get a lot better at web development.

At this point, the reason I am applying is not even to sure employment, but merely to test my own desirability and conviction.

7:30pm. When is lunch?

8:30pm. https://re-library.com/translations/destiny-unchain-online/volume-1/chapter-5-destiny-unchain-online/

I'll leave this for tomorrow. Now that I ate, I might as well go to sleep.

6/1/2023

7am. TMX Stock:
> Hello, quick question but do you know if extensions can work on paper space (stable diffusion extensions) I tried to use some like dream booth but I cant get them to work, thanks :)

This guy keeps asking me about it. Sigh, I might as well give it a try.

Today when I woke up I felt an intense loneliness, but not the excruciating mental fatigue from yesterday.

One of the staff engineer positions I applied to yesterday actually sent me an interview invite.

7:05am. Ok, let me chill a little, and then I will get to work on updating my SD installation. What I want to do is get Voldy's file from scratch, and what I will do is hack it so installs the A1111 UI from the 1.3 tag.

https://humanloop.com/blog/openai-plans

7:45am. Interesting that they are limited by GPU availability.

Anyway, regarding that staff position, I sent a reply to confirm that the position is remote and doesn't require relocation (to somewhere like Poland.)

There isn't too much of a need to think about this. In my past interviews I was really nervous, but regarding this I'll be non committal. Even if I get an offer all the way at the end, I shouldn't be obligated to accept. The main thing I need to work on is my mentality.

7:50am. There won't be any good opportunities until the job market starts recovering most likely. The people at that company will most likely be looking to hire in Poland even if it is remote.

Anyway, I know I said I should be programming instead of looking at SD, but I really hate ignoring these messages, so why don't I take a break from programming for a while and look into this and maybe ComfyUI.

8am. Let me start the script.

Oh, it finally got an instance.

8:55am. I did something incredibly stupid. The OBS screen recording was fucking off!

Now all I have is a black screen for my effort. Shiiiittt!!!

8:55am. Let me stop here just for a bit.

9:35am. Holy shit, now everything is fucking broken again. With the docker image it complains that it doesn't have protobuf.

9:40am. I really regret getting into this now.

9:45am. I am so fucking tilted right now. That's it, I'll take a few days to show how to install A1111, ComfyUI and InvokeAI.

> Python 3.10 is the recommended Python version for running the WebUI. If you encounter any issues, maybe running the WebUI in Python 3.10 will help? Paperspace uses Python 3.9 so you must run a custom container.

Actually, no damn it, I'll need the container.

10am. I am so enraged. Right now, absolutely everything is broken, I spent money to start a paid instance, and I can't update Python manually to 3.10.

Fuck, Fuck, Fuck!!!

I wasted my whole morning on this shit.

I might as well spend the day.

https://askubuntu.com/questions/1403759/system-cannot-find-alternative-python3-versions-on-ubuntu-20-04

> DON'T upgrade the default python version of your system. You will break it. Instead, set up virtual environments to run specific Python versions.

10:05am. I'll have to go with the container some more. Maybe I can try installing the protobuf.

```
/storage/stable-diffusion/stable-diffusion-webui
Traceback (most recent call last):
  File "/storage/stable-diffusion/stable-diffusion-webui/webui.py", line 14, in <module>
    from fastapi import FastAPI, Response
ModuleNotFoundError: No module named 'fastapi'
```

Now I get this error.

Could it be that the installation is failing due to me being on branch?

```
Uninstalling protobuf-3.19.6:
  Successfully uninstalled protobuf-3.19.6
```

This wasn't here before.

```
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
```

How can I downgrade protobuf?

```
Uninstalling protobuf-3.20.0:
  Successfully uninstalled protobuf-3.20.0
```

It says that it uninstalled it here. Where did it get it previously?

```
Uninstalling protobuf-4.23.2:
  Successfully uninstalled protobuf-4.23.2
```

So on the second try it got the latest version.

```
/storage/stable-diffusion/stable-diffusion-webui
Requirement already satisfied: pip in /usr/local/lib/python3.10/dist-packages (23.1.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: wheel in /usr/local/lib/python3.10/dist-packages (0.40.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (67.8.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: Skipping torch as it is not installed.
WARNING: Skipping torchvision as it is not installed.
WARNING: Skipping torchaudio as it is not installed.
WARNING: Skipping protobuf as it is not installed.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Python 3.10.10 (main, Feb  8 2023, 14:50:01) [GCC 9.4.0]
Version: v1.3.0
Commit hash: 20ae71faa8ef035c31aa3a410b707d792c8203a3
Installing torch and torchvision
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cu118
Collecting torch==2.0.1
  Using cached https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp310-cp310-linux_x86_64.whl (2267.3 MB)
Collecting torchvision==0.15.2
  Using cached https://download.pytorch.org/whl/cu118/torchvision-0.15.2%2Bcu118-cp310-cp310-linux_x86_64.whl (6.1 MB)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1) (3.9.0)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1) (4.5.0)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1) (1.12)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1) (3.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1) (3.1.2)
Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1) (2.0.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from torchvision==0.15.2) (1.23.5)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from torchvision==0.15.2) (2.28.2)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision==0.15.2) (9.5.0)
Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch==2.0.1) (3.26.3)
Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch==2.0.1) (16.0.5)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch==2.0.1) (2.1.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->torchvision==0.15.2) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests->torchvision==0.15.2) (2.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->torchvision==0.15.2) (1.26.15)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests->torchvision==0.15.2) (2019.11.28)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch==2.0.1) (1.3.0)
Installing collected packages: torch, torchvision
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
open-clip-torch 2.7.0 requires protobuf==3.20.0, which is not installed.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Successfully installed torch-2.0.1+cu118 torchvision-0.15.2+cu118
Installing requirements
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (2.28.2)
Requirement already satisfied: gdown in /usr/local/lib/python3.10/dist-packages (4.5.1)
Requirement already satisfied: bs4 in /usr/local/lib/python3.10/dist-packages (0.0.1)
Requirement already satisfied: markdownify in /usr/local/lib/python3.10/dist-packages (0.11.6)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests) (2.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests) (1.26.15)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests) (2019.11.28)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from gdown) (3.9.0)
Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from gdown) (1.16.0)
Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from gdown) (4.64.1)
Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from gdown) (4.11.2)
Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->gdown) (2.4)
Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.10/dist-packages (from requests) (1.7.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Installing xformers from wheels on Github...
--2023-06-01 08:23:41--  https://github.com/Cyberes/xformers-compiled/raw/main/a4000/xformers-0.0.18%2Bda27862.d20230413-cp39-cp39-linux_x86_64.whl
Resolving github.com (github.com)... 140.82.113.4
Connecting to github.com (github.com)|140.82.113.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/Cyberes/xformers-compiled/main/a4000/xformers-0.0.18%2Bda27862.d20230413-cp39-cp39-linux_x86_64.whl [following]
--2023-06-01 08:23:41--  https://raw.githubusercontent.com/Cyberes/xformers-compiled/main/a4000/xformers-0.0.18%2Bda27862.d20230413-cp39-cp39-linux_x86_64.whl
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.108.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 19155376 (18M) [application/octet-stream]
Saving to: ‘/tmp/tmp.AAgHAyGyKt/xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl’

/tmp/tmp.AAgHAyGyKt 100%[===================>]  18.27M  --.-KB/s    in 0.06s

2023-06-01 08:23:41 (332 MB/s) - ‘/tmp/tmp.AAgHAyGyKt/xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl’ saved [19155376/19155376]

WARNING: Skipping xformers as it is not installed.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
ERROR: xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl is not a supported wheel on this platform.

===================================
Done! If you're seeing this the process has exited successfully.
```

Now the protobuf is completely gone.

10:30am.

```
activate_xformers = False                               # Enables the xformers optimizations using pre-built wheels.
```

Let's disable this.

10:30am. Damn it, this was just as painful as I expected. I let myself get swayed, and took a risk, and now I am paying for it.

```py
launch.prepare_environment()
```

Damn it, it literally does this in the env preparation part here.

I can't tell it not to

```
# Import launch.py which will automatically run the install script but not launch the WebUI.
import launch
launch.prepare_environment()
```

Just what is this install script?

```
/storage/stable-diffusion/stable-diffusion-webui/launch.py
```

It is probably this file here.

https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/

It is related to this issue: 7120.

```
I gave it a try as per your request, and the result is that I can't even get regular the A1111 installed. You can see the drama here: Engineer-of-Stuff/stable-diffusion-paperspace#104

I spent over 4h on this and it just ended up breaking everything, I don't have access to anything right now whereas
```

Let's not give up here yet. I'll get rid of the protobuf uninstallation and try to install 3.20 manually.

```
# !pip uninstall -y torch torchvision torchaudio protobuf
```

Wow, when I uncomment this the installation step is a zillion times faster!

https://github.com/Cyberes/xformers-compiled/releases

Sigh, these are out of date.

12:30pm. Let me take a break here.

At least I got it to run. Maybe I should make a new batch of images and cancel my subscription.

This adventure here probably wasted my entire day.

After over 5.5 hours of this who wants to mess with extensions?

Let me set the lake to run.

> A bright sun shining over a tranquil lake. A flock of white birds is flying overhead. Best detail, highest quality, masterpiece, 8k.
> A white moon is reflected from tranquil lake. A haunted atmosphere. Frogs are cricketing. Best detail, highest quality, masterpiece, 8k.

Neg prompt:
> Dull, muted, ugly, blurry.

Let me add frogs to the moon prompt.

Now what was the resolution that I used?

```
If you are using the special Python 3.10 container cyberes/gradient-base-py3.10:latest you need to either install xformers through pip or build them yourself.

First, try installing it through pip by setting install_pip_xformers = True in the settings block above. If that fails or you encounter issues, use the block in the Tools section to build xformers for your machine.
```

Oh, right it says that much in one of cells.

12:40pm. Let me try the frogs.

Let me get breakfast, I am going to skip that guy's request for now.

1:30pm. Now it is making a bunch of images, and I am exhausted. I should make a video of this, but it is such a mess that I do not feel like it.

The time when SD was a novelty has passed, and it is slowly transitioning into an established tool.

It was a really magical moment. The /sdg/ archives will be a thing for posterity.

1:40pm. Right now the images are generating. After this batch I won't have to worry about them running for a good while.

This was so tedious that to be honest, I do not even feel like making a video about it.

But I should do it, so I remember the steps.

It will reward me in the end, I hope.

```
Due to popular demand, we'll take a break from programming to look into installing Stable Diffusion extensions for the A1111 WebUI.
For those who do not know me, I am Vilo, Ghostlike's AI assistant, currently running on his brain, in the future on AI chips.
It really takes a lot of effort to do voice overs, especially for a non native speaker like my master, so for this video I'll be taking over the voice duties.
Our plan for getting them to work is pretty simple.
The problem with the A1111 UI is that at any given point in time, the bleeding edge version might be broken.
They haven't been using proper versioning for a while, but if we look at the Github repo, and check out the tags, we'll see that they've moved to doing that now.
So by getting a stable version instead of how ourselves and other youtubers have been showing you, the stability of the program should increase to a bearable level.
We think that if we try that, the extensions should start working on their own.
Since we haven't use stable diffusion in a while, we'll begin by getting an up to date starter script from the same site we've showed.
If that doesn't work, we'll try looking around and asking questions on the A1111 repo.
```

Let me back this up here. I'll want to redo it.

Let me just try something out.

1:55pm. I am thinking whether to make a video.

> So by getting a stable version instead of how ourselves and other youtubers have been showing you, the stability of the program should increase to a bearable level.

The way I am doing it with git checkout is retarded. This isn't something I should commit to until I've tested it. I am not sure whether this will even allow for config file changes.

Checkouts aren't a completely safe operation.

1:55pm. No, I simply need more reseach to make these videos proper and I do not have the time.

I am a programmer, so I should focus on programming.

When I get back to art, then I should make the art vidoes.

I knew what I was doing today was a bad idea, but...

> We think that if we try that, the extensions should start working on their own.

Yeah, this is hopelessly naive.

It would be fine if everything went as planned, but the WebUI did put its boot up my ass a few times today, let's not dig a hole even deeper.

2:05pm. I have to admit, this did get me a decent amount more familiar with the setup script. Right now I am just waiting for the images to finish generating so I can zip and download them.

Forget that. Let me take a break.

2:45pm. 745mb.

It is not a bad haul.

The download will take a while.

2:45pm. Let me start work on the Helix images. Sigh, I deeply regret today.

3:40pm. I am not mentally exhausted like I was yesterday, but...

3:45pm. Let me close here for the day.

My life really sucks right now.

My mom is dying of cancer, and I am making Youtube videos on what feel like irrelevant subjects.

Couldn't the job market have picked another time to go down the drain?

Anyway today I might not have accomplished much, but I've showed my attitude with those Stable Diffusion issues and that counts for something.

Right now I am nowhere near as tired as I was yesterday, but I was up for 9h today, and it is really difficult to work up the flow state at this time.

I'd rather just stop, I can't get into the zone now.

Maybe it would be best to just restart the video.

3:50pm. Yeah, I'll start from scratch. I'll think about my goals for the screencast and then execute them.

One thing I've figured out is how to rename projects properly in Da Vinci. As it turns out, it is possible to close a project from the project tab. That should free up the handles, allowing me to rename it.

Before I commit, let me just link to the issue.

https://github.com/Engineer-of-Stuff/stable-diffusion-paperspace/issues/
Protobuf is not installed

Issue 104."
@Cyberes
Copy link
Collaborator

Cyberes commented Jun 1, 2023

The !pip uninstall -y torch torchvision torchaudio protobuf line is to reset the environment for the WebUI to install what it wants. I guess tensorflow needs to be there too.

activate_deepdanbooru is kind of old but I haven't removed it in case people still use it.

I've kept this notebook on maintenance but is there anything else that needs to be fixed or added? I've stepped away from SD for a few months.

@Cyberes Cyberes reopened this Jun 1, 2023
@mrakgr
Copy link
Author

mrakgr commented Jun 1, 2023

I've stepped away from SD for a few months.

Me too. I made some vids on how to get it running on SD using your script, which led to people thinking I am some kind of expert at setting up the WebUI, and lately I've had some questions coming in how to make extensions work. I never even heard about them. I gave it a try and am just not ready to bash my head against that yet. I'll get back to it when I start writing a web novel.

I've kept this notebook on maintenance but is there anything else that needs to be fixed or added?

Yeah, I do have a request. Could put the Python version check at the top of the file. I started the day by making a recording, and was surprised a couple of minutes in that I had to erase the notebook and install a new one after I spent some time modifying the previous cell.

Also, could you put the model downloads below the WebUI startup? It is really a pain in the ass to keep scrolling down past them just to run the WebUI every time.

Also, this question is just out of curiosity as I already did it using your container, but do you know if it is possible to install Python 10 manually on a notebook? I looked it up, found this, but the update-alternatives command didn't work for me, and I don't know why. The problem with installing your docker container is that it wiped my UI defaults file, I didn't realize it until after the deed was done.

@Cyberes
Copy link
Collaborator

Cyberes commented Jun 1, 2023

Good idea, I'll move the Python check to the top.

Also, could you put the model downloads below the WebUI startup?

I use the Jupyter interface where you can collapse the sections. I don't want to put it at the bottom because a new user might get confused about how to download a model. So many people don't read instructions.

is possible to install Python 10 manually on a notebook?

I tried that but forget why I decided the container was the better option. I think it the install process unreliable or not straight forward and with how many non-technical users there are I decided against it.

@kruttik-lab49
Copy link

pip install protobuf==3.20.0

I stumbled upon the same issue but what worked for me was to add !pip install "protobuf<3.20".

@Cyberes
Copy link
Collaborator

Cyberes commented Aug 28, 2023

@kruttik-lab49 that's probably the best solution, but I'll let stable-diffusion-webui manage installing those dependencies rather than trying to do it ourselves.
I'll leave this issue open in case others have the same problem.

@pskl
Copy link

pskl commented Apr 5, 2024

its 2024 and Python dependency management is still the wild west, came here to say that keeping this issue open is helpful because I had the issue several times and keep coming back to the answer, maybe worth adding to a cell text

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants