I have to really control the token usage precisely, I basically just went full retard in the last month

at least for me, it upsets me that I can't fully afford the best stuff

well it's a big difference

which is far worse than any model in the last year or so

since you're using the most powerful mode your GPU can handle

For me tokens > power usage

so you are constantly pulled back into the cloud where you have these massive blow outs where thousands of dollars disappears and then you realise you have to go local again but if you go local again, it's like, the last year of progress in AI, doesn't exist for you

For what I am doing, I don't need the most advanced. I need like mid tier or so.

so as AI advances you get left behind

the problem with local AI is that you will never run the best models

been really trying hard to control these costs, it's quite scary and it's really unfortunate because I really want to do X Y Z but then I realise it's just too expensive in terms of tokens

Yeah I have been looking at local AI running on regular hardware.

and I'll be running the jobs on this PC when I sleep as well as the two A6000's I have

so now I am trying to set up docker containers to localise as much AI usage as I can again

I may have a line on a second hand server soon. If so then I will have a unit with way more RAM than I need.

I some how blew like $4000 in AI tokens in one month in GCP

im trying to get my moneys worth out of it lol

I wish I could afford that much. lol

you can also get 96 GB ram like me and still have none left

I just do all that because i want it right down to the basics and then I can build it up from there as I like it.

Yeah that is down there pretty good.

Using

It was 8gb now i removed antivirus and copilot its now 5gb

You can get that RAM usage down pretty good.

Then I look through it and strip out all the unnecessary Windows junk as well.