Watt’s up with AI?: Saving Open Source | Lionel Ringenbach

0:00Incoming transmission from the future Vancouver AI community meetup. >> I'm about to invite Lionel Ringenbach up here. He has uh exhibited three things over the two years he's been with us. The first thing was a really cool AI prompting device where instead of prompting it, you put images in front of its eyeballs and then it would blend those things together and generate new things from it. Then he came back and he was like, "Yo, I'm not even sure about the power and impact the energy." He went for a vacation. He left his computer training. and he's like, I don't even actually know how much energy it's going to use. And so that began a really indepth long-term conversation in

0:36this community about the true impact of all this stuff. Um, not just in terms of power or water or culture as this comes to Anthony, but you know, jobs and all sorts of things. We've been asking the hard questions. So anyway, without too much further ado, I just want to draw a straight line. Come on up, Lionel. I want to draw a line between the conversation you invited us into around power and water and this panel that we're about to have here tonight. So, um, welcome buddy. I'm happy to have you back.

1:06I'm happy these, um, that these benches are here because you usually like to try to hide in the dark and I would like to encourage you to stand here in the light. >> All right, Lionel Ringenbach. >> Hello everyone. Um, so I'm Lionel. Um I'm going to talk tonight a bit a bit of an update from the last time on what is up with AI. So what with two T's like you know funny uh so and I'm going to talk about like open source a little bit also in that discussion and what kind of impact as AI on open source today.

1:40Um so that was the last time there was the episode two of WhatsApp with AI where I talked I talked about WhatsApp and um project that they develop which is a GitHub copilot version that uh actually tells you about and kind of memorize all the requests you've made and gives you a little bit of estimation of what is the equivalency of this energy use etc just so you have a bit of an idea um and to give a bit of a a summary of what is WhatsApp or the the goal of that project is to kind of measure the energy use of AI tools. So you can monitor inside tools that are either in a far distance or on your

2:22computer and also to build certain language to interpret what does that even mean? How do we make it tangible? And finally that will be the final goal of WhatsApp or really like the really the gold is is how do we hack on this and what do we do about it? Is there anything we can actually do? Um, so I did present that project at Mutek this year in uh in Montreal and uh that's where I developed a new tool that I will show you tonight.

2:52Um so this summer open router if you don't know it open router is one of the biggest provider for AI LLMs and even image generation now. Um, if you want to integrate with a plethora of LLM without to directly interact with OpenAI, then Open Router is a great choice because you can easily switch from one model to another. And one thing they did this summer is that they created a leaderboard. Um, no, no problem. This leaderboard here where we can see the number of tokens that have been used this week here. We see this week even they show for the past months across different providers and different models. and they'll even tell us over time what it what it looks like.

3:34And so I thought that's a great occasion to kind of maybe we can add some data on this. And so I built WhatsApp up for open router which uh basically scrapes their web page because they don't offer an API so I did what I could. sometimes it breaks. Um, and use the echolocit which is another tool open source tool that allows you to do estimation based on how many tokens were produced, how long was the request, what is the size of the model. So here this tool allows you to tell for example this month how many gawatt of energy were used and what is the projected gas emission CO2 emissions and soon you will also

4:18know about how much water has been potentially used for creating all this workload. So, it's really not about trying to blame or to make it or shame or to say this is bad, but to really say here's what it is and to actually be aware of it. Um, and to continue trying to work towards that. So, we have an idea is it is it getting better? Is it getting worse? Um, and in that tool also built in some equivalences so you we can have an idea. So, what does that even mean? There is other screens on it where you can see for example how many kilometers in a car it is or things that are a bit more relatable.

4:55Um can you go next screen and through this I started working a little bit on the second step of that project which is to be interpreting this data. Uh some other project already did some work. For example the AI impact tracker here that you can install in Chrome that tells you information about your JGP usage. is very specific to CHGP and continuously monitors your usage in Chrome. It stays local for you. And so from there they develop some kind of language of environmental equivalent. So we have an idea of what does that even mean? And so in WhatsApp for G pilot, I kind of started reusing that with um also like the famous how many hours of running a

5:37microwave and in WhatsApp for router for open router a bit more of equivalences. Uh knowing also that sometimes those equivalences is a bit difficult because there's an equivalency for the energy use an equivalency for the carbon emissions for the water that is used but they're not all precise. So I'm still kind of working through that to kind of really delineate and maybe build a framework for WhatsApp that could be used for all the tools.

6:04Um and here I'm going to talk about echolocit which is the tool that I use in open uh in open route in both of those tools. Um, Ecology

6:19uh, which also integrates water consumption and also they have really improved uh, the accuracy of the model estimation of what how many parameters in a model and also for open source model what is the actual energy use of these models. also including things such as like the the water usage of certain data center. So you have an idea also for example the power efficiency if it comes from Google if it comes from AWS you can have different efficiencies and a bit of a good news Google published an energy report recently as a paper which is measuring the environmental impact of delivering AI at Google at sca at Google scale finally. Yeah, that's great. It's a good

7:05move. That's a good move. It's actually a positive thing to see and hopefully we will see more and more companies doing that. So, it's the first Oh, it's the first methodology that's very holistic. It shows on the right side as part of the paper as a screenshot that it takes in account really a lot of the life cycle of a of a server that's running AI not just like the cheap utilization but also all the way to machine when machine are able or like the overhead of running a data center and this methodology can be used by other companies and other data centers to do the same and do a more precise meas measurement that is

7:46actually really taking the anti entire data center. It's not just about estimating almost the GPU use, but how much does it cost to run that whole thing and um there is some interesting part. They determine that it takes 0.24 24watt hour per prompt and five drops of water per prompt which is interesting but it's kind of problematic in some way because it those measurement that they define as standard are not comparable because it's per prompt means it's like very hard for you to compare to another company if they were doing doing it because a prompt is not the really unit of measurement of AI when it produces the token is the real measurement so hopefully we'll see new paper coming up

8:31with measurable and comparable values and hopefully companies don't continue using that value because that's not very useful to actually know if we're doing better. It's easy for you to you came here and compared them in terms of microwaves when you >> fair and another thing that um is it's great that they did this but while they're doing this they have Google view it's a really big model and this is one area where we really don't know can I go next please um and so open to three minutes Okay.

9:10Okay. So, open source is really the reason why we know all this. There there would be no data on on this matter and Google probably wouldn't step fair to do this otherwise and right now all the larger flagship models don't tell us anything about their use and they are the big ones the one that's really interesting and juicy. Um and there is almost no data on video models today and that's very important because those we have a quite a good in intuition I think that they would be using a lot more than text text models and absolutely nothing on world models and those are very cool models but also the energy use we have no idea and there's no open source world

9:49model as far as I know so we can't really tell and user privacy is kind of crumbling because we use more and more of that closed AI because think about what you the question that you were asking when you use a search and what kind of things you tell to chat GPT because it's a chat it's very different and so I'm going to talk a bit project open source project for example the curl project it's a very important project that runs the internet in so many ways is basically how a lot of software make requests to servers and recently the maintainer has wrote this article this by a southern slop where they started receiving on Akawan. Akawa 1 is a

10:29platform where you can post uh vulnerabilities that you find about software and they can give you money back for finding actual problems. But recently in July uh they spent eight time more time on vulnerability report but actually only 5% of those reports are legitimate. So they're basically spending time on reports that are absolutely not reproducible. They're completely hallucinated. So that's a bit of a problem.

10:56Um, and that's kind of like really overloading open source that is already running on low funs usually. >> So you're talking about hackers seeking bounties are using cloud code and like this to try to find, you know, vulnerabilities in curl, but they're not there. They're hallucinating them. They're submitting them anyway and they're wasting all the people resources of the open source project parsing between real commits and fake commits.

11:17>> Exactly. And the blog is very interesting. Please read it because it gives details about about it. It's really interesting. they converse on the aker one with actually AI and it's really weird. Um, can we go to next please? And that's another project ffmpeg. It's a different problem but it's also an open source project. It's a very important project because it runs all the streaming online. YouTube, Netflix, think about any streaming platform uses fmpg in some way. And Google big sleep which come from Google deep mind is a very big system that is very good at finding vulnerabilities and actually making real vulnerabilities true vulnerabilities actually and publishing them. But their policy is that if within 90 days you haven't fixed

11:58it we will reveal it. So now there are tons of projects that are open source that need have like this sticking bomb from Google that is a trillion dollar company where there's company like open source project that don't have much funds and certainly no time to execute so fast on vulnerabilities and all of a sudden they get 20 vulnerabilities. Some of them are actually not really exploitable and what do they do because in 90 days they will have actual revulnerabilities like revealed in product that we all use and they don't have a time to fix it. See how this could be very discouraging for open source maintainers to continue maintaining those project. So recently open AI had this token consuming award

12:41which is very interesting because they they actually rewarding people for consuming a trillion token. Uh is that the right incentive? what are we gamifying here? You know, what is this gamification? So, can we go to the next? And so, I I kind of done an estimation. I did an estimation. What does a 1 trillion token is actually? And so, here are the numbers. And for example, is the the energy use of 52,000 household daily energy use. That's one trillion token or 3 mill 3.5 million km driven in a car.

13:13Not saying it's good, it's bad, but here what it is. Probably some of you have received that award, but that's okay. I'm not blaming you. I understand. But it's it's like what are they trying to reward? It's kind of interesting. And we can go to the next and that will be it. That's the award. That's your one.

13:30>> No, I mean it's actually I want to both be conscientious of time but also expansive. And so it's like well you know what they're rewarding with that award. They're saying that there is work that is being done, valuable work of some sort that's being done. and they're trying to optimize. They're not just like who can blow a million tokens down the toilet and give you an award. Those people are trying to like through useful things such as science, research, healthcare, video games and entertainment.

13:59>> I agree with you. I'm not saying the award is is like completely bad or people using token is bad. What I'm saying is that what is it being rewarding? because they clearly took this from YouTube which is a democratic system where people actually subscribe to community like videos that they actually enjoy resist something that is actually paid literally paid for and now the number of tokens consumed does not bind to value delivered like a revenue of a company does not mean it's actually delivering value that is the thing and but definitely meaning that we're to using token to do something useful is true But is that the measurement of value? Probably not. I don't know what is the measurement of

14:41that value. Community can say it with different way. It's not revenue or token used. >> And it's as our conversations go deep on it's like until you can even see what you're using uh those million tokens for in terms of impact. How do you ever make a choice between a a better performing system in terms of impact or a worse performing system? Like you have no choice. You have no ability to even see those things. So monitoring your usage and is the first step I think towards hitting targets and reduction ultimately.

15:12>> Exactly. And that goes right into the award. So I was thinking about this thing. What if we turned this onto its head and actually created an award for saving tokens? So maybe we can do something with BCAI actually and no part of this. But the idea is like what if you here have found a way to save 1 million, 10 million, 100 million token. It doesn't really matter what the tier is, but let's say you saved 1 million token this month by doing something. Please let us know about it. Let's write about it. Let's talk about it and let's share it back to the community so we can actually make progress toward like not using less of AI but make a more

15:55sober use of it and not just like wasting tokens tokens but actually have an award that is worse that was going to actually like really contribute back to the community in a positive way. >> Yeah. In business talk they say if you want to change the outcomes you change the incentives. Well, you know, how about we give out awards for, you know, how many tokens you can save instead of how many tokens you can use. And >> nice laying out.

16:17>> Yeah. >> And there are some flyers on the outside. If you want to subscribe to the newsletter of WhatsApp or see the website, see how you can contribute or contact me, please pick up a flyer on the way out and question. >> Yes, sir. In the back. Uh hi, thanks for the presentation. I'm Ragav. I own an AI firm I founded in March of this year. So I'm I was kind of intrigued by one of the comments that you had in the slide that u user information is crumbling with closed AI.

16:54Uh how do you define closed eye first and then because your premise for the presentation was open source, how do you interpret open source to actually not do the same? So I guess there is many interpretation of open source that is true and when I talk about closed AI and privacy I talk about AI that you cannot run on your computer for example that you wouldn't be able to run on your own hardware that you have to actually send your data to a company for it to process it for you and gives you an answer as opposed to something you can ask locally and you can be sure that your privacy is preserved. Usually those models are open

17:32source. I don't know of any like private source model that ships on your computer so far even though there probably on your iPhone but these are a different type of AI. Yeah, I won't belleaguered the point but clearly you know open source software we can all write tools and monitoring stuff on top of it.

17:47Closed proprietary systems you never know what's going on under the hood. I mean traditional definition. Any other burning questions for Lionel comments? Yes, sir. Thank you sir. Um I have a a quick question and then also a comment. Uh first question when you did the the CO2 emissions uh calculation. I'm just curious like was there a way to like to figure out if uh the energy was actually coming from renewable source or or did you just assume it was like 100% like fossil fuel based and is there a way to kind of like change that depending on where it's actually coming from the data processing? And then uh I like the comment on incentivizing uh like saving tokens, but I think right

18:30now a lot of these companies are like uh that are selling like this computational time and capacity are making money from that. So they're incentivizing usage because they make more money from that. But I think if we're going to incentivize saving, then I guess it would be cool to find a way to like actually make monetize that saving as well. So there's a financial backing for it, too.

18:54Uh sorry remind me your first question was I >> CO2 emission >> CO2 emission okay how is it calculated so in this case um I use the world electricity mix the tool open router what's what's open router you actually have a setting somewhere where you can choose that energy mix I'm planning to improve it so that based on the different provider you have you know which energy mix is tapping into so you can say it's going to be maybe like anthropic is going to be USA and if different places you'll have different energy mix but there's a tool there's a way you can change that in the UI settings that you can try to simulate

19:29like in what's for open router I don't know each models parameters so I just defined it as 20 billion parameters in average for everything so if you crank this to 40 will be a lot more so that's how it's kind of evaluated there are some assumption of what energy mix or billion of parameters at the moment but we can improve this over time by trying to you know and your second question that I forgot was about >> it was a comment >> a comment on the monetization of like those award right or kind of like giving a good incentive and I agree with you like um everybody's running right now it's an actual global stampede of AI

20:08like you know like Google didn't want to run at first but open AI did it so Google started running everybody's running and we are all trying to >> let's go as fast as we can I don't know but let's go fast >> and and I hear a lot that this is inevitable, but I disagree with it. I think it's evitable. We just don't have to contribute to it. We can slow down.

20:28We're not going to miss anything. There's nothing to be missed on. Of course, if we think about money, we're going to miss on some money. That's for sure. But as humanity, I don't know if by going faster, we're going to earn something more. So, I don't have like a way to I'm just thinking about like, you know, this award, how can we make this award more interesting? But to me the reward is that we get to contribute to the larger community by sharing our findings, you know, of really trying to, you know, have know what we're doing with this.

20:58>> Yeah, man. >> Thank you. You've been challenging us since the very beginning. Lionel is a First Order programmer. He's a First Order artist. He's a First Order thinker. If you're interested in this stuff, connect with Lionel. Good job, man. Thank you, sir. Have you ever been the warm-up band for somebody else?

21:19Leon, have you ever been the warm-up band? What is a warmup band? >> The band that goes on before the main band to get the crowd in the right spirit. No, but I guess that's it. Congratulations.