This is neat. It's a little hard to compare the best-performing models ("~ 0.00006232g" vs. "~0.00013746g" is not easy at a glance) --- perhaps better to use scientific notation, milligrams, or CO2e per 10M tokens?
Did you include the environmental impact of training these models? I failed to find the information and that seems quite important to me (I'm a bit tired so maybe that's just me).
> Our proprietary methodology and algorithms are trade secrets and confidential business information
Sorry, but that's a no go IMHO. This stuff needs to be reproducible, transparent, ideally peer reviewed, and we must be able to study it. You are doing the equivalent of "here are facts, just trust us". That's not reliable.
What is it you are selling that such a core thing like your methodology is kept secret?
I tried to understand a bit what you did reading your methodology page [1] but didn't understand a thing. I looked for concrete stuff but didn't find any. Now, again, I'm a bit tired so I'm possibly missing something.
We need to see for ourselves all this stuff to determine if you are rigorous, if there are possible conflict interests (for what I know, you could be selling good grades), etc. The basic requirements of scientific research, basically.
This is neat. It's a little hard to compare the best-performing models ("~ 0.00006232g" vs. "~0.00013746g" is not easy at a glance) --- perhaps better to use scientific notation, milligrams, or CO2e per 10M tokens?
You're right! Fixed.
Did you include the environmental impact of training these models? I failed to find the information and that seems quite important to me (I'm a bit tired so maybe that's just me).
> Our proprietary methodology and algorithms are trade secrets and confidential business information
Sorry, but that's a no go IMHO. This stuff needs to be reproducible, transparent, ideally peer reviewed, and we must be able to study it. You are doing the equivalent of "here are facts, just trust us". That's not reliable.
What is it you are selling that such a core thing like your methodology is kept secret?
I tried to understand a bit what you did reading your methodology page [1] but didn't understand a thing. I looked for concrete stuff but didn't find any. Now, again, I'm a bit tired so I'm possibly missing something.
We need to see for ourselves all this stuff to determine if you are rigorous, if there are possible conflict interests (for what I know, you could be selling good grades), etc. The basic requirements of scientific research, basically.
[1] https://modelpilot.co/methodology
I agree with the rest of your comment, but w.r.t. training impact, it's not necessarily such a big factor in emissions[1]
[1] https://limited.systems/articles/climate-cost-of-ai-revoluti...