Replies: 3 comments 24 replies
-
|
Do you think using AI to assist open research for amateurs starting out a bad idea? Or would that be valid for those without a conventional academic background? https://github.com/orgs/openml/discussions/17#discussioncomment-11897058 |
Beta Was this translation helpful? Give feedback.
-
|
Hi @BradKML! Sorry for the delay - you can imagine we have been busy lately :) AI is a tool, in itself it is not bad or good. It's about learning about how and when to use it. Personally I use agents sparingly, mostly for brainstorming, feedback on text, or debugging, all situations where I have the knowledge to judge the output of the models. It is also possible that I might lack the experience working with LLMs to access their full potential, as so far results have been mostly underwhelming to me.1 I definitely think there is a place for AI in making open science more accessible to "amateurs" or the general public. It can help with programming, data visualization, feedback, and so on. However, there are also dangers. Current agents are frequently giving answers that are incorrect or incomplete, even for simple things like counting. This is why I would generally use the models when I can judge the output as one cannot trust agents blindly, or where accuracy of the output is not important. This means that novices probably should use the models more sparingly in some areas, such as learning new material.2 When it comes to doing "actual science", there is no substitute to knowing what you are doing. That does not mean you need a university degree, but it does mean finding reliable sources to learn the niche you are working on. The good news is that there is a lot of well-made information available for free, whether that's courses, youtube videos, books, and so on. It doesn't even mean you have to know everything when you start, everyone learns along the way, and you can also start with finding more experienced collaborators. Conversely, AI may also be used by the experts to help make their science more accessible to the general public, by e.g., letting it help come up with layman explanations to the research, as the experts will be able to judge the accuracy of the simplification! Footnotes
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the response @PGijsbers and I would like to add one more point: scientific education (including communication to the average populace) and "citizen science" (grassroots-first research) is seriously underexplored. Non-textbook STEM education resources (e.g. physical access for the poor, social labor for hands-on work, time-agnostic low-friction media) are increasingly underfunded in many places, and sometimes not even supported in others. |
Beta Was this translation helpful? Give feedback.
-
|
I hope I didn't come off wrong with the "actual science", it was very much in quotes a reason. Citizen science is very meaningful and impactful, and not in any way inferior to a researcher working in a lab -- in fact they can often end up working together! Allowing people to classify galaxies, count birds with their phones, analyze conjectures and many more examples show that there's so much scientific value while also increasing engagement and education of the general public. It's fantastic we live in an age where that's possible, and it's a wonderful thing if we can use AI to enhance or accelerate that - provided we exercise care. |
Beta Was this translation helpful? Give feedback.
-
|
That is true, the more concerning part is how "care" became a euphemism to limit citizen access through red tape. Streamlining safety controls and make that specifically more accessible would help in that regard. Rather than established fields like ecology and astronomy (hobbyist delegation), or places where it is bona-fide "compute donation" like mathematics or biochemistry (assumption of resources), it would be even more important to focus on frontier work, like social science and AI technology, lending voice to the masses and bridging the gaps in understanding. "Soft science" implies the tenderness of community care rather than bias on "lab coats". |
Beta Was this translation helpful? Give feedback.
-
|
It feels like openml was created with the goal of building a superior (to "standard" optimizers like Optuna, Hyperopt etc) metalearning-based HPO/HPT solution in mind, but nowhere on the site or in the docs that idea can't be traced. Is it entirely dropped or never existed? Also, for classification tasks, are raw predicted probabilities stored somewhere? run.predictions seems to only return labels (on the other hand, metrics like ROC AUC are there). Also, when trying to sign up for an API key, confirmation email is not arriving. |
Beta Was this translation helpful? Give feedback.
-
|
There's definitely been work in that area! As early as 2015, auto-sklearn used meta-learning to warm-start Bayesian optimization1. More recent example for generic warm-starting is work with building portfolios, this is about finding a set of generally good but complementary pipelines, as in e.g., tabrepo2 and version 2 of auto-sklearn3. The latter also uses meta-learning to self-configure the optimizer's hyperparameters. There's even work in transferring priors for Bayesian optimization4. I also came across this recently though haven't read it yet myself, but seems relevant to your interests. Footnotes
|
Beta Was this translation helpful? Give feedback.
-
|
Yes I was asking myself why auto-sklearn only uses meta-learning for a warm start. Imho it there's predictability of final performance depending on meta-features, no Bayesian optimization is needed. But look at this recent paper, curated by the famous Frank Hutter.
Such speed & accuracy gains I did not have even in my boldest meta-learning dreams ) |
Beta Was this translation helpful? Give feedback.
-
|
I agree, the results of TabPFN are remarkable. It's very interesting and impressive they can get such results from training on (cleverly) generated synthetic data. |
Beta Was this translation helpful? Give feedback.
-
|
Wondering what's the contributions of metalearning and transformer architecture to that achievements, percentwise. |
Beta Was this translation helpful? Give feedback.
-
|
@fingoldo maybe code agents and creative agents can work together towards something more efficient, data-to-paper or AI-scientist being two examples |
Beta Was this translation helpful? Give feedback.
-
|
I would like to publish a dataset, but the sign up process seems broken. I have not recieved a activation email. is that semi-automatic or currently broken? |
Beta Was this translation helpful? Give feedback.
-
|
ok, but the "edit dataset" api doesnt allow editing the dataset, but only the metadata. that was a bit confusing to me. i managed it. |
Beta Was this translation helpful? Give feedback.
-
|
Cool 💪 |
Beta Was this translation helpful? Give feedback.
-
Imho the documentation should state:edit_dataset:
create_dataset:
Version parameterI also think providing a "version" does not make a difference actually, because I accidentally provided the new upload version with the same version, but still the version on the website updated as expected. I think the version argument gets dropped and not used at all. maybe that parameter should also be documented accordingly. Function namesI think thats all my comments. For the next major version i think it would be good to change the names of the functions to "edit_dataset_metadata" and "upload_dataset" accordingly. |
Beta Was this translation helpful? Give feedback.
-
|
@stheid update: the upload via the website is also fixed |
Beta Was this translation helpful? Give feedback.
-
|
The 'version_label' parameter is a label you can freely choose. It's stored alongside OpenML's version (which is always an incrementing int). I see you indeed set "version_label": "0.1" for both dataset versions. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
The Q&A discussion board is for asking any OpenML questions! Whether they are about our organisation, hackathons, websites, or any of our programmatic interfaces! We're here to help 🤗 Just "create a discussion" and ask away!
Beta Was this translation helpful? Give feedback.
All reactions