Sebastian Pineda Arango

University of Freiburg

Author Statistics

Papers

Citation

H-Index

i-10 index

Research Trends

Author Order

Document Type

Co-Authors

Josif Grabocka

University of Freiburg

Hadi S. Jomaa

University of Hildesheim

Frank Hutter

University of Freiburg

Martin Wistuba

IBM Research - Ireland

Kiran Madhusudhanan

University of Hildesheim

Arlind Kadra

University of Freiburg

Lennart Purucker

Lars Schmidt-Thieme

University of Hildesheim

Maciej Janowski

University of Silesia in Katowice

Arber Zela

University of Freiburg

Cooperative Institutions

University of Florida

Amazon (Germany)

The University of Texas Health Science Center at San Antonio

Amazon (United States)

The University of Texas Health Science Center at Houston

University of Freiburg

Universidad EAFIT

University of Hildesheim

Florida College

Gainesville Obstetrics & Gynecology

Author Statistics

Papers

Citation

H-Index

i-10 index

Research Field

Breaking the Paradox of Explainable Deep Learning

arXiv (Cornell University) (2023)

Arlind Kadra Sebastian Pineda Arango Josif Grabocka

Deep Learning has achieved tremendous results by pushing the frontier of automation in diverse domains. Unfortunately, current neural network architectures are not explainable by design. In this paper, we propose a novel method that trains deep hypernetworks to generate explainable linear models. Our models retain the accuracy of black-box deep networks while offering free lunch explainability by design. Specifically, our explainable approach requires the same runtime and memory resources as black-box deep models, ensuring practical feasibility. Through extensive experiments, we demonstrate that our explainable deep networks are as accurate as state-of-the-art classifiers on tabular data. On the other hand, we showcase the interpretability of our method on a recent benchmark by empirically comparing prediction explainers. The experimental results reveal that our models are not only as accurate as their black-box deep-learning counterparts but also as interpretable as state-of-the-art explanation techniques.

Interpretability

Black box

Benchmark (surveying)

Deep Neural Networks

10.48550/arxiv.2305.13072

Cite

Citations (1)

Development of a BCL-xL and BCL-2 dual degrader with improved anti-leukemic activity,

Nature Communications (2021)

Dongwen Lv Pratik Pal Xingui Liu Yannan Jia Dinesh Thummuri

Abstract PROteolysis-TArgeting Chimeras (PROTACs) have emerged as an innovative drug development platform. However, most PROTACs have been generated empirically because many determinants of PROTAC specificity and activity remain elusive. Through computational modelling of the entire NEDD8-VHL Cullin RING E3 ubiquitin ligase (CRL VHL )/PROTAC/BCL-xL/UbcH5B(E2)-Ub/RBX1 complex, we find that this complex can only ubiquitinate the lysines in a defined band region on BCL-xL. Using this approach to guide our development of a series of ABT263-derived and VHL-recruiting PROTACs, we generate a potent BCL-xL and BCL-2 (BCL-xL/2) dual degrader with significantly improved antitumor activity against BCL-xL/2-dependent leukemia cells. Our results provide experimental evidence that the accessibility of lysines on a target protein plays an important role in determining the selectivity and potency of a PROTAC in inducing protein degradation, which may serve as a conceptual framework to guide the future development of PROTACs.

Cullin

NEDD8

Drug Development

Bcl-xL

10.1038/s41467-021-27210-x

Cite

Citations (99)

Deep Pipeline Embeddings for AutoML

arXiv (Cornell University) (2023)

Sebastian Pineda Arango Josif Grabocka

Automated Machine Learning (AutoML) is a promising direction for democratizing AI by automatically deploying Machine Learning systems with minimal human expertise. The core technical challenge behind AutoML is optimizing the pipelines of Machine Learning systems (e.g. the choice of preprocessing, augmentations, models, optimizers, etc.). Existing Pipeline Optimization techniques fail to explore deep interactions between pipeline stages/components. As a remedy, this paper proposes a novel neural architecture that captures the deep interaction between the components of a Machine Learning pipeline. We propose embedding pipelines into a latent representation through a novel per-component encoder mechanism. To search for optimal pipelines, such pipeline embeddings are used within deep-kernel Gaussian Process surrogates inside a Bayesian Optimization setup. Furthermore, we meta-learn the parameters of the pipeline embedding network using existing evaluations of pipelines on diverse collections of related datasets (a.k.a. meta-datasets). Through extensive experiments on three large-scale meta-datasets, we demonstrate that pipeline embeddings yield state-of-the-art results in Pipeline Optimization.

Bayesian Optimization

Representation

10.48550/arxiv.2305.14009

Cite

Citations (0)

HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML

arXiv (Cornell University) (2021)

Sebastian Pineda Arango Hadi S. Jomaa Martin Wistuba Josif Grabocka

Hyperparameter optimization (HPO) is a core problem for the machine learning community and remains largely unsolved due to the significant computational resources required to evaluate hyperparameter configurations. As a result, a series of recent related works have focused on the direction of transfer learning for quickly fine-tuning hyperparameters on a dataset. Unfortunately, the community does not have a common large-scale benchmark for comparing HPO algorithms. Instead, the de facto practice consists of empirical protocols on arbitrary small-scale meta-datasets that vary inconsistently across publications, making reproducibility a challenge. To resolve this major bottleneck and enable a fair and fast comparison of black-box HPO methods on a level playing field, we propose HPO-B, a new large-scale benchmark in the form of a collection of meta-datasets. Our benchmark is assembled and preprocessed from the OpenML repository and consists of 176 search spaces (algorithms) evaluated sparsely on 196 datasets with a total of 6.4 million hyperparameter evaluations. For ensuring reproducibility on our benchmark, we detail explicit experimental protocols, splits, and evaluation measures for comparing methods for both non-transfer, as well as, transfer learning HPO.

Benchmark (surveying)

Hyperparameter

Hyperparameter Optimization

Black box

Transfer of learning

10.48550/arxiv.2106.06257

Cite

Citations (2)

Multimodal Meta-Learning for Time Series Regression

arXiv (Cornell University) (2021)

Sebastian Pineda Arango Felix Heinrich Kiran Madhusudhanan Lars Schmidt-Thieme

Recent work has shown the efficiency of deep learning models such as Fully Convolutional Networks (FCN) or Recurrent Neural Networks (RNN) to deal with Time Series Regression (TSR) problems. These models sometimes need a lot of data to be able to generalize, yet the time series are sometimes not long enough to be able to learn patterns. Therefore, it is important to make use of information across time series to improve learning. In this paper, we will explore the idea of using meta-learning for quickly adapting model parameters to new short-history time series by modifying the original idea of Model Agnostic Meta-Learning (MAML) \cite{finn2017model}. Moreover, based on prior work on multimodal MAML \cite{vuorio2019multimodal}, we propose a method for conditioning parameters of the model through an auxiliary network that encodes global information of the time series to extract meta-features. Finally, we apply the data to time series of different domains, such as pollution measurements, heart-rate sensors, and electrical battery data. We show empirically that our proposed meta-learning method learns TSR with few data fast and outperforms the baselines in 9 of 12 experiments.

10.48550/arxiv.2108.02842

Cite

Citations (0)

Dynamic Post-Hoc Neural Ensemblers

arXiv (Cornell University) (2024)

Sebastian Pineda Arango Maciej Janowski Lennart Purucker Arber Zela Frank Hutter

Ensemble methods are known for enhancing the accuracy and robustness of machine learning models by combining multiple base learners. However, standard approaches like greedy or random ensembles often fall short, as they assume a constant weight across samples for the ensemble members. This can limit expressiveness and hinder performance when aggregating the ensemble predictions. In this study, we explore employing neural networks as ensemble methods, emphasizing the significance of dynamic ensembling to leverage diverse model predictions adaptively. Motivated by the risk of learning low-diversity ensembles, we propose regularizing the model by randomly dropping base model predictions during the training. We demonstrate this approach lower bounds the diversity within the ensemble, reducing overfitting and improving generalization capabilities. Our experiments showcase that the dynamic neural ensemblers yield competitive results compared to strong baselines in computer vision, natural language processing, and tabular data.

Post-hoc analysis

Post hoc

10.48550/arxiv.2410.04520

Cite

Citations (0)

Chronos: Learning the Language of Time Series

arXiv (Cornell University) (2024)

Abdul Fatir Ansari Lorenzo Stella Caner Türkmen Xiyuan Zhang Pedro Mercado

We introduce Chronos, a simple yet effective framework for pretrained probabilistic time series models. Chronos tokenizes time series values using scaling and quantization into a fixed vocabulary and trains existing transformer-based language model architectures on these tokenized time series via the cross-entropy loss. We pretrained Chronos models based on the T5 family (ranging from 20M to 710M parameters) on a large collection of publicly available datasets, complemented by a synthetic dataset that we generated via Gaussian processes to improve generalization. In a comprehensive benchmark consisting of 42 datasets, and comprising both classical local models and deep learning methods, we show that Chronos models: (a) significantly outperform other methods on datasets that were part of the training corpus; and (b) have comparable and occasionally superior zero-shot performance on new datasets, relative to methods that were trained specifically on them. Our results demonstrate that Chronos models can leverage time series data from diverse domains to improve zero-shot accuracy on unseen forecasting tasks, positioning pretrained models as a viable tool to greatly simplify forecasting pipelines.

10.48550/arxiv.2403.07815

Cite

Citations (16)

Deep Ranking Ensembles for Hyperparameter Optimization

arXiv (Cornell University) (2023)

Abdus Salam Khazi Sebastian Pineda Arango Josif Grabocka

Automatically optimizing the hyperparameters of Machine Learning algorithms is one of the primary open questions in AI. Existing work in Hyperparameter Optimization (HPO) trains surrogate models for approximating the response surface of hyperparameters as a regression task. In contrast, we hypothesize that the optimal strategy for training surrogates is to preserve the ranks of the performances of hyperparameter configurations as a Learning to Rank problem. As a result, we present a novel method that meta-learns neural network surrogates optimized for ranking the configurations' performances while modeling their uncertainty via ensembling. In a large-scale experimental protocol comprising 12 baselines, 16 HPO search spaces and 86 datasets/tasks, we demonstrate that our method achieves new state-of-the-art results in HPO.

Hyperparameter

Hyperparameter Optimization

Rank (graph theory)

Learning to Rank

10.48550/arxiv.2303.15212

Cite

Citations (0)

Multimodal Meta-Learning for Time Series Regression

Lecture notes in computer science (2021)

Sebastian Pineda Arango Felix Heinrich Kiran Madhusudhanan Lars Schmidt-Thieme

10.1007/978-3-030-91445-5_8

Cite

Citations (10)

Transformers Can Do Bayesian Inference

arXiv (Cornell University) (2021)

Samuel Müller Noah Hollmann Sebastian Pineda Arango Josif Grabocka Frank Hutter

Currently, it is hard to reap the benefits of deep learning for Bayesian methods, which allow the explicit specification of prior knowledge and accurately capture model uncertainty. We present Prior-Data Fitted Networks (PFNs). PFNs leverage large-scale machine learning techniques to approximate a large set of posteriors. The only requirement for PFNs to work is the ability to sample from a prior distribution over supervised learning tasks (or functions). Our method restates the objective of posterior approximation as a supervised classification problem with a set-valued input: it repeatedly draws a task (or function) from the prior, draws a set of data points and their labels from it, masks one of the labels and learns to make probabilistic predictions for it based on the set-valued input of the rest of the data points. Presented with a set of samples from a new supervised learning task as input, PFNs make probabilistic predictions for arbitrary other data points in a single forward propagation, having learned to approximate Bayesian inference. We demonstrate that PFNs can near-perfectly mimic Gaussian processes and also enable efficient Bayesian inference for intractable problems, with over 200-fold speedups in multiple setups compared to current methods. We obtain strong results in very diverse areas such as Gaussian process regression, Bayesian neural networks, classification for small tabular data sets, and few-shot image classification, demonstrating the generality of PFNs. Code and trained PFNs are released at https://github.com/automl/TransformersCanDoBayesianInference.

Leverage (statistics)

10.48550/arxiv.2112.10510

Cite

Citations (10)