In, Moosavi-Dezfooli, S., Fawzi, A., and Frossard, P. Deep-fool: a simple and accurate method to fool deep neural networks. . We'll see how to efficiently compute with them using Jacobian-vector products. Abstract. Negative momentum for improved game dynamics. Your search export query has expired. Datta, A., Sen, S., and Zick, Y. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. Thus, in the calc_img_wise mode, we throw away all grad_z Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. The more recent Neural Tangent Kernel gives an elegant way to understand gradient descent dynamics in function space. Liu, D. C. and Nocedal, J. calculates the grad_z values for all images first and saves them to disk. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. We'll use the Hessian to diagnose slow convergence and interpret the dependence of a network's predictions on the training data. On robustness properties of convex risk minimization methods for pattern recognition. To manage your alert preferences, click on the button below. Subsequently, A. the prediction outcomes of an entire dataset or even >1000 test samples. M. MacKay, P. Vicol, J. Lorraine, D. Duvenaud, and R. Grosse. Data poisoning attacks on factorization-based collaborative filtering. Check if you have access through your login credentials or your institution to get full access on this article. calculations, which could potentially be 10s of thousands. We have a reproducible, executable, and Dockerized version of these scripts on Codalab. thereby identifying training points most responsible for a given prediction. We have a reproducible, executable, and Dockerized version of these scripts on Codalab. ( , ?) We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. We show that even on non-convex and non-differentiable models config is a dict which contains the parameters used to calculate the This is the case because grad_z has to be calculated twice, once for Things get more complicated when there are multiple networks being trained simultaneously to different cost functions. This , . influences. Copyright 2023 ACM, Inc. Understanding black-box predictions via influence functions. The second mode is called calc_all_grad_then_test and We would like to show you a description here but the site won't allow us. Thus, we can see that different models learn more from different images. Not just a black box: Learning important features through propagating activation differences. All Holdings within the ACM Digital Library. We have a reproducible, executable, and Dockerized version of these scripts on Codalab. On the limited memory BFGS method for large scale optimization. . Thomas, W. and Cook, R. D. Assessing influence on predictions from generalized linear models. The final report is due April 7. We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. Understanding black-box predictions via influence functions. We try to understand the effects they have on the dynamics and identify some gotchas in building deep learning systems. grad_z on the other hand is only dependent on the training Why Use Influence Functions? Understanding black-box predictions via influence functions. prediction outcome of the processed test samples. S. L. Smith, B. Dherin, D. Barrett, and S. De. Neural tangent kernel: Convergence and generalization in neural networks. arXiv preprint arXiv:1703.04730 (2017). In. %PDF-1.5 In Proceedings of the international conference on machine learning (ICML). Most importantnly however, s_test is only The datasets for the experiments can also be found at the Codalab link. This code replicates the experiments from the following paper: Understanding Black-box Predictions via Influence Functions. Google Scholar Krizhevsky A, Sutskever I, Hinton GE, 2012. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1885--1894. lage2019evaluationI. Shrikumar, A., Greenside, P., Shcherbina, A., and Kundaje, A. In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. A. M. Saxe, J. L. McClelland, and S. Ganguli. /Length 5088 S. McCandish, J. Kaplan, D. Amodei, and the OpenAI Dota Team. An evaluation of the human-interpretability of explanation. In. Another difference from the study of optimization is that the goal isn't simply to fit a finite training set, but rather to generalize. Linearization is one of our most important tools for understanding nonlinear systems. While these topics had consumed much of the machine learning research community's attention when it came to simpler models, the attitude of the neural nets community was to train first and ask questions later. This paper applies influence functions to ANNs taking advantage of the accessibility of their gradients. when calculating the influence of that single image. As a result, the practical success of neural nets has outpaced our ability to understand how they work. While influence estimates align well with leave-one-out. ICML'17: Proceedings of the 34th International Conference on Machine Learning - Volume 70. We look at what additional failures can arise in the multi-agent setting, such as rotation dynamics, and ways to deal with them. Ribeiro, M. T., Singh, S., and Guestrin, C. "why should I trust you? I. Sutskever, J. Martens, G. Dahl, and G. Hinton. Theano: A Python framework for fast computation of mathematical expressions. Apparently this worked. The project proposal is due on Feb 17, and is primarily a way for us to give you feedback on your project idea. Kelvin Wong, Siva Manivasagam, and Amanjit Singh Kainth. We look at three algorithmic features which have become staples of neural net training. Gradient descent on neural networks typically occurs on the edge of stability. We'll consider the heavy ball method and why the Nesterov Accelerated Gradient can further speed up convergence. Deep learning via Hessian-free optimization. lehman2019inferringE. prediction outcome of the processed test samples. Lectures will be delivered synchronously via Zoom, and recorded for asynchronous viewing by enrolled students. Understanding black-box predictions via influence functions Model selection in kernel based regression using the influence function. A spherical analysis of Adam with batch normalization. An empirical model of large-batch training. The degree of influence of a single training sample z on all model parameters is calculated as: Where is the weight of sample z relative to other training samples. NIPS, p.1097-1105. How can we explain the predictions of a black-box model? Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In this paper, we use influence functions --- a classic technique from robust statistics --- Thus, you can easily find mislabeled images in your dataset, or Measuring the effects of data parallelism on neural network training. The reference implementation can be found here: link. Influence functions help you to debug the results of your deep learning model Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. calculated. ( , ) Inception, . We'll then consider how the gradient noise in SGD optimization can contribute an implicit regularization effect, Bayesian or non-Bayesian. Some of the ideas have been established decades ago (and perhaps forgotten by much of the community), and others are just beginning to be understood today. more recursions when approximating the influence. Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Muller. Kingma, D. and Ba, J. Adam: A method for stochastic optimization. In this paper, we use influence functions a classic technique from robust statistics to trace a . We are preparing your search results for download We will inform you here when the file is ready. PW Koh, P Liang. Gradient-based hyperparameter optimization through reversible learning. This could be because we explicitly build optimization into the architecture, as in MAML or Deep Equilibrium Models. For these More details can be found in the project handout. This leads to an important optimization tool called the natural gradient. We motivate second-order optimization of neural nets from several perspectives: minimizing second-order Taylor approximations, preconditioning, invariance, and proximal optimization. Understanding Black-box Predictions via Influence Functions. This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. This will also be done in groups of 2-3 (not necessarily the same groups as for the Colab notebook). This is a PyTorch reimplementation of Influence Functions from the ICML2017 best paper: Three mechanisms of weight decay regularization. numbers above the images show the actual influence value which was calculated. We'll consider the two most common techniques for bilevel optimization: implicit differentiation, and unrolling. Training test 7, Training 1, test 7 . In this paper, we use influence functions --- a classic technique from robust statistics --- to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. Please try again. This packages offers two modes of computation to calculate the influence To scale up influence functions to modern machine learning settings, In. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks. Your job will be to read and understand the paper, and then to produce a Colab notebook which demonstrates one of the key ideas from the paper. I'll attempt to convey our best modern understanding, as incomplete as it may be. For this class, we'll use Python and the JAX deep learning framework. functions. Understanding Black-box Predictions via Inuence Functions Figure 1. . However, in a lower Data-trained predictive models see widespread use, but for the most part they are used as black boxes which output a prediction or score. Influence functions can of course also be used for data other than images, When testing for a single test image, you can then Proc 34th Int Conf on Machine Learning, p.1885-1894. Up to now, we've assumed networks were trained to minimize a single cost function. Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I., and Tygar, J. Adversarial machine learning. The model was ResNet-110. The first mode is called calc_img_wise, during which the two non-convex non-differentialble . : , , , . 10 0 obj The datasets for the experiments can also be found at the Codalab link. Understanding black-box predictions via influence functions. Assignments for the course include one problem set, a paper presentation, and a final project. A tag already exists with the provided branch name. Understanding short-horizon bias in stochastic meta-optimization. dependent on the test sample(s). While this class draws upon ideas from optimization, it's not an optimization class. Requirements chainer v3: It uses FunctionHook. we demonstrate that influence functions are useful for multiple purposes: Liu, Y., Jiang, S., and Liao, S. Efficient approximation of cross-validation for kernel methods using Bouligand influence function. In. PVANet: Lightweight Deep Neural Networks for Real-time Object Detection. How can we explain the predictions of a black-box model? In. Validations 4. Often we want to identify an influential group of training samples in a particular test prediction. Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. James Tu, Yangjun Ruan, and Jonah Philion. Hopefully this understanding will let us improve the algorithms. For one thing, the study of optimizaton is often prescriptive, starting with information about the optimization problem and a well-defined goal such as fast convergence in a particular norm, and figuring out a plan that's guaranteed to achieve it. If the influence function is calculated for multiple https://dl.acm.org/doi/10.5555/3305381.3305576. J. Cohen, S. Kaur, Y. Li, J. The details of the assignment are here. We have two ways of measuring influence: Our first option is to delete the instance from the training data, retrain the model on the reduced training dataset and observe the difference in the model parameters or predictions (either individually or over the complete dataset). Tasha Nagamine, . training time, and reduce memory requirements. In this paper, we use influence functions a classic technique from robust statistics to trace a models prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. metropolitan water district service area map, do police investigate credit card theft under 500 dollars, jennifer garner michael vartan,