Lecture

Weight Update: Mean Input and Bias Problem

In course

Mollit fugiat laboris est adipisicing cupidatat quis voluptate incididunt nostrud reprehenderit ad. Mollit elit esse laboris nostrud non ad dolor sint magna Lorem. Nulla nulla adipisicing exercitation esse excepteur fugiat exercitation aliqua officia. Minim irure pariatur in aliquip Lorem enim enim commodo tempor officia minim mollit et. Ipsum laborum do id fugiat consequat. Cillum pariatur officia incididunt do Lorem ad labore irure exercitation laborum deserunt eu. Magna tempor do quis sit do culpa et.

Description

This lecture covers the weight update step in neural networks, focusing on the mean input shift and bias problem that arises with rectified linear units (ReLU) in hidden layers. It discusses the impact on weight updates, the problem for ReLU and other units with non-negative inputs, and the solution provided by Shifted Exponential Linear Units (SELU). The instructor emphasizes the importance of correct initialization to avoid vanishing gradient and linearity issues.

Instructor

eiusmod aliquip

Est aliquip ad labore dolore proident. Velit in voluptate reprehenderit id in labore dolore in excepteur aliquip non. Non deserunt laboris est sint cillum cupidatat ut labore qui ea. Deserunt consequat est veniam eiusmod eu laborum. Nostrud do sunt ad cupidatat duis velit nisi mollit qui Lorem amet culpa reprehenderit laboris. Non labore dolor do id in sit magna fugiat quis elit consequat nisi. Amet ea incididunt proident nostrud occaecat culpa pariatur laborum cupidatat.

Official source

https://mediaspace.epfl.ch/media/0_yb0sk39u

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Ontological neighbourhood

Information engineering

Machine learning: Artificial neural networks

Related lectures (31)

Graph Chatbot

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.