Applied Deep Learning Resources

 

Applied Deep Learning Resources

A collection of research articles, blog posts, slides and code snippets about deep learning in applied settings. Including trained models and simple methods that can be used out of the box. Mainly focusing on Convolutional Neural Networks (CNN) but Recurrent Neural Networks (RNN), deep Q-Networks (DQN) and other interesting architectures will also be listed.

CNN

Latest overview of the CNNs can be found from the paper "Deep learning for visual understanding: A review" [link, PDF]

Another decent overview in Nature by LeCun, Bengio and Hinton: "Deep learning" [link, PDF]

ImageNet

ImageNet is the most important image classification and localization competition. Other data sets with results can be found from here: "Discover the current state of the art in objects classification." [link].

Prediction error of the ImageNet competition has been decreasing rapidly over the last 5 years: 

Main network architectures on ImageNet

AlexNet

Original paper: "ImageNet Classification with Deep Convolutional Neural Networks" [PDF]

Properties: 8 weight layers (5 convolutional and 2 fully connected), 60 million parameters, Rectified Linear Units (ReLUs), Local Response Normalization, Dropout

VGG

Original paper: "Very Deep Convolutional Networks for Large-Scale Image Recognition" [arxiv]

Properties: 19 weight layers, 144m parameters, 3x3 convolution filters, L2 regularised, Dropout, No Local Response Normalization

GoogLeNet

Original paper: "Going deeper with convolutions" [arxiv]

Lates upgrade to the model achieves even better scores with models and import to Torch: "Rethinking the Inception Architecture for Computer Vision" [arxiv], "Torch port of Inception V3" [github]

Properties: 22 layers, 7m parameters, Inception modules, 1x1 conv layers, ReLUs, Dropout, Mid-level outputs

Inception modules:

ResNet

Original paper: "Deep Residual Learning for Image Recognition" [arxiv]

Very nice slides: "Deep Residual Learning" [PDF]

Github: [github]

Properties: 152 layers, ReLUs, Batch Normalization (See "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" [arxiv]), less hacks (no dropout), more stable (different number of layers work as well) and lower complexity than VGG.

Main building block of the network:

Features are also very good and transferable with (faster) R-CNNs (see below):

Other architectures

  • Deep Learning for 3D shapes: "3D ShapeNets: A Deep Representation for Volumetric Shapes" [PDF]
  • Code and a model for faces: "Free and open source face recognition with deep neural networks." [github]
  • Fast neural networks which can perform arbitrary filters for images: "Deep Edge-Aware Filters" [PDF]
  • Lot's of different models in Caffe's "Model Zoo" [github]

Feature learning and object detection

  • "CNN Features off-the-shelf: an Astounding Baseline for Recognition" [arxiv]
  • First paper about R-CNN: "Rich feature hierarchies for accurate object detection and semantic segmentation" [PDF, slides]
  • "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks" [arxiv, github, Slides]

  • "An Empirical Evaluation of Deep Learning on Highway Driving" [arxiv]

  • "Object Detectors Emerge in Deep Scene CNNs" [arxiv]

  • Faster and better features: "Efficient Deep Feature Learning and Extraction via StochasticNets" [arxiv]

Other

  • Code and models for automatic captions of images: "Deep Visual-Semantic Alignments for Generating Image Descriptions"[web poster, PDF, github]

  • Google Deep Dream or neural networks on LSD: "Inceptionism: Going Deeper into Neural Networks" [link,deepdreamer.io/]

Deep dreaming from noise:

  • "Automatic Colorization" and it includes a pre-trained model [Link]

  • "Learning visual similarity for product design with convolutional neural networks" [PDF]

  • Using images and image descriptions to improve search results: "Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank" [arxiv]
  • "How Google Translate squeezes deep learning onto a phone" [post]

  • "What a Deep Neural Network thinks about your #selfie" [blog]

Top selfies according to the ConvNet:

  • "Recommending music on Spotify with deep learning" [github]
  • "DeepStereo: Learning to Predict New Views from the World's Imagery" [arxiv]

  • Classifying street signs: "The power of Spatial Transformer Networks" [blog] with "Spatial Transformer Networks" [arxiv]

  • "Pedestrian Detection with RCNN" [PDF]

DQN

  • Original paper: "Playing Atari with Deep Reinforcement Learning" [arxiv]
  • My popular science article about DQN: "Artificial General Intelligence that plays Atari video games: How did DeepMind do it?" [link]
  • DQN for RoboCup: "Deep Reinforcement Learning in Parameterized Action Space" [arxiv]

RNN

  • Original paper of the best RNN architecture: "Long short-term memory" [PDF]
  • Very good tutorial-like introduction to RNNs by Andrej Karpathy: "The Unreasonable Effectiveness of Recurrent Neural Networks" [link]
  • "Visualizing and Understanding Recurrent Networks" [arxiv]
  • "Composing Music With Recurrent Neural Networks" [blog]

Other promising or useful architectures

  • HTMs by Jeff Hawkins: "Continuous online sequence learning with an unsupervised neural network model"​ [arxiv]
  • Word2vec: "Efficient Estimation of Word Representations in Vector Space" [arxiv, Google code]
  • "Feedforward Sequential Memory Networks: A New Structure to Learn Long-term Dependency" [arxiv]

Framework benchmarks

  • "Comparative Study of Caffe, Neon, Theano and Torch for deep learning" [arxiv]

Their summary: From our experiments, we observe that Theano and Torch are the most easily extensible frameworks. We observe that Torch is best suited for any deep architecture on CPU, followed by Theano. It also achieves the best performance on the GPU for large convolutional and fully connected networks, followed closely by Neon. Theano achieves the best performance on GPU for training and deployment of LSTM networks. Finally Caffe is the easiest for evaluating the performance of standard deep architectures.

  • Very good qualitative analysis: zer0n/deepframeworks: [github]
  • Just performance comparison: soumith/convnet-benchmarks: [github]
  • "Deep Learning Libraries by Language" [link]

Other resources

Credits

Most of the snippets have come to my attention via internal mailing lists of Computational Neuroscience Lab at University of Tartu and London-based visual search company Dream It Get It. I am also reading a weekly newsletter by Data Elixir and checking research papers of the two main deep learning conferences: ICML and NIPS.

 

时间： 2024-12-26 06:29:37

Applied Deep Learning Resources的相关文章

(转) Deep Learning Resources

    转自:http://www.jeremydjacksonphd.com/category/deep-learning/   Deep Learning Resources Posted on May 13, 2015   Videos Deep Learning and Neural Networks with Kevin Duh: course page NY Course by Yann LeCun: 2014 version, 2015 version NIPS 2015 Deep

QA Systems and Deep Learning Technologies – Part 2

Introduction This is the second article in a two part series about QA Systems and Deep Learning. You can read part 1 here. Deep Learning is a subfield of machine learning, and aims at using machines for data abstraction with the help of multiple proc

(转) Learning Deep Learning with Keras

Learning Deep Learning with Keras  Piotr Migdał - blog Projects Articles Publications Resume About Photos Learning Deep Learning with Keras 30 Apr 2017 • Piotr Migdał • [machine-learning] [deep-learning] [overview] I teach deep learning both for a li

Machine and Deep Learning with Python

Machine and Deep Learning with Python Education Tutorials and courses Supervised learning superstitions cheat sheet Introduction to Deep Learning with Python How to implement a neural network How to build and run your first deep learning network Neur

QA Systems and Deep Learning Technologies – Part 1

1. Introduction The automatic question and answering (QA) system has been in use for decades now. However, Siri's and Watson's success in 2011 has captured the whole industry's attention. Since the success of these two technologies, the automatic QA

Deep Learning vs. Machine Learning vs. Pattern Recognition

Introduction: Deep learning, machine learning, and pattern recognition are highly relevant topics commonly used in the field of robotics with artificial intelligence. Despite the overlapping similarities, these concepts are not identical. In this art

(zhuan) Where can I start with Deep Learning?

Where can I start with Deep Learning? By Rotek Song, Deep Reinforcement Learning/Robotics/Computer Vision/iOS | 03/01/2017       If you are a newcomer to the Deep Learning area, the first question you may have is "Which paper should I start reading f

(转) Deep Learning Research Review Week 2: Reinforcement Learning

  Deep Learning Research Review Week 2: Reinforcement Learning   转载自: https://adeshpande3.github.io/adeshpande3.github.io/Deep-Learning-Research-Review-Week-2-Reinforcement-Learning This is the 2nd installment of a new series called Deep Learning Res

（转）The 9 Deep Learning Papers You Need To Know About (Understanding CNNs Part 3)

Adit Deshpande CS Undergrad at UCLA ('19) Blog About The 9 Deep Learning Papers You Need To Know About (Understanding CNNs Part 3) Introduction Link to Part 1Link to Part 2                 In this post, we'll go into summarizing a lot of the new and