sem/CVML-MachineLearning
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add all files

Browse Source
This commit is contained in:
Sem van der Hoeven
2021-05-26 15:12:05 +02:00
parent 5c3be10247
commit d979ca38f5
44 changed files with 43574 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

206
course/0_Check_Environment.ipynb Normal file
View File

@@ -0,0 +1,206 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Check Environment\n",
"This notebook checks that you have correctly created the environment and that all packages needed are installed."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Environment\n",
"\n",
"The next command should return a line like (Mac/Linux):\n",
"\n",
" /<YOUR-HOME-FOLDER>/anaconda/envs/ztdl/bin/python\n",
"\n",
"or like (Windows 10):\n",
"\n",
" C:\\\\<YOUR-HOME-FOLDER>\\\\Anaconda3\\\\envs\\\\ztdl\\\\python.exe\n",
"\n",
"In particular you should make sure that you are using the python executable from within the course environment.\n",
"\n",
"If that's not the case do this:\n",
"\n",
"1. close this notebook\n",
"2. go to the terminal and stop jupyer notebook\n",
"3. make sure that you have activated the environment, you should see a prompt like:\n",
"\n",
" (ztdl) $\n",
"4. (optional) if you don't see that prompt activate the environment:\n",
" - mac/linux:\n",
" \n",
" conda activate ztdl\n",
"\n",
" - windows:\n",
"\n",
" activate ztdl\n",
"5. restart jupyter notebook"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import sys\n",
"sys.executable"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Python 3.7\n",
"\n",
"The next line should say that you're using Python 3.7.x from Anaconda. At the time of publication it looks like this (Mac/Linux):\n",
"\n",
" Python 3.7.3 (default, Mar 27 2019, 22:11:17)\n",
" [GCC 7.3.0] :: Anaconda, Inc. on linux\n",
" Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n",
"\n",
"or like this (Windows 10):\n",
"\n",
" Python 3.7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32\n",
" Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n",
"\n",
"but date and exact version of GCC may change in the future.\n",
"\n",
"If you see a different version of python, go back to the previous step and make sure you created and activated the environment correctly."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"sys.version"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Jupyter\n",
"\n",
"Check that Jupyter is running from within the environment. The next line should look like (Mac/Linux):\n",
"\n",
" /<YOUR-HOME-FOLDER>/anaconda/envs/ztdl/lib/python3.6/site-packages/jupyter.py'\n",
"\n",
"or like this (Windows 10):\n",
"\n",
" C:\\\\Users\\\\<YOUR-USER>\\\\Anaconda3\\\\envs\\\\ztdl\\\\lib\\\\site-packages\\\\jupyter.py"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import jupyter\n",
"jupyter.__file__"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Other packages\n",
"\n",
"Here we will check that all the packages are installed and have the correct versions. If everything is ok you should see:\n",
" \n",
" Using TensorFlow backend.\n",
" \n",
" Houston we are go!\n",
"\n",
"If there's any issue here please make sure you have checked the previous steps and if it's all good please send us a question in the Q&A forum."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pip\n",
"import numpy\n",
"import jupyter\n",
"import matplotlib\n",
"import sklearn\n",
"import scipy\n",
"import pandas\n",
"import PIL\n",
"import seaborn\n",
"import tensorflow\n",
"\n",
"\n",
"def check_version(pkg, version):\n",
" actual = pkg.__version__.split('.')\n",
" if len(actual) == 3:\n",
" actual_major = '.'.join(actual[:2])\n",
" elif len(actual) == 2:\n",
" actual_major = '.'.join(actual)\n",
" else:\n",
" raise NotImplementedError(pkg.__name__ +\n",
" \"actual version :\"+\n",
" pkg.__version__)\n",
" try:\n",
" assert(actual_major == version)\n",
" except Exception as ex:\n",
" print(\"{} {}\\t=> {}\".format(pkg.__name__,\n",
" version,\n",
" pkg.__version__))\n",
" raise ex\n",
"\n",
"check_version(pip, '21.0')\n",
"check_version(numpy, '1.19')\n",
"check_version(matplotlib, '3.3')\n",
"check_version(sklearn, '0.24')\n",
"check_version(scipy, '1.6')\n",
"check_version(pandas, '1.2')\n",
"check_version(PIL, '8.2')\n",
"check_version(seaborn, '0.11')\n",
"check_version(tensorflow, '2.5')\n",
"\n",
"print(\"Houston we are go!\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

190
course/1 First Deep Learning Model-Copy1.ipynb Normal file
View File

@@ -0,0 +1,190 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# First Deep Learning Model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np # import the numpy library and assign the name np to it\n",
"%matplotlib inline # magic function that sets the backend of matplotlib to the inline backend\n",
"import matplotlib.pyplot as plt # import the matplotlib.pyplot and assign the name plt to it"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.datasets import make_circles"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X, y = make_circles(n_samples=1000,\n",
" noise=0.1,\n",
" factor=0.2,\n",
" random_state=0)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.figure(figsize=(5, 5))\n",
"plt.plot(X[y==0, 0], X[y==0, 1], 'ob', alpha=0.5)\n",
"plt.plot(X[y==1, 0], X[y==1, 1], 'xr', alpha=0.5)\n",
"plt.xlim(-1.5, 1.5)\n",
"plt.ylim(-1.5, 1.5)\n",
"plt.legend(['0', '1'])\n",
"plt.title(\"Blue circles and Red crosses\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.models import Sequential\n",
"from tensorflow.keras.layers import Dense\n",
"from tensorflow.keras.optimizers import SGD"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = Sequential()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.add(Dense(4, input_shape=(2,), activation='tanh'))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.add(Dense(1, activation='sigmoid'))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.compile(SGD(learning_rate=0.5), 'binary_crossentropy', metrics=['accuracy'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.fit(X, y, epochs=20)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"hticks = np.linspace(-1.5, 1.5, 101)\n",
"vticks = np.linspace(-1.5, 1.5, 101)\n",
"aa, bb = np.meshgrid(hticks, vticks)\n",
"ab = np.c_[aa.ravel(), bb.ravel()]\n",
"c = model.predict(ab)\n",
"cc = c.reshape(aa.shape)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.figure(figsize=(5, 5))\n",
"plt.contourf(aa, bb, cc, cmap='bwr', alpha=0.2)\n",
"plt.plot(X[y==0, 0], X[y==0, 1], 'ob', alpha=0.5)\n",
"plt.plot(X[y==1, 0], X[y==1, 1], 'xr', alpha=0.5)\n",
"plt.xlim(-1.5, 1.5)\n",
"plt.ylim(-1.5, 1.5)\n",
"plt.legend(['0', '1'])\n",
"plt.title(\"Blue circles and Red crosses\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

190
course/1 First Deep Learning Model.ipynb Normal file
View File

@@ -0,0 +1,190 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# First Deep Learning Model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"%matplotlib inline\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.datasets import make_circles"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X, y = make_circles(n_samples=1000,\n",
" noise=0.1,\n",
" factor=0.2,\n",
" random_state=0)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.figure(figsize=(5, 5))\n",
"plt.plot(X[y==0, 0], X[y==0, 1], 'ob', alpha=0.5)\n",
"plt.plot(X[y==1, 0], X[y==1, 1], 'xr', alpha=0.5)\n",
"plt.xlim(-1.5, 1.5)\n",
"plt.ylim(-1.5, 1.5)\n",
"plt.legend(['0', '1'])\n",
"plt.title(\"Blue circles and Red crosses\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.models import Sequential\n",
"from tensorflow.keras.layers import Dense\n",
"from tensorflow.keras.optimizers import SGD"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = Sequential()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.add(Dense(4, input_shape=(2,), activation='tanh'))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.add(Dense(1, activation='sigmoid'))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.compile(SGD(learning_rate=0.5), 'binary_crossentropy', metrics=['accuracy'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.fit(X, y, epochs=20)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"hticks = np.linspace(-1.5, 1.5, 101)\n",
"vticks = np.linspace(-1.5, 1.5, 101)\n",
"aa, bb = np.meshgrid(hticks, vticks)\n",
"ab = np.c_[aa.ravel(), bb.ravel()]\n",
"c = model.predict(ab)\n",
"cc = c.reshape(aa.shape)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.figure(figsize=(5, 5))\n",
"plt.contourf(aa, bb, cc, cmap='bwr', alpha=0.2)\n",
"plt.plot(X[y==0, 0], X[y==0, 1], 'ob', alpha=0.5)\n",
"plt.plot(X[y==1, 0], X[y==1, 1], 'xr', alpha=0.5)\n",
"plt.xlim(-1.5, 1.5)\n",
"plt.ylim(-1.5, 1.5)\n",
"plt.legend(['0', '1'])\n",
"plt.title(\"Blue circles and Red crosses\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

1892
course/2 Data.ipynb Normal file
View File

File diff suppressed because one or more lines are too long

1177
course/3 Machine Learning.ipynb Normal file
View File

File diff suppressed because one or more lines are too long

560
course/4 Deep Learning Intro.ipynb Normal file
View File

@@ -0,0 +1,560 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Deep Learning Intro"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import matplotlib.pyplot as plt\n",
"import pandas as pd\n",
"import numpy as np"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Shallow and Deep Networks"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.datasets import make_moons\n",
"\n",
"X, y = make_moons(n_samples=1000, noise=0.1, random_state=0)\n",
"plt.plot(X[y==0, 0], X[y==0, 1], 'ob', alpha=0.5)\n",
"plt.plot(X[y==1, 0], X[y==1, 1], 'xr', alpha=0.5)\n",
"plt.legend(['0', '1'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train, X_test, y_train, y_test = train_test_split(X, y,\n",
" test_size=0.3,\n",
" random_state=42)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.models import Sequential\n",
"from tensorflow.keras.layers import Dense\n",
"from tensorflow.keras.optimizers import SGD, Adam"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Shallow Model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = Sequential()\n",
"model.add(Dense(1, input_shape=(2,), activation='sigmoid'))\n",
"model.compile(Adam(learning_rate=0.05), 'binary_crossentropy', metrics=['accuracy'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.fit(X_train, y_train, epochs=200, verbose=0)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"results = model.evaluate(X_test, y_test)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"results"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(\"The Accuracy score on the Train set is:\\t{:0.3f}\".format(results[1]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def plot_decision_boundary(model, X, y):\n",
" amin, bmin = X.min(axis=0) - 0.1\n",
" amax, bmax = X.max(axis=0) + 0.1\n",
" hticks = np.linspace(amin, amax, 101)\n",
" vticks = np.linspace(bmin, bmax, 101)\n",
" \n",
" aa, bb = np.meshgrid(hticks, vticks)\n",
" ab = np.c_[aa.ravel(), bb.ravel()]\n",
" \n",
" c = model.predict(ab)\n",
" cc = c.reshape(aa.shape)\n",
"\n",
" plt.figure(figsize=(12, 8))\n",
" plt.contourf(aa, bb, cc, cmap='bwr', alpha=0.2)\n",
" plt.plot(X[y==0, 0], X[y==0, 1], 'ob', alpha=0.5)\n",
" plt.plot(X[y==1, 0], X[y==1, 1], 'xr', alpha=0.5)\n",
" plt.legend(['0', '1'])\n",
" \n",
"plot_decision_boundary(model, X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Deep model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = Sequential()\n",
"model.add(Dense(4, input_shape=(2,), activation='tanh'))\n",
"model.add(Dense(2, activation='tanh'))\n",
"model.add(Dense(1, activation='sigmoid'))\n",
"model.compile(Adam(learning_rate=0.05), 'binary_crossentropy', metrics=['accuracy'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.fit(X_train, y_train, epochs=100, verbose=0)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.evaluate(X_test, y_test)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.metrics import accuracy_score, confusion_matrix"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_train_pred = model.predict_classes(X_train)\n",
"y_test_pred = model.predict_classes(X_test)\n",
"\n",
"print(\"The Accuracy score on the Train set is:\\t{:0.3f}\".format(accuracy_score(y_train, y_train_pred)))\n",
"print(\"The Accuracy score on the Test set is:\\t{:0.3f}\".format(accuracy_score(y_test, y_test_pred)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_decision_boundary(model, X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Multiclass classification\n",
"\n",
"### The Iris dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv('../data/iris.csv')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import seaborn as sns\n",
"sns.pairplot(df, hue=\"species\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X = df.drop('species', axis=1)\n",
"X.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"target_names = df['species'].unique()\n",
"target_names"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"target_dict = {n:i for i, n in enumerate(target_names)}\n",
"target_dict"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y= df['species'].map(target_dict)\n",
"y.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.utils import to_categorical"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_cat = to_categorical(y)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_cat[:10]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train, X_test, y_train, y_test = train_test_split(X.values, y_cat,\n",
" test_size=0.2)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = Sequential()\n",
"model.add(Dense(3, input_shape=(4,), activation='softmax'))\n",
"model.compile(Adam(learning_rate=0.1),\n",
" loss='categorical_crossentropy',\n",
" metrics=['accuracy'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.fit(X_train, y_train, epochs=20, validation_split=0.1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_pred = model.predict(X_test)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_pred[:5]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_test_class = np.argmax(y_test, axis=1)\n",
"y_pred_class = np.argmax(y_pred, axis=1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.metrics import classification_report"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(classification_report(y_test_class, y_pred_class))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"confusion_matrix(y_test_class, y_pred_class)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise 1"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"The [Pima Indians dataset](https://archive.ics.uci.edu/ml/datasets/diabetes) is a very famous dataset distributed by UCI and originally collected from the National Institute of Diabetes and Digestive and Kidney Diseases. It contains data from clinical exams for women age 21 and above of Pima indian origins. The objective is to predict based on diagnostic measurements whether a patient has diabetes.\n",
"\n",
"It has the following features:\n",
"\n",
"- Pregnancies: Number of times pregnant\n",
"- Glucose: Plasma glucose concentration a 2 hours in an oral glucose tolerance test\n",
"- BloodPressure: Diastolic blood pressure (mm Hg)\n",
"- SkinThickness: Triceps skin fold thickness (mm)\n",
"- Insulin: 2-Hour serum insulin (mu U/ml)\n",
"- BMI: Body mass index (weight in kg/(height in m)^2)\n",
"- DiabetesPedigreeFunction: Diabetes pedigree function\n",
"- Age: Age (years)\n",
"\n",
"The last colum is the outcome, and it is a binary variable.\n",
"\n",
"In this first exercise we will explore it through the following steps:\n",
"\n",
"1. Load the ..data/diabetes.csv dataset, use pandas to explore the range of each feature\n",
"- For each feature draw a histogram. Bonus points if you draw all the histograms in the same figure.\n",
"- Explore correlations of features with the outcome column. You can do this in several ways, for example using the `sns.pairplot` we used above or drawing a heatmap of the correlations.\n",
"- Do features need standardization? If so what stardardization technique will you use? MinMax? Standard?\n",
"- Prepare your final `X` and `y` variables to be used by a ML model. Make sure you define your target variable well. Will you need dummy columns?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise 2"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"Build a fully connected NN model that predicts diabetes. Follow these steps:\n",
"\n",
"1. Split your data in a train/test with a test size of 20% and a `random_state = 22`\n",
"- define a sequential model with at least one inner layer. You will have to make choices for the following things:\n",
" - what is the size of the input?\n",
" - how many nodes will you use in each layer?\n",
" - what is the size of the output?\n",
" - what activation functions will you use in the inner layers?\n",
" - what activation function will you use at output?\n",
" - what loss function will you use?\n",
" - what optimizer will you use?\n",
"- fit your model on the training set, using a validation_split of 0.1\n",
"- test your trained model on the test data from the train/test split\n",
"- check the accuracy score, the confusion matrix and the classification report"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise 3\n",
"Compare your work with the results presented in [this notebook](https://www.kaggle.com/futurist/d/uciml/pima-indians-diabetes-database/pima-data-visualisation-and-machine-learning). Are your Neural Network results better or worse than the results obtained by traditional Machine Learning techniques?\n",
"\n",
"- Try training a Support Vector Machine or a Random Forest model on the exact same train/test split. Is the performance better or worse?\n",
"- Try restricting your features to only 4 features like in the suggested notebook. How does model performance change?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise 4\n",
"\n",
"[Tensorflow playground](http://playground.tensorflow.org/) is a web based neural network demo. It is really useful to develop an intuition about what happens when you change architecture, activation function or other parameters. Try playing with it for a few minutes. You don't need do understand the meaning of every knob and button in the page, just get a sense for what happens if you change something. In the next chapter we'll explore these things in more detail.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

1055
course/5 Gradient Descent.ipynb Normal file
View File

File diff suppressed because it is too large Load Diff

988
course/6 Convolutional Neural Networks.ipynb Normal file
View File

@@ -0,0 +1,988 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Convolutional Neural Networks"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Machine learning on images"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"%matplotlib inline\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### MNIST"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.datasets import mnist"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"(X_train, y_train), (X_test, y_test) = mnist.load_data('/tmp/mnist.npz')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_test.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.imshow(X_train[0], cmap='gray')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train = X_train.reshape(-1, 28*28)\n",
"X_test = X_test.reshape(-1, 28*28)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train = X_train.astype('float32')\n",
"X_test = X_test.astype('float32')\n",
"X_train /= 255.0\n",
"X_test /= 255.0"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.utils import to_categorical"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_train_cat = to_categorical(y_train)\n",
"y_test_cat = to_categorical(y_test)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_train[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_train_cat[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_train_cat.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_test_cat.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Fully connected on images"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.models import Sequential\n",
"from tensorflow.keras.layers import Dense\n",
"import tensorflow.keras.backend as K\n",
"\n",
"# K.clear_session()\n",
"\n",
"model = Sequential()\n",
"model.add(Dense(512, input_dim=28*28, activation='relu'))\n",
"model.add(Dense(256, activation='relu'))\n",
"model.add(Dense(128, activation='relu'))\n",
"model.add(Dense(32, activation='relu'))\n",
"model.add(Dense(10, activation='softmax'))\n",
"model.compile(loss='categorical_crossentropy',\n",
" optimizer='rmsprop',\n",
" metrics=['accuracy'])\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"h = model.fit(X_train, y_train_cat, batch_size=128, epochs=10, verbose=1, validation_split=0.3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.plot(h.history['accuracy'])\n",
"plt.plot(h.history['val_accuracy'])\n",
"plt.legend(['Training', 'Validation'])\n",
"plt.title('Accuracy')\n",
"plt.xlabel('Epochs');"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"test_accuracy = model.evaluate(X_test, y_test_cat)[1]\n",
"test_accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Tensor Math"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A = np.random.randint(10, size=(2, 3, 4, 5))\n",
"B = np.random.randint(10, size=(2, 3))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A[0, 1, 0, 3]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"B"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### A random colored image"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"img = np.random.randint(255, size=(4, 4, 3), dtype='uint8')\n",
"img"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.figure(figsize=(5, 5))\n",
"plt.subplot(221)\n",
"plt.imshow(img)\n",
"plt.title(\"All Channels combined\")\n",
"\n",
"plt.subplot(222)\n",
"plt.imshow(img[:, : , 0], cmap='Reds')\n",
"plt.title(\"Red channel\")\n",
"\n",
"plt.subplot(223)\n",
"plt.imshow(img[:, : , 1], cmap='Greens')\n",
"plt.title(\"Green channel\")\n",
"\n",
"plt.subplot(224)\n",
"plt.imshow(img[:, : , 2], cmap='Blues')\n",
"plt.title(\"Blue channel\");"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Tensor operations"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"2 * A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A + A"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"A.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"B.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.tensordot(A, B, axes=([0, 1], [0, 1]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.tensordot(A, B, axes=([0], [0])).shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1D convolution"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"a = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0], dtype='float32')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"b = np.array([-1, 1], dtype='float32')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"c = np.convolve(a, b)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"a"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"b"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"c"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.subplot(211)\n",
"plt.plot(a, 'o-')\n",
"\n",
"plt.subplot(212)\n",
"plt.plot(c, 'o-');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Image filters with convolutions"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from scipy.ndimage.filters import convolve\n",
"from scipy.signal import convolve2d\n",
"from scipy import misc"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"img = misc.ascent()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"img.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.imshow(img, cmap='gray');"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"h_kernel = np.array([[ 1, 2, 1],\n",
" [ 0, 0, 0],\n",
" [-1, -2, -1]])\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"plt.imshow(h_kernel, cmap='gray');"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"res = convolve2d(img, h_kernel)\n",
"\n",
"plt.imshow(res, cmap='gray');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Convolutional neural networks"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.layers import Conv2D"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"img.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.figure(figsize=(5, 5))\n",
"plt.imshow(img, cmap='gray');"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"img_tensor = img.reshape((1, 512, 512, 1))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = Sequential()\n",
"model.add(Conv2D(1, (3, 3), strides=(2,1), input_shape=(512, 512, 1)))\n",
"model.compile('adam', 'mse')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"img_pred_tensor = model.predict(img_tensor)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"img_pred_tensor.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"img_pred = img_pred_tensor[0, :, :, 0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.imshow(img_pred, cmap='gray');"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"weights = model.get_weights()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"weights[0].shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.imshow(weights[0][:, :, 0, 0], cmap='gray');"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"weights[0] = np.ones(weights[0].shape)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.set_weights(weights)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"img_pred_tensor = model.predict(img_tensor)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"img_pred = img_pred_tensor[0, :, :, 0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.imshow(img_pred, cmap='gray');"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = Sequential()\n",
"model.add(Conv2D(1, (3, 3), input_shape=(512, 512, 1), padding='same'))\n",
"model.compile('adam', 'mse')\n",
"\n",
"img_pred_tensor = model.predict(img_tensor)\n",
"\n",
"\n",
"img_pred_tensor.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pooling layers"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.layers import MaxPool2D, AvgPool2D"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = Sequential()\n",
"model.add(MaxPool2D((5, 5), input_shape=(512, 512, 1)))\n",
"model.compile('adam', 'mse')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"img_pred = model.predict(img_tensor)[0, :, :, 0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.imshow(img_pred, cmap='gray')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = Sequential()\n",
"model.add(AvgPool2D((5, 5), input_shape=(512, 512, 1)))\n",
"model.compile('adam', 'mse')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"img_pred = model.predict(img_tensor)[0, :, :, 0]\n",
"plt.imshow(img_pred, cmap='gray');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Final architecture"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train = X_train.reshape(-1, 28, 28, 1)\n",
"X_test = X_test.reshape(-1, 28, 28, 1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.layers import Flatten, Activation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"K.clear_session()\n",
"\n",
"model = Sequential()\n",
"\n",
"model.add(Conv2D(32, (3, 3), input_shape=(28, 28, 1)))\n",
"model.add(MaxPool2D(pool_size=(2, 2)))\n",
"model.add(Activation('relu'))\n",
"\n",
"model.add(Flatten())\n",
"\n",
"model.add(Dense(128, activation='relu'))\n",
"\n",
"model.add(Dense(10, activation='softmax'))\n",
"\n",
"model.compile(loss='categorical_crossentropy',\n",
" optimizer='rmsprop',\n",
" metrics=['accuracy'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.summary()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.fit(X_train, y_train_cat, batch_size=128,\n",
" epochs=2, verbose=1, validation_split=0.3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.evaluate(X_test, y_test_cat)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Exercise 1\n",
"You've been hired by a shipping company to overhaul the way they route mail, parcels and packages. They want to build an image recognition system capable of recognizing the digits in the zipcode on a package, so that it can be automatically routed to the correct location.\n",
"You are tasked to build the digit recognition system. Luckily, you can rely on the MNIST dataset for the intial training of your model!\n",
"\n",
"Build a deep convolutional neural network with at least two convolutional and two pooling layers before the fully connected layer.\n",
"\n",
"- Start from the network we have just built\n",
"- Insert a `Conv2D` layer after the first `MaxPool2D`, give it 64 filters.\n",
"- Insert a `MaxPool2D` after that one\n",
"- Insert an `Activation` layer\n",
"- retrain the model\n",
"- does performance improve?\n",
"- how many parameters does this new model have? More or less than the previous model? Why?\n",
"- how long did this second model take to train? Longer or shorter than the previous model? Why?\n",
"- did it perform better or worse than the previous model?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 2\n",
"\n",
"Pleased with your performance with the digits recognition task, your boss decides to challenge you with a harder task. Their online branch allows people to upload images to a website that generates and prints a postcard that is shipped to destination. Your boss would like to know what images people are loading on the site in order to provide targeted advertising on the same page, so he asks you to build an image recognition system capable of recognizing a few objects. Luckily for you, there's a dataset ready made with a collection of labeled images. This is the [Cifar 10 Dataset](http://www.cs.toronto.edu/~kriz/cifar.html), a very famous dataset that contains images for 10 different categories:\n",
"\n",
"- airplane \t\t\t\t\t\t\t\t\t\t\n",
"- automobile \t\t\t\t\t\t\t\t\t\t\n",
"- bird \t\t\t\t\t\t\t\t\t\t\n",
"- cat \t\t\t\t\t\t\t\t\t\t\n",
"- deer \t\t\t\t\t\t\t\t\t\t\n",
"- dog \t\t\t\t\t\t\t\t\t\t\n",
"- frog \t\t\t\t\t\t\t\t\t\t\n",
"- horse \t\t\t\t\t\t\t\t\t\t\n",
"- ship \t\t\t\t\t\t\t\t\t\t\n",
"- truck\n",
"\n",
"In this exercise we will reach the limit of what you can achieve on your laptop and get ready for the next session on cloud GPUs.\n",
"\n",
"Here's what you have to do:\n",
"- load the cifar10 dataset using `keras.datasets.cifar10.load_data()`\n",
"- display a few images, see how hard/easy it is for you to recognize an object with such low resolution\n",
"- check the shape of X_train, does it need reshape?\n",
"- check the scale of X_train, does it need rescaling?\n",
"- check the shape of y_train, does it need reshape?\n",
"- build a model with the following architecture, and choose the parameters and activation functions for each of the layers:\n",
" - conv2d\n",
" - conv2d\n",
" - maxpool\n",
" - conv2d\n",
" - conv2d\n",
" - maxpool\n",
" - flatten\n",
" - dense\n",
" - output\n",
"- compile the model and check the number of parameters\n",
"- attempt to train the model with the optimizer of your choice. How fast does training proceed?\n",
"- If training is too slow (as expected) stop the execution and move to the next session!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.datasets import cifar10"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

565
course/8 Recurrent Neural Networks.ipynb Normal file
View File

@@ -0,0 +1,565 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Recurrent Neural Networks"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"%matplotlib inline\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Time series forecasting"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv('../data/cansim-0800020-eng-6674700030567901031.csv',\n",
" skiprows=6, skipfooter=9,\n",
" engine='python')\n",
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pandas.tseries.offsets import MonthEnd"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df['Adjustments'] = pd.to_datetime(df['Adjustments']) + MonthEnd(1)\n",
"df = df.set_index('Adjustments')\n",
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.plot()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"split_date = pd.Timestamp('01-01-2011')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train = df.loc[:split_date, ['Unadjusted']]\n",
"test = df.loc[split_date:, ['Unadjusted']]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ax = train.plot()\n",
"test.plot(ax=ax)\n",
"plt.legend(['train', 'test'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.preprocessing import MinMaxScaler\n",
"\n",
"sc = MinMaxScaler()\n",
"\n",
"train_sc = sc.fit_transform(train)\n",
"test_sc = sc.transform(test)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_sc[:4]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train = train_sc[:-1]\n",
"y_train = train_sc[1:]\n",
"\n",
"X_test = test_sc[:-1]\n",
"y_test = test_sc[1:]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Fully connected predictor"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.models import Sequential\n",
"from tensorflow.keras.layers import Dense\n",
"import tensorflow.keras.backend as K\n",
"from tensorflow.keras.callbacks import EarlyStopping"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"K.clear_session()\n",
"\n",
"model = Sequential()\n",
"model.add(Dense(12, input_dim=1, activation='relu'))\n",
"model.add(Dense(1))\n",
"model.compile(loss='mean_squared_error', optimizer='adam')\n",
"model.summary()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"early_stop = EarlyStopping(monitor='loss', patience=1, verbose=1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.fit(X_train, y_train, epochs=200,\n",
" batch_size=2, verbose=1,\n",
" callbacks=[early_stop])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_pred = model.predict(X_test)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.plot(y_test)\n",
"plt.plot(y_pred)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Recurrent predictor"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.layers import LSTM"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#3D tensor with shape (batch_size, timesteps, input_dim)\n",
"X_train[:, None].shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train_t = X_train[:, None]\n",
"X_test_t = X_test[:, None]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"K.clear_session()\n",
"model = Sequential()\n",
"\n",
"model.add(LSTM(6, input_shape=(1, 1)))\n",
"\n",
"model.add(Dense(1))\n",
"\n",
"model.compile(loss='mean_squared_error', optimizer='adam')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.fit(X_train_t, y_train,\n",
" epochs=100, batch_size=1, verbose=1,\n",
" callbacks=[early_stop])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_pred = model.predict(X_test_t)\n",
"plt.plot(y_test)\n",
"plt.plot(y_pred)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Windows"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_sc.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_sc_df = pd.DataFrame(train_sc, columns=['Scaled'], index=train.index)\n",
"test_sc_df = pd.DataFrame(test_sc, columns=['Scaled'], index=test.index)\n",
"train_sc_df.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for s in range(1, 13):\n",
" train_sc_df['shift_{}'.format(s)] = train_sc_df['Scaled'].shift(s)\n",
" test_sc_df['shift_{}'.format(s)] = test_sc_df['Scaled'].shift(s)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_sc_df.head(13)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train = train_sc_df.dropna().drop('Scaled', axis=1)\n",
"y_train = train_sc_df.dropna()[['Scaled']]\n",
"\n",
"X_test = test_sc_df.dropna().drop('Scaled', axis=1)\n",
"y_test = test_sc_df.dropna()[['Scaled']]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train = X_train.values\n",
"X_test= X_test.values\n",
"\n",
"y_train = y_train.values\n",
"y_test = y_test.values"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Fully Connected on Windows"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"K.clear_session()\n",
"\n",
"model = Sequential()\n",
"model.add(Dense(12, input_dim=12, activation='relu'))\n",
"model.add(Dense(1))\n",
"model.compile(loss='mean_squared_error', optimizer='adam')\n",
"model.summary()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.fit(X_train, y_train, epochs=200,\n",
" batch_size=1, verbose=1, callbacks=[early_stop])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_pred = model.predict(X_test)\n",
"plt.plot(y_test)\n",
"plt.plot(y_pred)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### LSTM on Windows"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train_t = X_train.reshape(X_train.shape[0], 1, 12)\n",
"X_test_t = X_test.reshape(X_test.shape[0], 1, 12)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train_t.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"K.clear_session()\n",
"model = Sequential()\n",
"\n",
"model.add(LSTM(6, input_shape=(1, 12)))\n",
"\n",
"model.add(Dense(1))\n",
"\n",
"model.compile(loss='mean_squared_error', optimizer='adam')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.summary()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.fit(X_train_t, y_train, epochs=100,\n",
" batch_size=1, verbose=1, callbacks=[early_stop])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_pred = model.predict(X_test_t)\n",
"plt.plot(y_test)\n",
"plt.plot(y_pred)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise 1\n",
"\n",
"In the model above we reshaped the input shape to: `(num_samples, 1, 12)`, i.e. we treated a window of 12 months as a vector of 12 coordinates that we simultaneously passed to all the LSTM nodes. An alternative way to look at the problem is to reshape the input to `(num_samples, 12, 1)`. This means we consider each input window as a sequence of 12 values that we will pass in sequence to the LSTM. In principle this looks like a more accurate description of our situation. But does it yield better predictions? Let's check it.\n",
"\n",
"- Reshape `X_train` and `X_test` so that they represent a set of univariate sequences\n",
"- retrain the same LSTM(6) model, you'll have to adapt the `input_shape`\n",
"- check the performance of this new model, is it better at predicting the test data?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Exercise 2\n",
"\n",
"RNN models can be applied to images too. In general we can apply them to any data where there's a connnection between nearby units. Let's see how we can easily build a model that works with images.\n",
"\n",
"- Load the MNIST data, by now you should be able to do it blindfolded :)\n",
"- reshape it so that an image looks like a long sequence of pixels\n",
"- create a recurrent model and train it on the training data\n",
"- how does it perform compared to a fully connected? How does it compare to Convolutional Neural Networks?\n",
"\n",
"(feel free to run this exercise on a cloud GPU if it's too slow on your laptop)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

872
course/9 Improving performance.ipynb Normal file
View File

@@ -0,0 +1,872 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 9 Improving performance"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"%matplotlib inline\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Learning curves"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.datasets import load_digits"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"digits = load_digits()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X, y = digits.data, digits.target"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for i in range(8):\n",
" plt.subplot(1,8,i+1)\n",
" plt.imshow(X.reshape(-1, 8, 8)[i], cmap='gray')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.models import Sequential\n",
"from tensorflow.keras.layers import Dense\n",
"from tensorflow.keras.utils import to_categorical\n",
"import tensorflow.keras.backend as K\n",
"from tensorflow.keras.callbacks import EarlyStopping"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"K.clear_session()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = Sequential()\n",
"model.add(Dense(16, input_shape=(64,), activation='relu'))\n",
"model.add(Dense(10, activation='softmax'))\n",
"model.compile('adam', 'categorical_crossentropy', metrics=['accuracy'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# store the initial random weights\n",
"initial_weights = model.get_weights()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_cat = to_categorical(y, 10)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train, X_test, y_train, y_test = train_test_split(X, y_cat,\n",
" test_size=0.3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_sizes = (len(X_train) * np.linspace(0.1, 0.999, 4)).astype(int)\n",
"train_sizes"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_scores = []\n",
"test_scores = []\n",
"\n",
"for train_size in train_sizes:\n",
" X_train_frac, _, y_train_frac, _ = \\\n",
" train_test_split(X_train, y_train, train_size=train_size)\n",
" \n",
" # at each iteration reset the weights of the model\n",
" # to the initial random weights\n",
" model.set_weights(initial_weights)\n",
" \n",
" h = model.fit(X_train_frac, y_train_frac,\n",
" verbose=0,\n",
" epochs=300,\n",
" callbacks=[EarlyStopping(monitor='loss', patience=1)])\n",
"\n",
" r = model.evaluate(X_train_frac, y_train_frac, verbose=0)\n",
" train_scores.append(r[-1])\n",
" \n",
" e = model.evaluate(X_test, y_test, verbose=0)\n",
" test_scores.append(e[-1])\n",
" \n",
" print(\"Done size: \", train_size)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.plot(train_sizes, train_scores, 'o-', label=\"Training score\")\n",
"plt.plot(train_sizes, test_scores, 'o-', label=\"Test score\")\n",
"plt.legend(loc=\"best\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Batch Normalization"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.layers import BatchNormalization"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def repeated_training(X_train,\n",
" y_train,\n",
" X_test,\n",
" y_test,\n",
" units=512,\n",
" activation='sigmoid',\n",
" optimizer='sgd',\n",
" do_bn=False,\n",
" epochs=10,\n",
" repeats=3):\n",
" histories = []\n",
" \n",
" for repeat in range(repeats):\n",
" K.clear_session()\n",
"\n",
" model = Sequential()\n",
" \n",
" # first fully connected layer\n",
" model.add(Dense(units,\n",
" input_shape=X_train.shape[1:],\n",
" kernel_initializer='normal',\n",
" activation=activation))\n",
" if do_bn:\n",
" model.add(BatchNormalization())\n",
"\n",
" # second fully connected layer\n",
" model.add(Dense(units,\n",
" kernel_initializer='normal',\n",
" activation=activation))\n",
" if do_bn:\n",
" model.add(BatchNormalization())\n",
"\n",
" # third fully connected layer\n",
" model.add(Dense(units,\n",
" kernel_initializer='normal',\n",
" activation=activation))\n",
" if do_bn:\n",
" model.add(BatchNormalization())\n",
"\n",
" # output layer\n",
" model.add(Dense(10, activation='softmax'))\n",
" \n",
" model.compile(optimizer,\n",
" 'categorical_crossentropy',\n",
" metrics=['accuracy'])\n",
"\n",
" h = model.fit(X_train, y_train,\n",
" validation_data=(X_test, y_test),\n",
" epochs=epochs,\n",
" verbose=0)\n",
" histories.append([h.history['accuracy'], h.history['val_accuracy']])\n",
" print(repeat, end=' ')\n",
"\n",
" histories = np.array(histories)\n",
" \n",
" # calculate mean and standard deviation across repeats:\n",
" mean_acc = histories.mean(axis=0)\n",
" std_acc = histories.std(axis=0)\n",
" print()\n",
" \n",
" return mean_acc[0], std_acc[0], mean_acc[1], std_acc[1]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"mean_acc, std_acc, mean_acc_val, std_acc_val = \\\n",
" repeated_training(X_train, y_train, X_test, y_test, do_bn=False)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"mean_acc_bn, std_acc_bn, mean_acc_val_bn, std_acc_val_bn = \\\n",
" repeated_training(X_train, y_train, X_test, y_test, do_bn=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def plot_mean_std(m, s):\n",
" plt.plot(m)\n",
" plt.fill_between(range(len(m)), m-s, m+s, alpha=0.1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_mean_std(mean_acc, std_acc)\n",
"plot_mean_std(mean_acc_val, std_acc_val)\n",
"plot_mean_std(mean_acc_bn, std_acc_bn)\n",
"plot_mean_std(mean_acc_val_bn, std_acc_val_bn)\n",
"plt.ylim(0, 1.01)\n",
"plt.title(\"Batch Normalization Accuracy\")\n",
"plt.xlabel('Epochs')\n",
"plt.ylabel('Accuracy')\n",
"plt.legend(['Train', 'Test', 'Train with Batch Normalization', 'Test with Batch Normalization'], loc='best');\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Weight Regularization & Dropout"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.layers import Dropout"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = Sequential()\n",
"model.add(Dropout(0.2, input_shape=X_train.shape[1:]))\n",
"# first fully connected layer\n",
"model.add(Dense(512, kernel_initializer='normal',\n",
" kernel_regularizer='l2', activation='sigmoid'))\n",
"model.add(Dropout(0.4))\n",
"model.add(Dense(10, activation='softmax'))\n",
"\n",
"model.compile('sgd',\n",
" 'categorical_crossentropy',\n",
" metrics=['accuracy'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data augmentation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
"\n",
"generator = ImageDataGenerator(rescale = 1./255,\n",
" width_shift_range=0.1,\n",
" height_shift_range=0.1,\n",
" rotation_range = 20,\n",
" shear_range = 0.3,\n",
" zoom_range = 0.3,\n",
" horizontal_flip = True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train = generator.flow_from_directory('../data/generator',\n",
" target_size = (128, 128),\n",
" batch_size = 32,\n",
" class_mode = 'binary')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.figure(figsize=(12, 12))\n",
"for i in range(16):\n",
" img, label = train.next()\n",
" plt.subplot(4, 4, i+1)\n",
" plt.imshow(img[0])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Embeddings"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.layers import Embedding"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = Sequential()\n",
"model.add(Embedding(input_dim=100, output_dim=2))\n",
"model.compile(loss='binary_crossentropy',\n",
" optimizer='adam',\n",
" metrics=['accuracy'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.summary()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"emb = model.predict(np.array([[81, 1, 96, 79],\n",
" [17, 47, 69, 50],\n",
" [49, 3, 12, 88]]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"emb.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"emb"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sentiment prediction on movie Reviews"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.datasets import imdb"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"(X_train, y_train), (X_test, y_test) = imdb.load_data('/tmp/imdb.npz',\n",
" num_words=None,\n",
" skip_top=0,\n",
" maxlen=None,\n",
" start_char=1,\n",
" oov_char=2,\n",
" index_from=3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train[1]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"idx = imdb.get_word_index()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"max(idx.values())"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"idx"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rev_idx = {v+3:k for k,v in idx.items()}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rev_idx"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rev_idx[0] = 'padding_char'\n",
"rev_idx[1] = 'start_char'\n",
"rev_idx[2] = 'oov_char'\n",
"rev_idx[3] = 'unk_char'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rev_idx[3]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_train[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"example_review = ' '.join([rev_idx[word] for word in X_train[0]])\n",
"example_review"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"len(X_train[0])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"len(X_train[1])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"len(X_train[2])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"len(X_train[3])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
"from tensorflow.keras.layers import LSTM"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"maxlen = 100\n",
"\n",
"X_train_pad = pad_sequences(X_train, maxlen=maxlen)\n",
"X_test_pad = pad_sequences(X_test, maxlen=maxlen)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train_pad.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train_pad[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"max_features = max([max(x) for x in X_train_pad] + \n",
" [max(x) for x in X_test_pad]) + 1\n",
"max_features"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y_train"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = Sequential()\n",
"model.add(Embedding(max_features, 128))\n",
"model.add(LSTM(64, dropout=0.2, recurrent_dropout=0.2))\n",
"model.add(Dense(1, activation='sigmoid'))\n",
"\n",
"model.compile(loss='binary_crossentropy',\n",
" optimizer='adam',\n",
" metrics=['accuracy'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.fit(X_train_pad, y_train,\n",
" batch_size=32,\n",
" epochs=2,\n",
" validation_split=0.3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"score, acc = model.evaluate(X_test_pad, y_test)\n",
"print('Test score:', score)\n",
"print('Test accuracy:', acc)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise 1\n",
"\n",
"- Reload the IMDB data keeping only the first 20000 most common words\n",
"- pad the reviews to a shorter length (eg. 70 or 80), this time make sure you keep the first part of the review if it's longer than the maximum length\n",
"- re run the model (remember to set max_features correctly)\n",
"- does it train faster this time?\n",
"- do you get a better performance?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise 2\n",
"\n",
"- Reload the digits data as above\n",
"- define a function repeated_training_reg_dropout that adds regularization and dropout to a fully connected network\n",
"- compare the performance with/witouth dropout and regularization like we did for batch normalization\n",
"- do you get a better performance?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise 3\n",
"\n",
"This is a very long and complex exercise, that should give you an idea of a real world scenario. Feel free to look at the solution if you feel lost. Also, feel free to run this on Floyd with a GPU, in which case you don't need to download the data.\n",
"\n",
"If you are running this locally, download and unpack the male/female pictures from [here](https://www.dropbox.com/s/nov493om2jmh2gp/male_female.tgz?dl=0). These images and labels were obtained from [Crowdflower](https://www.crowdflower.com/data-for-everyone/).\n",
"\n",
"Your goal is to build an image classifier that will recognize the gender of a person from pictures.\n",
"\n",
"- Have a look at the directory structure and inspect a couple of pictures\n",
"- Design a model that will take a color image of size 64x64 as input and return a binary output (female=0/male=1)\n",
"- Feel free to introduce any regularization technique in your model (Dropout, Batch Normalization, Weight Regularization)\n",
"- Compile your model with an optimizer of your choice\n",
"- Using `ImageDataGenerator`, define a train generator that will augment your images with some geometric transformations. Feel free to choose the parameters that make sense to you.\n",
"- Define also a test generator, whose only purpose is to rescale the pixels by 1./255\n",
"- use the function `flow_from_directory` to generate batches from the train and test folders. Make sure you set the `target_size` to 64x64.\n",
"- Use the `model.fit_generator` function to fit the model on the batches generated from the ImageDataGenerator. Since you are streaming and augmenting the data in real time you will have to decide how many batches make an epoch and how many epochs you want to run\n",
"- Train your model (you should get to at least 85% accuracy)\n",
"- Once you are satisfied with your training, check a few of the misclassified pictures. Are those sensible errors?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}