| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
| Name | Name | Last commit date | ||
|---|---|---|---|---|
We used Python and R to build our solution. Particularly, for Python we used Keras for neural networks and sklearn for logistic regression. In R we mainly used XGBoost for Gradient boosting algorithms and the base function optim was used for model ensembling. In the Chapters 2 and 3 model descriptions and instructions on how to build them are given. In Chapter 4 we show the ensembling approach of our best no leak submission. Chapter 5 gives the details on the best leak submission. Only the models in Chapter 5 use information about the leak.
Duplicate rows were removed and all files were saved as .RData files to reload faster. The different approaches for xgboost and keras are explained below. Out of fold predictions and test predictions were made for each model.
Two different models were trained for devices with and without events. To make the predictions for devices without events we trained on all devices. The only features were phone brand, device model and event flag (has event: yes/no).
We only trained on the subset of devices with events to predict IDs with events. We can subdivide the features in three groups: (1) features related to categories, (2) to apps and to (3) event information. For each device
Only categories with >25 non-zero entries and the 1000 most frequent apps were used for training. Those thresholds achieved the best cv score. Further, phone brand and device model were included. In total, we have 1423 columns.
The same kind of 2-stage model was fitted for devices with/without events, just with different parameters.
The clue is that we do not need the gender in the test set to make the prediction, because we have to make two predictions for each ID in the test set anyway. First, we assume that the user is female and predict, then we assume that the user is male and predict again. Using the definition of conditional probability, we combine the predictions for gender and age groups to get the probability for each group:
$P(A_i, F) = P(A_i| F) P(F)$ for $i = 1,\dots,6$ and $P(A_i, M) = P(A_i| M) P(M)$ for $i = 1,\dots,6$,
where $A_i$ denote the age groups 1 to 6, and $F$ and $M$ denote female and male, respectively.
The models keras_1 and keras_3 were trained on the whole training set, just with different parameters. For the keras_2 model, we used the same splitting schema as for the xgboost model and trained a 2-stage model for devices without events. However, splitting between devices with/without events did not improve the score.
All features were one-hot encoded. The features were event flag (has event: yes/no), category features (has category: yes/no), app features (has app: yes/no), phone brand and device model. No feature selection was performed. In total, the training set had 15772 columns.
Make sure that all training files are saved in danijel/input. Open a R console and set the working directory to danijel. At first, run "create_rdata.R" and afterwards "for_py_create_train_test.R". To build the xgb model, run all scripts in the folder danijel/xgb. Of importance, the script "xgb_merge.R" has to be run at last. To build all python models, open a command line, change directory to danijel and run all scripts in the folders danijel/keras_1, danijel/keras_2 and danijel/keras_3. Of importance, the R script "keras_2_merge.R" has again to be run after all other scripts in danijel/keras_2. Train and test predictions were all saved in danijel/preds.
Our best "single" model was a combination of a simple logistic regression for devices without events and a neural network model for devices with events. It was trained using one hot encoding.
After removing features with zero variance the number of features was 18.150 for training devices with events and 3.408 for training devices without events
This model was fit using 10 folds. Each fold model predicted on the test set and the predictions were averaged to get the final prediction.
The model parameters for this keras model were:
model = Sequential() model.add(Dropout(0.4, input_shape=(num_columns,))) model.add(Dense(75)) model.add(PReLU()) model.add(Dropout(0.30)) model.add(Dense(50, init='normal', activation='tanh')) model.add(PReLU()) model.add(Dropout(0.20)) model.add(Dense(12, init='normal', activation='softmax')) # Compile model model.compile(loss='categorical_crossentropy', optimizer='adadelta', metrics=['accuracy'])
The number of epochs was 20 and the samples per epoch was the size of the train set (for the corresponding fold)
This Keras/Logistic regression model achieved a score of 2.23452 on public Leaderboard and 2.23822 on private Leaderboard. On our final blending a new version of this model was included but for folds 5, and also the predictions were averaged by running the same script several times with different seeds
In our best submission blend we also noticed a small improvement by adding some additional models with lower performances. Those "second order" models included:
Make sure that all training files are saved in matias/input, including the fold_10.csv and fold_5.csv files (already provided) For R files, open a R console and set the working directory to matias. For Python files, open Idle or your favorite IDE (I used Pycharm community) and set the project directory to matias. Then run the scripts in the following order:
######Preprocessing data
######Generating model predictions in R 3. cv5_shityGlmnet_1hot_d1-d31.R # Train an elastic net model 4. cv5_shityXgb_1hot_d1-d31_bagged.R # Train an xgboost model 5. cv5_shityXgb_1hot_d1-d31_bagged_seed2.R # Train the same xgboost model with different seed 6. cv5_shityXgb_2_1hot_d1-d31_bagged.R # Train an xgboost model 7. cv5_shityXgbLinear_1hot_d1-d31_bagged.R # Train an xgboost model using gblinear
######Generating model predictions in Python 8. keras_cv5_2_bagging_split.py # Train a keras model 9. keras_cv10_with_bags5.py # Train a keras model 10. keras_cv10_with_bags5_wEvents_AllData2.py # Train a keras model 11. keras_cv10_with_bags5_wEvents_AllData2_seed1.py # Train the same keras with different seed 12. keras_cv10_with_bags5_wEvents_AllData2_seed2.py # Train the same keras with different seed
Train and test predictions were all saved in matias/preds.
######Blending predictions using the optim package in R At this point you should have all the prediction files in danijel/preds and matias/preds. Then the following script needs to run:
The final blending will be generated in the folder: no_leak_sub. This folder lies at the same level than matias and danijel folders. This script will optimize the weights for the previous 15 predictions generated. The script performs two independent optimizations: One for for devices with events and another one for devices with events.
We observed that the row leak only improved the score of devices without events. For devices with events, we got slightly worse scores. Therefore, the predictions of the leak solution are the same as the no leak solution for devices with events. Additionally, Keras performed worse than Xgboost with the leak. So we only trained the same Xgboost model for no event devices, just with the normalized row as an additional feature included. This gave us 2.16835 on public LB and 2.17584 on private LB. The final score of 2.14223 on public LB and 2.14949 on private LB was achieved with a matching script where we tried to exactly match IDs in the train set with IDs in the test set using the normalized rows.
Open a R console and set the working directory to the root folder. The scripts have to be run in the following order: 1) 'create_rdata_leak.R' 2) 'xgb_test_leak.R' 3) 'match_ids_leak.R'. The final submission will be saved in the folder leak_sub.
#####Hardware and Software
######Danijel's R scripts were run using the following environment:
######Danijel's Python scripts were run using the following environment:
######Matias' R scripts were run using the following environment:
######Matias' Python scripts were run using the following environment:
| Back | FazBrowse Home | New Git URL |