Feature Engineering

　　在机器学习应用中，特征工程扮演着重要的角色。数据和特征决定了机器学习算法的上限，而模型、算法的选择和优化只是不断逼近这个上限而已。特征工程 (Feature Engineering) 介于“数据”和“模型”之间，是利用数据的专业领域知识和现有数据，从源数据中抽取出来对预测结果有用的信息，用在机器学习算法上的过程。美国计算机科学家 Peter Norvig 有两句经典的名言：

“基于大量数据的简单模型胜过基于少量数据的复杂模型。”
“更多的数据胜过聪明的算法，而好的数据胜过多的数据。”

　　吴恩达更是说过“应用机器学习基本上就是特征工程”。对于工业界来说，大部分复杂模型的算法精进都是资深的数据科学家的任务，大部分人员的工作还是跑数据、map-reduce，hive SQL，数据仓库搬砖，做一些业务分析、数据清洗、特征工程（找特征）的工作。当然在实际工作中，特征工程旨在去除原始数据中的杂质和冗余，设计更高效的特征以刻画求解的问题与预测模型之间的关系。

　　特征工程面对的数据一般分为两大类：

结构化数据。
- 结构化数据可以看做关系型数据库的一张表，每列都有清晰的定义，包含了数值型、类别性两种基本类型。
- 每一行数据表示一个样本信息。
非结构化数据。
- 非结构化数据主要包括文本、图像、音频、视频数据，其包含的信息无法用一个简单的数值表示，也没有清晰的定义，并且每条数据的大小各不相同。

　　特征工程一般分成特征提取 (Feature Extraction) 和特征选择 (Feature Selection) 两个方面。

　　在特征工程中有一个比较棘手的问题是，如何保证训练集、验证集和测试集的输入一直？

　　为了不在训练中掺入测试数据，我们要求最好先单独对训练数据进行特征提取，其中包含归一化等操作，再对验证集或者测试集进行操作，但是问题是在验证集和测试集中对同一个特征做完特征提取后的结果维度可能不一样，这要怎么处理？

One could also perform analysis of residuals or log-odds (for linear model) to check for strong nonlinearities.
Create a feature which captures the frequency of the occurrence of each level of the categorical variable. For high cardinality, this helps a lot. One might use ratio/percentage of a particular level to all the levels present.
For every possible value of the variable, estimate the mean of the target variable; use the result as an engineered feature.
Encode a variable with the ratio of the target variable.
Take the two most important variables and throw in second order interactions between them and the rest of the variables - compare the resulting model to the original linear one
if you feel your solutions should be smooth, you can apply a radial basis function kernel . This is like applying a smoothing transform.
If you feel you need covariates , you can apply a polynomial kernel, or add the covariates explicitly
High cardinality features : convert to numeric by preprocessing: out-of-fold average two variable combinations
Additive transformation
difference relative to baseline
Multiplicative transformation : interactive effects
divisive : scaling/normalisation
thresholding numerical features to get boolean values
Cartesian Product Transformation
Feature crosses: cross product of all features – Consider a feature A, with two possible values {A1, A2}. Let B be a feature with possibilities {B1, B2}. Then, a feature-cross between A & B (let’s call it AB) would take one of the following values: {(A1, B1), (A1, B2), (A2, B1), (A2, B2)}. You can basically give these ‘combinations’ any names you like. Just remember that every combination denotes a synergy between the information contained by the corresponding values of A and B.
Normalization Transformation: – One of the implicit assumptions often made in machine learning algorithms (and somewhat explicitly in Naive Bayes) is that the the features follow a normal distribution. However, sometimes we may find that the features are not following a normal distribution but a log normal distribution instead. One of the common things to do in this situation is to take the log of the feature values (that exhibit log normal distribution) so that it exhibits a normal distribution.If the algorithm being used is making the implicit/explicit assumption of the features being normally distributed, then such a transformation of a log-normally distributed feature to a normally distributed feature can help improve the performance of that algorithm.
Quantile Binning Transformation
whitening the data
Windowing – If points are distributed in time axis, previous points in the same window are often very informative
Min-max normalization : does not necessarily preserve order
sigmoid / tanh / log transformations
Handling zeros distinctly – potentially important for Count based features
Decorrelate / transform variables
Reframe Numerical Quantities
Map infrequent categorical variables to a new/separate category.

28.Sequentially apply a list of transforms.

One Hot Encoding
Target rate encoding

Hash Trick Multivariate:

PCA
MODEL STACKING
compressed sensing

34..guess the average” or “guess the average segmented by variable X”

Projection : new basis

Hack projection:

Perform clustering and use distance between points to the cluster center as a feature PCA/SVD – Useful technique to analyze the interrelationships between variables and perform dimensionality reduction with minimum loss of information (find the axis through the data with highest variance / repeat with the next orthogonal axis and so on , until you run out of data or dimensions; Each axis acts a new feature) 36.Sparse coding – choose basis : evaluate the basis based on how well you can use it to reconstruct the input and how sparse it is take some sort of gradient step to improve that evaluation

efficient sparse coding algorithms deep auto encoders 37 :Random forest: train bunch of decision trees :use each leaf as a feature

References