| FazBrowse Wikipedia Viewer | Main Page | | |
| Viewing: https://en.wikipedia.org/wiki/Multitask_optimization |
Multi-task learning (MTL) is a subfield of machine learning in which multiple learning tasks are solved at the same time, while exploiting commonalities and differences across tasks. This can result in improved learning efficiency and prediction accuracy for the task-specific models, when compared to training the models separately. Inherently, multi-task learning is a multi-objective optimization problem having trade-offs between different tasks. Early versions of MTL were called "hints".
In a widely cited 1997 paper, Rich Caruana gave the following characterization:
Multitask Learning is an approach to inductive transfer that improves generalization by using the domain information contained in the training signals of related tasks as an inductive bias. It does this by learning tasks in parallel while using a shared representation; what is learned for each task can help other tasks be learned better.
In the classification context, MTL aims to improve the performance of multiple classification tasks by learning them jointly. One example is a spam-filter, which can be treated as distinct but related classification tasks across different users. To make this more concrete, consider that different people have different distributions of features which distinguish spam emails from legitimate ones, for example an English speaker may find that all emails in Russian are spam, not so for Russian speakers. Yet there is a definite commonality in this classification task across users, for example one common feature might be text related to money transfer. Solving each user's spam classification problem jointly via MTL can let the solutions inform each other and improve performance.𝒮t={(xit,yit)}i=1nt, with t∈1,...,T. Let
| f:𝒳→𝒴T having components f:𝒳→ℝT is a symmetric matrix-valued function 1 has the form:
f(x)=\\sum _{t=1}^T \\sum _{i=1}^{n_t} \\Gamma(x,x_i^t)c_i^t </math>"},"3":{"wt":"{{EquationRef|3}}"}},"i":0}}]}'>Γ induces both the representation of the feature space and structures the output across tasks. A natural simplification is to choose a separable kernel, which factors into separate kernels on the input space X and on the tasks fs is given by Γ(xi,xj)=k(xi,xj)A, where T×T matrix. Henceforth denote Γ is a current field of research.
For the separable case, the representation theorem is reduced to K is the C is the 1 can be rewritten as \\min _{C\\in \\mathbb{R}^{n\\times T}} V(Y,KCA) + \\lambda tr(KCAC^{\\top})</math>"},"3":{"wt":"{{EquationRef|P}}"}},"i":0}}]}'>V is a (weighted) average of KCA. (The weight is zero if ‖f‖ℋ2=⟨∑i=1nk(⋅,xi)Aci,∑j=1nk(⋅,xj)Acj⟩ℋ=∑i,j=1n⟨k(⋅,xi)Aci,k(⋅,xj)Acj⟩ℋ(bilinearity)=∑i,j=1n⟨k(xi,xj)Aci,cj⟩ℝT(reproducing property)=∑i,j=1nk(xi,xj)ci⊤Acj=tr(KCAC⊤)Known task structureeditTask structure representationseditThere are three largely equivalent ways to represent task structure: through a regularizer; through an output metric, and through an output mapping. ||f||^2_\\mathcal{H} = \\sum_{s,t=1}^T A^\\dagger _{t,s} \\langle f_s, f_t \\rangle _{\\mathcal H_k} </math>, where <math>A^\\dagger _{t,s} </math> is the <math> t,s </math> element of the pseudoinverse of <math> A </math>, and <math>\\mathcal H_k </math> is the RKHS based on the scalar kernel <math> k </math>, and <math display=\"inline\"> f_t(x)=\\sum _{i=1} ^n k(x,x_i)A_t^\\top c_i </math>. This formulation shows that <math>A^\\dagger _{t,s} </math> controls the weight of the penalty associated with <math display=\"inline\">\\langle f_s, f_t \\rangle _{\\mathcal H_k} </math>. (Note that <math display=\"inline\">\\langle f_s, f_t \\rangle _{\\mathcal H_k} </math> arises from <math display=\"inline\">||f_t||_{\\mathcal H_k} = \\langle f_t, f_t \\rangle _{\\mathcal H_k} </math>.)\n\n{{Proof|\n<math>\\begin{align}\n\\|f\\|^2_\\mathcal{H} &= \\left\\langle \\sum _{i=1} ^n \\gamma ((x_i,t_i),\\cdot )c_i^{t_i}, \\sum _{j=1} ^n \\gamma ((x_j,t_j), \\cdot )c_j^{t_j}\\right\\rangle_{\\mathcal H } \\\\\n&=\\sum _{i,j=1} ^n c_i^{t_i} c_j^{t_j} \\gamma ((x_i,t_i),(x_j,t_j)) \\\\\n&=\\sum _{i,j=1} ^n \\sum _{s,t=1} ^T c_i^{t} c_j^{s} k(x_i,x_j)A_{s,t} \\\\\n&=\\sum _{i,j=1} ^n k(x_i,x_j) \\langle c_i, A c_j\\rangle_{\\mathbb R^T} \\\\\n&=\\sum _{i,j=1} ^n k(x_i,x_j) \\langle c_i, A A^\\dagger A c_j\\rangle_{\\mathbb R^T} \\\\\n&=\\sum _{i,j=1} ^n k(x_i,x_j) \\langle Ac_i, A^\\dagger A c_j\\rangle_{\\mathbb R^T} \\\\\n&=\\sum _{i,j=1} ^n \\sum _{s,t=1} ^T (Ac_i)^t (A c_j)^s k(x_i,x_j) A^\\dagger_{s,t} \\\\\n&= \\sum _{s,t=1} ^T A^\\dagger_{s,t} \\langle \\sum _{i=1} ^n k(x_i,\\cdot )(Ac_i)^t, \\sum _{j=1} ^n k(x_j,\\cdot )(A c_j)^s \\rangle _{\\mathcal H_k} \\\\\n&= \\sum _{s,t=1} ^T A^\\dagger_{s,t} \\langle f_t, f_s \\rangle _{\\mathcal H_k} \n\\end{align}</math>\n}}"}},"i":0}}]}'>Regularizer—With the separable kernel, it can be shown (below) that t,s element of the pseudoinverse of k, and ⟨fs,ft⟩ℋk. (Note that ‖f‖ℋ2=⟨∑i=1nγ((xi,ti),⋅)citi,∑j=1nγ((xj,tj),⋅)cjtj⟩ℋ=∑i,j=1nciticjtjγ((xi,ti),(xj,tj))=∑i,j=1n∑s,t=1Tcitcjsk(xi,xj)As,t=∑i,j=1nk(xi,xj)⟨ci,Acj⟩ℝT=∑i,j=1nk(xi,xj)⟨ci,AA†Acj⟩ℝT=∑i,j=1nk(xi,xj)⟨Aci,A†Acj⟩ℝT=∑i,j=1n∑s,t=1T(Aci)t(Acj)sk(xi,xj)As,t†=∑s,t=1TAs,t†⟨∑i=1nk(xi,⋅)(Aci)t,∑j=1nk(xj,⋅)(Acj)s⟩ℋk=∑s,t=1TAs,t†⟨ft,fs⟩ℋk Learning tasks together with their structureeditLearning problem
|