Gradient Descent: Cost Function












2














I'm trying to implement the gradient descent method for the problem of minimising the following function:



$$f(x) = frac{1}{2}(x-m)^{T}A(x-m)-sumlimits_{i=1}^nlogleft(x_i^{2}right),$$



where $x in R^n$ is a vector; $m in R^n$ is a fixed vector; and $A$ is a fixed positive definite matrix.



The only applications of gradient descent I have come across is for linear regression! So, as a starting point for helping me to solve this, I'd like to know in what situations this cost function would be applied. Does anyone out there recognise it?






gradient-descent







share|cite|improve this question
























  • Your objective function is a special case of a quadratic cost function with a regularization term that uses the $log$ of the coefficients. $L_1$ or $L_2$ regularization cost functions are more typical, but you can impose a gentler penalty with a $log$ cost function.
    – Aditya Dua
    Nov 12 at 22:25






  • 1




    Thanks @AdityaDua. This response has been useful to me
    – Barton
    Nov 14 at 21:52












  • @AdityaDua do you know what the role of the matrix A would be in this cost function?
    – Barton
    Nov 17 at 17:48












  • I'd suggest reading up on "generalised least squares". The matrix A allows you to deal with correlated errors and unequal variances (Heteroscedasticity).
    – Aditya Dua
    Nov 18 at 3:09
















2














I'm trying to implement the gradient descent method for the problem of minimising the following function:



$$f(x) = frac{1}{2}(x-m)^{T}A(x-m)-sumlimits_{i=1}^nlogleft(x_i^{2}right),$$



where $x in R^n$ is a vector; $m in R^n$ is a fixed vector; and $A$ is a fixed positive definite matrix.



The only applications of gradient descent I have come across is for linear regression! So, as a starting point for helping me to solve this, I'd like to know in what situations this cost function would be applied. Does anyone out there recognise it?






gradient-descent







share|cite|improve this question
























  • Your objective function is a special case of a quadratic cost function with a regularization term that uses the $log$ of the coefficients. $L_1$ or $L_2$ regularization cost functions are more typical, but you can impose a gentler penalty with a $log$ cost function.
    – Aditya Dua
    Nov 12 at 22:25






  • 1




    Thanks @AdityaDua. This response has been useful to me
    – Barton
    Nov 14 at 21:52












  • @AdityaDua do you know what the role of the matrix A would be in this cost function?
    – Barton
    Nov 17 at 17:48












  • I'd suggest reading up on "generalised least squares". The matrix A allows you to deal with correlated errors and unequal variances (Heteroscedasticity).
    – Aditya Dua
    Nov 18 at 3:09














2












2








2







I'm trying to implement the gradient descent method for the problem of minimising the following function:



$$f(x) = frac{1}{2}(x-m)^{T}A(x-m)-sumlimits_{i=1}^nlogleft(x_i^{2}right),$$



where $x in R^n$ is a vector; $m in R^n$ is a fixed vector; and $A$ is a fixed positive definite matrix.



The only applications of gradient descent I have come across is for linear regression! So, as a starting point for helping me to solve this, I'd like to know in what situations this cost function would be applied. Does anyone out there recognise it?






gradient-descent







share|cite|improve this question















I'm trying to implement the gradient descent method for the problem of minimising the following function:



$$f(x) = frac{1}{2}(x-m)^{T}A(x-m)-sumlimits_{i=1}^nlogleft(x_i^{2}right),$$



where $x in R^n$ is a vector; $m in R^n$ is a fixed vector; and $A$ is a fixed positive definite matrix.



The only applications of gradient descent I have come across is for linear regression! So, as a starting point for helping me to solve this, I'd like to know in what situations this cost function would be applied. Does anyone out there recognise it?






gradient-descent




gradient-descent






share|cite|improve this question















share|cite|improve this question













share|cite|improve this question




share|cite|improve this question








edited Nov 13 at 13:31









Davide Giraudo

125k16150259




125k16150259










asked Nov 12 at 20:13









Barton

133




133












  • Your objective function is a special case of a quadratic cost function with a regularization term that uses the $log$ of the coefficients. $L_1$ or $L_2$ regularization cost functions are more typical, but you can impose a gentler penalty with a $log$ cost function.
    – Aditya Dua
    Nov 12 at 22:25






  • 1




    Thanks @AdityaDua. This response has been useful to me
    – Barton
    Nov 14 at 21:52












  • @AdityaDua do you know what the role of the matrix A would be in this cost function?
    – Barton
    Nov 17 at 17:48












  • I'd suggest reading up on "generalised least squares". The matrix A allows you to deal with correlated errors and unequal variances (Heteroscedasticity).
    – Aditya Dua
    Nov 18 at 3:09


















  • Your objective function is a special case of a quadratic cost function with a regularization term that uses the $log$ of the coefficients. $L_1$ or $L_2$ regularization cost functions are more typical, but you can impose a gentler penalty with a $log$ cost function.
    – Aditya Dua
    Nov 12 at 22:25






  • 1




    Thanks @AdityaDua. This response has been useful to me
    – Barton
    Nov 14 at 21:52












  • @AdityaDua do you know what the role of the matrix A would be in this cost function?
    – Barton
    Nov 17 at 17:48












  • I'd suggest reading up on "generalised least squares". The matrix A allows you to deal with correlated errors and unequal variances (Heteroscedasticity).
    – Aditya Dua
    Nov 18 at 3:09
















Your objective function is a special case of a quadratic cost function with a regularization term that uses the $log$ of the coefficients. $L_1$ or $L_2$ regularization cost functions are more typical, but you can impose a gentler penalty with a $log$ cost function.
– Aditya Dua
Nov 12 at 22:25




Your objective function is a special case of a quadratic cost function with a regularization term that uses the $log$ of the coefficients. $L_1$ or $L_2$ regularization cost functions are more typical, but you can impose a gentler penalty with a $log$ cost function.
– Aditya Dua
Nov 12 at 22:25




1




1




Thanks @AdityaDua. This response has been useful to me
– Barton
Nov 14 at 21:52






Thanks @AdityaDua. This response has been useful to me
– Barton
Nov 14 at 21:52














@AdityaDua do you know what the role of the matrix A would be in this cost function?
– Barton
Nov 17 at 17:48






@AdityaDua do you know what the role of the matrix A would be in this cost function?
– Barton
Nov 17 at 17:48














I'd suggest reading up on "generalised least squares". The matrix A allows you to deal with correlated errors and unequal variances (Heteroscedasticity).
– Aditya Dua
Nov 18 at 3:09




I'd suggest reading up on "generalised least squares". The matrix A allows you to deal with correlated errors and unequal variances (Heteroscedasticity).
– Aditya Dua
Nov 18 at 3:09










1 Answer
1






active

oldest

votes


















0














From the question, we have that f is a convex quadratic function



$$f(x) = frac{1}{2}(x-m)^{mathrm T} mathrm A (x-m)-sumlimits_{i=1}^nlogleft(x_i^{2}right),$$



where $x in R^n$ is a vector; $m in R^n$ is a fixed vector, and $A$ is a fixed positive definite matrix (symmetric and positive semidefinite).



$$nabla f ( x) = frac{mathrm T+1}{2} mathrm A (x-m)^{mathrm T}-sumlimits_{i=1}^nleft(frac{2x}{x_i^{2}ln(10)}right)$$



Using gradient descent with step $mu$,



$$ x_{k+1} = x_k - mu nabla f ( x_k)$$



Choose $mu$ such that $f(x_k) < f(x_{k+1}) $,then do a loop until we find $x^{*}: f(x^{*}_k) - f(x^{*}_{k+1}) sim 0$



This is a general way to take gradient descent for a convex quadratic function in n-dimensional space. Hope it is helpful.






share|cite|improve this answer























    Your Answer





    StackExchange.ifUsing("editor", function () {
    return StackExchange.using("mathjaxEditing", function () {
    StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
    StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
    });
    });
    }, "mathjax-editing");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "69"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    noCode: true, onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2995797%2fgradient-descent-cost-function%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    From the question, we have that f is a convex quadratic function



    $$f(x) = frac{1}{2}(x-m)^{mathrm T} mathrm A (x-m)-sumlimits_{i=1}^nlogleft(x_i^{2}right),$$



    where $x in R^n$ is a vector; $m in R^n$ is a fixed vector, and $A$ is a fixed positive definite matrix (symmetric and positive semidefinite).



    $$nabla f ( x) = frac{mathrm T+1}{2} mathrm A (x-m)^{mathrm T}-sumlimits_{i=1}^nleft(frac{2x}{x_i^{2}ln(10)}right)$$



    Using gradient descent with step $mu$,



    $$ x_{k+1} = x_k - mu nabla f ( x_k)$$



    Choose $mu$ such that $f(x_k) < f(x_{k+1}) $,then do a loop until we find $x^{*}: f(x^{*}_k) - f(x^{*}_{k+1}) sim 0$



    This is a general way to take gradient descent for a convex quadratic function in n-dimensional space. Hope it is helpful.






    share|cite|improve this answer




























      0














      From the question, we have that f is a convex quadratic function



      $$f(x) = frac{1}{2}(x-m)^{mathrm T} mathrm A (x-m)-sumlimits_{i=1}^nlogleft(x_i^{2}right),$$



      where $x in R^n$ is a vector; $m in R^n$ is a fixed vector, and $A$ is a fixed positive definite matrix (symmetric and positive semidefinite).



      $$nabla f ( x) = frac{mathrm T+1}{2} mathrm A (x-m)^{mathrm T}-sumlimits_{i=1}^nleft(frac{2x}{x_i^{2}ln(10)}right)$$



      Using gradient descent with step $mu$,



      $$ x_{k+1} = x_k - mu nabla f ( x_k)$$



      Choose $mu$ such that $f(x_k) < f(x_{k+1}) $,then do a loop until we find $x^{*}: f(x^{*}_k) - f(x^{*}_{k+1}) sim 0$



      This is a general way to take gradient descent for a convex quadratic function in n-dimensional space. Hope it is helpful.






      share|cite|improve this answer


























        0












        0








        0






        From the question, we have that f is a convex quadratic function



        $$f(x) = frac{1}{2}(x-m)^{mathrm T} mathrm A (x-m)-sumlimits_{i=1}^nlogleft(x_i^{2}right),$$



        where $x in R^n$ is a vector; $m in R^n$ is a fixed vector, and $A$ is a fixed positive definite matrix (symmetric and positive semidefinite).



        $$nabla f ( x) = frac{mathrm T+1}{2} mathrm A (x-m)^{mathrm T}-sumlimits_{i=1}^nleft(frac{2x}{x_i^{2}ln(10)}right)$$



        Using gradient descent with step $mu$,



        $$ x_{k+1} = x_k - mu nabla f ( x_k)$$



        Choose $mu$ such that $f(x_k) < f(x_{k+1}) $,then do a loop until we find $x^{*}: f(x^{*}_k) - f(x^{*}_{k+1}) sim 0$



        This is a general way to take gradient descent for a convex quadratic function in n-dimensional space. Hope it is helpful.






        share|cite|improve this answer














        From the question, we have that f is a convex quadratic function



        $$f(x) = frac{1}{2}(x-m)^{mathrm T} mathrm A (x-m)-sumlimits_{i=1}^nlogleft(x_i^{2}right),$$



        where $x in R^n$ is a vector; $m in R^n$ is a fixed vector, and $A$ is a fixed positive definite matrix (symmetric and positive semidefinite).



        $$nabla f ( x) = frac{mathrm T+1}{2} mathrm A (x-m)^{mathrm T}-sumlimits_{i=1}^nleft(frac{2x}{x_i^{2}ln(10)}right)$$



        Using gradient descent with step $mu$,



        $$ x_{k+1} = x_k - mu nabla f ( x_k)$$



        Choose $mu$ such that $f(x_k) < f(x_{k+1}) $,then do a loop until we find $x^{*}: f(x^{*}_k) - f(x^{*}_{k+1}) sim 0$



        This is a general way to take gradient descent for a convex quadratic function in n-dimensional space. Hope it is helpful.







        share|cite|improve this answer














        share|cite|improve this answer



        share|cite|improve this answer








        edited Nov 18 at 16:45

























        answered Nov 18 at 16:39









        AnNg

        355




        355






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Mathematics Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid …



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            Use MathJax to format equations. MathJax reference.


            To learn more, see our tips on writing great answers.





            Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


            Please pay close attention to the following guidance:


            • Please be sure to answer the question. Provide details and share your research!

            But avoid …



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2995797%2fgradient-descent-cost-function%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            -

            Popular posts from this blog

            AnyDesk - Fatal Program Failure

            Image
            up vote 0 down vote favorite Good day all. I have a window that pops up saying Fatal Program Failure (AnyDesk). This is what I have done so far: Uninstalled AnyDesk and reinstalled it. ( The window still appeared ) I disabled AnyDesk start up in Msconfig and the window seems to have disapeared. Lastly I made sure that AnyDesk worked by remoting into it, and it worked. Today the same window appeared. How do I get rid of this window? remote-desktop share | improve this question edited 10 hours ago Tomasz Jakub Rup 552 3 13 ...
            Read more

            How to calibrate 16:9 built-in touch-screen to a 4:3 resolution?

            Image
            up vote 0 down vote favorite I need help with an issue we're facing in a project where we are moving from a Windows-based POS environment to one running on CentOS7. The problem consists of a number of factors, please find the details below: The supplier of our POS application will only deliver an application that is hard-coded to run in a 1024:768 4:3 format. The hardware we are using is a POS unit from HP that has a built-in 16:9 touch-screen. The employees using this solution is not supposed to see nor have access to anything but the POS application. Our sales organization demands that touch is an available feature when going live with the new POS environment. Going further into this, allow me to explain what we have encountered on the way trying to achieve the goal of having a touch-screen: We've simply change...
            Read more

            QoS: MAC-Priority for clients behind a repeater

            Image
            1 1 I've setup a wireless network using a Linksys-Router (DD-WRT) as an AP and a TP-Link repeater for range extension. To manage bandwith for specific users, I've setup QoS-Rules on the AP (DD-WRT) using MAC-Priority: However the AP shows only the Repeater's MAC and the MACs of users inside the AP-Range. The MAC addresses of users behind the Repeater are not listed : The purple MAC is not listed in the AP-Clientlist. Its related to a device connected via the TP-Link repeater. Q: How can I setup MAC-Priority for repeater-clients? Do I simply add the purple MAC in AP-MAC priority list although it's not listed as a client? I want to setup specific priority instead of giving one priority for all clients behind the repeater by adding the repeater's MAC in the MAC priority list in the AP as ...
            Read more