Minimizing Jensen-Shannon Divergence with constraints











up vote
0
down vote

favorite












I am trying to minimize the following function :



$$J(p) = JSD(p_u || p)$$
with constraints :
$$ int p = 1 $$
$$ p(x) geq hat{pi} p_p(x) $$
where $JSD$ is the Jensen Shannon Divergence, $p_u = pi p_p + (1-pi) p_n$, $hat{pi} > pi$, and $p_p$ and $p_n$ are probability distributions with disjoint supports.



I formulated the problem as a minimization problem with constraints to try to solve it with a Lagrangian, but I don't really know how (and if) this works on a functional space.



edit:
Let L be the Lagrangian: $$L(p,lambda,mu) = JSD(p_u||p)-lambda int p -1 - int_x mu(x) (hat{pi}p_p - p)(x)dx$$



Functional derivative yields:
$$frac{partial L}{partial p} = ln(frac{2p}{p+p_u})-lambda+mu p = 0$$
After a discussion on the value of $mu$ (0 or not), I show that as I suspected :



On $supp(p_p)$, $p(x) = hat{pi}p_p$



On $supp(p_n)$, $p(x) = (1-hat{pi})p_n$



Two questions remains :




  1. How to prove rigorously the necessary condition on the minimum (with the Lagrangian...) ?


  2. How to prove there is a global minimum ? (sufficient condition)











share|cite|improve this question




























    up vote
    0
    down vote

    favorite












    I am trying to minimize the following function :



    $$J(p) = JSD(p_u || p)$$
    with constraints :
    $$ int p = 1 $$
    $$ p(x) geq hat{pi} p_p(x) $$
    where $JSD$ is the Jensen Shannon Divergence, $p_u = pi p_p + (1-pi) p_n$, $hat{pi} > pi$, and $p_p$ and $p_n$ are probability distributions with disjoint supports.



    I formulated the problem as a minimization problem with constraints to try to solve it with a Lagrangian, but I don't really know how (and if) this works on a functional space.



    edit:
    Let L be the Lagrangian: $$L(p,lambda,mu) = JSD(p_u||p)-lambda int p -1 - int_x mu(x) (hat{pi}p_p - p)(x)dx$$



    Functional derivative yields:
    $$frac{partial L}{partial p} = ln(frac{2p}{p+p_u})-lambda+mu p = 0$$
    After a discussion on the value of $mu$ (0 or not), I show that as I suspected :



    On $supp(p_p)$, $p(x) = hat{pi}p_p$



    On $supp(p_n)$, $p(x) = (1-hat{pi})p_n$



    Two questions remains :




    1. How to prove rigorously the necessary condition on the minimum (with the Lagrangian...) ?


    2. How to prove there is a global minimum ? (sufficient condition)











    share|cite|improve this question


























      up vote
      0
      down vote

      favorite









      up vote
      0
      down vote

      favorite











      I am trying to minimize the following function :



      $$J(p) = JSD(p_u || p)$$
      with constraints :
      $$ int p = 1 $$
      $$ p(x) geq hat{pi} p_p(x) $$
      where $JSD$ is the Jensen Shannon Divergence, $p_u = pi p_p + (1-pi) p_n$, $hat{pi} > pi$, and $p_p$ and $p_n$ are probability distributions with disjoint supports.



      I formulated the problem as a minimization problem with constraints to try to solve it with a Lagrangian, but I don't really know how (and if) this works on a functional space.



      edit:
      Let L be the Lagrangian: $$L(p,lambda,mu) = JSD(p_u||p)-lambda int p -1 - int_x mu(x) (hat{pi}p_p - p)(x)dx$$



      Functional derivative yields:
      $$frac{partial L}{partial p} = ln(frac{2p}{p+p_u})-lambda+mu p = 0$$
      After a discussion on the value of $mu$ (0 or not), I show that as I suspected :



      On $supp(p_p)$, $p(x) = hat{pi}p_p$



      On $supp(p_n)$, $p(x) = (1-hat{pi})p_n$



      Two questions remains :




      1. How to prove rigorously the necessary condition on the minimum (with the Lagrangian...) ?


      2. How to prove there is a global minimum ? (sufficient condition)











      share|cite|improve this question















      I am trying to minimize the following function :



      $$J(p) = JSD(p_u || p)$$
      with constraints :
      $$ int p = 1 $$
      $$ p(x) geq hat{pi} p_p(x) $$
      where $JSD$ is the Jensen Shannon Divergence, $p_u = pi p_p + (1-pi) p_n$, $hat{pi} > pi$, and $p_p$ and $p_n$ are probability distributions with disjoint supports.



      I formulated the problem as a minimization problem with constraints to try to solve it with a Lagrangian, but I don't really know how (and if) this works on a functional space.



      edit:
      Let L be the Lagrangian: $$L(p,lambda,mu) = JSD(p_u||p)-lambda int p -1 - int_x mu(x) (hat{pi}p_p - p)(x)dx$$



      Functional derivative yields:
      $$frac{partial L}{partial p} = ln(frac{2p}{p+p_u})-lambda+mu p = 0$$
      After a discussion on the value of $mu$ (0 or not), I show that as I suspected :



      On $supp(p_p)$, $p(x) = hat{pi}p_p$



      On $supp(p_n)$, $p(x) = (1-hat{pi})p_n$



      Two questions remains :




      1. How to prove rigorously the necessary condition on the minimum (with the Lagrangian...) ?


      2. How to prove there is a global minimum ? (sufficient condition)








      functional-analysis optimization calculus-of-variations lagrange-multiplier






      share|cite|improve this question















      share|cite|improve this question













      share|cite|improve this question




      share|cite|improve this question








      edited Nov 16 at 12:36

























      asked Nov 7 at 11:40









      Mathieu

      12




      12



























          active

          oldest

          votes











          Your Answer





          StackExchange.ifUsing("editor", function () {
          return StackExchange.using("mathjaxEditing", function () {
          StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
          StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
          });
          });
          }, "mathjax-editing");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "69"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          noCode: true, onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














           

          draft saved


          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2988426%2fminimizing-jensen-shannon-divergence-with-constraints%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown






























          active

          oldest

          votes













          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes
















           

          draft saved


          draft discarded



















































           


          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2988426%2fminimizing-jensen-shannon-divergence-with-constraints%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          QoS: MAC-Priority for clients behind a repeater

          Ивакино (Тотемский район)

          Can't locate Autom4te/ChannelDefs.pm in @INC (when it definitely is there)