Energy landscape for large average submatrix detection problems in Gaussian random matrices

Research output: Contribution to journalArticle

Abstract

The problem of finding large average submatrices of a real-valued matrix arises in the exploratory analysis of data from a variety of disciplines, ranging from genomics to social sciences. In this paper we provide a detailed asymptotic analysis of large average submatrices of an n× n Gaussian random matrix. The first part of the paper addresses global maxima. For fixed k we identify the average and the joint distribution of the k× k submatrix having largest average value. As a dual result, we establish that the size of the largest square sub-matrix with average bigger than a fixed positive constant is, with high probability, equal to one of two consecutive integers that depend on the threshold and the matrix dimension n. The second part of the paper addresses local maxima. Specifically we consider submatrices with dominant row and column sums that arise as the local optima of iterative search procedures for large average submatrices. For fixed k, we identify the limiting average value and joint distribution of a k× k submatrix conditioned to be a local maxima. In order to understand the density of such local optima and explain the quick convergence of such iterative procedures, we analyze the number Ln(k) of local maxima, beginning with exact asymptotic expressions for the mean and fluctuation behavior of Ln(k). For fixed k, the mean of Ln(k) is Θ (nk/ (log n) ( k - 1 ) / 2) while the standard deviation is Θ(n2k2/(k+1)/(logn)k2/(k+1)). Our principal result is a Gaussian central limit theorem for Ln(k) that is based on a new variant of Stein’s method.

LanguageEnglish (US)
Pages919-983
Number of pages65
JournalProbability Theory and Related Fields
Volume168
Issue number3-4
DOIs
StatePublished - Aug 1 2017

Fingerprint

Energy Landscape
Random Matrices
Joint Distribution
Exploratory Analysis
Value Distribution
Social Sciences
Iterative Procedure
Energy
Asymptotic Analysis
Central limit theorem
Standard deviation
Genomics
Consecutive
Limiting
Fluctuations
Integer

Keywords

  • Central limit theorem
  • Energy landscape
  • Extreme value theory
  • Stein’s method

ASJC Scopus subject areas

  • Analysis
  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Cite this

Energy landscape for large average submatrix detection problems in Gaussian random matrices. / Bhamidi, Shankar; Dey, Partha S.; Nobel, Andrew B.

In: Probability Theory and Related Fields, Vol. 168, No. 3-4, 01.08.2017, p. 919-983.

Research output: Contribution to journalArticle

@article{22bf221fa2d04b71a7bd90a745340a72,
title = "Energy landscape for large average submatrix detection problems in Gaussian random matrices",
abstract = "The problem of finding large average submatrices of a real-valued matrix arises in the exploratory analysis of data from a variety of disciplines, ranging from genomics to social sciences. In this paper we provide a detailed asymptotic analysis of large average submatrices of an n× n Gaussian random matrix. The first part of the paper addresses global maxima. For fixed k we identify the average and the joint distribution of the k× k submatrix having largest average value. As a dual result, we establish that the size of the largest square sub-matrix with average bigger than a fixed positive constant is, with high probability, equal to one of two consecutive integers that depend on the threshold and the matrix dimension n. The second part of the paper addresses local maxima. Specifically we consider submatrices with dominant row and column sums that arise as the local optima of iterative search procedures for large average submatrices. For fixed k, we identify the limiting average value and joint distribution of a k× k submatrix conditioned to be a local maxima. In order to understand the density of such local optima and explain the quick convergence of such iterative procedures, we analyze the number Ln(k) of local maxima, beginning with exact asymptotic expressions for the mean and fluctuation behavior of Ln(k). For fixed k, the mean of Ln(k) is Θ (nk/ (log n) ( k - 1 ) / 2) while the standard deviation is Θ(n2k2/(k+1)/(logn)k2/(k+1)). Our principal result is a Gaussian central limit theorem for Ln(k) that is based on a new variant of Stein’s method.",
keywords = "Central limit theorem, Energy landscape, Extreme value theory, Stein’s method",
author = "Shankar Bhamidi and Dey, {Partha S.} and Nobel, {Andrew B.}",
year = "2017",
month = "8",
day = "1",
doi = "10.1007/s00440-017-0766-0",
language = "English (US)",
volume = "168",
pages = "919--983",
journal = "Probability Theory and Related Fields",
issn = "0178-8051",
publisher = "Springer New York",
number = "3-4",

}

TY - JOUR

T1 - Energy landscape for large average submatrix detection problems in Gaussian random matrices

AU - Bhamidi,Shankar

AU - Dey,Partha S.

AU - Nobel,Andrew B.

PY - 2017/8/1

Y1 - 2017/8/1

N2 - The problem of finding large average submatrices of a real-valued matrix arises in the exploratory analysis of data from a variety of disciplines, ranging from genomics to social sciences. In this paper we provide a detailed asymptotic analysis of large average submatrices of an n× n Gaussian random matrix. The first part of the paper addresses global maxima. For fixed k we identify the average and the joint distribution of the k× k submatrix having largest average value. As a dual result, we establish that the size of the largest square sub-matrix with average bigger than a fixed positive constant is, with high probability, equal to one of two consecutive integers that depend on the threshold and the matrix dimension n. The second part of the paper addresses local maxima. Specifically we consider submatrices with dominant row and column sums that arise as the local optima of iterative search procedures for large average submatrices. For fixed k, we identify the limiting average value and joint distribution of a k× k submatrix conditioned to be a local maxima. In order to understand the density of such local optima and explain the quick convergence of such iterative procedures, we analyze the number Ln(k) of local maxima, beginning with exact asymptotic expressions for the mean and fluctuation behavior of Ln(k). For fixed k, the mean of Ln(k) is Θ (nk/ (log n) ( k - 1 ) / 2) while the standard deviation is Θ(n2k2/(k+1)/(logn)k2/(k+1)). Our principal result is a Gaussian central limit theorem for Ln(k) that is based on a new variant of Stein’s method.

AB - The problem of finding large average submatrices of a real-valued matrix arises in the exploratory analysis of data from a variety of disciplines, ranging from genomics to social sciences. In this paper we provide a detailed asymptotic analysis of large average submatrices of an n× n Gaussian random matrix. The first part of the paper addresses global maxima. For fixed k we identify the average and the joint distribution of the k× k submatrix having largest average value. As a dual result, we establish that the size of the largest square sub-matrix with average bigger than a fixed positive constant is, with high probability, equal to one of two consecutive integers that depend on the threshold and the matrix dimension n. The second part of the paper addresses local maxima. Specifically we consider submatrices with dominant row and column sums that arise as the local optima of iterative search procedures for large average submatrices. For fixed k, we identify the limiting average value and joint distribution of a k× k submatrix conditioned to be a local maxima. In order to understand the density of such local optima and explain the quick convergence of such iterative procedures, we analyze the number Ln(k) of local maxima, beginning with exact asymptotic expressions for the mean and fluctuation behavior of Ln(k). For fixed k, the mean of Ln(k) is Θ (nk/ (log n) ( k - 1 ) / 2) while the standard deviation is Θ(n2k2/(k+1)/(logn)k2/(k+1)). Our principal result is a Gaussian central limit theorem for Ln(k) that is based on a new variant of Stein’s method.

KW - Central limit theorem

KW - Energy landscape

KW - Extreme value theory

KW - Stein’s method

UR - http://www.scopus.com/inward/record.url?scp=85014692118&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85014692118&partnerID=8YFLogxK

U2 - 10.1007/s00440-017-0766-0

DO - 10.1007/s00440-017-0766-0

M3 - Article

VL - 168

SP - 919

EP - 983

JO - Probability Theory and Related Fields

T2 - Probability Theory and Related Fields

JF - Probability Theory and Related Fields

SN - 0178-8051

IS - 3-4

ER -