Indian Institute of Technology Palakkad&nbsp;(IIT Palakkad or IIT PKD) is a public autonomous&nbsp;engineering&nbsp;and&nbsp;research&nbsp;institute located in&nbsp;Palakkad,&nbsp;Kerala. It was&nbsp;one of the five new&nbsp;IITs&nbsp;proposed in the&nbsp;2014 Union budget of India. The campus was inaugurated on 3 August 2015. The institute has 94 faculty, 978 students, and 63 non-teaching staffs.

IT Palakkad currently offers four-year Bachelor of Technology (B.Tech) and two years MS programmes in 8 different engineering disciplines including&nbsp;Computer Science, Electrical, Mechanical, Civil among others.

&nbsp;

IIT Palakkad

World University root node

Department of Mathematics

arXiv e-prints, arXiv: 2104.

Hybrid and Generalized Bayesian Cramér-Rao Inequalities via Information Geometry

We extend projection theorems concerning Hellinger and Jones et al. divergences to the continuous case. These projection theorems reduce certain estimation problems on generalized exponential models to linear problems. We introduce the notion of regularity for generalized exponential models and show that the projection theorems in this case are similar to the ones in discrete and canonical case. We also apply these ideas to solve certain estimation problems concerning Student and Cauchy distributions.

Fulltext

Journal of Multivariate Analysis

Projection theorems and estimating equations for power-law models

We examine the role of information geometry in the context of classical Cram{\'{e}}r–Rao (CR) type inequalities. In particular, we focus on Eguchi's theory of obtaining dualistic geometric structures from a divergence function and then applying Amari–Nagoaka's theory to obtain a CR type inequality. The classical deterministic CR inequality is derived from Kullback–Leibler (KL) divergence. We show that this framework could be generalized to other CR type inequalities through four examples: $\alpha$-version of CR inequality, generalized CR inequality, Bayesian CR inequality and Bayesian $\alpha$-CR inequality. These are obtained from, respectively, I$\alpha$-divergence (or relative $\alpha$-entropy), generalized Csisz{\'{a}}r divergence, Bayesian KL divergence and Bayesian I$\alpha$-divergence.

Handbook of Statistics

Information geometry and classical Cram{\'{e}}r–Rao-type inequalities

The relative $\alpha$-entropy is the R{\'{e}}nyi analog of relative entropy and arises prominently in information-theoretic problems. Recent information geometric investigations on this quantity have enabled the generalization of the Cram{\'{e}}r-Rao inequality, which provides a lower bound for the variance of an estimator of an escort of the underlying parametric probability distribution. However, this framework remains unexamined in the Bayesian framework. In this paper, we propose a general Riemannian metric based on relative $\alpha$-entropy to obtain a generalized Bayesian Cram{\'{e}}r-Rao inequality. This establishes a lower bound for the variance of an unbiased estimator for the $\alpha$-escort distribution starting from an unbiased estimator for the underlying distribution. We show that in the limiting case when the entropy order approaches unity, this framework reduces to the conventional Bayesian Cram{\'{e}}r-Rao inequality. Further, in the absence of priors, the same framework yields the deterministic Cram{\'{e}}r-Rao inequality.

2020 54th Annual Conference on Information Sciences and Systems, CISS 2020

Generalized Bayesian Cram{\'{e}}r-Rao Inequality via Information Geometry of Relative $\alpha$-Entropy

Information Geometry

Cram{\'{e}}r–Rao lower bounds arising from generalized Csisz{\'{a}}r divergences

This paper focuses on test procedures under corrupted data.We assume that the observations Zi are mismeasured, due to the presence of measurement errors. Thus, instead of Zi for i = 1, . . ., n, we observe Xi = Zi + √ $\delta$Vi, with an unknown parameter $\delta$ and an unobservable random variable Vi. It is assumed that the random variables Zi are i.i.d., as are the Xi and the Vi. The test procedure aims at deciding between two simple hyptheses pertaining to the density of the variable Zi, namely f0 and g0. In this setting, the density of the Vi is supposed to be known. The procedure which we propose aggregates likelihood ratios for a collection of values of d. A new definition of least-favorable hypotheses for the aggregate family of tests is presented and a relation with the Kullback-Leibler divergence between the sets (f$\delta$)$\delta$ and (g$\delta$)$\delta$ is presented. Finite-sample lower bounds for the power of these tests are presented, both through analytical inequalities and through simulation under the least-favorable hypotheses. Since no optimality holds for the aggregation of likelihood ratio tests, a similar procedure is proposed, replacing the individual likelihood ratio by some divergence based test statistics. It is shown and discussed that the resulting aggregated test may perform better than the aggregate likelihood ratio procedure.

Entropy

Composite tests under corrupted data

In [12], it was shown that a generalized maximum likelihood estimation problem on a (canonical) \alpha -power-law model (\mathbb{M}^{(\alpha)} -family) can be solved by solving a system of linear equations. This was due to an orthogonality relationship between the \mathbb{M}^{(\alpha)} -family and a linear family with respect to the relative \alpha -entropy (or the \mathscr{I}-{\alpha} -divergence). Relative \alpha -entropy is a generalization of the usual relative entropy (or the Kullback-Leibler divergence). \mathbb{M}^{(\alpha)} -family is a generalization of the usual exponential family. In this paper, we first generalize the \mathbb{M}^{(\alpha)}- family including the multivariate, continuous case and show that the Student-t distributions fall in this family. We then extend the above stated result of [12] to the general \mathbb{M}^{(\alpha)} -family. Finally we apply this result to the Student-t distribution and find generalized estimators for its parameters.

IEEE International Symposium on Information Theory - Proceedings

Generalized Estimating Equation for the Student-t Distributions

Information geometry describes a framework where probability densities can be viewed as differential geometry structures. This approach has shown that the geometry in the space of probability distributions that are parameterized by their covariance matrices is linked to the fundamental concepts of estimation theory. In particular, prior work proposes a Riemannian metric - the distance between the parameterized probability distributions - that is equivalent to the Fisher Information Matrix and helpful in obtaining the deterministic Cram{\'{e}}r-Rao lower bound (CRLB). Recent work in this framework has led to establishing links with several practical applications. However, classical CRLB is useful only for unbiased estimators and inaccurately predicts the mean square error in low signal-to-noise (SNR) scenarios. In this paper, we propose a general Riemannian metric that, at once, is used to obtain both Bayesian CRLB and deterministic CRLB along with their vector parameter extensions. We also extend our results to the Barankin bound, thereby enhancing their applicability to low SNR situations.

Information Geometric Approach to Bayesian Lower Error Bounds

Projection theorems of divergences enable us to find reverse projection of a divergence on a specific statistical model as a forward projection of the divergence on a different but rather "simpler" statistical model, which, in turn, results in solving a system of linear equations. Reverse projection of divergences are closely related to various estimation methods such as the maximum likelihood estimation or its variants in robust statistics. We consider projection theorems of three parametric families of divergences that are widely used in robust statistics, namely the R\'enyi divergences (or the Cressie-Reed power divergences), density power divergences and the relative $\alpha$-entropy (or the logarithmic density power divergences). We explore these projection theorems from the usual likelihood maximization approach and from the principle of sufficiency. In particular, we show the equivalence of solving the estimation problems by the projection theorems of the respective divergences and by directly solving the corresponding estimating equations. We also derive the projection theorem for the density power divergences.

arXiv

Projection theorems of divergences and likelihood maximization methods

This paper studies forward and reverse projections for the Renyi divergence of order $\alpha$ ∈ (0, ∞) on $\alpha$-convex sets. The forward projection on such a set is motivated by some works of Tsallis et al. in statistical physics and the reverse projection is motivated by robust statistics. In a recent work, van Erven and Harremoes proved a Pythagorean inequality for Renyi divergences on $\alpha$-convex sets under the assumption that the forward projection exists. Continuing this study, a sufficient condition for the existence of a forward projection is proved for probability measures on a general alphabet. For $\alpha$ ∈ (1, ∞), the proof relies on a new Apollonius theorem for the Hellinger divergence and for $\alpha$ ∈ (0,1), the proof relies on the Banach-Alaoglu theorem from the functional analysis. Further projection results are then obtained in the finite alphabet setting. These include a projection theorem on a specific $\alpha$-convex set, which is termed an $\alpha$-linear family, generalizing a result by Csiszar to $\alpha$ ≠ 1. The solution to this problem yields a parametric family of probability measures, which turns out to be an extension of the exponential family and it is termed an $\alpha$-exponential family. An orthogonality relationship between the $\alpha$-exponential and $\alpha$-linear families is established and it is used to turn the reverse projection on an $\alpha$-exponential family into a forward projection on an $\alpha$-linear family. This paper also proves a convergence result of an iterative procedure used to calculate the forward projection on an intersection of a finite number of $\alpha$-linear families.

IEEE Transactions on Information Theory

Projection Theorems for the Renyi Divergence on $\alpha$-Convex Sets

In part I of this two-part work, certain minimization problems based on a parametric family of relative entropies (denoted ℐ α ) were studied. Such minimizers were called forward ℐ α -projections. Here, a complementary class of minimization problems leading to the so-called reverse ℐ α -projections are studied. Reverse ℐ α -projections, particularly on log-convex or power-law families, are of interest in robust estimation problems (α &gt; 1) and in constrained compression settings (α &lt;; 1). Orthogonality of the power-law family with an associated linear family is first established and is then exploited to turn a reverse ℐ α -projection into a forward ℐ α -projection. The transformed problem is a simpler quasi-convex minimization subject to linear constraints.

Minimization Problems Based on Relative α -Entropy II: Reverse Projection

We study minimization of a parametric family of relative entropies, termed relative $\alpha$-entropies (denoted I$\alpha$(P,Q)). These arise as redundancies under mismatched compression when cumulants of compressed lengths are considered instead of expected compressed lengths. These parametric relative entropies are a generalization of the usual relative entropy (Kullback-Leibler divergence). Just like relative entropy, these relative $\alpha$-entropies behave like squared Euclidean distance and satisfy the Pythagorean property. Minimization of I$\alpha$(P,Q) over the first argument on a set of probability distributions that constitutes a linear family is studied. Such a minimization generalizes the maximum R{\'{e}}nyi or Tsallis entropy principle. The minimizing probability distribution (termed I$\alpha$-projection) for a linear family is shown to have a power-law.

2015 21st National Conference on Communications, NCC 2015

Relative $\alpha$-entropy minimizers subject to linear statistical constraints

Minimization problems with respect to a one-parameter family of generalized relative entropies are studied. These relative entropies, which we term relative $\alpha$-entropies (denoted I$\alpha$), arise as redundancies under mismatched compression when cumulants of compressed lengths are considered instead of expected compressed lengths. These parametric relative entropies are a generalization of the usual relative entropy (Kullback-Leibler divergence). Just like relative entropy, these relative $\alpha$-entropies behave like squared Euclidean distance and satisfy the Pythagorean property. Minimizers of these relative $\alpha$-entropies on closed and convex sets are shown to exist. Such minimizations generalize the maximum R{\'{e}}nyi or Tsallis entropy principle. The minimizing probability distribution (termed forward I$\alpha$-projection) for a linear family is shown to obey a power-law. Other results in connection with statistical inference, namely subspace transitivity and iterated projections, are also established. In a companion paper, a related minimization problem of interest in robust statistics that leads to a reverse I$\alpha$-projection is studied.