A Heuristic Fast Gradient Descent Method for Unimodal Optimization
George Anescu *
Power Plant Engineering Faculty, Polytechnic University of Bucharest, Romania.
*Author to whom correspondence should be addressed.
Abstract
The known gradient descent optimization methods applied to convex functions are using the gradient's magnitude in order to adaptively determine the current step size. The paper is presenting a new heuristic fast gradient descent (HFGD) approach, which uses the change in gradient's direction in order to adaptively determine the current step size. The new approach can be applied to solve classes of unimodal functions more general than the convex functions (e.g., quasi-convex functions), or as a local optimization method in multimodal optimization. Testing conducted on a testbed of 16 test functions showed an overall much better eciency and an overall better success rate of the proposed HFGD method when compared to other three known first order gradient descent methods.
Keywords: Optimization, unimodal/multimodal functions, convex optimization, quasi-convex optimization, golden ratio, fibonacci sequence, first-order optimization algorithms, Heuristic Fast Gradient Descent (HFGD), Backtracking Line Search (BLS), Accelerated Gradient Descent (AGD), Fast Iterative Shrinkage/Thresholding Algorithm (FISTA).