Random Search (Random Search) is what, an article to see and understand

Definition of random search

Random Search (Random Search) is a hyperparametric optimization method that finds the optimal configuration by randomly sampling candidate points in the parameter space. Random Search uses a probabilistic sampling strategy to explore the parameter space, based on a key realization that the performance of most machine learning models is sensitive to only a few hyperparameters. Randomized search can cover a wider range of parameters within a fixed computational budget and has a higher probability of finding superior solutions. In practice, it is necessary to first define a search distribution for each hyperparameter, such as a uniform distribution, a log-uniform distribution, or a specific probability distribution, and then randomly sample parameter combinations from these distributions for model training and evaluation. The number of random samples is usually predefined and determined based on the available computational resources. Due to the introduction of randomness, multiple runs of the same configuration may produce different results, but this randomness happens to help the algorithm jump out of the local optimal region. Both theoretical analysis and empirical validation show that in most cases, random search is more efficient than grid search, and the advantage is especially obvious in high-dimensional parameter spaces.

The Core Idea of Randomized Search

• Probabilistic exploration mechanism: Randomly sampling parameter combinations based on probability distributions to avoid computational waste from systematic traversal. This method focuses more on exploring breadth rather than local fine search.
• The Principle of Dimensional Efficiency: In high-dimensional parameter spaces, random sampling is more likely to reach performance-critical regions than grid sampling. Most model performance depends on only a small number of key parameters.
• Calculate budget optimization: Maximize search efficiency within limited computational resources, prioritizing the discovery of potential regions rather than exhausting all possibilities. Each randomized trial provides new information value.
• Going beyond local optimality: Randomness helps the algorithm to avoid falling into the local optimality trap and increases the chance of discovering the global optimum or near-global optimum.
• Simple and Effective Philosophy: Achieve satisfactory optimization results through a simple stochastic mechanism without relying on complex heuristic rules. Simplicity brings convenience in practical applications.

Workflow of randomized search

• Parameter space definition: Determine the hyperparameters to be optimized and the range or distribution of their values. Continuous parameters define upper and lower bounds, discrete parameters enumerate possible values.
• Search Distribution Settings: Specify the type of sampling distribution for each parameter, e.g., uniform, normal, or log-uniform. The distribution choice affects the search efficiency.
• Determination of the number of samples: Set the total number of random samples according to the computational budget. It is usually necessary to balance the breadth and depth of the search, and it is recommended that at least a few dozen samples be taken.
• Random sampling loop: Randomly draw parameter combinations from a specified distribution, train the model and evaluate performance. Record the parameters and results for each trial.
• Results Analysis Options: Compare all test results and select the parameter configuration with the best performance. A further fine search in the optimal region can be performed.

Advantageous features of random search

• High-dimensional spatial efficiency: excels in high-dimensional parameter spaces and avoids the dimensional catastrophe problem. Random sampling is not affected by exponential growth in the number of parameters.
• Simplicity of implementation: The algorithm is logical and clear, and the code is simple to implement without complex mathematical derivation. Various programming languages can be easily realized.
• Computing Resource Friendly: Flexible control of computational cost, interrupt at any time and obtain the current optimal solution. Suitable for resource-constrained scenarios.
• Parallelization facilitation: Individual randomized trials are independent of each other and naturally support parallel computing. Distributed computing resources can be fully utilized.
• Explore the breadth of priorities: Prioritize the exploration of broad regions in unknown problems to avoid prematurely falling into localized regions. Suitable for scenarios where the problem characteristics are unknown.

Scenarios for randomized search

• hyperparameter tuning: Machine learning model hyperparameter optimization, especially deep learning network parameter tuning. Models with high computational cost are particularly suitable.
• Algorithm Configuration Optimization: Optimize the internal parameter configuration of an algorithm to improve its performance on a particular problem. Such as optimizer parameters, regularization parameters, etc.
• Resource-constrained environments: Quickly obtain usable solutions when computing resources or time are limited. Obtain satisfactory results faster than systematic searches.
• preliminary phase of exploration: Rapid understanding of parameter influence patterns in the study of new problems. Provide direction guidance for subsequent fine optimization.
• Multi-peak problem optimization: Increase the probability of finding the global optimum in problems where multiple local optima exist. Randomness helps to cross local barriers.

Parameterization of random search

• Sampling times setting: Typically set up for 50-200 randomized trials, depending on the parameter space dimension and computational budget. Important parameters may increase the sampling density.
• Distribution type selection: Uniform distributions are commonly used for continuous parameters and logarithmic uniform distributions are recommended for scale parameters. Categorical parameters use uniform categorical distribution.
• Parameter range determination: Set reasonable parameter ranges based on domain knowledge, too wide reduces efficiency, too narrow may miss the optimal solution. The range can be set in segments.
• Randomized seed management: Fixed random seeds ensure reproducible results, and changing seeds checks the stability of the results. Different seeds may find different local optima.
• Early Stop Strategy Design: Set performance thresholds or improve stall conditions to terminate hopeless experiments early. Save computational resources for more promising searches.

Practical Tips for Randomized Search

• parameter space transformation: Different sampling strategies are used for parameters of varying importance, with increased sampling density for key parameters. A priori knowledge guides sampling distribution design.
• Analysis of results records: Record the parameters and performance of each test in detail and analyze the pattern of relationship between parameters and performance. Accumulate experience for subsequent optimization.
• Gradual refinement strategy: Coarse search over a wide area, followed by a fine search in promising areas. Multi-level searches balance breadth and depth.
• parallelization: Conduct multiple trials simultaneously utilizing multi-core CPUs or distributed clusters. Dramatically reduces total search time.
• Visualization Aids: Plot parameter versus performance to visualize and understand parameter impact. Helps to adjust search strategy and scope.

Improvements to randomized search

• Adaptive Random Search: Dynamically adjust the sampling distribution based on preliminary results, concentrating on high-performing regions. Improved targeting of searches.
• Hybrid Search Strategy: Combine random search with other optimization methods, e.g., random search to determine the general range before local fine search. Utilizing the advantages of each.
• Intelligent Initialization: Initialize the search distribution based on historical experiments or domain knowledge to avoid complete randomness. Accelerate convergence to good regions.
• Multi-fidelity optimization: Start with a simple model or a small amount of data for a quick evaluation, and then use the full evaluation for promising candidates. Layered evaluation saves computational resources.
• Meta-learning guide: Learning parameter distribution patterns from similar problems guides the search distribution setup for new problems. Transfer learning to improve search efficiency.

Limitations of Randomized Search

• Convergence guarantee missing: There is no guarantee that a globally optimal solution will be found, and the results are somewhat random. Multiple runs may yield different results.
• Localized improvement difficulties: Difficult to fine-tune when close to optimal solution, random perturbations may miss the exact optimum. Not suitable for fine optimization in the final stage.
• Parameter correlation ignored: Sampling each parameter independently does not capture interactions between parameters. May miss parameter combinations that synergize well.
• High performance fluctuations: The results are less stable between different runs and need to be run several times to get the optimum. Increase additional computational cost.

Comparison of randomized search with other methods

• Comparison with Grid Search: Stochastic search usually finds better solutions for the same computational budget. The advantage is more pronounced in higher dimensional spaces, avoiding dimensionality catastrophe.
• Comparison with Bayesian optimization: Random search is simpler and easier to implement, Bayesian optimization of samples is more efficient but has high computational overhead. Both have comparable performance for small budgets.
• Comparison with genetic algorithms: Random search is simpler and more straightforward, genetic algorithms may find better solutions through evolutionary mechanisms but require more parameter tuning.
• Scenario Differences: Randomized search is suitable for initial exploration and simple problems, complex problems may require more advanced methods. Choose the appropriate method according to the characteristics of the problem.