Special Issue: Machine Learning for Engineering Design

Abstract

Modern machine learning (ML) techniques are transforming many disciplines ranging from transportation to healthcare by uncovering patterns in data, developing autonomous systems that mimic human abilities, and supporting human decision-making. Modern ML techniques, such as deep neural networks, are fueling the rapid developments in artificial intelligence. Engineering design researchers have increasingly used and developed ML techniques to support a wide range of activities from preference modeling to uncertainty quantification in high-dimensional design optimization problems. This special issue brings together fundamental scientific contributions across these areas.The special issue consists of 24 papers spread over two issues of the Journal of Mechanical Design. The papers use various ML techniques, including artificial neural networks, Gaussian processes, reinforcement learning, clustering techniques, and natural language processing. Based on their research objective, the papers can be broadly classified into four groups: (i) ML to support surrogate modeling, design exploration, and optimization, (ii) ML for design synthesis, (iii) ML for extracting human preferences and design strategies, and (iv) comparative studies of ML techniques and research platforms to help design researchers. The papers are summarized in Secs. 1–4. An analysis of the themes covered in the special issue and the potential opportunities for future research in ML for Engineering Design are presented in Sec. 5.In the paper titled Multifidelity Physics-Constrained Neural Network and Its Application in Materials Modeling, Liu and Yang address how to incorporate multifidelity, physics-based constraints into neural network predictions. The paper contributes two key insights. First, the paper extends existing Physics-Constraints Neural Network architectures by imposing a multifidelity constraint scheme wherein an auxiliary network minimizes discrepancies between low and high fidelity models—essentially learning how to correct the low-fidelity one. Second, it proposes an adaptive weighting scheme to control the convergence of individual losses among the different fidelities. They demonstrate the impact of these improvements on several fundamental multiscale material modeling challenges including two-dimensional heat transfer, phase transition, and dendritic growth problems. On these problems, the proposed multifidelity, physics-based constraints decrease the prediction error up to order of magnitude compared with networks without such constraints. This achieves comparable accuracy to that of direct numerical solutions of the underlying equations.Sarkar et al. present a multifidelity modeling and information-theoretic sequential sampling strategy for optimization in their paper titled Multifidelity and Multiscale Bayesian Framework for High-Dimensional Engineering Design and Calibration. The approach is based on modeling of the varied fidelity information sources via Gaussian processes, augmented with efficient active learning strategies that involve sequential selection of optimal points in a multiscale architecture. The strategy is demonstrated using the design optimization of a compressor rotor and calibration of a microstructure prediction model.In the paper titled A Case Study of Deep Reinforcement Learning for Engineering Design: Application to Microfluidic Devices for Flow Sculpting, Lee et al. address how to design micro-fluidic flow sculpting devices by overcoming some of the key weaknesses of evolutionary optimization-based methods, namely, poor sample efficiency and slow optimization convergence. The paper adapts deep reinforcement learning (DRL) techniques to the flow sculpting task and also studies the effectiveness of transfer learning on accelerating the design of target flow shapes. The paper demonstrates that DRL is able to match 90% of the target flow shapes using significantly fewer sculpting pillars than comparable GA models as well as provides a means to interpret the learned model (using Principal Components) that existing approaches to fluidic sculpting do not provide.Lynch et al., in their paper Machine Learning to Aid Tuning of Numerical Parameters in Topology Optimization, present an ML-based meta-learning framework to determine tuning parameters in topology optimization. The parameters are learned from similar optimization problems carried out in the past and adjusted for the problem at hand. This helps in avoiding costly trial-and-error involved in manual parameter tuning.In the paper Data-Driven Design Space Exploration and Exploitation for Design for Additive Manufacturing, Xiong et al. present a data-driven approach for design search and optimization at successive stages in the design process. They use Bayesian network classifier in the embodiment design stage and Gaussian process regression in the detailed design phase. The approach is illustrated in the paper through the design of a customized ankle brace design.Odonkor and Lewis apply data-driven design to the design of operational strategies of complex systems, specifically distributed energy resources. The paper is titled Data-Driven Design of Control Strategies for Distributed Energy Systems. The problem of maximizing arbitrage value is formulated as an optimization problem and solved using reinforcement learning. The approach is demonstrated for shared distributed energy resources in multi-building residential clusters.In Globally Approximate Gaussian Processes for Big Data With Application to Data-Driven Metamaterials Design by Bostanabad et al., a globally approximate Gaussian process (GAGP) is introduced for the purpose of handling large datasets. A GAGP is constructed by pooling several Gaussian processes using identical hyperparameters but built from different subsets of the training data. The predictive capability of GAGPs is shown to be at least as good as state-of-the-art supervised learning methods. It is demonstrated on the unit-cell design of metamaterials through inverse optimization.Liu et al. present a method for the design for crashworthiness involving categorical multimaterial structures in their paper titled Design for Crashworthiness of Categorical Multimaterial Structures Using Cluster Analysis and Bayesian Optimization. Following a topology optimization, the dimensionality of the problem is reduced through clustering followed by a Bayesian optimization to assign a given material to a specific cluster. The approach is applied to the maximization of absorbed energy of an S-rail.Garriga et al. propose a framework to assist the optimization of aircraft systems at the early design stages. The approach in their paper titled A Machine Learning Enabled Multifidelity Platform for the Integrated Design of Aircraft Systems is based on the screening of designs using clustering followed by an identification of the best candidate on a Pareto front. The framework enables the use of models of various fidelities and is demonstrated on a primary flight control system and a landing gear.In Synthesizing Designs With Interpart Dependencies Using Hierarchical Generative Adversarial Networks, Chen and Fuge present a method for synthesizing hierarchical designs with inter-part dependencies using generative models learned from examples. The method constructs multiple generative models using generative adversarial networks (GANs) while satisfying the dependencies through part dependency graphs. The paper lays the foundation for extending the use of generative models from creative individual parts to more realistic engineering systems.The objective in Evolving a Psycho-Physical Distance Metric for Generative Design Exploration of Diverse Shapes by Khan et al. is to incorporate humans’ psychological perceptions about design into the design exploration process. A psycho-physical distance metric is proposed that enables the augmentation of CAD designs based on feedback from users. Results reveal that the proposed method generates more distinct variations of CAD designs compared with a baseline Euclidean distance method.Oh et al. in their paper titled Deep Generative Design: Integration of Topology Optimization and Generative Models present a design framework for creating diverse aesthetic designs that are optimized for engineering performance. The framework integrates topology optimization and generative adversarial networks (GANs) to generate large numbers of design options from limited previous design data. The approach is validated using a 2D wheel design problem.Deshpande and Purwar, in their paper Computational Creativity Via Assisted Variational Synthesis of Mechanisms Using Deep Generative Models, present an approach for variational synthesis of mechanisms and an End-to-End synthesis pipeline that accepts raw, high-level input from users and provides them with distinct concept solutions. The approach is based on learning the probability distribution of linkage parameters and their interdependence to perform tasks such as input conditioning, imputation, and variational synthesis. The approach is a step in the direction of enhancing users’ computational creativity for engineering design.Stump et al. in their paper, Spatial Grammar-Based Recurrent Neural Network for Design Form and Behavior Optimization, present a method for simultaneous optimization of form and behavior through a combination of physics-based models and ML techniques. Specifically, they use character-Recurrent Neural Networks to embody spatial grammars and reinforcement learning to optimize the behavior. The design of a modular multi-hull sailing craft is used as a demonstration problem.Suryadi and Kim utilize machine-learning algorithms for customer choice modeling in their paper titled A Data-Driven Methodology to Construct Customer Choice Sets Using Online Data and Customer Reviews. They present an approach that utilizes publicly available online data and customer reviews from e-commerce websites to construct customer choice sets in the absence of both an actual choice set and customer sociodemographic data. The approach consists of clustering (i) products based on their attributes and (ii) customers based on their reviews, and constructing the choice-sets based on a sampling probability scenario that relies on product and customer clusters. The approach generates choice models with higher predictive ability than randomly constructed choice sets.In their paper Extracting Customer Perceptions of Product Sustainability From Online Reviews, El Dehaibi et al. seek to extract perceived sustainable design features from online reviews. Annotators from Amazon’s Mechanical Turk are used to annotate product reviews and develop a natural language processing model that predicts the positive/negative sentiment of sustainable phrases. The results reveal that the model is more efficient at predicting positive sentiment pertaining to sustainable product features compared with negative sentiments.Raina et al. take a step toward transfer learning from human designers to computational agents in their paper titled Transferring Design Strategies From Human To Computer and Across Design Problems. They present an approach where design strategies are represented using a probabilistic model that provides a general mechanism to transfer strategies from human designers to computational design agents and to generate new designs. The approach is illustrated using a configuration design problem.The goal in Learning to Design From Humans: Imitating Human Designers Through Deep Learning by Raina et al. is to teach computational agents to generate designs without the need for explicit information about objective or performance metrics. A deep learning model is proposed that learns from historical human data and identifies the important regions of a design space. The results reveal that the machine learning agent learns to create designs that are comparable to human-generated ones, despite not having the same explicit feedback that humans do to guide them through the design exploration process.He et al. address the challenge of mining large numbers of design ideas generated from the crowd in their paper titled Mining and Representing the Concept Space of Existing Ideas for Directed Ideation. The authors use natural language processing to extract keywords as elementary concepts and represent the concepts in a way that they can be recombined to generate new ideas.In the paper titled A Data-Driven Approach to Product Usage Context Identification From Online Customer Reviews, Suryadi and Kim use machine learning and natural language processing to identify and cluster usage contexts from a large volume of customer reviews. The methodology also captures sentiments toward a particular usage context in a sentence. The methodology enables designers to effectively use online product reviews by focusing on several specific reviews regarding particular usage contexts and potentially to identify market opportunities for new products that excel in specific usage contexts.Sharpe et al. illuminate differences between Supervised Learning algorithms in terms of how and where different algorithms may apply to different Engineering Design applications. Their paper titled A Comparative Evaluation of Supervised Machine Learning Classification Techniques for Engineering Design Applications does this by comparing four common supervised learning approaches—Support Vector Machines, Random Forests, Gaussian Näive Bayes, and shallow depth Neural Networks—across six example problems that demonstrate different facets or challenges classifiers may face within the engineering design. The results from the work are multifaceted with different algorithms performing better or worse under different conditions and performance measures. However, this leads to the general notion of strong problem dependence for the classifier choice and highlights the importance of understanding appropriate benchmark problems within the engineering design that can shed light on such issues in the future.The availability of data enables not just designers but also design researchers. Rahman et al., in their paper A Computer-Aided Design Based Research Platform for Design Thinking Studies, present a research platform to support data-driven design-thinking and decision-making research. Through the use of fine-grained design action data and unsupervised clustering methods in conjunction with design process models, the authors show how the platform enables data-driven research studies on designers’ sequential decision-making behaviors.In Design Repository Effectiveness for 3D Convolutional Neural Networks: Application to Additive Manufacturing, Williams et al. address the question of whether or not a data repository is useful for training effective ML tools. The authors experimentally test the effects of changes in CAD datasets on the precision and generalizability of trained convolutional neural networks (CNNs) for additive manufacturing applications. The study sheds light on how standardization of design repositories can influence the performance of ML tools.Cunningham et al. study the construction of a performance surrogate based on 3D point cloud representations. In their paper titled An Investigation of Surrogate Models for Efficient Performance-Based Decoding of 3D Point Clouds, a radial basis function (RBF) surrogate is used to link performance and cloud representation mapped onto a latent vector. The proposed RBF-based approach was found to be more efficient and accurate than traditional neural network-based approaches.In the call for proposals for this special issue, the guest editors posed three primary questions: How to effectively use ML for new design applications that are not well-supported by existing ML practice or tools?How to leverage the unique aspects of engineering design in creating new ML approaches?How to share benchmark problems or datasets that can measure ML progress in design?Looking back at the papers collectively within this Special Issue helps shed light on the areas that are receiving significant attention within the design research community and the areas where there are opportunities for future design research community a strong of using machine learning. techniques have used for data-driven techniques such as Gaussian process regression and neural networks have applications in learning complex between design and performance natural language processing used for mining customer and reinforcement learning is an part of control systems design. of these techniques are in the papers for the special issue Bostanabad et et Suryadi and Xiong et of the deep learning techniques such as convolutional neural networks (CNNs) and generative adversarial networks (GANs) are their way into engineering design research and and and et papers in this special issue address diverse applications including modeling, additive distributed energy systems, and topology optimization, synthesis, mechanism preference modeling, and learning from human of the areas where there is potential for ML techniques, but are not well represented in this special issue, modeling human design of market systems, of products with humans or the design for use of data from product usage to using data from of the product or data, or across different aspects of the are opportunities for design challenges in engineering such as and and that from the products and ML can also be used to support engineering design for supporting studies and the generalizability of research terms of understanding and of machine learning methods, the papers in the special issue on the the of multifidelity or multiple data or with the common surrogate modeling and physics-based constraints into ML and ML-based models of human preferences and of the papers in the special issue present surrogate modeling this an active of research for the past two the approaches are using ML techniques. A of papers issues that ML models in multifidelity and et The availability of multifidelity models is in engineering design. In some designers’ are to construct multiple and fidelities of models or of a system as to with it at the appropriate This is not that ML systems are to In this of do multiple or important to to this special issue and is an active of of the approaches on existing with changes that constraints. of the is that many ML systems are not used to many of the problems that need for engineering design. in et al., how the between form and behavior the generated or in and Lewis the between Control and Design these of between and within Engineering and existing ML approaches do not need to for papers the of design or constraints in a constraints by or and hierarchical or et Chen and and constraints and This is important the of many ML systems relies on their to but specific of into the This is where engineering design researchers are well to are opportunities for research in engineering with ML models, such as physics-based models from or via more system models, and studies a problem and as a In the can ML be used with a between design and A is learning from multiple of a or learning among design with multiple data structures and of the papers to this special issue used supervised or reinforcement learning where data or available via used Learning clustering and dimensionality in natural and have from unsupervised learning of using deep learning or and this be a for future work in design. there are opportunities for approaches to or for in uncertainty with or data. and calibration of ML models an papers on the of ML for optimization and specifically how to about generative models of two of approach to on the of the the approach Optimization as as by approaches that used or latent methods et al. and et or Optimization as such as that formulated the problem using reinforcement learning or inverse problem et are active and areas of in to for optimization that not in this special issue, such as direct inverse design papers how to best or behavior into an ML This is a are many opportunities for human information and strategies into ML models, and models of and computational or for behavior. This to approaches have used to models into the same be for of models, such as involving human or to ML models of human behavior designers or the design research community on in both and and this is within the special However, there are constraints that engineering design on of a notion that be to function and to that out as a of approaches to do not such aspects and this a that the Engineering Design community can This also for are the fundamental of designers as conditions and does this datasets have ML research and a common for performance. collectively the the among have collectively of of They have in and This special issue set out to to new datasets for engineering design papers in the special issue propose datasets or platforms optimization problems et 2D and 3D shapes et Chen and and manufacturing et while not datasets do use platforms such as to data or these papers within the Special there is a wide to be by future work that can useful datasets or for machine learning within engineering design. are some of the areas that papers in the Special Issue and have used ML but for good benchmark datasets datasets for human or including how designers or for more complex than optimization do not have the the model of for Engineering problems such as or that link or use multiple of design representations. datasets a CAD and a by the do not datasets have ML approaches in in the have in for and and datasets for approaches to specific in do not have a good of engineering problems and a given is to This is not unique to Engineering Design. How do or the and that it a realistic benchmark for Computer In some to be is the design community on of or the of where and design datasets apply and how their results to datasets and understanding of their or be for future of how to such models within Engineering Design. how and of ML models within this on the of or ML models influence the design of a How does this Design are some of the and the papers in this special issue the of research opportunities in this The papers in the special issue many points from future researchers and may set guest the for their in the and feedback to the Special to and for their help with the for the process of of this special issue, and for the