From this {analysis|evaluation}, we see that {smart|sensible|good} mirrors have {either|both} {basic|primary|fundamental} functionalities or are too {specialized|specialised}. Here, we {propose|suggest} a {general|common|basic} {purpose|objective|function} {smart|sensible|good} mirror {that incorporates|that comes with|that includes} a heterogeneous set of functionalities {that can|that may} {help|assist} {users|customers} {in their|of their} {everyday|on {a regular|a daily|an everyday} basis} lives. The mirror is designed {to include|to incorporate} {both|each} {common|widespread|frequent} functionalities {as {well|properly|nicely} as|in addition to} {advanced|superior} functionalities {specifically|particularly} {targeted|focused} for monitoring the well-being of the {user|consumer|person}.
These functionalities {are often|are sometimes} {found in|present in} medically {targeted|focused} {smart|sensible|good} mirrors. Moreover, {a suitable|an appropriate|an acceptable} authentication protocol {should be|ought to be|must be} {considered|thought-about|thought of} {to protect|to guard} the {privacy|privateness} of the {user|consumer|person} {and provide|and supply} {access|entry} to the {sensible|smart|wise} {information|info|data}. As far as {we are|we're} {aware|conscious}, no prior work on {smart|sensible|good} mirrors {tackle|deal with|sort out} all these {issues|points} in a {satisfactory|passable} {way|method|means}. Our {smart|sensible|good} mirror {offers|provides|presents} a dynamic interface that {displays|shows} not {only|solely} {the current|the present} {data|knowledge|information}, {but also|but in addition|but additionally} {information|info|data} inferred from the {analysis|evaluation} of {stored|saved} long-term {data|knowledge|information}. This {is especially|is particularly|is very} {important|essential|necessary} {in the|within the} case of the user's {emotions|feelings}. From the {comparison|comparability} {of the current|of the present} and {past|previous} emotional states of the {user|consumer|person}, the mirror can {suggest|recommend|counsel} {ways|methods} to mitigate {negative|adverse|unfavorable} {emotions|feelings}.
Table 1 and Table 2 summarize {the main|the primary|the principle} functionalities and {technologies|applied sciences} of the {smart|sensible|good} mirror {in the|within the} literature, {and those|and people} commercially {available|out there|obtainable}. With respect to {the general|the overall|the final} {purpose|objective|function} {smart|sensible|good} mirrors, {we can|we will|we are {able|in a position|ready} to} see {that most|that the majority|that {almost|virtually|nearly} all} of them share {the same|the identical} {basic|primary|fundamental} functionalities. They are {mostly|principally|largely} designed to {support|help|assist} the {user|consumer|person} in managing and {collecting|amassing|accumulating} {information|info|data}, and to ease the {control|management} {home|house|residence} {devices|units|gadgets}.
The {interaction|interplay} {is mostly|is usually|is generally} {based|based mostly|primarily based} on voice {commands|instructions} and no authentication protocols are included. Fashion mirror are very {specialized|specialised} {and often|and sometimes|and infrequently} {include|embrace|embody} some {form of|type of} augmented {reality|actuality} {to engage|to interact|to have interaction} the {user|consumer|person} and {recommend|advocate|suggest} or {try|attempt|strive} {virtual|digital} {clothes|garments} and {accessories|equipment}. The mirrors {targeted|focused} to the medical or {health|well being} fields {include|embrace|embody} some {form of|type of} direct or {indirect|oblique} {health|well being} monitoring {often|typically|usually} {using|utilizing} {physical|bodily} sensors. Finally, for the {commercial|business|industrial} mirrors, {we can|we will|we are {able|in a position|ready} to} see that {they provide|they supply} many functionalities for {services|providers|companies} {in the|within the} cloud. Little {information|info|data} {can be|could be|may be} {found|discovered} {about the|concerning the|in regards to the} underlying {technologies|applied sciences}. Following the users' {feedback|suggestions} and interactions with the mirror {during|throughout} usability {tests|checks|exams}, hardware {improvement|enchancment} is {under|beneath|underneath} investigation.
For {example|instance}, {to increase|to extend} the {image|picture} {quality|high quality} even with poor lightning, {a night|an evening} {vision|imaginative and prescient} {camera|digital camera|digicam} {could be|might be|could {possibly|probably|presumably} be} added to the mirror. We are {currently|presently|at present} devising {how to|the {way|method|means} to|tips on how to} incorporate this {within the|inside the|throughout the} processing workflow. Also the microphone {will be|shall be|might be} {replaced|changed} with {a better|a greater} performing one. The {data|knowledge|information} {provided|offered|supplied} from the emotion recognition {features|options} are {important|essential|necessary} for monitoring and understanding the user's long-term emotional progress.
At the {moment|second}, this {feature|function|characteristic} is {under|beneath|underneath} {development|improvement|growth}, and {it will be|it {is going|goes} to be|will {probably|in all probability|most likely} be} {integrated|built-in} {via|by way of|through} an ad-hoc {data|knowledge|information} {management|administration} module and {visual|visible} {application|software|utility} to browse and analyze the recorded {data|knowledge|information}. Previous {performance|efficiency} {analysis|evaluation} {shows|exhibits|reveals} that the developed recognition {methods|strategies} for audio emotion recognition are {effective|efficient}. In {general|common|basic}, the modules developed for video and audio {interaction|interplay} represented the state-of-the-art {at the|on the} time of {the development|the event} of the mirror.
More {robust|strong|sturdy} {methods|strategies} for {both|each} video and audio identification {were|have been|had been} {recently|lately|just lately} proposed . At {the same|the identical} time, {more effective|simpler|more practical} face attribute estimation and speaker emotion recognition {methods|strategies} {were|have been|had been} {presented|introduced|offered}. Therefore, {to meet|to satisfy|to fulfill} the {growing|rising} {security|safety} {needs|wants} and {improve|enhance} the {user|consumer|person} {experience|expertise}, we plan to {further|additional} {improve|enhance} our {methods|strategies} {in the future|sooner or later}. Figure 1.Functional {requirements|necessities} of the proposed {smart|sensible|good} mirror. Color of {boxes|bins|packing containers} {indicates|signifies} relationship between processing modules {and information|and knowledge|and data} displayed by {different|totally different|completely different} {applications|purposes|functions}. For {example|instance}, to implement user-specific {applications|purposes|functions}, {user|consumer|person} authentication {through|via|by way of} face and speaker recognition are required.
Applications {such as|similar to|corresponding to} {weather|climate} forecast, calendar, and music {player|participant} {depend on|depend upon|rely upon} integration of mirror with Alexa Skills. Smart mirrors are {also|additionally} exploited for monitoring the {users|customers} as {smart|sensible|good} healthcare {systems|methods|techniques}. The {device|system|gadget} is used to {record|document|report} the user's {health|well being} {conditions|circumstances|situations}, remind about {medications|drugs|medicines}, {suggest|recommend|counsel} {exercises|workouts|workout routines}, and, {generally|usually|typically}, monitor the well-being of the {user|consumer|person}. The paper by Silapasuphakornwong et al. is a {recent|current|latest} {example|instance} of {this {kind|type|sort} of|this {type|sort|kind} of|this {sort|type|kind} of} {smart|sensible|good} IoT {device|system|gadget}. The authors designed {a smart|a sensible|a wise} mirror {that can|that may} {talk|speak|discuss} and have a {conversation|dialog} with {elderly|aged} {users|customers} {so that|in order that} {they do not|they don't} {feel|really feel} lonely. It {monitors|screens|displays} their {mental|psychological} well-being by analyzing their {emotions|feelings} and {behavior|conduct|habits}.
Another {example|instance} is the LUX system , which analyzes {user|consumer|person} sentiment and manages sentiment {feedback|suggestions} by exploiting speech-to-text, NLP, and deep {learning|studying} {techniques|methods|strategies}. Currently, LUX {is {able|in a position|ready} to|is {ready|prepared} to} {manage|handle} Korean {text|textual content} and {provides|offers|supplies} {feedback|suggestions} {in the|within the} {form of|type of} reciting {wise|sensible|clever} sayings, empathizing, and {playing|enjoying|taking {part|half} in} music. For the audio {part|half}, the MFCC {representation|illustration} are first extracted {using|utilizing} the Librosa library, {and then|after which} this {representation|illustration} {is sent|is shipped|is distributed} to the speaker and emotion recognition modules. The {results|outcomes} of the {previous|earlier} modules are returned to the Visual-Audio Manager.
From the {analysis|evaluation} of the responses, {we can|we will|we are {able|in a position|ready} to} say that the {smart|sensible|good} mirror was {well|properly|nicely} {received|acquired|obtained} {overall|general|total}. The {users|customers} perceived the system as being very responsive and {useful|helpful}. Some {users|customers} {pointed out|identified} some {problems|issues} with the Alexa service {where|the place} the system {could not|couldn't} {fully|absolutely|totally} {understand|perceive} the spoken name, which delayed the registration {phase|part|section}.
This {is probably|might be} {due to the|because of the|as a {result|end result|outcome} of} microphone used, which was not of {sufficient|enough|adequate} {quality|high quality} for the environments {in which|by which|during which} the mirror was {located|situated|positioned}. Although this {feature|function|characteristic} was positively {received|acquired|obtained}, one {female|feminine} {user|consumer|person} complained that the system created a male avatar {instead|as an alternative|as a substitute} of a {female|feminine} one. A {solution|answer|resolution} {would be|can be|could be} {to train|to coach} the module with a {more|extra} {diverse|numerous|various} set of {subjects|topics}. The {users|customers} {did not|didn't} have any {issues|points} with {the other|the opposite} functionalities. From {the first|the primary} 10 questions, a SUS {score|rating} that represents {the overall|the general} usability of the {device|system|gadget}, {including|together with} how {easy|straightforward|simple} {it is|it's} {to use|to make use of} and {learn|study|be taught}, was calculated.
The {average|common} SUS {score|rating}, computed from the surveys of 500 {products|merchandise}, {is considered|is taken into account} to be {68|sixty eight}. Our {smart|sensible|good} mirror achieved {82|eighty two}.88, which is above {average|common} and {in the|within the} {range|vary} {of good|of excellent|of fine} {systems|methods|techniques}. In Figure 12, {it is|it's} {possible|potential|attainable} to view the {users|customers} responses for {each|every} {one of the|one of many} 20 questions {in the|within the} survey. The {main|primary|major} {component|element|part} of the Visual-Audio Manager is the Coordinator.
It is {in {charge|cost} of|in command of|in {control|management} of} {handling|dealing with} the input-output streams for {both|each} the {visual|visible} and audio {signals|alerts|indicators}. In the {input|enter} {data|knowledge|information} {flow|circulate|move}, the Coordinator acts as an {intermediary|middleman} for the Data Processing Manager. It receives the video frames acquired with the Webcam, and the audio {signal|sign} recorded {through|via|by way of} the Microphone. It then forwards the Visual-Audio {raw|uncooked} {data|knowledge|information} to the Data Processing Manager, which returns the processed {data|knowledge|information}. The Coordinator {is also|can {also|additionally} be|can be} {in {charge|cost} of|in command of|in {control|management} of} avoiding overloads and conflicts. This {problem|drawback|downside} can {arise|come up} {due to the|because of the|as a {result|end result|outcome} of} availability of {only one|just one} {camera|digital camera|digicam} and one microphone, the presence of asynchronous {services|providers|companies} (i.e., Alexa Voice Service).
Whenever a {feature|function|characteristic} requires {one of|certainly one of|considered one of} these {devices|units|gadgets}, the Coordinator {changes|modifications|adjustments} {status|standing} to "locked" and blocks {access|entry} to the {resources|assets|sources} {until|till} the devices' proprietary {function|perform|operate} unlocks it. Finally, the Coordinator manages determines which {screen|display|display screen} {has to be|needs to be|must be} displayed from the Graphical User Interface Manager. This {section|part} {is dedicated|is devoted} to {commercial|business|industrial} {smart|sensible|good} mirrors {that are|which are|which {might|may|would possibly} be} developed {outside|outdoors|exterior} the research laboratory and the DIY {community|group|neighborhood}. The {smart|sensible|good} mirror {is still|continues to be|remains to be} a {growing|rising} market, {and companies|and corporations|and firms} are {still|nonetheless} {developing|creating|growing} {their own|their very own} product. However, {we can|we will|we are {able|in a position|ready} to} {find|discover} some {smart|sensible|good} mirror {products|merchandise} {on the market|available on the market|in the marketplace}. There are few general-purpose {commercial|business|industrial} {smart|sensible|good} mirrors.
Ekko {offers|provides|presents} {basic|primary|fundamental} {features|options} {such as|similar to|corresponding to} {general|common|basic} {information|info|data} and {personalized|personalised|customized} profile and {can be|could be|may be} {controlled|managed} with hand gestures. Griffin Technology developed the Connected Mirror, {a smart|a sensible|a wise} mirror {showing|displaying|exhibiting} {general|common|basic} {information|info|data} , and updates from {other|different} Griffin {devices|units|gadgets}. Toshiba created a mirror {that can|that may} {help|assist} the {user|consumer|person} {prepare|put together} recipes and work as {a personal|a private} {fitness|health} monitor {through|via|by way of} a {connection with|reference to} a smartphone. Chakra Groups {released|launched} {a smart|a sensible|a wise} mirror {offering|providing} health-related {features|options} (e.g., {tracking|monitoring} for weight, {calories|energy}, sleep, exercise) by connecting to Apple Health or Fitbit.
The {popularity|reputation|recognition} of {smart|sensible|good} mirrors {rapidly|quickly} {increased|elevated}, {and many|and {lots|tons|heaps} of|and {plenty|lots|a lot} of} articles on {the subject|the topic} are featured {in the|within the} literature. Most of {the existing|the prevailing|the present} work describes the design of a {general|common|basic} {type|sort|kind} of {smart|sensible|good} mirror {that includes|that features} {only|solely} {simple|easy} {basic|primary|fundamental} {functions|features|capabilities} {such as|similar to|corresponding to} {news|information}, {weather|climate}, alarm, time, {etc|and so on|and so forth}. However, there are works involving {smart|sensible|good} mirrors developed for medical or {fashion|style|trend} {purposes|functions}. In this {section|part}, we briefly {review|evaluate|evaluation} the state-of-the-art in {smart|sensible|good} mirrors {both|each} {in the|within the} research/prototype stage and {commercial|business|industrial} ones. For a {summary|abstract} of the functionalities and {technologies|applied sciences} {of these|of those} {smart|sensible|good} mirrors, see Table 1 and Table 2.
For {the other|the opposite} modules (i.e., {those|these} {relating to|referring to|regarding} {visual|visible} interaction), we refer the reader to our {previous|earlier} papers . Most of the mirror {features|options} are {based|based mostly|primarily based} on on-the-fly {data|knowledge|information} generated {during|throughout} {user|consumer|person} {interaction|interplay}. However, {there is|there's|there {may|might|could} be} some {information|info|data} that {need to be|have to be|must be} {stored|saved} {in the|within the} system, {they can be|they are often} recovered and analyzed {at the|on the} {appropriate|applicable|acceptable} time. The {features|options} and identities of the enrolled {users|customers} are {stored|saved} by the system and used for identification. Monitoring the user's long-term emotional state {is helpful|is useful} in understanding his emotional progress and {possibly|probably|presumably} diagnose {disorders|issues|problems}.
For {example|instance}, the emotional {symptoms|signs} of {distress|misery} {can be|could be|may be} traced {back|again} to {long|lengthy} {periods|durations|intervals} of depression/sadness, {anxiety|nervousness|anxiousness}, and anger. To implement the {previous|earlier} {feature|function|characteristic}, we {store|retailer} the emotional {information|info|data} {that is|that's} {automatically|mechanically|routinely} gathered {each|every} time the {user|consumer|person} {interact|work together} with the mirror. The proposed {smart|sensible|good} mirror {provides|offers|supplies} the {user|consumer|person} with an interactive interface that {can be used|can be utilized} comfortably {in the|within the} {home|house|residence} {environment|surroundings|setting}. The mirror can {acquire|purchase} {information|info|data} {relating to|referring to|regarding} the user's emotional state and react accordingly.
Users can {access|entry} {personal|private} notes, reminders, calendars, {news|information}, {weather|climate} {information|info|data}, and {more|extra}. The following sections describe how the mirror interacts with {external|exterior} stimuli and what {technologies|applied sciences} are used to develop the prototype. More {sophisticated|refined|subtle} mirrors can {link|hyperlink} with sensors, wearable {devices|units|gadgets}, or {other|different} {smart|sensible|good} {products|merchandise} {to collect|to gather} physiological {signals|alerts|indicators}. In , the authors proposed {a smart|a sensible|a wise} {fitness|health} mirror {to monitor|to watch|to observe} the user's {health|well being}. Load and ultrasonic sensors {were|have been|had been} used to detect the user's weight and {height|peak|top}, electrode plated for bio-electrical impedance {analysis|evaluation}, {finally|lastly}, an IR temperature sensor to measure the {user|consumer|person} temperature.
Bluetooth and LoRa modules {were|have been|had been} {used in|utilized in} to measure {basic|primary|fundamental} {information|info|data} {such as|similar to|corresponding to} a temperature, and humidity of the room. The user's avatar is {showed|confirmed} {together with|along with} a joybar that measures {the current|the present} user's valence gathered from the face emotion recognition module. Figure {4|four} {shows|exhibits|reveals} {the combination|the mixture|the mix} of avatar and joybar on the user's {page|web page}. The {display|show} monitor {is also|can {also|additionally} be|can be} {responsible for|liable for|answerable for} {showing|displaying|exhibiting} {information|info|data} and {feedback|suggestions} from {the various|the varied|the assorted} {active|lively|energetic} modules.
Apart from recognizing {user|consumer|person} {emotions|feelings}, the system {can also|also can|can even} estimate facial traits {such as|similar to|corresponding to} {hairstyle|coiffure} and {color|colour|shade}, {skin|pores and skin} {color|colour|shade}, and worn {accessories|equipment} (e.g., glasses and hats). The {model|mannequin} used for face attribute estimation is the one proposed in . It {is based|is predicated|relies} on a ResNet-50, whose predictions are conditioned by a layer that takes {into account|under consideration|into consideration} the correlation between the attributes. The {model|mannequin} is {trained|educated|skilled} {using|utilizing} a binary cross-entropy loss on the CelebA dataset . This dataset {contains|incorporates|accommodates} over 200,000 {images|pictures|photographs} of 10,000 {different|totally different|completely different} identities and {a list|an inventory|a listing} of {40|forty} attributes that describes {general|common|basic} traits for {each|every} facial {image|picture} . These attributes are used to create an avatar with the user's detected facial traits.
Some {smart|sensible|good} {fashion|style|trend} mirrors {include|embrace|embody} {recommendation|suggestion|advice} {systems|methods|techniques}. For {example|instance}, in to {suggest|recommend|counsel} {the ideal|the perfect|the best} outfit basing on the user's {mood|temper}, or in , the {makeup|make-up} that {best suits|most {accurately|precisely} fits|most {closely|intently|carefully} fits} the user's face. The use of Augmented Reality and Virtual Reality in {smart|sensible|good} mirrors {to improve|to enhance} {the customer|the client|the shopper} {experience|expertise} is {increasingly|more and more} widespread.
In {fact|reality|truth}, {both|each} AR and VR {can help|might help|may help} make retail {stores|shops} {more|extra} interactive and {the online|the web|the net} {experience|expertise} {more|extra} {real|actual} . In Section 2, we {comment on|touch upon} {recent|current|latest} works {related|associated} to {smart|sensible|good} mirrors, {both|each} {academic|educational|tutorial} and {commercial|business|industrial}. In Section {3|three}, we describe the design of our {smart|sensible|good} mirror, from the functionalities to the hardware and {software|software program} implementation. In Section {4|four}, we report some quantitative {results|outcomes} on some developed modules {incorporated|included|integrated} into the mirror. We {also|additionally} report on {the results|the outcomes} from a subjective {study|research|examine} on the usability of {the whole|the entire} system.
The Data Processing Manager {contains|incorporates|accommodates} {most of the|a lot of the|many of the} {smart|sensible|good} {functions|features|capabilities} of the mirror and {specifically|particularly} all {those|these} {relating to|referring to|regarding} recognition from {both|each} video and audio streams. These {functions|features|capabilities} are {the most|probably the most|essentially the most} computationally {expensive|costly}. Two {components|elements|parts}, one for {the recognition|the popularity} from the video {signal|sign} and {the other|the opposite} for {the recognition|the popularity} from the audio {signal|sign}, are {implemented|carried out|applied}.
Each {component|element|part} executes {one or more|a {number|quantity} of} {software|software program} modules that {process|course of} {a data|a knowledge|an information} stream. General {purpose|objective|function} {smart|sensible|good} mirrors {can be|could be|may be} grouped primarily by the {input|enter} sensors with which {they are|they're} {equipped|outfitted|geared up}. Vision-based {smart|sensible|good} mirrors are {equipped|outfitted|geared up} with an RGB {camera|digital camera|digicam}. They {commonly|generally} activate {when they|once they|after they} {recognize|acknowledge} the framed {user|consumer|person} and {show|present} contents {dedicated|devoted} to the variation of the user's expressions . When the mirror is in standby mode, the face detection module is executed {in the|within the} background. It is a {fundamental|elementary|basic} {function|perform|operate} of the mirror, which {also|additionally} triggers {the subsequent|the next|the following} modules, {namely|specifically|particularly}, the face identification, the facial emotion recognition, and the face attribute estimation modules.
In the medical {field|area|subject}, {smart|sensible|good} mirrors can {improve|enhance} {both|each} {clinical|medical|scientific} and at-home healthcare. Several proposals {rely on|depend on} the {analysis|evaluation} of the face and facial expressions for {daily|every day|day by day} {personal|private} checkups . Some mirrors complement the {diagnosis|analysis|prognosis} with {alternative|various|different} {medicine|drugs|medication} {treatments|remedies|therapies} {such as|similar to|corresponding to} music {therapy|remedy} and {color|colour|shade} {therapy|remedy} .
Average (±standard deviation) precision, recall, F1-score, and accuracy {across|throughout} 24 actors of RAVDESS dataset. In {each|every} column, {the best|one of the best|the most effective} {results|outcomes} are marked in boldface. From a technical {point of view|perspective|viewpoint}, {since the|because the|for {the reason|the rationale|the explanation} that} mirror is {used in|utilized in} {domestic|home} environments, we developed it to have low {power|energy} consumption. As {previously|beforehand} described, heavy computing {is dedicated|is devoted} to an {external|exterior} server, devoting the {smart|sensible|good} mirror {almost|virtually|nearly} {exclusively|solely|completely} to I/O operations. Thus, the {resulting|ensuing} {power|energy} consumption of the mirror {relies|depends} {only|solely} on the Raspberry and the {display|show} monitor.
When the mirror is {in the|within the} idle state, {the power|the facility|the ability} consumption is {approximately|roughly} 23 W. This consumption is {split|cut up|break up} between 15–20 W for the {display|show} monitor , {and 2|and {a couple|a pair} of|and a pair of}.6–2.{8|eight} W for the Raspberry . In the working state, {the power|the facility|the ability} consumption {increases|will increase} to about 25 W.
The consumption of the Raspberry rises to {4|four}.7–4.{8|eight} W, {while|whereas} that of the {display|show} monitor is unchanged. This {page|web page} {shows|exhibits|reveals} the {weather|climate} for the user's {current|present} location and {other|different} generic {information|info|data}. To {access|entry} this {page|web page}, the {user|consumer|person} is required not {only|solely} to authenticate {but also|but in addition|but additionally} {to show|to {point|level} out|to indicate} a {positive|constructive|optimistic} emotional {value|worth} in {front|entrance} of the mirror .
This {is done|is completed|is finished} not {only|solely} as an initiative to ask the {user|consumer|person} {to give|to offer|to provide} a {positive|constructive|optimistic} stimulus, {but also|but in addition|but additionally} as a therapeutic {treatment|remedy|therapy} for the {user|consumer|person} to be {positive|constructive|optimistic} {more|extra} {often|typically|usually}. This {procedure|process} {allows|permits} us {to easily|to simply} {extend|prolong|lengthen} the set of the identities {recognized|acknowledged} by the mirror {without|with out} requiring a retraining of the {network|community} in an end-to-end {fashion|style|trend}. The {features|options} used for classification of the identities are collected {during the|through the|in the course of the} registration {phase|part|section}. As {part of|a {part|half} of} the {smart|sensible|good} mirror for {fashion|style|trend} {category|class}, Memories {uses|makes use of} a multilayer AR and AI engine.
